| title | Propagating taints from Cluster API Machines to Kubernetes Nodes | ||||
|---|---|---|---|---|---|
| authors |
|
||||
| reviewers |
|
||||
| creation-date | 2025-05-13 | ||||
| last-updated | 2025-06-06 | ||||
| status | provisional | ||||
| see-also |
|
- Glossary
- Summary
- Motivation
- Proposal
- User Stories
- Requirements (Optional)
- Implementation Details/Notes/Constraints
- Security Model
- Risks and Mitigations
- Alternatives
- Upgrade Strategy
- Additional Details
- Implementation History
Refer to the Cluster API Book Glossary.
This proposal introduces taints as a first-class citizen in Cluster API's core types, enabling users to declaratively manage Node taints through higher-level Cluster API resources such as Cluster, ClusterClass, MachineSet, MachineDeployment and MachinePool.
The proposal defines two different kind of propagation modes for taints:
- Always: Taints are continuously reconciled and maintained on nodes
- OnInitialization: Taints are set once during node initialization and then left unmanaged
Note: This new proposal has been created rather than updating the prior in-place metadata propagation proposal because taints are not yet part of the Core Provider's API types and are different enough from labels or annotations that a different set of constraints will need to be considered. Very early versions of Kubernetes tracked taints as annotations, but they have long since been promoted to their own API type
As stated in the goals of the project, Cluster API tries to "define common operations" and "manage the lifecycle [...] of Kubernetes-conformant Clusters using a declarative API" [0]. Users of Cluster API can currently update labels and annotations on Cluster API objects and have those values propagate from their high level resources all the way down to nodes (see the related proposal for more context). While this is useful, it does not provide a way to, for example, reserve a set of nodes for specific workloads like GPU or network functions.
As of today defining custom taints for nodes can be done via bootstrap providers, e.g. via CABPK's nodeRegistration.taints field in a KubeadmConfig
[1]
or by adding them manually after Node creation.
In case of CABPK updating taints currently also requires a rollout.
This proposal aims to standardize the way users can define taints for nodes in workload Clusters as well as how they get propagated in case of changes.
Note: CAPI already manages the node.cluster.x-k8s.io/uninitialized and node.cluster.x-k8s.io/outdated-revision taints today. This proposal does not plan to change this.
- Make taints a first-class citizen in core Cluster API types.
- Define how taints get propagated to corresponding
Node's. - Define how a migration from a bootstrap provider's implementation of taints to the new feature could look like (taking CABPK as example).
- Taints managed by Cluster API should not interfere with taints applied by other actors.
- Supporting taints on individual devices via Dynamic Resource Allocation. This may be added in the future, but is currently out of scope.
- Change how CAPI manages the
node.cluster.x-k8s.io/uninitializedandnode.cluster.x-k8s.io/outdated-revisiontaints.
In Kubernetes Clusters, taints and tolerations are what allow a workload to be scheduled to a specific Node. As a user, I would like to use this community-standard mechanism within the framework that Cluster API provides.
As a user, I wish to use Cluster API to manage a set of Machines that have very specific characteristics for targeting workloads. Some examples of this might be:
- Designating Nodes as
edgeNodes and steering locality-critical workloads only toedgeNodes. - Designating Nodes as having a particular hardware capability, such as high performance GPUs
As a user, I wish to have autoscaling capabilities using Kubernetes and Cluster API resources and conventions. I would like to define taints on a Cluster API resource representing some collection (including but not limited to MachineSets, MachinePools, and MachineDeployments). This is especially useful in scale-from-zero scenarios, where that autoscaling technology can reference taints on a collection to make decisions about the cloud resources available.
As a user, I would like to update taint metadata on my collection resources without forcing a complete replacement of an owned resource, such as a Machine or Node.
As a user, I want to define a taint that is applied only once so I could ensure workload is scheduled after a custom initialization process initialized the node and remove that taint.
Users should be able to define Taints on collection resources and have the taints propagate to the owned resources.
This would start at a ClusterClass or Cluster level, and ultimately be written to a Node.
Users should be able to remove Taints managed by Cluster API without removing taints that Cluster API does not manage.
Note: This only applies to taints defined below as Always. It does not apply for OnInitialization taints.
Cluster API already supports propagating labels and annotations downward in its resource hierarchy. This support is implemented such that when these fields are updated, the underlying compute resources are not replaced.
Taints present a challenge to this, because they are defined as an "atomic" field by Kubernetes. (see Kubernetes taints API discussions for more context).
This means that when updating Taints on a Node, all Taints are replaced; it is not possible to add and replace individual elements like for labels and annotations.
As a concrete example, if a Node has 3 taints and some client submits a patch request with only one, the end result is one taint on the Node.
It also means that Server-Side Apply ownership rules could not be applied to individual taints, which could present conflicts between controllers or users trying to modify Taints on the resultant Node.
For Cluster API to support propagating Taints, it will need to:
- implement its own mechanism for tracking what
Taints it owns. This will be similar to the implementation for annotations and labels. See Changes to the Machine controller for more details. - ensure there are no conflicts with other actors by always setting the
metadata.resourceVersionon API calls changing the taints on a Node. In the code this can be done by using theclient.MergeFromWithOptimisticLock{}option.
A taint for a Node may be defined for different use-cases:
- Taints supposed to stay on the Node to ensure only certain workload runs on such a
Nodeaka.Always:- These taints are supposed to be set on the
Nodeobject as long as it is defined on its parent core CAPI object. - Example: Nodes where only GPU related workload should run
- Reconciliation behavior:
Alwaystaint added to the machine or exists during initialization: reconciliation will add the taint to the node.Alwaystaint removed from machine: reconciliation will remove the taint from the node, if it did add it in the past.Alwaystaint not changed: reconciliation takes care that the taint still exists on the node.
- These taints are supposed to be set on the
- Taint's supposed to get added once to a Node aka.
OnInitialization- These taints are supposed to be set once by Cluster API on a
Nodeobject. - Example: Ensure that no workload gets scheduled to a
Nodeunless the taint got removed to e.g. install a GPU driver before allowing workload. - Cluster API should once set the taint on the Node and not add it again if it got removed.
- These taints are supposed to be set once by Cluster API on a
If a taint exists on a node and was changed from Always to OnInitialization, Cluster API should drop ownership of the taint, but does not remove it, so the taint behaves as it would have been defined as OnInitialization from the beginning.
If a taint was changed from OnInitialization to Always, Cluster API should ensure the taint is set on the node and track ownership.
In CABPK the taint field has special behavior depending on what is set. This comes from how kubeadm handles the relevant field and is as documented in the following table:
| CABPK | Behavior |
|---|---|
| Set | add only the taints configured |
| Not set | add the default taint *[1] |
empty / [] |
not add any taints |
[*1]: Per default kubeadm adds the taint node-role.kubernetes.io/control-plane:NoSchedule to control plane nodes.
The proposed changes should not influence the behavior for CABPK. The following table shows the resulting behavior, depending on where taints are set or not set.
Note: The taints field on machine's will only allow being set or not being set, there will be no "empty / []" option.
| Machine | CABPK | Result for CABPK |
|---|---|---|
| Set | Set | CABPK and Machine taints, on same key + effect use the value from the Machine defined taint |
| Set | Not set | CABPK default and Machine taints, on same key + effect use the value from the Machine defined taint |
| Set | empty / [] |
Machine taints |
| Not set | Set | CABPK taints |
| Not set | Not set | CABPK default taint |
| Not set | empty / [] |
no taints |
So the desired behavior of the new taints field does not change the behavior for taints configured via CABPK.
In future CABPK could consider deprecating the related fields and propose using the machine taints instead.
The following defines a new struct which should be used to define taints at the corresponding API types.
It replicates the upstream corev1.Taint specification and extends it by a field called propagation, which will define the propagation mechanism to use for the taint.
Using a type definition allows to be extensible and add additional propagation mechanisms when necessary.
// MachineTaint defines a taint equivalent to corev1.Taint, but additionally having a propagation field.
type MachineTaint struct {
// key is the taint key to be applied to a node.
// +required
// +kubebuilder:validation:MinLength=1
// +kubebuilder:validation:MaxLength=253
Key string `json:"key,omitempty"`
// value is the taint value corresponding to the taint key.
// +optional
// +kubebuilder:validation:MaxLength=63
Value *string `json:"value,omitempty"`
// effect is the effect for the taint. Valid values are NoSchedule, PreferNoSchedule and NoExecute.
// +required
// +kubebuilder:validation:Enum=NoSchedule;PreferNoSchedule;NoExecute
Effect corev1.TaintEffect `json:"effect,omitempty"`
// propagation defines how this taint should be propagated to nodes.
// Always: The taint will be continuously reconciled. If it is not set for a node, it will be added during reconciliation.
// OnInitialization: The taint will be added during node initialization. If it gets removed from the node later on it will not get added again.
// +required
Propagation MachineTaintPropagation `json:"propagation,omitempty"`
}
// MachineTaintPropagation defines when a taint should be propagated to nodes.
// +kubebuilder:validation:Enum=Always;OnInitialization
type MachineTaintPropagation string
const (
// TaintPropagationAlways means the taint should be continuously reconciled and kept on the node.
// - If an Always taint is added to the Machine, the taint will be added to the node.
// - If an Always taint is removed from the Machine, the taint will be removed from the node.
// - If an OnInitialization taint is changed to Always, the Machine controller will ensure the taint is set on the node.
// - If an Always taint is removed from the node, it will be re-added during reconciliation.
TaintPropagationAlways MachineTaintPropagation = "Always"
// TaintPropagationOnInitialization means the taint should be set once during initialization and then
// left alone.
// - If an OnInitialization taint is added to the Machine, the taint will only be added to the node on initialization.
// - If an OnInitialization taint is removed from the Machine nothing will be changed on the node.
// - If an Always taint is changed to OnInitialization, the taint will only be added to the node on initialization.
// - If an OnInitialization taint is removed from the node, it will not be re-added during reconciliation.
TaintPropagationOnInitialization MachineTaintPropagation = "OnInitialization"
)Proper validation on the new field has to:
- ensure the
keyandvaluefields are validated as done in the upstream node validation [3] [4]. - block taints with a key of
node.cluster.x-k8s.io/uninitializedornode.cluster.x-k8s.io/outdated-revision.- These taints are managed by Cluster API and providers.
- block taints with the key prefix
node.kubernetes.io/, exceptnode.kubernetes.io/out-of-service.- With the exception of
node.kubernetes.io/out-of-service, this taints are managed by the node controller or the kubelet.
- With the exception of
- block taints with the key prefix
node.cloudprovider.kubernetes.io/- This taints are either managed by the kubelet or by a cloud-controller-manager's node-lifecycle-controller
- block the taint
node-role.kubernetes.io/control-planefor worker nodes (identified by absence of the labelcluster.x-k8s.io/control-plane) and for MachineDeployments, MachineSets and MachinePools.- This taints are dedicated to be set on control-plane machines, not for workers.
- block the taint
node-role.kubernetes.io/masterwhich is deprecated since 1.24
If in the future we are introducing new taints that users should not be able to set, ratcheting may be used.
Note: Other taints normally set by kubeadm should be able to get set by Cluster API too and not be blocked on to allow more flexibility and use-cases.
This proposes to add a field array to the MachineSpec struct.
This implicitly leads to adding the field to the following types:
| API Type | New field |
|---|---|
Machine |
.spec.taints |
MachineSet |
.spec.template.spec.taints |
MachineDeployment |
.spec.template.spec.taints |
MachinePool |
.spec.template.spec.taints |
type MachineSpec struct{
// taints are the node taints that Cluster API will manage.
// This list does not necessarily reflect the full list of taints on a Node: other Kubernetes components may add or remove other taints.
// Only the taints defined in this list will be added or removed by core Cluster API controllers.
//
// NOTE: This list is implemented as a "map" type, meaning that individual elements can be managed by different owners.
// As of Kubernetes 1.33, this is different from the implementation on corev1.NodeSpec, but provides a more flexible API for components building on top of Cluster API.
// +optional
// +listType=map
// +listMapKey=key
// +listMapKey=effect
// +kubebuilder:validation:MinItems=1
// +kubebuilder:validation:MaxItems=64
Taints []MachineTaint `json:"taints,omitempty"`
// Other fields...
}This proposes to add a separate struct as field array to ClusterClass and the topology section of Cluster for the controlPlane and workers sections.
The following table summarizes the new fields:
| API Type | New field |
|---|---|
ClusterClass |
spec.controlPlane.taints |
spec.workers.machineDeployments[].taints |
|
spec.workers.machinePools[].taints |
|
Cluster |
spec.topology.controlPlane.taints |
spec.topology.workers.machineDeployments[].taints |
|
spec.topology.workers.machinePools[].taints |
The propagation of the fields should follow the prior art and is summarized in the following table and picture:
| ClusterClass | Cluster | Result |
|---|---|---|
| Set | Set | Cluster taints (ClusterClass taints are ignored) |
| Set | Not set | ClusterClass taints |
| Not set | Set | Cluster taints |
| Not set | Not set | No taints from ClusterClass or Cluster |
Type definitions:
// ClusterMachineTaint defines a taint equivalent to corev1.Taint, but additionally having a propagation field.
type ClusterMachineTaint struct {
// All fields are identical to MachineTaint.
}
// ClusterClassMachineTaint defines a taint equivalent to corev1.Taint, but additionally having a propagation field.
type ClusterClassMachineTaint struct {
// All fields are identical to MachineTaint.
}Example type changes for the Cluster type:
type ControlPlaneTopology struct {
// taints are the node taints that Cluster API will manage.
// ...
Taints []ClusterMachineTaint `json:"taints,omitempty"`
// Other fields...
}
// Same for MachineDeploymentTopology and MachinePoolTopology.Example type changes for the ClusterClass type:
type ControlPlaneClass struct {
// taints are the node taints that Cluster API will manage.
// ...
Taints []ClusterClassMachineTaint `json:"taints,omitempty"`
// Other fields...
}
// Same for MachineDeploymentClass and MachinePoolClass.To support taints for control plane Machines, the contract for ControlPlanes should be extended.
Similar to the extension for ReadinessGates at the Machines section, an additional optional FooControlPlaneMachineTemplate rule should be added for supporting taints.
Support and usage of that field would result in having the defined taints set on the Machines created by the ControlPlane provider and the Machine controller being responsible to behave as for worker Machines.
The Machine controller as of today already manages the lifecycle of the taint node.cluster.x-k8s.io/uninitialized as well as label and annotation propagation and uses a single patch operation for this in internal/controllers/machine.patchNode.
This is a good fit to also reconcile the taints set on a machine.
Due to how the .spec.taints field on a Node is defined, additionally to patching the taints field, the controller needs to keep track of the taints it added.
To keep track of which Always taints are owned by the machine controller, the controller adds the annotation cluster.x-k8s.io/taints-from-machine to the Node.
This follows the convention established by cluster.x-k8s.io/labels-from-machine.
This allows the machine controller to identify an Always taint it:
- has added in the past which now should be removed
- has added in the past which now should not be tracked anymore (transition to
OnInitialization) - has added in the past which should be kept (does not require the annotation)
- has not added yet and should be added (does not require the annotation)
The taints will be concatenated with , and the serialization will look like this given the field keys of the listMap:
cluster.x-k8s.io/taints-from-machine: "<key1>:<effect1>,<key2>:<effect2>"
Additionally, the fact that this annotation does not exist yet should be used as signal that the controller has to add the OnInitialization taints.
This also means if no Always taints are configured on the machine, the controller should add the annotation with an empty value.
This ensures that a node ever only gets these taints added once, at the time of finishing initialization of a Node.
If a Node re-registers itself, the controller would add the OnInitialization taints again in addition to the Always taints because the tracking annotation does not exist.
Note: If a bootstrap provider supports the node.cluster.x-k8s.io/uninitialized taint as documented in Contract rules for BootstrapConfig, the implementation ensures that all OnInitialization taints are applied in the same call through which the node.cluster.x-k8s.io/uninitialized taint is getting removed. This prevents that workload without proper tolerations could be scheduled to the node in the time between node creation and Cluster API adding the OnInitialization taints.
Note: A taint added by a bootstrap provider (e.g. CABPK) will be treated as a taint added by any other actor and will not get special handling. Especially for an Always taint this means the machine controller would adopt the taint and start tracking it.
The MachinePool controller changes depend on whether the provider implements MachinePool Machines or not.
When using MachinePool Machines, no changes are required to the MachinePool controller itself. The taint propagation will be handled by the Machine controller as described above, ensuring consistent behavior between standalone Machines and Machines created by MachinePools.
For MachinePools that directly manage infrastructure without creating Machine resources, the MachinePool controller will need to implement logic similar to the Machine controller to propagate taints to Nodes. This includes handling both OnInitialization and Always taints, applying the tracking annotation, and managing taint lifecycle.
It's worth noting that the MachinePool controller as of today does not implement label or annotation propagation, so taint propagation represents a new capability for this controller. Additionally the MachinePool controller should not remove the node.cluster.x-k8s.io/uninitialized taint if the MachinePool uses Machines, as this responsibility then falls to the Machine controller to ensure consistent initialization semantics across all Machine resources.
The Cluster's topology controller will properly calculate and in-place propagate the newly introduced taint fields down to the ControlPlane object (if it complies with the optional contract part), MachineDeployment and/or MachinePool resources.
The MachineDeployment and MachineSet controllers will watch their associated MachineSpec for updates to the spec.taints field and update their owned objects (MachineSet or Machine) to reflect the new values.
This should be done similar to how the existing in-place mutable fields like ReadinessGates, NodeDrainTimeout, etc. for a Machine are handled today.
The cluster-autoscaler implementation for CAPI as of today consumes the capacity.cluster-autoscaler.kubernetes.io/taints (see Pre-defined labels and taints on nodes scaled from zero).
It should be adjusted to also consider the configured taints.
Users who can define Taints that get placed on Nodes will be able to steer workloads, possibly to malicious hosts in order to extract sensitive data.
However, users who can define Cluster API resources already have this capability - an attacker who receives the permissions to update a MachineTemplate could alter the definition in a similar manner.
This proposal therefore does not present any heightened security requirements on top of the ones that Cluster API already has.
With the introduction of the new fields, bootstrap providers could start deprecating their equivalent fields.
For CABPK, adding a taint at a KubeadmConfigTemplate at .spec.template.spec.joinConfiguration.nodeRegistration.taints is almost equivalent to adding an OnInitialization typed taint via the new API.
The only differences are that the taint does get added by the controllers after the Node joined the Cluster and the new field has no special behavior to add the control plane related taint as e.g. CABPK does via kubeadm.
This is considered okay, because workloads should not be able to be scheduled on a Node unless the node.cluster.x-k8s.io/uninitialized taint was removed and the implementation should take care to add the OnInitialization taints when removing this taint.
Managing the Taints on Nodes is considered a highly privileged action in Kubernetes; it even has its own top level kubectl taint command.
The Kubernetes scheduler uses taints rather than Conditions to decide when to evict workloads.
Updating these in place could then evict workloads unintentionally, or disrupt other systems that rely on taints being present.
This risk can be mitigated by ensuring Cluster API only modifies taints that it owns on nodes, as described above.
Deciding whether to reconcile taint changes continuously has been a challenge for the Cluster API.
Historically, the v1alpha1 API included a Machine.Taints field.
However, since this field was mostly used in cluster bootstrapping, it was later extracted into bootstrap provider implementations.
There is no standard API in bootstrap provider's though for taints.
Moving forward, two broad alternatives that have been explored in light of this: adding taints only at bootstrap time, and making taints immutable at the Machine level. Both methods would require that a Node be replaced in order to make any changes to the taints.
While this simplifies the implementation logic for Cluster API, it may be surprising to many users, since the Kubernetes documentation presents taints as a mutable field on a Node. This would also mean that there are two different behaviors when modifying metadata within Cluster API, which could again be very confusing. There is already precedent for leaving infrastructure in-place when Kubernetes-only fields are modified, and this proposal seeks to align with the established function.
An alternative to define the taints as written above is using slices per type as follows:
type MachineTaints struct{
Always []corev1.Taint `json:"always,omitempty"`
OnInitialization []corev1.Taint `json:"onInitialization,omitempty"`
}However this was considered as not being as expressive and extensible as the above.
Taint support is a net-new field, and therefore must be optional and not affect upgrades.
[*2]: It is worth noting that there has been discussion about making the taints on a Node a "map" list type and allowing for ownership of individual taints. As of this writing, the pull request and issue remain open. This proposal should be unaffected regardless of any upstream change to the handling of taints, except that using a "map" type would simplify the implementation and allow us to cooperate with other field managers.
- There should be proper unit and integration test coverage when introducing the changes.
- Unit test coverage for the matrix of:
- Type of taint (
OnInitializationandAlways) - Type of operation (adding a taint, removing a taint, transition to other type)
- Type of taint (
- Unit test coverage for the matrix of:
- An e2e test should validate the high-level user-experience
- Usage of an
OnInitializationandAlwaystaint - Adding or removing an
AlwaysandOnInitializationtaint on day 2
- Usage of an
The feature should be implemented behind a feature gate MachineTaintPropagation.
If deactivated, the corresponding controllers should block setting the fields via webhooks and not run the corresponding logic in the controllers.
This allows to gain experience and the ability to do adjustments and graduate the feature over time.
- 2025-01-15: First discussions at the community meeting
- 2025-06-06: Open proposal PR
- 2025-10-13: Reworked the proposal based on feedback
- 2025-10-21: Review feedback and discussions
- 2025-11-12: Implementation for Machine, MachineSet and MachineDeployments