{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Determine Pod Escape (ipynb)\n",
    "\n",
    "**References**\n",
    "- [Bad Pods: Kubernetes Pod Privilege Escalation](https://bishopfox.com/blog/kubernetes-pod-privilege-escalation)\n",
    "- [Pod Security Context](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podsecuritycontext-v1-core)\n",
    "- [Container Security Context](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#securitycontext-v1-core)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the dependencies `ipywidgets`, `pandas` and `kubectl`. Skip the next cell if they had already been installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install ipywidgets, pandas\n",
    "!pip3 install ipywidgets pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install kubectl\n",
    "!gcloud components install kubectl --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7fb1bd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "import json\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "from IPython.display import HTML, display\n",
    "\n",
    "# extend width of widgets\n",
    "display(HTML('''<style>\n",
    "    .widget-label { min-width: 18ex !important; font-weight:bold; }\n",
    "</style>'''))\n",
    "# extend width of cells\n",
    "display(HTML(\"<style>.container { width:100% !important; }</style>\"))\n",
    "display(HTML(\"<style>.output_result { max-width:100% !important; }</style>\"))\n",
    "\n",
    "# extend width and max rows of pandas output\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "pd.set_option('display.max_rows', None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# [OPTIONAL] authenticate using your service account\n",
    "!gcloud auth activate-service-account --key-file <json_key_file>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca0c5b6",
   "metadata": {},
   "source": [
    "## Define Environment Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Specify the following information**\n",
    "| Fields      | Description |\n",
    "| ----------- | ----------- |\n",
    "| `Source Project`   | Project id of target project that contains the k8s cluster   |\n",
    "| `Cluster Name`   | Name of k8s cluster     |\n",
    "| `Cluster Type`   | Type of k8s cluster (i.e. Regional or Zonal)    |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba8f19e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create text boxes for user input\n",
    "src_project = widgets.Text(description = \"Source Project: \", disabled=False)\n",
    "cluster_name = widgets.Text(description = \"Cluster Name: \", disabled=False)\n",
    "cluster_type = widgets.Dropdown(options=['Regional', \"Zonal\"], value='Zonal', description=\"Cluster Type: \", disabled=False)\n",
    "\n",
    "display(src_project, cluster_name, cluster_type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `Cluster Type` is `Regional`, specify the `Cluster Region` (e.g. `asia-southeast1`).  \n",
    "Else, if `Cluster Type` is `Zonal`, specify the `Cluster Zone` (e.g. `asia-southeast1-b`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94a90614",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create text boxes for user input\n",
    "if cluster_type.value == 'Regional':\n",
    "    cluster_region = widgets.Text(description = \"Cluster Region: \", disabled=False)\n",
    "    display(cluster_region)\n",
    "elif cluster_type.value == 'Zonal':\n",
    "    cluster_zone = widgets.Text(description = \"Cluster Zone: \", disabled=False)\n",
    "    display(cluster_zone)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f0efea9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# store user input in environment variables for use in subsequent comamnds\n",
    "os.environ['SRC_PROJECT'] = src_project.value\n",
    "os.environ['CLUSTER_NAME'] = cluster_name.value\n",
    "\n",
    "if cluster_type.value == 'Regional':\n",
    "    os.environ['CLUSTER_REGION'] = cluster_region.value\n",
    "elif cluster_type.value == 'Zonal':\n",
    "    os.environ['CLUSTER_ZONE'] = cluster_zone.value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf20cd88",
   "metadata": {},
   "source": [
    "## Get Cluster `nodeConfig`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a2894e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "if cluster_type.value == 'Regional':\n",
    "    !gcloud container clusters describe $CLUSTER_NAME --region $CLUSTER_REGION --project $SRC_PROJECT --format='json' > cluster_descr.json\n",
    "elif cluster_type.value == 'Zonal':\n",
    "    !gcloud container clusters describe $CLUSTER_NAME --zone $CLUSTER_ZONE --project $SRC_PROJECT --format='json' > cluster_descr.json\n",
    "\n",
    "with open('./cluster_descr.json') as infile:\n",
    "    cluster_descr = json.load(infile)\n",
    "cluster_descr_df = pd.json_normalize(cluster_descr['nodeConfig'])\n",
    "\n",
    "columns = ['metadata.disable-legacy-endpoints', 'serviceAccount', 'oauthScopes']\n",
    "display(cluster_descr_df[columns]\n",
    "       .rename(columns={'metadata.disable-legacy-endpoints': 'disable-legacy-endpoints'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3adfa48",
   "metadata": {},
   "source": [
    "`disable-legacy-endpoints`\n",
    "- Ensure that value is `true`\n",
    "- When `true`, config requires specified header is present when querying GCP metadata service, and disable querying of `v1beta1` endpoints\n",
    "\n",
    "`serviceAccount`\n",
    "- Service account attached to cluster\n",
    "- Default value is `default`, which is the `<12-digit>-compute@developer.gserviceaccount.com`\n",
    "- If not `default`, worthwhile to check the IAM roles/permissions granted to this service account\n",
    "\n",
    "`oauthScopes`\n",
    "- Scope of service account attached to cluster\n",
    "- Ensure that it **IS NOT** https://www.googleapis.com/auth/cloud-platform, which enables the authentication to any API function and leverage the full powers of IAM permissions assigned to the service account\n",
    "- Default is `devstorage.read_only`, `logging.write`, `monitoring`, `servicecontrol`, `service.management.readonly`, `trace.append`, which prevent the leveraging of full powers of IAM permissions assigned to the service account\n",
    "- If **NOT** the above, scope is user-customised\n",
    "- Scope **DOES NOT** matter if the access token of the service account is obtained from the metadata service and used outside of the cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b68c55af",
   "metadata": {},
   "source": [
    "## Connect to Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dcdee34",
   "metadata": {},
   "outputs": [],
   "source": [
    "if cluster_type.value == 'Regional':\n",
    "    !gcloud container clusters get-credentials $CLUSTER_NAME --region $CLUSTER_REGION --project $SRC_PROJECT\n",
    "elif cluster_type.value == 'Zonal':\n",
    "    !gcloud container clusters get-credentials $CLUSTER_NAME --zone $CLUSTER_ZONE --project $SRC_PROJECT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb4a463f",
   "metadata": {},
   "source": [
    "## Get Pods' Security Context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fed9853",
   "metadata": {},
   "outputs": [],
   "source": [
    "def highlight_not_na(value):\n",
    "    if pd.isna(value):\n",
    "        return None\n",
    "    else:\n",
    "        return 'color:white; background-color:purple'\n",
    "\n",
    "!kubectl get pods -A --output=json > pods_sc.json\n",
    "\n",
    "with open('./pods_sc.json') as infile:\n",
    "    pods_sc = json.load(infile)\n",
    "pods_sc_df = pd.json_normalize(pods_sc['items'], max_level=3)\n",
    "\n",
    "desired_columns=['metadata.name', 'metadata.namespace', 'spec.securityContext.runAsNonRoot', 'spec.securityContext.runAsGroup', 'spec.securityContext.runAsUser', 'spec.securityContext.seLinuxOptions']\n",
    "columns = list(set(pods_sc_df.columns) & set(desired_columns))\n",
    "pods_sc_df_formatted = pods_sc_df[columns].rename(columns={'metadata.name': 'Pod Name', \n",
    "                     'metadata.namespace': 'Namespace',\n",
    "                     'spec.securityContext.runAsNonRoot': 'runAsNonRoot',\n",
    "                     'spec.securityContext.runAsGroup': 'runAsGroup',\n",
    "                     'spec.securityContext.runAsUser': 'runAsUser',\n",
    "                     'spec.securityContext.seLinuxOptions': 'seLinuxOptions'}).sort_index(axis=1)\n",
    "        \n",
    "unwanted_columns = ['Namespace', 'Pod Name']\n",
    "columns = [x for x in list(pods_sc_df_formatted.columns) if x not in unwanted_columns]\n",
    "display(pods_sc_df_formatted\n",
    "        .dropna(thresh=3)\n",
    "        .style.format(precision=0).applymap(highlight_not_na, subset=pd.IndexSlice[:, columns]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "034b7fe8",
   "metadata": {},
   "source": [
    "Due to the potential overwhelming output that the `kubectl` command could return, the output had been parsed to return only values that are not `NA`. Check against the following documentation to determine if these values are of concern.\n",
    "\n",
    "`runAsNonRoot` - Indicates that the container must run as a **non-root** user\n",
    "\n",
    "`runAsGroup`\n",
    "- GID to run the entrypoint of the container process\n",
    "- Uses runtime default if unset\n",
    "- Often set up in conjunction with volume mounts containing files that have the same ownership IDs\n",
    "- In GKE, it is normal for `event-exporter-gke`, `konnectivity-agent` and `konnectivity-agent-autoscaler` to have `runAsGroup` value of `1000`\n",
    "\n",
    "`runAsUser`\n",
    "- UID to run the entrypoint of the container process\n",
    "- Defaults to user specified in image metadata if unspecified\n",
    "- Enables the viewing of environment variables or file descriptors of processes with the specified UID\n",
    "- Often set up in conjunction with volume mounts containing files that have the same ownership IDs\n",
    "- Check `/etc/passwd` of host/node to map uid to username\n",
    "- In GKE, it is normal for `event-exporter-gke`, `konnectivity-agent` and `konnectivity-agent-autoscaler` to have `runAsUser` value of `1000`\n",
    "\n",
    "`seLinuxOptions`\n",
    "- SELinux is a policy driven system to control access to apps, processes and files on a Linux system\n",
    "- Implements the Linux Security Modules framework in the Linux kernel\n",
    "- Based on the concept of labels - it applies these labels to all the elements in the system which group elements together\n",
    "- Labels are also known as the security context (not to be confused with the Kubernetes `securityContext`)\n",
    "- Labels consist of a user, role, type, and an optional field level in the format `user:role:type:level`\n",
    "- SELinux then uses policies to define which processes of a particular context can access other labelled objects in the system\n",
    "- SELinux can be strictly enforced, in which case access will be denied, or it can be configured in permissive mode where it will log access\n",
    "- In containers, SELinux typically labels the container process and the container image in such a way as to restrict the process to only access files within the image\n",
    "- Changing the SELinux labeling for a container could potentially allow the containerized process to escape the container image and access the host filesystem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "122650ba",
   "metadata": {},
   "source": [
    "## Get Containers' Security Context (Precedence over Pods')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5801bcb6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def highlight_not_na(value):\n",
    "    if pd.isna(value):\n",
    "        return None\n",
    "    else:\n",
    "        return 'color:white; background-color:purple'\n",
    "\n",
    "with open('./pods_sc.json') as infile:\n",
    "    ctrs_sc = json.load(infile)\n",
    "\n",
    "frames = list()\n",
    "for item in ctrs_sc['items']:\n",
    "    for ctr in item['spec']['containers']:\n",
    "        ctr_series = dict()\n",
    "        ctr_series['Namespace'] = item['metadata']['namespace']\n",
    "        ctr_series['Pod Name'] = item['metadata']['name']\n",
    "        ctr_series['Container Name'] = ctr['name']\n",
    "        if 'securityContext' in ctr:\n",
    "            securityContext = ctr['securityContext']\n",
    "            if 'privileged' in securityContext: ctr_series['privileged'] = securityContext['privileged']\n",
    "            if 'allowPrivilegeEscalation' in securityContext: ctr_series['allowPrivilegeEscalation'] = securityContext['allowPrivilegeEscalation']\n",
    "            if 'capabilities' in securityContext: ctr_series['capabilities'] = securityContext['capabilities']\n",
    "            if 'procMount' in securityContext: ctr_series['procMount'] = securityContext['procMount']\n",
    "            if 'readOnlyRootFilesystem' in securityContext: ctr_series['readOnlyRootFilesystem'] = securityContext['readOnlyRootFilesystem']\n",
    "            if 'runAsGroup' in securityContext: ctr_series['runAsGroup'] = securityContext['runAsGroup']\n",
    "            if 'runAsNonRoot' in securityContext: ctr_series['runAsNonRoot'] = securityContext['runAsNonRoot']\n",
    "            if 'runAsUser' in securityContext: ctr_series['runAsUser'] = securityContext['runAsUser']\n",
    "            if 'seLinuxOptions' in securityContext: ctr_series['seLinuxOptions'] = securityContext['seLinuxOptions']\n",
    "            if 'windowsOptions' in securityContext: ctr_series['windowsOptions'] = securityContext['windowsOptions']   \n",
    "        ctr_series = pd.Series(ctr_series)\n",
    "        frames.append(ctr_series)\n",
    "ctrs_sc_df = pd.DataFrame(frames)\n",
    "\n",
    "unwanted_columns = ['Namespace', 'Pod Name', 'Container Name']\n",
    "columns = [x for x in list(ctrs_sc_df.columns) if x not in unwanted_columns]\n",
    "display(ctrs_sc_df\n",
    "        .dropna(thresh=4)\n",
    "        .style.format(precision=0).applymap(highlight_not_na, subset=pd.IndexSlice[:, columns]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90b47ab7",
   "metadata": {},
   "source": [
    "Due to the overwhelming output that the `kubectl` command could return, the output had been parsed to return only values that are not `NA`. Amongst the displayed output are pods in `kube-system` namespace which come with the GKE cluster by default and they can be ignored. For others, check against the following to determine if the values are of concern. \n",
    "\n",
    "`privileged`\n",
    "- Runs container in privileged mode\n",
    "- Processes in privileged containers are essentially equivalent to root on the node/host\n",
    "- Provides access to `/dev` on the host, which enables the mounting of the node/host filesytem to the privileged pod\n",
    "    - But provides a limited view of the filesystem - files that require privilege escalation (e.g. to root) are not accessible\n",
    "- Enables multiple options to gaining RCE with root privileges on the node/host\n",
    "\n",
    "`allowPrivilegeEscalation`\n",
    "- Controls whether a process can gain more privileges than its parent process\n",
    "- This bool directly controls if the `no_new_privs` flag will be set on the container process\n",
    "- Always `true` when the container is: 1) run as `privileged` 2) has `CAP_SYS_ADMIN`\n",
    "\n",
    "`capabilities`\n",
    "- Kernel level permissions that allow for more granular controls over kernel call permissions than simply running as root\n",
    "- Capabilities include things like the ability to change file permissions, control the network subsystem, and perform system-wide administration functions\n",
    "- Can be configured to `drop` or `add` capabilities\n",
    "\n",
    "`procMount`\n",
    "- By default, container runtimes mask certain parts of the `/proc` filesystem from inside a container in order to prevent potential security issues\n",
    "- However, there are times when access to those parts of `/proc` is required; particularly when using nested containers as is often used as part of an in-cluster build process\n",
    "- There are only two valid options for this entry:\n",
    "    - `Default`, which maintains the standard container runtime behavior, or\n",
    "    - `Unmasked`, which removes all masking for the /proc filesystem.\n",
    "\n",
    "`readOnlyRootFilesystem`\n",
    "- Default is `false` (represented by `nan` in the output)\n",
    "- If `true`, limits the actions that an attacker can perform on the container filesystem\n",
    "\n",
    "`windowsOptions`\n",
    "- Windows specific settings applied to all containers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5072cf74",
   "metadata": {},
   "source": [
    "## Check Pods' `hostPID`, `hostIPC`, `hostNetwork` Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "261e2d7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods -A --field-selector=metadata.namespace!=kube-system \\\n",
    "    -o custom-columns=Name:.metadata.name,Namespace:.metadata.namespace,HostPID:.spec.hostPID,HostIPC:.spec.hostIPC,HostNetwork:.spec.hostNetwork"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb985d68",
   "metadata": {},
   "source": [
    "`hostPID`\n",
    "* **Unable** to get privileged code execution on the host directly with only `hostPID: true`\n",
    "* If `true`, possible options for attacker\n",
    "    * View processes on host, including processes running in each pod\n",
    "    * View environment variables for each pod on the host (which may contain credentials)\n",
    "        * Applies only to processes running within pods that share the same UID as the `hostPID` pod\n",
    "        * To get the environment variables from processes that do not share the same UID, `hostPID` pod needs to run with the `runAsUser` set to the desired UID\n",
    "    * View file descriptors for each pod on the host (which may contain credentials)\n",
    "        * Permissions about environment variables above applies here as well\n",
    "    * Kill process on the node\n",
    "\n",
    "`hostIPC`\n",
    "* **Unable** to get privileged code execution on the host directly with only `hostIPC: true`\n",
    "* If any process on the host or any processes in a pod uses the host’s inter-process communication mechanisms (shared memory, semaphore arrays, message queues, etc), these mechanisms can be read or written to\n",
    "* If `true`, possible options for attacker\n",
    "    * Access data used by any pods that also use the host’s IPC namespace by inspecting `/dev/shm`\n",
    "        * `/dev/shm` is shared between any pod with `hostIPC: true` and the host\n",
    "        * Look for any files in this shared memory location\n",
    "    * Inspect existing IPC facilities - Check to see if any IPC facilities are being used with `/usr/bin/ipcs -a`\n",
    "\n",
    "`hostNetwork`\n",
    "* **Unable** to get privileged code execution on the host directly with only `hostNetwork: true`\n",
    "* If `true`, possible options for attacker\n",
    "    * Sniff traffic - Use tcpdump to sniff unencrypted traffic on any interface on the host\n",
    "    * Access services bound to localhost - Can reach services that only listen on the host’s loopback interface or that are otherwise blocked by network policies\n",
    "    * Bypass network policy - Pod would be bound to the host’s network interfaces and not the pods/namspaces’"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad966a01",
   "metadata": {},
   "source": [
    "## Check Pods' `hostpath` Config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0481e0af",
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get pods -A --field-selector=metadata.namespace!=kube-system \\\n",
    "    -o custom-columns=Name:.metadata.name,Namespace:.metadata.namespace,HostPath:.spec.volumes[].hostPath"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdcf13c6",
   "metadata": {},
   "source": [
    "* No results returned if there are no pods with `hostpath` configured\n",
    "* If the administrator had not limited what can be mounted, the entire host’s filesystem can be mounted\n",
    "* Provides read/write access on the host’s filesystem (limited to what the administrator defined)\n",
    "* If configured, possible options for attacker\n",
    "    * Look for kubeconfig files on the host filesystem (may find a cluster-admin config with full access to everything)\n",
    "        * **Not applicable** to GKE as GKE by default **DOES NOT** store kubeconfig files (i.e. `.kube/config`) on the node hosting the pod\n",
    "    * Add persistence \n",
    "        * Add own SSH key\n",
    "        * Add own CRON job\n",
    "    * Crack hashed passwords in `/etc/shadow`\n",
    "* Mount point can be found with\n",
    "    * `kubectl describe pod hostpath-exec-pod | sed -ne '/Mounts/,/Conditions/p'`"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e774977668b7c0ae8309835a5187aa7fbf7669e7d0bb59755bc63e573643edcd"
  },
  "kernelspec": {
   "display_name": "Python 3.7.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}