1. Overview

The Kubernetes auditing system records the activities of users, administrators, and other components, that have affected the cluster. Prisma Cloud can ingest, analyze, and alert on security-relevant events. Write custom rules or leverage Prisma Cloud Labs prewritten rules to assess the incoming audit stream and surface suspicious activity.

To get started, configure your cluster to forward audits to Prisma Cloud. Prisma Cloud supports both Google Kubernetes Engine (GKE) and self-managed clusters.

2. Rule library

Custom rules are stored in a central library, where they can be reused. Besides your own rules, Prisma Cloud Labs also distributes rules via the Intelligence Stream. These rules are shipped in a disabled state by default. You can review, and optionally apply them at any time.

Your Kubernetes audit policy is defined in Defend > Access > Kubernetes, and formulated from the rules in your library. There are four types of rules, but the only one relevant to the audit policy is the kubernetes-audit type. Custom rules are written and managed in Console under Defend > Runtime > Custom Rules with an online editor. The compiler checks for syntax errors when you save a rule.

3. Expression grammar

Expressions let you examine contents of a Kubernetes audit. Expressions have the following grammar:

expression: term (op term | in )*

term

integer | string | keyword | event | '(' expression ')' | unaryOp term

in

'(' integer | string (',' integer | string)*)?

op

and | or | > | < | >= | ⇐ | = | !=

unaryOp

not

keyword

startswith | contains

string

Strings must be enclosed in double quotes

integer

int

event

process, file system, or network

4. Kubernetes audit events

When Prisma Cloud receives an audit, it is assessed against your policy. Like all policies in Prisma Cloud, rule order is important. Rules are processed top to bottom, and processing stops at the first match. When a rule matches, an alert is raised.

Write rules to surface audits of interest. Rules are written with the jpath function. The jpath function extracts fields from JSON objects, which is the format of a Kubernetes audit. The extracted string can then be compared against strings of interest. The primary operators for jpath expressions are '=', 'in', and 'contains'. For non-trivial examples, look at the Prisma Cloud Lab rules.

The argument to jpath is a single string. The right side of the expression must also be a string. A basic rule with a single jpath expressions has the following form:

jpath("path.in.json.object") = "something"

Let’s look at some examples using the following JSON object as our example audit.

Example Kubernetes audit
{
   "user":{
      "uid":"1234",
      "username":"some-user-name",
      "groups":[
         "group1",
         "group2"
      ]
   },
   "stage":"ResponseComplete"
}

To examine user’s UID, use the following syntax. This expression evaluates to true.

jpath("user.uid") = "1234"

To examine username, use the following syntax:

jpath("user.username") = "some-user-name"

To examine the stage field, use the following syntax:

jpath("stage") = "ResponseComplete"

To examine the groups list field, use the following syntax:

jpath("user.groups") contains "group1"

Or alternatively:

jpath("user.groups") in ("group1","group2")

5. Integrating with self-managed clusters

Prisma Cloud supports any self-managed cluster based on Kubernetes version 1.13, or later. You can deploy clusters with any number of tools, including kubeadm.

Prerequisites: You’ve already deployed a Kubernetes cluster.

5.1. Configure the API server

Configure the API server.

  1. Open /etc/kubernetes/manifests/kube-apiserver.yaml for editing.

  2. Add the following flags. Changes to kube-apiserver.yaml trigger the API server to restart.

    --audit-dynamic-configuration
    --feature-gates=DynamicAuditing=true
    --runtime-config=auditregistration.k8s.io/v1alpha1=true
  3. Verify the server has restarted with the right flags.

    $ ps -ef | grep kube-apiserver

5.2. Configure OpenShift 4.2+

OpenShift 4.2 uses Kubernetes 1.14 dynamic audit backend to configure webhook backends through an AuditSink API object.

  1. Authenticate to the OpenShift cluster as cluster-admin using the OC cli.

  2. Issue the command to patch the API server to allow the creation of dynamic audit backends.

    $ oc patch kubeapiserver cluster --type=merge -p '{"spec":{"unsupportedConfigOverrides":{"apiServerArguments":{"audit-dynamic-configuration":["true"],"feature-gates":["DynamicAuditing=true"],"runtime-config":["auditregistration.k8s.io/v1alpha1=true"]}}}}'

5.3. Configure your cluster to forward audits to Prisma Cloud

Configure your cluster to forward audits to Prisma Cloud.

  1. Open Console.

  2. Go to Defend > Access > Kubernetes.

  3. Set Kubernetes auditing to Enabled.

  4. Click Go to settings.

    1. Set Deployment type to Default.

    2. Copy the webhook URL. This where your cluster will send audits.

  5. Configure the webhook in the cluster with an AuditSink object.

    To route audits over HTTP: Create a file named auditsink.yaml. Paste the following listing into it. Replace WEBHOOK-URL with the URL you copied from Prisma Cloud Console.

    apiVersion: auditregistration.k8s.io/v1alpha1
    kind: AuditSink
    metadata:
      name: twistlock-sink
    spec:
      policy:
        level: Request
        stages:
        - ResponseComplete
      webhook:
        throttle:
          qps: 10
          burst: 15
        clientConfig:
          url: "WEBHOOK-URL"

    To route audits over HTTPS: Create a file named auditsink.yaml. Paste the following listing into it. Replace WEBHOOK-URL with the URL you copied from Prisma Cloud Console. Replace CA-BUNDLE with a PEM-encoded CA bundle, which the cluster can use to validate Prisma Cloud’s certificate.

    apiVersion: auditregistration.k8s.io/v1alpha1
    kind: AuditSink
    metadata:
      name: twistlock-sink
    spec:
      policy:
        level: Request
        stages:
        - ResponseComplete
      webhook:
        throttle:
          qps: 10
          burst: 15
        clientConfig:
          url: "WEBHOOK-URL"
          caBundle: CA-BUNDLE
  6. Create the AuditSink object in your cluster.

    $ kubectl apply -f auditsink.yaml

    Your cluster now forwards audits to Prisma Cloud Console.

6. Integrating with GKE

On GKE, the master node isn’t accessible, so you cannot directly configure the API server with the --audit-dynamic-configuration flag to push audits to Prisma Cloud. Instead, Prisma Cloud retrieves audits from Stackdriver, polling it every 10 minutes for new data.

Note that there can be some delay between the time an event occurs in the cluster and when it appears in Stackdriver. Due to Twistock’s polling mechanism, there’s another delay between the time an audit arrives in Stackdriver and it appears in Prisma Cloud.

For testing purposes, you might not want to wait for the 10 minute polling period to see audits in Prisma Cloud. After setting up the integation in Prisma Cloud by providing your GCP credentials, you can force Prisma Cloud to immediately poll Stackdriver by disabling then re-enabling the Kuberenetes audit feature in Defend > Access > Kubernetes.

Prisma Cloud supports GKE clusters version 1.11.6-gke.3, or later.

Prerequisites: You’ve created a service account with one of the following authorization scopes:

  • https://www.googleapis.com/auth/logging.read

  • https://www.googleapis.com/auth/logging.admin

  • https://www.googleapis.com/auth/cloud-platform.read-only

  • https://www.googleapis.com/auth/cloud-platform

  1. Open Console.

  2. Go to Defend > Access > Kubernetes.

  3. Set Kubernetes auditing to Enabled.

  4. Click Go to settings.

    1. Set Deployment type to GKE.

    2. Select a GCP credential with the minimum authorization scope described in the prerequisites. If there are no accounts to select, add one to the credentials store.

    3. (Optional) Set an advanced filter to reduce the amount of data transferred from Stackdriver. If your project has more than one cluster, consider filtering my cluster name with resource.labels.cluster_name="YOUR_CLUSTER_NAME".

      Do not use the resource.type or timestamp filters because Prisma Cloud uses them internally.

    4. Click Save.

7. CA bundle

If you’re sending audit data to Prisma Cloud’s webhook over HTTPS, you must specify a CA bundle in the AuditSink object.

If you’ve customized Console’s certificate, you can get a copy from Manage > Authentication > System-certificates > TLS certificate for Console. Paste the certificate into a file named server-cert.pem, then run the following command:

$ openssl base64 -in server-cert.pem -out base64-output -A

In the AuditSingle object, set the value of caBundle to the contents of the base64-output file.

8. Testing your setup

Write a new rule, or select a prewritten rule from the inventory, and add it your audit policy. This setup installs a rule that fires when privileged pods are created in the cluster.

  1. Open Console, and go to Defend > Access > Kubernetes.

  2. Add a Prisma Cloud Labs prewritten rule.

    1. Click Select rules.

    2. If you’re integrated with a managed cluster, select Prisma Cloud Labs - Privileged pod creation. If you’re integrated with GKE, select Prisma Cloud Labs - GKE - privileged pod creation.

      There are separate rules for standard Kubernetes and GKE because the structure of the audits are different. Therefore, the logic for parsing the audit JSON is different.
    3. Click Save.

  3. Create a pod deployment file named priv-pod.yaml, and enter the following contents.

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        securityContext:
          privileged: true
  4. Create the privileged pod.

    $ kubectl apply -f priv-pod.yaml
  5. Verify an audit was created.

    Go to Monitor > Events, and select the Kubernetes Audits filter.

    kubernetes auditing

    If you’re integrated with GKE, and you don’t see the audit, force Prisma Cloud to poll Stackdriver immediately by disabling then re-enabling the feature.