1. Overview

"Crash Loop Back-off" error message is more relevant to orchestrator type deployment such as Kubernetes or Openshift. There can be many reasons for a pod to result in this state. In this article, we cover one of the reasons for this error message to appear on Console deployment - setting lower cgroup/cpu limits. When fewer than 100 Defenders are connected, Console requires 1GB of RAM and 10GB of storage. Our recommendation is to meet default system requirements and don’t set additional cpu or cgroup limits on Console. Frequent API queries and large data set processing can result in slowing down Console and in some cases cause OOM if lower limits are applied to it.

2. Error messages

  1. Check the twistlock pod inside your Kubernetes/Openshift environment

    $ kubectl get pods -n twistlock

    Where 'twistlock' is the namespace in which Prisma Cloud pods are deployed.

  2. If using UI such as in Openshift platform, a Crash Loop Back-off error message would look like this:

    crash loop back off

3. Steps to confirm the issue

  1. oc / kubectl logs on the pod would show error messages like this:

    Exit code 137: OOM killed
  2. Example yaml configuration that sets limits on cpu:

    $ kubectl describe pods -n twistlock

    Look for "limits" section in the yaml:

    limits:
      - type: "Container"
        default:
          cpu: "500m"
          memory: "1024Mi"
        defaultRequest:
          cpu: "500m"
          memory: "1024Mi"

4. Troubleshooting steps

Remove custom limits on Console deployment. Completely remove the "limits" section from Prisma Cloud Console yaml.

resources: # See: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-requests-are-scheduled
          limits:
            memory: "512Mi"
            cpu: "900m"
          requests:
            cpu: "256m"

4.1. Additional information:

In Openshift, resource limits can be scoped to projects. For more information, refer to this article from Openshift: https://docs.openshift.com/container-platform/3.10/dev_guide/compute_resources.html

4.2. Prisma Cloud version

Applies to all versions

5. Cautionary notes

If this is a universal setting for all containers, you may want to check with your system admin before making changes to it. For any questions with making above changes, contact Prisma Cloud Support.