Argo Workflows pods missing cpu/memory resources

Question

I'm running into a missing resources issue when submitting a Workflow. The Kubernetes namespace my-namespace has a quota enabled, and for whatever reason the pods being created after submitting the workflow are failing with:

pods "hello" is forbidden: failed quota: team: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

I'm submitting the following Workflow,

apiVersion: "argoproj.io/v1alpha1"
kind: "Workflow"
metadata:
  name: "hello"
  namespace: "my-namespace"
spec:
  entrypoint: "main"
  templates:
  - name: "main"
    container:
      image: "docker/whalesay"
      resources:
        requests:
          memory: 0
          cpu: 0
        limits:
          memory: "128Mi"
          cpu: "250m"

Argo is running on Kubernetes 1.19.6 and was deployed with the official Helm chart version 0.16.10. Here are my Helm values:

controller:
  workflowNamespaces:
  - "my-namespace"
  resources:
    requests:
      memory: 0
      cpu: 0
    limits:
      memory: 500Mi
      cpu: 0.5
  pdb:
    enabled: true
  # See https://argoproj.github.io/argo-workflows/workflow-executors/
  # docker container runtime is not present in the TKGI clusters
  containerRuntimeExecutor: "k8sapi"
workflow:
  namespace: "my-namespace"
  serviceAccount:
    create: true
  rbac:
    create: true
server:
  replicas: 2
  secure: false
  resources:
    requests:
      memory: 0
      cpu: 0
    limits:
      memory: 500Mi
      cpu: 0.5
  pdb:
    enabled: true
executer:
  resources:
    requests:
      memory: 0
      cpu: 0
    limits:
      memory: 500Mi
      cpu: 0.5

Any ideas on what I may be missing? Thanks, Weldon

Update 1: I tried another namespace without quotas enabled and got past the missing resources issue. However I now see: Failed to establish pod watch: timed out waiting for the condition. Here's what the spec looks like for this pod. You can see the wait container is missing resources. This is the container causing the issue reported by this question.

spec:
  containers:
  - command:
    - argoexec
    - wait
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ARGO_CONTAINER_RUNTIME_EXECUTOR
      value: k8sapi
    image: argoproj/argoexec:v2.12.5
    imagePullPolicy: IfNotPresent
    name: wait
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /argo/podmetadata
      name: podmetadata
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-v4jlb
      readOnly: true
  - image: docker/whalesay
    imagePullPolicy: Always
    name: main
    resources:
      limits:
        cpu: 250m
        memory: 128Mi
      requests:
        cpu: "0"
        memory: "0"

Harsh Manvar Harsh Manvar · Accepted Answer · 2021-05-26T18:11:23

try deploying the workflow on another namespace if you can, and verify if it's working or not.

if you can try with removing the quota for respective namespace.

instead of quota you can also use the

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limit-range
spec:
  limits:
  - default:
      memory: 512Mi
      cpu: 250m
    defaultRequest:
      cpu: 50m
      memory: 64Mi
    type: Container

so any container have not resource request, limit mentioned that will get this default config of 50m CPU & 64 Mi Memory.

https://kubernetes.io/docs/concepts/policy/limit-range/

Argo Workflows pods missing cpu/memory resources

1 Answers