GKE Pod not scheduled in different namespace

Question

I want to deploy a monstache deployment in my already existing namespace "test-namespace". When I deploy it in "default" namespace it works but when I deploy it in "test-namespace" the pod does not schedule.

kubectl get pods -n test-namespace monstache-74466dc7-5tnrr -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
monstache-74466dc7-5tnrr   0/1     Pending   0          57m   <none>   <none>   <none>           <none>

and:

kubectl describe pods -n test-namespace monstache-74466dc7-5tnrr
Name:           monstache-74466dc7-5tnrr
Namespace:      test-namespace
Priority:       0
Node:           <none>
Labels:         app=monstache
                pod-template-hash=74466dc7
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/monstache-74466dc7
Containers:
  monstache:
    Image:      rwynn/monstache:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/monstache
      -f
      /app/monstache.test.config.toml
    Environment:
      MONSTACHE_DIRECT_READ_NS:    xxx.XXX
      MONSTACHE_CHANGE_STREAM_NS:  xxx.XXX
      MONSTACHE_MONGO_URL:         mongodb://xxx?replicaSet=rs0
      MONSTACHE_ES_USER:           elastic
      MONSTACHE_ES_PASS:           XXX
    Mounts:
      /app from monstache-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qmcwm (ro)
Volumes:
  monstache-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      monstache-config
    Optional:  false
  default-token-qmcwm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qmcwm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

and:

kubectl get events -n test-namespace
LAST SEEN   TYPE     REASON              OBJECT                          MESSAGE
55m         Normal   SuccessfulCreate    replicaset/monstache-74466dc7   Created pod: monstache-74466dc7-snrdb
55m         Normal   SuccessfulCreate    replicaset/monstache-74466dc7   Created pod: monstache-74466dc7-5tnrr
55m         Normal   ScalingReplicaSet   deployment/monstache            Scaled up replica set monstache-74466dc7 to 1
55m         Normal   ScalingReplicaSet   deployment/monstache            Scaled up replica set monstache-74466dc7 to 1

and:

This is my monstache deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: monstache
  namespace: test-namespace
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: monstache
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: monstache
    spec:
      containers:
      - command:
        - /bin/monstache
        - -f
        - /app/monstache.test.config.toml
        env:
        - name: MONSTACHE_DIRECT_READ_NS
          value: xxx.xxx
        - name: MONSTACHE_CHANGE_STREAM_NS
          value: xxx.xxx
        - name: MONSTACHE_MONGO_URL
          value: mongodb://mongodb-service:27017/xxx?replicaSet=rs0
        - name: MONSTACHE_ES_USER
          value: elastic
        - name: MONSTACHE_ES_PASS
          value: XXXX
        image: rwynn/monstache:latest
        imagePullPolicy: Always
        name: monstache
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /app
          name: monstache-config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: monstache-config
        name: monstache-config
---
apiVersion: v1
data:
  monstache.test.config.toml: |
    resume = true

    gzip = true

    elasticsearch-urls = ["https://elasticsearch:9200"]

    elasticsearch-max-conns = 10

    elasticsearch-max-seconds = 1

    elasticsearch-max-docs = 1

    #namespace-regex = '*'

    verbose = false

    enable-http-server = true

    elasticsearch-validate-pem-file = false

    [[mapping]]
    namespace = "XXX.XXX"
    index = "XXX"
kind: ConfigMap
metadata:
  name: monstache-config
  namespace: test-namespace

Few more Things to know:

• I already have pods scheduled in that namespace
• I tried to delete the deployment an re-create
• I even created a new nodepool and tried to schedule the deployment there - also didn't work.
• I searched for a pod count limit and pod quota, and it does not conflict.
• I have 12 namespaces in that GKE cluster
• I have total 113 pods in that GKE cluster
• I have some successfully scheduled monstache deployments in other namespaces in that cluster.
• It happens in the 2 most recent namespaces I've created.

Any clues?

Answer 1

It was a bug. Re-deploying it with GKE version 1.17.14-gke.1200 solved the problem.