Azure Function with KEDA Not scaling out more than 1 pod

1
votes

Pods come down to 0 but it doesnt spin up more than 1 pod. Even if more than 5mgs are there in the subscription Updated code as per suggestions.

Added TriggerAuthentication and used connectionString as authentication. Tried to send around 25messages, but still pod is not scaling up.

Only one pod is handling all the messages. Do I need to update anything

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-servicebus-auth
spec:
  secretTargetRef:
    - parameter: connection
      name: mdsp-secret-infra
      key: service_bus_conn_str
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: xyz-test
  labels:
    app: xyz-test
spec:
  selector:
    matchLabels:
      app: xyz-test
  template:
    metadata:
      labels:
        app: xyz-test
    spec:
      initContainers:
        - name: mdsp-java-init
          image: aksacrname.azurecr.io/ops/mdsp-init-java:1.0
          imagePullPolicy: Always
          resources: {}
          volumeMounts:
            - name: azure
              mountPath: /mnt/azure
            - name: cert-storage
              mountPath: /certs
      volumes:
        - name: azure
          azureFile:
            shareName: certs
            secretName: mdsp-secret-iots-sa
            readOnly: true
        - name: cert-storage
          emptyDir: {}
        - name: "logging-volume-azure-file"
          persistentVolumeClaim:
            claimName: "pvc-azure-file-logging"
      containers:
        - name: xyz-test
          image: mdspaksdevacr.azurecr.io/iots/xyz-test:aksacrversion
          resources:
            limits:
              cpu: 3
              memory: 2Gi
            requests:
              cpu: 1
              memory: 1Gi
          imagePullPolicy: Always
          env:
            - name: mdsp_product_line
              value: "iiotservices"
            - name: mdsp_application_name
              value: "xyz-test"
            - name: eh_connection_string
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: LOGGING_EH_CON_STRING
            - name: log_base_path
              value: "/mnt/logging"
            - name: app.logging.console.enabled
              value: "true"
            - name: ai_instrumentation_key
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: APPINSIGHT_KEY
            - name: MDSP_HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: MDSP_HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: APPLICATION_NAME
              value: "xyz-test"
            - name: logging.level.com.siemens.mindsphere
              value: "INFO"
            - name: PRODUCTLINE_NAME
              value: "iiotServices"
            - name: environment
              value: "azEnvironment"
            - name: CONNECTION_TIMEOUT_MS
              value: "10000"
            - name: GATEWAY_URI
              value: "https://gateway.core.mindsphere.local"
            - name: UPDATES_SERVICE_BUS_TOPIC_NAME
              value: "internal-iiot-meta-data-updates-topic"
            - name: ASSET_CLIENTID
              value: asset
            - name: AM_BASE_URL
              value: "https://gateway.core.mindsphere.local/api/assetmanagement/v3"
            - name: CORE_AUTH_TOKEN_URL
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: OAUTH_TOKEN_URI
            - name: amServiceBusConnectionString
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: advs_servicebus_conn_str
            - name: feedbackServiceBusConnectionString
              value: "Endpoint=sb://iot-test-rc.servicebus.windows.net/;SharedAccessKeyName=Testkey;SharedAccessKey=test"
            - name: UPDATES_SERVICE_BUS_CONN_STR
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: iots_service_bus_conn_str
            - name: AZURE_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_ID
            - name: AZURE_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_KEY
            - name: AZURE_SUBSCRIPTION_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: SUBSCRIPTION_ID
            - name: AZURE_TENANT_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: TENANT_ID
            - name: KV_CLIENT_ID
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_ID
            - name: KV_CLIENT_SECRET
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-kv
                  key: KEYVAULT_CLIENT_KEY
            - name: JAVA_OPTS
              value: >-
                -Djavax.net.ssl.trustStore=/certs/cacerts
                -Xmx2500m
            - name: KEYVAULT_URI
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOTS_KEYVAULT_URI
            - name: COSMOS_URL
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOT_COSMOS_URI
            - name: default_key
              valueFrom:
                secretKeyRef:
                  name: mdsp-iots-secret-infra
                  key: iots_cosmos_db_conn_key
            - name: COSMOS_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOT_COSMOS_DB_AM
            - name: COSMOS_CONTAINER_NAME
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: IOT_COSMOS_CON_AM
            - name: default_secret_name
              valueFrom:
                secretKeyRef:
                  name: mdsp-secret-iots-tu
                  key: iots_bulk_data_stream
            - name: MINDSPHERE_SECURITY_OAUTH2_CLIENT_ASSET_ACCESSTOKENURI
              valueFrom:
                configMapKeyRef:
                  name: mdsp-iots-configmap
                  key: OAUTH_TOKEN_URI
            - name: MINDSPHERE_SECURITY_OAUTH2_CLIENT_ASSET_ID
              value: "iottest"


          volumeMounts:
            - name: cert-storage
              mountPath: /certs
            - name: "logging-volume-azure-file"
              mountPath: "/mnt/logging"
      imagePullSecrets:
        - name: mdsp-secret-iots
      dnsPolicy: ClusterFirst
      terminationGracePeriodSeconds: 30
---

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: xyz-test-scaler-dev
  labels:
    app: xyz-test
    deploymentName: xyz-test
spec:
  scaleTargetRef:
    deploymentName: xyz-test
  minReplicaCount: 0
  maxReplicaCount: 3
  pollingInterval: 1
  triggers:
  - type: azure-servicebus
    metadata:
      # Required
      topicName: am-removal-test
      subscriptionName: am-subs
      namespace: iot-sbus-rc
      # Optional, can use TriggerAuthentication as well
      connection: feedbackServiceBusConnectionString
      # This must be a connection string for a queue itself, and not a namespace level (e.g. RootAccessPolicy) connection string [#215](https://github.com/kedacore/keda/issues/215)
      # Optional
      queueLength: "2"
    authenticationRef:
      name: azure-servicebus-auth
---

Getting error :

Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric iots-dev/queueLength/&LabelSelector{MatchLabels:map[string]string{deploymentName: xyz,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get queueLength.external.metrics.k8s.io)
Events:
  Type     Reason                   Age                       From                       Message
  ----     ------                   ----                      ----                       -------
  Warning  FailedGetExternalMetric  2m17s (x2591 over 5h53m)  horizontal-pod-autoscaler  unable to get external metric iots-dev/queueLength/&LabelSelector{MatchLabels:map[string]string{deploymentName: xyz,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get queueLength.external.metrics.k8s.io)

After running command :

```

apiVersion: apiregistration.k8s.io/v1

166kind: APIService

167metadata:

168 annotations:

169 kubectl.kubernetes.io/last-applied-configuration: |

170 {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.external.metrics.k8s.io"},"spec":{"group":"external.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"custom-metrics-azure-apiserver","namespace":"custom-metrics"},"version":"v1beta1","versionPriority":100}}

171 creationTimestamp: "2020-09-15T15:48:18Z"

172 name: v1beta1.external.metrics.k8s.io

173 resourceVersion: "13008780"

174 selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io

175 uid: a6a72678-001f-4378-ba10-cbff7d8c0ad2

176spec:

177 group: external.metrics.k8s.io

178 groupPriorityMinimum: 100

179 insecureSkipTLSVerify: true

180 service:

181 name: custom-metrics-azure-apiserver

182 namespace: custom-metrics

183 port: 443

184 version: v1beta1

185 versionPriority: 100

186status:

187 conditions:

188 - lastTransitionTime: "2020-10-27T12:19:19Z"

189 message: service/custom-metrics-azure-apiserver in "custom-metrics" is not present

190 reason: ServiceNotFound

191 status: "False"

192 type: Available

1
azurekubernetesazure-functionskeda
what are logs of the keda operator pod? – paltaa
edited my question – sar
Well you can see that you miscunfigured the trigger auth – paltaa
@paltaa could you please elaborate – sar
ill post a working example i use with rabbitmq, sec – paltaa

1 Answers

1
votes

So, my working example of KEDA autoscaler with rabbitmq:

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: job-export-autoscaler
  namespace: thor
spec:
  scaleTargetRef:
    deploymentName: job-export
  pollingInterval: 5 # Time after each check
  minReplicaCount: 0   # Optional. Default: 0
  maxReplicaCount: 10 # Optional. Default: 100
  cooldownPeriod: 60 # Cooldown since last trigger to outoscale down
  triggers:
  - type: rabbitmq
    metadata:
      queueName: report-export
      queueLength: "1"
    authenticationRef:
      name: trigger-auth-rabbitmq

Now the triggerAuth mentioned in the comments:

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-rabbitmq
  namespace: thor
spec:
  secretTargetRef:
  - parameter: host
    name: rabbitmq-host-secret
    key: rabbitmq-connectionstring
---
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-host-secret
  namespace: thor
  labels:
    app: rabbitmq
data:
  rabbitmq-connectionstring: (base64 encoded rabbitmq connection string)

So, for using Azure service bus:

Documentation: https://keda.sh/docs/1.4/scalers/azure-service-bus/#authentication-parameters

Example:

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: azure-servicebus-auth
spec:
  podIdentity:
    provider: azure
---
apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    deploymentName: azure-servicebus-queue-function
  triggers:
  - type: azure-servicebus
    metadata:
      # Required: queueName OR topicName and subscriptionName
      queueName: functions-sbqueue
      # or
      topicName: functions-sbtopic
      subscriptionName: sbtopic-sub1
      # Required: Define what Azure Service Bus to authenticate to with Managed Identity
      namespace: service-bus-namespace
      # Optional
      queueLength: "5" # default 5
    authenticationRef:
        name: azure-servicebus-auth # authenticationRef would need either podIdentity or define a connection parameter

So, you are missing the TriggerAuthentication object with the connection string