Istio 1.5.7 installation fails on kubernetes 1.18.6 with centos 7.8(3.10.0-1127.el7.x86_64) with calico network CNI

Question

I have installed fresh k8s 1.18.6 on multinode ( 12GB RAM & 4 CPU) on centos-7.8(3.10.0-1127.el7.x86_64) with docker version (19.03.6) & runc version 1.0.0-rc10.

I used calico(3.11.1) network plugin and till here everything worked fine. When i am trying to install istio 1.5.7 on it i am facing issue as below

Issue:

Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
error installer Failed to wait for resource: resources not ready after 10m0s: timed out waiting for the condition Deployment/istio-system/istiod

kubectl get pods -n istio-system NAME READY STATUS RESTARTS AGE grafana-5f6f8cbf75-lngb7 1/1 Running 0 41s istio-egressgateway-cbd86567c-5x6rk 0/1 ContainerCreating 0 44s istio-ingressgateway-799d5b8875-4ztq8 0/1 ContainerCreating 0 42s istio-tracing-9dd6c4f7c-vv64n 1/1 Running 0 41s istiod-b7d8f955b-mtqgb 0/1 CrashLoopBackOff 5 10m kiali-869c6894c5-pw7sm 1/1 Running 0 41s prometheus-7d697b95b-2rjvn 0/2 ContainerCreating 0 41s

istiod-pod-logs: info No certificates specified, skipping DNS certificate controller info CRD controller watching namespaces "" info Ingress controller watching namespaces "" warn Config Store &{0xc00020c6c0 cluster.local 0xc00026b1e0 0xc000795e00 0xc00079eea0 []} cannot track distribution in aggregate info Adding Kubernetes registry adapter info Service controller watching namespace "" for services, endpoints, nodes and pods, refresh 1m0s info JWT policy is first-party-jwt info Use self-signed certificate as the CA certificate info pkica Failed to get secret (error: Get https://10.96.0.1:443/api/v1/namespaces/istio-system/secrets/istio-ca-secret: dial tcp 10.96.0.1:443: i/o timeout), will create one Error: failed to create discovery service: enableCA: failed to create a self-signed Citadel: failed to create CA due to secret write error error pkica Failed to write secret to CA (error: Post https://10.96.0.1:443/api/v1/namespaces/istio-system/secrets: dial tcp 10.96.0.1:443: i/o timeout). Abort. error failed to create discovery service: enableCA: failed to create a self-signed Citadel: failed to create CA due to secret write error

From what I see in the logs you provided your cluster does not support third party JWT authentication. Could you try with --set values.global.jwtPolicy=first-party-jwt when you install istio? There is related part about that in istio documentation. Also worth to note that there is already istio version 1.8.2, consider installing the new version due to the improvements. — Jakub
Hi Jakub, Thank you for replying on my post. I tried with the flag as you suggested but i am still failing with the same issue. Its like this flag has no affect after adding it in command I used it as below= "istioctl manifest apply --set profile=demo --set values.global.jwtPolicy=first-party-jwt" — Shailendra Pratap Singh
That's weird, is there a chance you could add 4GB of ram to your setup? If I remember correctly istio requires at least 16GB of ram, maybe it's just crushing because of that. — Jakub

Jakub Jakub · Accepted Answer · 2021-02-01T07:06:32

I would start with checking if your cluster supports third party tokens, more about it here.

Because the properties of the first party token are less secure, Istio will default to using third party tokens. However, this feature is not enabled on all Kubernetes platforms.

If you are using istioctl to install, support will be automatically detected. This can be done manually as well, and configured by passing --set values.global.jwtPolicy=third-party-jwt or --set values.global.jwtPolicy=first-party-jwt.

To determine if your cluster supports third party tokens, look for the TokenRequest API. If this returns no response, then the feature is not supported:

kubectl get --raw /api/v1 | jq '.resources[] | select(.name | index("serviceaccounts/token"))'
{
    "name": "serviceaccounts/token",
    "singularName": "",
    "namespaced": true,
    "group": "authentication.k8s.io",
    "version": "v1",
    "kind": "TokenRequest",
    "verbs": [
        "create"
    ]
}

Then I would check the requirements, according to istio documentation you should have at least 16384 MB of memory and 4 CPUs to run istio.

There is another stackoverflow case where community member had same problem with 12GB of RAM.

I would also check the versions, as mentioned in documentation

Istio 1.5 has been tested with Kubernetes releases 1.14, 1.15, 1.16.

As you use Kubernetes 1.18.6 version, it might not work with Istio 1.5 version.

So I would recommend to go with this approach:

Add 4GB of RAM to your cluster,
Use newer version of istio,
Install istio with --set values.global.jwtPolicy=first-party-jwt.

Istio 1.5.7 installation fails on kubernetes 1.18.6 with centos 7.8(3.10.0-1127.el7.x86_64) with calico network CNI

2 Answers