Setting up release pipeline of kubernetes+istio with terraform+helm gives forbidden error

Question

I have tried now so many times setting up this pipeline in Azure devops where I want to deploy a AKS cluster and place istio on top.

Deploying the AKS using Terraform works great.

After this I try to install istio using helm but the command I use gives forbidden error.

helm.exe install --namespace istio-system --name istio-init --wait C:\Istio\install\kubernetes\helm\istio

I used the local path since this was the only good way I could find for helm to find the istio chart i have on the build agent.

The error message

Error: release istio-init failed: clusterroles.rbac.authorization.k8s.io "istio-galley-istio-system" is forbidden: attempt to grant extra privileges: [{[*] [admissionregistration.k8s.io] [validatingwebhookconfigurations] [] []} {[get] [config.istio.io] [*] [] []} {[list] [config.istio.io] [*] [] []} {[watch] [config.istio.io] [*] [] []} {[get] [*] [deployments] [istio-galley] []} {[get] [*] [endpoints] [istio-galley] []}] user=&{system:serviceaccount:kube-system:tillerserviceaccount 56632fa4-55e7-11e9-a4a1-9af49f3bf03a [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found, clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "cluster-admin" not found, clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "system:discovery" not found, clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]]

The serviceaccount I use (system:serviceaccount:kube-system:tillerserviceaccount as you can see in error message) are configured using this rbac config:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tillerserviceaccount
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tillerbinding
roleRef:
  apiGroup: ""
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tillerserviceaccount
    namespace: kube-system

Still the error message says in the ruleResolutionErrors that it looks for cluster-admin but it is not found.

I even tried the extreme and set all service accounts as cluster admins to test:

kubectl create clusterrolebinding serviceaccounts-admins --clusterrole=cluster-admin --group=system:serviceaccounts

But even after that I get the same error with the same ruleResolutionErrors.

I am stuck and appriciate any help in what I can do differently.

pretty sure you need to run istio init first, not just run istio and call it istio init — 4c74356b41
I am pretty new to this.. When I downloaded the install pack for istio I got istio charts and istio-remote charts. no init. or you mean any other type of init. I am running like helm init to set up service account and such. — JohanSellberg
Reading up on github dokumentation the istio charts says: "The istio-init chart must be run to completion prior to install the istio chart." So ill download the init chart from there and run it first. — JohanSellberg
yeah, its in the same folder as istio, but called istio-init: C:\Istio\install\kubernetes\helm\istio-init - should be here. you should be using 1.1+ — 4c74356b41
Tried the istio-init now but got the forbidder error still. "clusterroles.rbac.authorization.k8s.io "istio-init-istio-system" is forbidden..." I can update the question with full error if needed. — JohanSellberg

4c74356b41 4c74356b41 · Accepted Answer · 2019-04-03T09:15:22

this is the role binding we are using in dev clusters:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tillerbinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tillerserviceaccount
    namespace: kube-system

edit: in this case the error was due to AKS created without RBAC.

Setting up release pipeline of kubernetes+istio with terraform+helm gives forbidden error

1 Answers