Google Cloud Run Authentication Service-to-Service

Question

I have two services (APIs) deployed on GCP Cloud Run. Call them service-one.myDomain.com and service-two.myDomain.com. I would like service-one to be authenticated in calling service-two independently of what any user is doing.

I've read and implemented the instructions from GCP Cloud Run docs on Authenticating service-to-service (https://cloud.google.com/run/docs/authenticating/service-to-service) but service-one.myDomain.com is unsuccessful in calling service-two.myDomain.com receiving a 401:Unauthorized response.

Any thoughts on how to get service-one to successfully call service-two?

Here's my setup:

IAM and service accounts:

On google IAM, I created two service accounts and granted them both the "Cloud Run Invoker" (roles/run.invoker) role: service-one@myproject.iam.gserviceaccount.com service-two@myproject.iam.gserviceaccount.com

Inside Cloud Run I changed the service account from the "Default compute service account" to the service accounts I created. I assigned service-one@myproject.iam.gserviceaccount.com for service-one.myDomain.com and service-two@myproject.iam.gserviceaccount.com for service-two.myDomain.com

OIDC Auth token:

In service-one.myDomain.com I make a call to the metadata server to get a token (jwt) from the following url: http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://service-two.myDomain.com with a request header set as {'Metadata-Flavor': 'Google'} The request is successful and the token I receive is decoded to have the following payload:

{
  "alg": "RS256",
  "kid": "9cef5340642b157fa8a4f0d874fe7543872d82db",
  "typ": "JWT"
}

{
  "aud": "https://service-two.mydomain.com",
  "azp": "100959068407876085761",
  "email": "service-one@myproject.iam.gserviceaccount.com",
  "email_verified": true,
  "exp": 1572806540,
  "iat": 1572802940,
  "iss": "https://accounts.google.com",
  "sub": "100953168404568085761"
}

Http request:

Using the token I make a request from service-one.myDomain.com to an http endpoint on service-two.myDomain.com. I set the request header with {'Authorization': 'Bearer {token}'} ({token} is value of token).

Http Response:

The response is a 401 Unauthorized and my logs show the response headers to include:

{'WWW-Authenticate': 'Bearer error="invalid_token" error_description="The access token could not be verified"'}

With a content of:

"
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>401 Unauthorized</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Unauthorized</h1>
<h2>Your client does not have permission to the requested URL <code>/health</code>.</h2>
<h2></h2>
</body></html>
"

I'm stumped.... any ideas on what I'm missing to get service-one to authenticate to service-two?

Can you perform this command: gcloud beta run services get-iam-policy <service-two> (replace service-two by the real name)? And also this command gcloud iam service-accounts get-iam-policy <service-one@myproject.iam.gserviceaccount.com> (again with the right replacement)? Final question, do you use custom domain ? — guillaume blaquiere
So I feel like a bonehead, @guillaumeblaquiere your comment lead me to the answer. I am using custom domains. It does say in the documentation that custom domains aren't supported. After switching out my custom domain for the cloud run generated url, service-one was able to successfully make a request to service-two. — teaMonkeyFruit

teaMonkeyFruit teaMonkeyFruit · Accepted Answer · 2019-11-04T20:22:36

The answer was to use the gcp cloud run generated Url as the audience in the OIDC token request. And relatedly the "aud" field in the jwt.

My discovery was that Service-to-Service authentication in cloud run does not support custom domains (myDomain.com). I was using my custom domain.

(I feel like a bonehead) thanks @guillaumeblaquiere