Cloud Run: Service-to-Service 401 while curl works

Question

This is an issue I've been trying to figure out for a few days now.

Service-to-service communication always results in 401 for my Cloud Run applications.

For token retrieval in service I've been using this code snippet:

tokenURL := fmt.Sprintf("/instance/service-accounts/default/identity?audience=%s", c.serviceURL)

var err error
token, err = metadata.Get(tokenURL)
if err != nil {
    return nil, fmt.Errorf("metadata.Get: failed to query id_token: %+v", err)
}

I have also tried to supply a Service account key to my Cloud Run service and using the token that is returned from google.JWTAccessTokenSourceFromJSON to no avail.

The problem is that the JWT that is returned does work when used from cURL, but does not work when used in a service that runs on Cloud Run.

Both the Cloud Run service account and the external service account (for the key) has the roles/run.invoker IAM binding. And the audience is the Cloud Run issued service URL.

I have tried the following:

Using the metadata service for token retrieval (401 inside Cloud Run, but works when the token is used in cURL and Postman)
Using the JWTAccessTokenSourceFromJSON for token retrieval (401 inside Cloud Run, but works when the token is used in cURL and Postman)
Tried adding a delay before using issued token (in case the token is not propagated)
Tried manually pinging the Cloud Run service a few times with given token after the delay (using http.Client)

Nothing seems to work. Every request when run inside Cloud Run is returning 401 and the logs show The request was not authorized to invoke this service. Read more at https://cloud.google.com/run/docs/securing/authenticating. While using cURL or Postman I am able to access the service.

I've tried these methods to invoke the service:

tokenURL := fmt.Sprintf("/instance/service-accounts/default/identity?audience=%s", c.serviceURL)

var err error
token, err := metadata.Get(tokenURL)
if err != nil {
    return nil, fmt.Errorf("metadata.Get: failed to query id_token: %+v", err)
}

// First one (ping)
req, _ := http.NewRequest("GET", "{{serviceURL}}", nil)
req.Header.Set("Authorization", "Bearer: "+token)
l, _ := cr.Do(req)
m, _ := ioutil.ReadAll(l.Body)
logrus.Println(l.Header)
logrus.Println(string(m))

// RoundTripper
r.Header.Set("Authorization", "Bearer: "+token)
return c.r.RoundTrip(r)

I appreciate any answer.

Thanks!

How do you call your service in your code? How/where do you pass the token? — guillaume blaquiere
The token is passed in as Authorization: Bearer {token}. I’ve tested with invalid tokens and that returns a 403, but with a Google issued JWT returns 401 in Cloud Run while it works using cURL. In the code I’m using a RoundTripper that adds an Authorization header using req.Header.Set. I’ve also tried with an ordinary http.Request and sent it with http.Client — Mustafa Gezen
And when you use this same token, generated by Cloud Run, with curl it worked? — guillaume blaquiere
So, share all your code from token generation to request sending, the problem is somewhere in what you do! — guillaume blaquiere

Mustafa Gezen Mustafa Gezen · Accepted Answer · 2020-07-13T23:40:04

While posting code examples I just realized I added an extra colon(:) after Bearer which makes this request fail. Can't believe I spent days to solve this!

Cloud Run: Service-to-Service 401 while curl works

1 Answers