Cloud Run Service requests always fail with 401 when using JWT returned for another service. The same request works with JWT returned for end-user

Question

I have an app consisting of multiple Cloud Run Services. Each service is open only to a specific set of other services.

Let's say I have 2 services with URLs https://api-service-123.run.app and https://data-provider-service-123.run.app. The first service also has a custom URL https://api.my.domain.

api-service needs access to data-provider-service. So, following the documentation, I created two per-service user-managed service-accounts api-service-account@domain and data-provider-service-account@domain. I added api-service-account@domain into data-provider-service with Cloud Run Invoker role.

Now, I have trouble accessing the account with ID Token returned from Google. I tried several ways. Firstly, I utilized slightly adjusted code from the docks

export const getToken = async (url: string, targetAUD?: string) => {
    const auth = new GoogleAuth();
    const request  = async () => {
        if (!targetAUD) {
            targetAUD = new URL(url).origin;
        }
        console.info(`request ${url} with target audience ${targetAUD}`);
        const client = await auth.getIdTokenClient(targetAUD);
        const res = await client.request({url});
        console.info(res.data);
        return res.data;
    }
    try {
       return await request();
    } catch (err) {
        console.error(err);
    }
}

When the function above is called as getToken('http://data-provider-service-123.run.app');, the request fails with 401 Unauthorized. I also tried to call it as getToken('https://data-provider-service-123.run.app'); and got a response 403 Forbidden. The following entry is added to data-provider-service logs for each such request

    {
    httpRequest: {9}insertId: "60fcb7e0000c12346fe37579"
    logName: "projects/my-project/logs/run.googleapis.com%2Frequests"
    receiveTimestamp: "2021-07-25T01:01:20.806589957Z"
    resource: {2}
    severity: "WARNING"
    textPayload: "The request was not authorized to invoke this service. Read more at 
     https://cloud.google.com/run/docs/securing/authenticating"
    timestamp: "2021-07-25T01:01:20.800918Z"
    trace: "projects/my-project/traces/25b3c13890aad01234829711d4e9f25"
    }

I also tried to obtain and use JWT from compute metadata server (also a way advertised in the docs) by running curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=http://data-provider-servive-123.run.app" \ -H "Metadata-Flavor: Google".

When I execute this code in Cloud Shell, I get JWT that looks like this

{
"iss": "accounts.google.com",
"azp": "123456789-9r9s1c4alg36erliucho9t52n12n6abc.apps.googleusercontent.com",
"aud": "123456789-9r9s1c4alg36erliucho9t52n12n6abc.apps.googleusercontent.com",
"sub": "106907196154567890477",
"email": "[email protected]",
"email_verified": true,
"at_hash": "dQpctkQE2Sy1as1qv_w",
"iat": 1627173901,
"exp": 1627177501,
"jti": "448d660269d4c7816ae3zd45wrt89sb9f166452dce"
}

Then I make a request using postman to https://data-provider-service/get-data with a header Authorization: "Bearer <the_jwt_from_above>" and everything works fine

However, if I make the same request from my api-service, the JWT returned is

{
"aud": "http://data-provider-service-123.run.app",
"azp": "111866132465987679716",
"email": "api-service-account@domain",
"email_verified": true,
"exp": 1627179383,
"iat": 1627175783,
"iss": "https://accounts.google.com",
"sub": "111866132465987679716"
}

If I make the same request with the postman and place this JWT into the Authorization header, the 401 Unauthorized is returned.

I have spent this week trying to solve this issue. I triple-checked the permissions, redeployed the services several times, but nothing helps.

The problem is the incorrect aud (audience). Compare the working token with the failing one. — John Hanley
@JohnHanley, I'm not sure, but I guess the audience from the working token refers to the scope of the entire project, as I get it from Shell while having the Owner permissions. Actually, I wonder why the working JWT has a broader aud than requested, but it's another question. I guess this quote from docs supports my thoughts "Fetch a Google-signed ID token with the audience claim (aud) set to the URL of the receiving service." — AntoineRNT
The aud for the working Identity Token is "aud": "http://data-provider-service-123.run.app",. Do you notice what that represents? That is the service that you are calling. If the audience is wrong, permission denied. — John Hanley
@JohnHanley, Conversely, the working token obtained from Shell is 123456789-9r9s1c4alg36erliucho9t52n12n6abc.apps.googleusercontent.com. And the aud of not-working token matches the data-provider-service, yet it returns 401. — AntoineRNT
Simplify your question to a) show the code; b) show the decoded token; c) show the exact call using the token using a tool such as curl; d) show the error. Your other code that works is not helpful. Something is wrong in the information you have included which is confusing/masking the real problem. In other words, start over with a concise problem that I can reproduce. — John Hanley

AntoineRNT AntoineRNT · Accepted Answer · 2021-07-25T06:38:02

I changed http to https in the URL of the target service when asking computing metadata server, and looks like it solved the problem. Requesting token like this with google-auth-library still fails with 403 error.

I'll update that answer if the solution is sustainable.

Cloud Run Service requests always fail with 401 when using JWT returned for another service. The same request works with JWT returned for end-user

1 Answers