I'm using the following stack for my backend:
- Cloud Endpoints
- Cloud Run Gateway with ESPv2 => check API keys and validate JWTs
- Cloud Run gRPC server implementation
- When a change of interest happen I publish a message on pubsub
Now, attached to that pubsub topic I have a push subscription that triggers a Cloud Run method with a service account and a specified audience.
Setup:
Now I need to configure the Cloud Run Gateway to validate the JWT that pubsub generates for the specified service account. This is done in the api_config.yaml as per documentation here: https://cloud.google.com/pubsub/docs/push#jwt_format
The issuer is https://accounts.google.com and the audience in the one I specified in the pubsub subscription.
authentication:
providers:
- id: gcloud
jwks_uri: https://www.googleapis.com/robot/v1/metadata/x509/[email protected]
issuer: https://accounts.google.com
audiences: project.clounrun.admin
rules:
- selector: project.v2.Events.*
requirements:
- provider_id: gcloud
The jwks for this service account can be found here:
https://www.googleapis.com/robot/v1/metadata/x509/SERVICE-ACCOUNT-ID
In my case:
https://www.googleapis.com/robot/v1/metadata/x509/[email protected]
I've activated debug logs for the ESPv2 and this is what I get when it tries to validate the JWT:
13:07:37.027 request headers complete (end_stream=false):\n\':authority\', \'project-gateway-v2-dev-cuvfttrlpq-de.a.run.app\'\n\':path\', \'/v2/sessions:event\'\n\':method\', \'POST\'\n\'content-type\', \'application/json\'\n\'authorization\', \'Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImQwNWVmMjBjNDUxOTFlZmY2NGIyNWQzODBkNDZmZGU1NWFjMjI5ZDEiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOiJ5aXRub3cuY2xvdW5ydW4uYWRtaW4iLCJhenAiOiIxMDk0NDIwMDAxNjU1ODQ3Nzc4MDciLCJlbWFpbCI6Ijg3NTUxNzUyMzgyNS1jb21wdXRlQGRldmVsb3Blci5nc2VydmljZWFjY291bnQuY29tIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImV4cCI6MTYwMzgwNzAwMSwiaWF0IjoxNjAzODAzNDAxLCJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJzdWIiOiIxMDk0NDIwMDAxNjU1ODQ3Nzc4MDcifQ.PwLvaBz-_dM3_5VjgzVlaueoRhacUE39FdFuVSfQI2w4V3OD79tIA6t_0cuvE1-kIpNrfDda5RSavVcs4SV4Y5P8AvW5jDtHdCELP3yb8HzVT9nCJlLac-v5ZKuv06syBN9F2Ve7VtZHHZOE2VS4B7uw0Q__1rIVzIllYWBYHkYGUBP2mZ3VhRw9VXARMr-EICanXfETe_MMfoKsX4202L_O4LffPdv16pA5hYtwzKi67gFYuubKI1XNkVQVatQieYQrhkz5jMyNyhVKy8ZY5a2UXagQL7erdsm-uPJo6ujoq0Yxtl8iKMdRv4XfrQSLyFZHCYdO6n2LHJle_FQzCQ\'\n\'content-length\', \'335\'\n\'accept\', \'application/json\'\n\'from\', \'[email protected]\'\n\'user-agent\', \'APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)\'\n\'x-cloud-trace-context\', \'c3027b54c6aad09d85ce75f6dcaf07d5/8778433264575104852\'\n\'x-forwarded-for\', \'66.249.82.169\'\n\'x-forwarded-proto\', \'https\'\n\'forwarded\', \'for=\"66.249.82.169\";proto=https\'\n\'accept-encoding\', \'gzip,deflate,br\'
13:07:37.027 Called Filter : setDecoderFilterCallbacks
13:07:37.027 matched operation: project.v2.Events.OnSessionEvent
13:07:37.027 Called Filter : decodeHeaders
13:07:37.027 use filter state value project.v2.Events.OnSessionEvent to find verifier.
13:07:37.028 extract authorizationBearer
13:07:37.028 extract x-goog-iap-jwt-assertion
13:07:37.028 gcloud: JWT authentication starts (allow_failed=false), tokens size=1
13:07:37.028 gcloud: startVerify: tokens size 1
13:07:37.028 gcloud: Verifying JWT token of issuer https://accounts.google.com
13:07:37.028 gcloud: JWT token verification completed with: Jwks doesn\'t have key to match kid or alg from Jwt
13:07:37.028 Called Filter : check complete Jwks doesn\'t have key to match kid or alg from Jwt
13:07:37.028 Sending local reply with details jwt_authn_access_denied
As you can see the jwks doesn't contain the JWT kid. And this is true.
The header of this JWT is this:
{
"alg": "RS256",
"kid": "d05ef20c45191eff64b25d380d46fde55ac229d1",
"typ": "JWT"
}
and the payload is this:
{
"aud": "project.clounrun.admin",
"azp": "109442000165584777807",
"email": "[email protected]",
"email_verified": true,
"exp": 1603807001,
"iat": 1603803401,
"iss": "https://accounts.google.com",
"sub": "109442000165584777807"
}
This indicates that pubsub does indeed use that service account but it looks like that specific kid is missing from the jwks response.
{
"4366ae10d4a79728de14f6f89a628b4fe640140f": "-----BEGIN CERTIFICATE-----\nMIIC+jCCAeKgAwIBAgIIapShdpj8y6QwDQYJKoZIhvcNAQEFBQAwIDEeMBwGA1UE\nAxMVMTA5NDQyMDAwMTY1NTg0Nzc3ODA3MB4XDTIwMDQzMDIyNDE0N1oXDTMwMDQy\nODIyNDE0N1owIDEeMBwGA1UEAxMVMTA5NDQyMDAwMTY1NTg0Nzc3ODA3MIIBIjAN\nBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArvpvYFbpJiXk4cEOo2xXzIFjLZEq\nIe1753Oe00IXmq5u/Glf6H0TdShqSn/mHd283UOeDGyjcz/AZO3iKyGv+GndSfiq\ny9TbXfeinCUoVtLUU500P32Ciej/t8Hf4UZYl6XlBVSMZK5ZVCqdWHs9vfPH8k6w\nSJm456BwjL3xty5AjuBooTSHec92SGe2DYSpMJL9NHGELdSnNRoxEaXpEUBV93vr\nTKfbBKa/1WaumVvIn54rAIMkaFq7dJRFr98U2yfHFvUhMtqAwX7HkdvgM74sjfV0\nduVfVz3T/m/oG/7lCllpI4LhVHpxxuNhimFo5quXjShJLPNEjFbESndMaQIDAQAB\nozgwNjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDAWBgNVHSUBAf8EDDAK\nBggrBgEFBQcDAjANBgkqhkiG9w0BAQUFAAOCAQEAZqBZCbFswKBxa4S+fvC6qb5c\nF/29sxM4WijC4e+/Ti+e6tcRrf3LyOF0jIpHbHWcwjTiYXX+D+UUmMrgh0ZVLCOR\nTDj29JSIUEhzYsGmBzBBcPLfMO2zl6c0aMdUkO+3vXQoTHUNjcs8UoN6BlPo3oIG\n2BjOXhEmuuUA3BQVDsMIM5g4G5r28WaprV7GaTa4fsyCh9oRquTtqL34CZLxiLXv\ncqK+oRMFU4tsLKvZjcfTeKp3fbXDpo7R1R7/+SyxAJTQOe1uPeAc5qhlVK6Ky/Zy\nTv3SUzAifJ3BDz1eNKYTqQNWiXi3QnX6qwebHgHKcJ01qnbCqMqbv6HicpULdQ==\n-----END CERTIFICATE-----\n",
"e0b3368c1646eb88d93b9d0b2d65e2d6fbae27b7": "-----BEGIN CERTIFICATE-----\nMIIDNDCCAhygAwIBAgIIQ2si/YIYDg0wDQYJKoZIhvcNAQEFBQAwPTE7MDkGA1UE\nAxMyODc1NTE3NTIzODI1LWNvbXB1dGUuZGV2ZWxvcGVyLmdzZXJ2aWNlYWNjb3Vu\ndC5jb20wHhcNMjAwOTMwMTM1NjM5WhcNMjIxMDEzMjAxNTU3WjA9MTswOQYDVQQD\nEzI4NzU1MTc1MjM4MjUtY29tcHV0ZS5kZXZlbG9wZXIuZ3NlcnZpY2VhY2NvdW50\nLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALejPchRlDY8EE5D\nvD1CuPyGsbCoXHDnJpaA4P23CLNUmooBydxxzfV611vTUTBCFjq5Pcg3fpGusoMt\nyF9TqQVq4bGZxrXv+yxVs24uFdHAB7mY9JUE8GKN5i7IMP3egDcns4LmNWsB0iKN\niK1gK5q7gZIISjo3igLrup1G6wM02qym2VS6raKn/12WY+pa/PiZrO79eAkYPyqr\n73AvLdcLsik9U7lNDfxiev3/IE+tP0B68Uo5Ff+Wai+RDnNmDX3Fy50hv8vfniRe\nB/b57Kn6SMtMz4IMD4BeNQpOXdZe780cubuwS1oPLcRLHCXTqEzsGVIMLoMxrxF9\nhRK743kCAwEAAaM4MDYwDAYDVR0TAQH/BAIwADAOBgNVHQ8BAf8EBAMCB4AwFgYD\nVR0lAQH/BAwwCgYIKwYBBQUHAwIwDQYJKoZIhvcNAQEFBQADggEBAKF+dNSK+GFN\ndKFBs5PZufLUMlyaIXYysDnZgCk9oWMBiBpa6WDg00SllAz5z9koVYFr3ySQq6mK\nddkJLUXVt+NkR6Jh4k4OYshBV4v6q/BOXEjpP6/+jNV9qD5vVuq+w1eReouCl8eO\ng/rfUAivAPpU1srXSmhs6Uw2E4jDf8ArJWLsPfHjoqWLJEICGBG0i1nlJBRpGNWj\nyBfN1cEp64pYynpnVY9kuIEKdBCue8QEXVhsKURHGcOfCWP9+vVerhlyDatL1tQW\n5RAjvzS8NGLe8QnMdy+63TwP4qKGkWSEPTxP0fpQpxuLbqKHHsSeA7WL6nS6zSwx\nRXHDLiUfU0A=\n-----END CERTIFICATE-----\n",
"c0191d2d00f89eb1905886a04335a79a124885e1": "-----BEGIN CERTIFICATE-----\nMIIDNDCCAhygAwIBAgIIaTwbvrNhTkYwDQYJKoZIhvcNAQEFBQAwPTE7MDkGA1UE\nAxMyODc1NTE3NTIzODI1LWNvbXB1dGUuZGV2ZWxvcGVyLmdzZXJ2aWNlYWNjb3Vu\ndC5jb20wHhcNMjAxMDI4MDA0NDEwWhcNMjAxMTEzMTI1OTEwWjA9MTswOQYDVQQD\nEzI4NzU1MTc1MjM4MjUtY29tcHV0ZS5kZXZlbG9wZXIuZ3NlcnZpY2VhY2NvdW50\nLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKJZDpcJAHyot7z9\n0EAOOEC6+dwlkxduThH5FQXlaCUoHRAoCtTHrxeITSsZg55cXGXKzTTf1Nv+UE9E\nTY5nhU70nEK77B5FbxwzX9Q/OkOhP3NQnl0U0O6nedfCJvOtCMdopHnrRa+ZIWhG\nPoW2RKQTv7gr4bGFJnQshcYDtrahH6Xv/RfyyTnI9AUzCX6eVO3g7odLSdcv+qnV\niqsZLCqz2lflKky+Rti/1f8LGXySTs+r7bdvqCdpbIm6f79WqQ720hmx+4JaIuS3\nYmYztK5J6JwsDLqULyyMvZs9vVA6cK1+580Vb76pkF9MHqTZdZ2gPP1qoKCO0ezI\nK3lYfGECAwEAAaM4MDYwDAYDVR0TAQH/BAIwADAOBgNVHQ8BAf8EBAMCB4AwFgYD\nVR0lAQH/BAwwCgYIKwYBBQUHAwIwDQYJKoZIhvcNAQEFBQADggEBAHB+6GiylFKU\nq9nyzrvlTRe0tph7kT6D8WhlnxGDgnDOn9xAB360xh+KPczq7+8DJczSb5145No/\nefrCUpvgkraVe/eTsu8EAjDjk/XPOeV4117EU+PRVV5nitYiVGJ0Z+3V3kHCnqLT\ntTxQlKDRpJ02GjhQRkW03fhLEFr7eu79fiwfjfaFSoAAIrNxBi3mtrhcidu43zD1\n/VXAReH26S129UoYz7fPlsnULX/oUyZPidaKIZ2Fl5N2QTgXnd1PnU3HgesU4HWs\nYJeiq6Z1pFxTw9192VD3MgDSDVQAeHXs+wTQsUHftMcNl7nPWl9YfiPjzD4sNWL3\n8whVkhIN8r4=\n-----END CERTIFICATE-----\n"
}
We have:
4366ae10d4a79728de14f6f89a628b4fe640140fe0b3368c1646eb88d93b9d0b2d65e2d6fbae27b7c0191d2d00f89eb1905886a04335a79a124885e1
But no
d05ef20c45191eff64b25d380d46fde55ac229d1
If pubsub is using a different key that where can I find the jwks?
UPDATE: I verified all my services accounts JWKS links and none of them contain that kid. So this is very strange.
