AWS API Gateway forbids consecutive requests

2
votes

I'm using AWS API Gateway and Lambda to upload images to an S3 bucket. I have setup API Gateway to use a custom authoriser.

I'm currently passing the base64 encoded image in the JSON payload (not ideal but for the moment it is fine for what I need).

This is how I call my API (note you need jq and httpie installed to issue this call):

base64 <my_image> | jq -R '{image: .}' | http https://<api_gw_url>/dev/upload 'Authorization:Bearer <my_auth_token>'

If I issue more than once the same request, API Gateway returns a 200 the first time (and the document is correctly uploaded), but for any subsequent request (with the same auth token to the same URI) a 403 Forbidden is returned with the following error in the body:

{
"Message": "User is not authorized to access this resource"
}

After about 5min the request is accepted again :/

I didn't find any mention of default rate limiting in the documentation.

I have also tried to add a Usage Plan, with an API Key associated, to be able to tweak the rate limiting, but it didn't make any difference.

Did anybody else experienced this?

Edit

Just wanted to add some more info about my architecture to make sure the issue is better explained.

My setup includes an API Gateway, a Lambda that takes care of the authorization with JWT and another Lambda that does the upload to S3. What happens is the following:

First call

>base64 <my_image> | jq -R '{image: .}' | http https://<api_gw_url>/dev/upload 'Authorization:Bearer <my_auth_token>'

Request hits:

  • API Gateway
  • Authorization Lambda
  • Document Upload Lambda

Response code: 200

Second call (issued less than 5 min after the first one)

>base64 <my_image> | jq -R '{image: .}' | http https://<api_gw_url>/dev/upload 'Authorization:Bearer <my_auth_token>'

Request hits:

  • API Gateway

Request does NOT hit:

  • Authorization Lambda
  • Document Upload Lambda

Response code: 403

1
amazon-web-servicesaws-lambdaimage-uploadingaws-api-gatewayrate-limiting
It allows you to issue a new request only after the first one is finished? Maybe there's a per-user rate limit of one consecutive request? - Renato Byrro
That sounds a bit like you may have a bug in your Lambda code that clears itself up when a container is destroyed and you get a new one, though 5 minutes is a little short for that to happen. - Michael - sqlbot
@RenatoByrro I have never tried concurrent requests. What I have been doing was sending a request at a time and only the first one was allowed (i.e.: 200). It sounds strange that the user can issue the same request just once every ~5min. Imagine the user wants to override the document he/she just uploaded, he/she would have to wait 5 min every time :/ - barbasa
Isn't there anything else in CloudWatch to help debug? I'd put some logs throughout the code manually to narrow down where specifically the code is failing. I had an issue with Lambda running Python, in the past. It worked well some times, others it failed. It only manifested the issue with specific types of hardware. Lambda has thousands of servers as an underlying infrastructure and each execution may be served by a machine with a different configuration... Not sure if that's your case, it seems more what Michael pointed out above. In either case, manual logs will help narrow down the source - Renato Byrro
@RenatoByrro I can exclude a bug in the Authorization Lambda since it only gets invoked the first time, hence it looks like it is something related to API Gateway. Sorry I should have made it clearer since the beginning. I'm going to update the post. - barbasa

1 Answers

2
votes

It turns out it was a mix of a bug in my code and a misunderstanding on how API Gateway works.

The issue I was having was the following:

  • 1st call: PUT /entity/UUID1 with JWT_Token_1
    • Policy to access UUID1 cached for JWT_Token_1
  • 2nd call: PUT /entity/UUID2 with JWT_Token_1
    • Retrieve cached policy for JWT_Token_1, BUT tries to call resource with UUID2 then "User is not authorized to access this resource"

I was basically reusing the same JWT Token to access different resources. In my use case this shouldn't be allowed.