AWS Lambda nodejs long running process

Question

I'm working on a project (Serverless, Lambda, Nodejs, MongoDB, SQS) where I need to make a price chart, there is an API /api/products?country=countryCode that returns data about the product and the price (see response sample below), prices can be different for each country, for example, for NL the price can be $12.99, and for AU €13.99 (all other properties do not change)

I have a collection of countries in MongoDb, there are about 225+ countries in the collection. I have a cron job that triggers a lambda function once daily. For each country, I need to call the API that returns the product data (see sample response below) that needs to be processed. After fetching all the data, I insert products and prices at the same time (products are unique, price count = product count * country count * days). To avoid price differences between countries, you need to insert the data at once. For example yesterday the price of the Sword of Bladur was $17.99, today it is $18.99, We have updated the price for NL, but AU is in process.

Please help me solve one of these problems:

Problem 1 (Cron -> Handler): 1 API request takes about 6-8 seconds. To process all countries, it takes ~30 (225 * 8 / 60) minutes, however, there is one small problem, lambda function has timeout limit (max: 15 minutes), of course this time is not enough to finish the job.
Problem 2 (Cron -> Handler -> SQS Handler): I rewrote my code and instead of running one lambda function, I send each country to AWS SQS (Consumer/Producer), which triggers a lambda function that fetches data from API and processes, however, here I have a problem with the fact that I do not know when the cycle will end to insert prices at once.

P.S. In my opinion I should use Producer/Consumer to avoid losing country data, so I will be glad if someone has solutions for the second problem

MongoDB Product model
{
  id: String,
  title: String,
  ...
}

MongoDB Price model
{
  country: String,
  productId: String,
  price: Number
}

API response sample:

GET /api/products?country=NL
[
  {
    "id": "37071265-7f98-4e32-ae45-c23f83e7c7a2",
    "title": "Dusty Book",
    "country": "NL",
    "price": 1299,
    ...
  },
  {
    "id": "49701bcf-c076-4064-b331-0952aee21deb",
    "title": "Sword of Bladur",
    "country": "NL",
    "price": 1799,
    ...
  },
  ...
]
-----
GET /api/products?country=AU
[
  {
    "id": "37071265-7f98-4e32-ae45-c23f83e7c7a2",
    "title": "Dusty Book",
    "country": "AU",
    "price": 1549,
    ...
  },
  {
    "id": "49701bcf-c076-4064-b331-0952aee21deb",
    "title": "Sword of Bladur",
    "country": "AU",
    "price": 1799,
    ...
  },
  ...
]

Could you insert each price to a temp table as you get them, and after each insert, see if the temp table has all countries in it, if so, copy all to the final location and empty the temp table? — Anon Coward

Jason Wadsworth Jason Wadsworth · Accepted Answer · 2021-04-02T02:51:20

Have you considered using Step Functions? You could one a single function that gets a list of the countries. The results of that could be passed to a map state that would call a lambda for each country, getting the required data. Once all of those functions are done it can call another lambda that would perform the final updates in the database.

AWS Lambda nodejs long running process

1 Answers