Is it advisable to use amazon-sqs-java-messaging-lib Library for SQS consumer?

0
votes

Requirement: Need to consume batches of messages from sqs queue. Series of operation needs to be applied on the data while processing. The message frequency might be high during some time, so the application should be able to handle it by scaling accordingly. We are developing the application for a longer run, once developed there shouldn't be any major development changes.

Project is based on drop-wizard based framework.

Options:

  1. Lambda - Ruled out this option as huge load of messages in sqs leads to more failures(end up in dlq) in case of lambda. Also i need to do more than one operation on the data, where lambda suits for only one operation on it.
  2. AWS SQS SDK - Need to write down the thread logic for this (For polling and message handling)
  3. AWS provided jms based sqs sdk (amazon-sqs-java-messaging-lib) - Seems like a prefect fit for my use case. As it will handle the polling and concurrency logic. But the problem is that the latest release for this jar was from April 2019. Its been nearly 2 years, still no update for this library. Also there is still many open issues unanswered.

I'm afraid aws will decommission this library and also My project should be built in such a way so there shouldn't be no major dev change like change of dependency in future.

Can someone please enlighten me whether i can go for this library or its better to just stick with vanilla aws sqs sdk? If there's any other options Please do mention that also.

--------------------------------------------------------UPDATE------------------------------------------------------------- Thanks for the suggestion of using lambda or step by increasing the number of concurrent limit to tackle the heavy load of 1000 tps. I Have also did some more analysis on my project use case and decided to go with containerized APP using plain vanilla AWS SQS sdk for interacting with SQS.

Reasons for this decision:

SQS -> My_APP -> Another APP

  1. As you can see the flow, from My_APP i need to interact with another app which can support certain tps. If i have control over polling logic, i can take care of this issue.
  2. Also My_APP - Its not doing a simple process. more than 1 Process will be happening here. (Transformation, some mapping,...)
  3. The reason i posted this ques is to get some update on the amazon-sqs-java-messaging-lib Library. I ended up not proceeding with this path as I am not sure on the support of aws on this library in future.
1
javaamazon-web-servicesjmsamazon-sqs
Why would lambda lead to failure? How much is your "huge load of messages "? – Marcin
During peak time, Expecting around 1200 tps. – mariz
Not that much. You can get lambda to do it. You can ask support to increase you default concurency limit of 1000 to 2000 lets say. – Marcin
Thanks for pointing it out. I forgot to add one more point - Here the processing will be like series of operations on the data. Whereas lambda is optimum for addressing only one operation. Also i cant afford 1 lambda for each of this operation - which might lead to high cost. Thats the reason i'm looking into some container app in ec2 using aws provided sdk. – mariz

1 Answers

0
votes

I don't understand "lambda is optimum for addressing only one operation". What makes you say that? The only limit is the amount of time it takes to execute the series of operations. If you consistently get timeouts while running multiple operations, you could use step functions. See https://docs.aws.amazon.com/step-functions/latest/dg/connect-lambda.html.