AWS Lambda and SQS are some of the most popular services that Amazon provides. Yes, we do love them but lambda’s concurrency handling with SQS is silly. In this blog, I am going to share my experience working with them.
AWS Lambda provides a reserved concurrency of 1000 per region. It can be further raised by raising a ticket requesting additional concurrencies.
To keep it short, recently I was working on a project where 100’s of messages will be pushed into an SQS queue. This queue will be polled by a lambda which has a concurrency limit of 10 with a batch size of 1. What I expected was lambda would process the first 10 messages and the remaining messages would stay in SQS waiting for lambda to get picked.
I noticed that all 100 messages were in flight which means every message is being polled and processed. But unfortunately, that did not happen and the remaining 90 messages came back to the queue after the visibility timeout since it was not deleted.
After reaching the maxReceiveCount, those messages will be moved to DLQ (Dead Letter Queue) if configured. If you do not have a DLQ configured messages will be lost. Also, keep in mind that the maximum delay time for a message is 15 minutes.
What Did I Do?
My lambda would run for at least 8 minutes and I removed the SQS from the design. So I gave a buffer time and configured an event bridge rule that triggers the lambda at 15 minutes intervals with the appropriate parameter. Though this worked I would still be happy if AWS could do something about this.
- As said early configure a DLQ to the queue so you can use it in case of failure.
- Set the maxReceiveCount to at least 5. By doing this you can extend your chances for message to not reach the DLQ.
- Set the visibility timeout to at least 5 times your lambda time.
- You can try using API Gateway if it suits your need.