SQS Lambda ESM Scaling Behaviour - Experiment ESM vs Lambda Concurrency Uploads
#Test
Important
This page only contains the experiments that have been done for the: SQS Lambda ESM Scaling Behaviour blog post.
We keep the processing time at 50ms and vary ESM concurrency. We then reduce the ESM concurrency in steps below the recomended (observed unbound concurrency), which was recorded as 100. An SQS generator function is called locally to send messages to the queue at a constant rate. For these we update the Lambda source code every 3 minutes while generating SQS traffic locally. We are using small Lambda function with some added non JS files to inflate the package size to ~30MB.
#Results
| Condition | Lambda Concurrent Steady State | Lambda Concurrent Max | Logged Cold Start Max | Lambda Throttle Max |
|---|---|---|---|---|
| 50ms | 100 | 173 | 139 | 0 |
| 50ms 120 ESM | 100 | 120 | 117 | 0 |
| 50ms 90 ESM | 90 | 90 | 90 | 0 |
| 50ms 50 ESM | 50 | 50 | 50 | 0 |
| 50ms 25 ESM | 25 | 50 | 50 | 0 |
| 400ms | 500 | 194 | 187 | 0 |
| 400ms 120 ESM | 120 | 128 | 127 | 0 |
| 400ms 90 ESM | 90 | 91 | 90 | 0 |
| 400ms 60 ESM | 60 | 60 | 60 | 0 |
| 50ms 60 ESM Lambda 70 | 60 | 70 | 58 | 56 |
- All of these are taken over 1 minute intervals
#Processing 50ms
#Processing 50ms, no limits
#Processing 50ms, ESM 120
#Processing 50ms, ESM 90
#Processing 50ms, ESM 50
#Processing 50ms, ESM 25
- The first datapoint here is not correct. I did not wait long enough between experiments I believe.
- We did have a queue backlog of about 1m and 20k messages at the 15 mins mark.
#Processing 400ms
#Processing 400ms, no limits
#Processing 400ms, ESM 120
#Processing 400ms, ESM 90
This experiment did form a queue backlog.