Accepting, processing and sending an event are separate stages
What happens during a peak
The source must deliver notifications to users. In the first 10 seconds, 500 unique events/s arrive, then 100/s for 20 seconds, then input stops. There are 30 more seconds to finish. Worker capacity is measured in processing attempts, provider capacity in deliveries. Adding a queue increases neither capacity.
How a queue helps
A queue keeps work that workers cannot take immediately. Once input drops, spare capacity drains the backlog. But buffers and waiting time are limited. Two workers with a suitable queue can spread a short peak; four can handle it without a queue. Both are valid if all requirements are met.
Where duplicates come from
A retry can arrive even after the first event was processed. Here every tenth event arrives twice. The deduplicator recognizes an event by its ID and prevents resending. Duplicate detection still costs work, but the second user-visible effect disappears. This does not promise universal exactly-once delivery. For example, 500 unique events create 550 processing attempts. With deduplication, workers process all 550 attempts but only 500 sends reach the provider.
What it means to finish the task
Accepting every event into a queue and showing zero loss is not enough. All must be delivered by the end, with no backlog or duplicate sends. Peak wait is queue size divided by processing rate. Real FIFO timings and network retries are not reproduced.
Task
First 10 s: 500 new notifications/s; next 20 s: 100/s; then 30 s to finish. Every tenth notification arrives again. Choose a queue and workers; deliver everything without duplicates or waits longer than 12 s.
60 one-second steps, with 7,000 unique incoming events at the set peak of 500/s. Plus 10% retry attempts that require worker effort. A worker handles 150 attempts/s. The provider accepts 600 deliveries/s; deduplication removes repeats before sending but after worker processing. Queue capacity counts raw attempts; without a queue, each step’s remainder is lost.
- Add a deduplicator and choose worker capacity.
- If using a queue, size it for the peak and drain time.
- Check losses, duplicates, backlog and wait; compare a setup without a queue.