How to limit incoming traffic before the service overloads
Why requests sometimes need to be rejected
If a service can handle 400 requests/s, 800 exceeds its capacity. You can admit everything and get failures inside the service. Or limit input in advance so only work it can handle reaches it.
A limiter decides whether to admit a request. Rejected requests do not occupy the service, but users still receive no operation result. A solution that protects the service by discarding almost all useful work cannot be considered good.
How token bucket works
Think of a token bucket as a box of permits. One request consumes one token. With a token, it passes; with an empty box, it is rejected. Tokens refill at a chosen rate, but their stock never exceeds bucket capacity.
The algorithm has two distinct settings. Refill rate controls sustained admitted traffic. Capacity provides reserve for short bursts. The experiment starts full. Rejected requests do not wait: there is no separate queue.
The risk of too much stored capacity
At 300 tokens/s and a reserve of 60, more than 300 requests/s can pass briefly. Once reserves run out, sustained traffic approaches refill rate. Unused permits accumulate only up to 60.
If the token reserve is very large, the limiter may admit more at the start of a burst than the service can handle. Average rate alone is therefore insufficient: short-term admission matters too. Here, calculations use 100-millisecond steps.
Distinguishing two kinds of rejection
Rejected at entry means requests without enough tokens. Service failures are requests admitted by the limiter that the overloaded service could not complete. The analysis shows these separately.
Vary refill and capacity to preserve enough successful work without overloading the service. A limiter does not add service capacity. It changes where and how many excess requests are stopped.