Rate Limits That Protect Expensive Work, Not Just Request Counts

One request can consume ten milliseconds or ten minutes of compute. A single requests-per-second limit treats those requests as equivalent and can leave the expensive path almost unprotected.

For an export API, I would separate admission from execution. Admission establishes who may request work and how much budget the tenant has. Execution controls how much work can run concurrently and how long it may occupy resources.

Budget the scarce resource

A token bucket can regulate admission:

available = min(capacity, previous + elapsed * refill_rate)
admit only if available >= estimated_cost
new_available = available - estimated_cost

That calculation must be atomic for a shared bucket. Independent workers doing read-modify-write can overspend the budget. Cost estimates should come from validated server-side properties, such as bounded row counts or export classes, rather than a client-provided “cheap” flag.

Estimates will be imperfect. Combine them with hard ceilings on execution time, bytes produced, database work, and concurrent jobs. Reconcile measured usage where practical, but never rely on after-the-fact billing to protect a resource that can already be exhausted.

Preserve fairness under overload

Use per-tenant concurrency limits and a global capacity bound. Otherwise a large tenant can fill the queue before smaller tenants submit anything. Separate interactive operations from bulk work so a burst of exports does not prevent users from signing in or cancelling a job.

Bound queue length and age. A queue that accepts hours of work while promising a quick response hides overload rather than solving it. Reject early with a stable error contract and a retry policy appropriate to the operation. Clients need backoff and jitter; immediate synchronized retries can amplify the load.

Google’s SRE chapter on handling overload explains why load shedding and retry behavior must be designed together. My proposed tenant cost budget adds an application-specific fairness boundary to that reliability baseline.

Treat limiter failure as a design decision

If the shared limiter is unavailable, unrestricted admission may turn a small dependency failure into a large outage. A bounded local fallback can preserve limited service, but its aggregate allowance grows with the number of instances. Calculate that upper bound before relying on it.

For especially expensive or sensitive operations, defer admission until the authoritative budget returns. The product owner should understand that availability tradeoff and provide a usable customer message.

I would ask platform engineering to own the limiter’s availability, service teams to own cost models and execution ceilings, and security to test evasion across identities and endpoints. Load tests should include a tenant issuing many small requests, a tenant issuing a few expensive ones, and a limiter outage.

Track completed useful work, queue age, tenant-level rejection patterns, and resource saturation together. A low rejection rate can mean a generous customer experience or a control that is allowing the service to collapse. Leadership is choosing the service guarantee and funding the capacity and controls needed to keep it.