Worked solution · 10 min read
Design an inference request scheduler on GPUs
Separate admission, placement and batching — they optimise different things. Use continuous batching bounded by memory rather than request count, schedule against per-class deadlines instead of arrival order, and shed load explicitly at saturation rather than letting latency drift.
One defensible approach, not a model answer or a marking scheme. No interviewer said any of this — the value is in the reasoning, so read it after attempting the question rather than instead of.
01Walkthrough
Establish which number you are optimising
Throughput and tail latency pull in opposite directions, and batching is the knob that trades one for the other. An answer that does not name the target is guessing.
- Per-tenant SLAs, or best-effort? Guaranteeing a p99 for one tenant on shared hardware is a materially harder problem.
- How variable are output lengths? High variance is what makes fixed batching fail.
- Is preemption allowed once a request starts? If not, one long request can hold capacity for a long time.
Split the three decisions
Conflating these is the most common structural mistake. Each has a different objective and a different time horizon.
- Admission: should we accept this request at all, given current load and its deadline? Protects the system.
- Placement: which GPU or replica should serve it, given memory and model residency? Optimises utilisation.
- Batching: which in-flight requests execute together this step? Optimises throughput against latency.
Use continuous batching, bounded by memory
Fixed-size batches force short requests to wait for a window to fill and are wrong at both peak and trough. Continuous batching admits new requests into the running batch each step, so a short request is not stuck behind a long one.
- Bound concurrency by KV-cache memory, not request count. Output length drives memory unpredictably, so a count-based limit either wastes capacity or triggers out-of-memory.
- Reserve headroom for the worst-case growth of in-flight requests rather than admitting to the limit.
- Group by model to avoid reloading weights; a request for a cold model is a placement decision, not a batching one.
Schedule against deadlines
Give each tenant or latency class its own queue with a deadline derived from its SLA, then pick work by earliest deadline rather than arrival order. This is what stops a strict-latency tenant queueing behind a batch job.
- Weighted fair queueing across tenants prevents one flooding the pool.
- Cap per-tenant in-flight work so a single caller cannot consume the whole pool regardless of how much it submits.
- For heterogeneous hardware, route latency-critical work to faster GPUs and treat older ones as best-effort capacity.
Degrade honestly under saturation
When demand exceeds capacity, something has to give. The choice is whether it gives visibly or silently. Silent degradation means unbounded queues and latency that climbs until callers time out, wasting the work already done.
- Reject at admission with a clear signal once the projected wait exceeds the deadline. Failing fast beats a timeout after computing half an answer.
- Consider degraded service — a smaller model or shorter output cap — rather than outright rejection where the product allows.
- Surface queue depth and projected wait so callers can make their own decisions.
Trade-offs worth naming out loud
Saying what you rejected, and why, is most of what separates a senior answer from a correct one.
Continuous batching
instead of Fixed-window batching
Fixed windows are simpler and more predictable, but add a fixed wait to every request and behave badly when load or output length varies. Continuous batching is more complex to implement and much better under real traffic.
Memory-bounded concurrency
instead of Request-count limits
Counts are easy to reason about but do not correlate with actual GPU memory, so you either under-utilise or hit out-of-memory during a long-output burst.
Reject at admission
instead of Queue everything
Queueing feels more generous but converts a capacity problem into a latency problem and wastes compute on requests whose callers have already given up.
What gets read as strong or weak
Strong signals
- Naming the throughput-versus-tail tension unprompted and choosing a position on it.
- Bounding on memory rather than request count, which shows familiarity with how inference actually fails.
- Treating admission control as part of the design rather than an afterthought.
- Handling heterogeneous hardware as a routing decision rather than pretending the pool is uniform.
Loses points
- Optimising average latency and never mentioning p99.
- Assuming a fixed batch size throughout.
- Reasoning only about compute and never about memory.
- Letting queues grow without bound and calling that graceful.