Design an inference request scheduler on GPUs
The question
Design a scheduler placing inference requests onto a GPU pool shared by models with different latency targets. Cover how you batch without inflating the tail, what stops one large request starving small ones, and how the design changes when the pool is heterogeneous. Then discuss admission control once demand exceeds capacity.
Draft write-up. The title is what was reported. Everything below it was written by us to flesh the question out, so treat the specifics as illustrative rather than as the interviewer’s words.
What's being tested
Whether you can hold two goals in tension. Batching raises throughput and raises latency at the same time, and the interesting answers are about where you choose to sit on that curve rather than pretending one side wins.
Clarify first
- Is the target throughput, p50, or p99? They pull in different directions.
- Do requests have per-tenant SLAs, or is the pool best-effort?
- How variable are input and output lengths?
- Can a request be preempted once it has started, or must it run to completion?
A concrete version
One pool of 64 GPUs, a dozen models, request rates swinging 10x between peak and trough, output lengths from a few tokens to several thousand. A fixed batch size that suits the average is wrong at both ends.
Structuring an answer
- 01Separate admission (do we accept this?) from placement (which GPU?) from batching (who runs together?) — they have different objectives.
- 02Prefer continuous batching over fixed windows so a short request is not held behind a long one.
- 03Give each model or tenant class its own queue with a deadline, and schedule against deadlines rather than arrival order.
- 04Cap the work in flight per GPU by memory, not by request count, since long outputs consume KV cache unpredictably.
- 05Shed or queue explicitly at saturation, and return that state to the caller rather than letting latency drift upward silently.
An outline, not a model answer. Interviewers are listening for how you reason, so working through it yourself beats reciting this.
Follow-ups
- Half the pool is an older GPU generation. How does placement change?
- One tenant floods the queue. What stops the others degrading?
- You must guarantee a p99 for one tenant and best-effort for the rest on shared hardware. How?
Common mistakes
- Optimising average latency and never mentioning the tail.
- Assuming a fixed batch size, which fails at both peak and trough.
- Ignoring memory as the real constraint and reasoning only about compute.
1 person reported this question. One more independent report moves it to corroborated.
I was asked this too