Worked solution · 14 min read · 27 min video

Design GPU billing for a distributed cluster

The output is money, so correctness and reproducibility beat freshness — but you were also asked for a live spend view, which wants the opposite. What you are really building is one pipeline with two serving paths and two consistency contracts. Everything downstream follows from saying that out loud early.

One defensible approach, not a model answer or a marking scheme. No interviewer said any of this — the value is in the reasoning, so read it after attempting the question rather than instead of.

27 min walkthrough. The written solution below covers the same ground if you would rather read it.

Where the difficulty actually is

Be precise about where the difficulty lives, because it is easy to aim at the wrong thing. It is not throughput: a million GPUs sampled every minute is around 60k events per second at peak, which is unremarkable and solved. The difficulty is that the output is money.

Money means correctness beats freshness, and it means a number must be reproducible years after you first produced it.
But the prompt also asks for a live spend view, which wants exactly the opposite trade.
So: one pipeline, two serving paths, two consistency contracts. Say this in the first three minutes — everything else is downstream of it.

01Requirements

Functional

01Tenants receive an accurate invoice for each billing period.
02Tenants can see their near-real-time spend for the period in progress.
03Tenants and support can trace any invoice line back to the raw usage that produced it.

Non-functional

Billing path is CP: correct even when delayed, and within 0.01% of an independent recomputation.
Auditable — any invoice reproducible byte-for-byte for seven years.
Late tolerance — usage up to 3 days late absorbed silently, up to 90 days handled as explicit adjustments.
Spend view is AP: always answers, may be stale, under 5 minutes of freshness.

Below the line

Budgets and alerts · Payment collection and dunning · Tax · Contract negotiation · Revenue recognition · Refund workflows

State the scale assumptions out loud before quantifying anything: roughly 1M GPUs, ~20 data centres, 60-second telemetry, 100k tenants, monthly invoicing. Note also that 'handle late and out-of-order events' appears in the prompt beside the features, but it belongs here rather than in functional — nobody asks for late-event handling as a capability. It is a condition the world imposes, and it threatens all three functional requirements. Treating lateness as a feature leads to a special-case path, and special-case paths are where correctness goes to die.

02Capacity estimation

One calculation, and only because it changes a decision. Skip arithmetic that decides nothing.

Samples per day: ~1.4B
Average rate: ~17k/s
Peak rate: ~60k/s
Raw telemetry: ~700 GB/day

Pricing every sample commits you to ~17k ledger writes per second forever. But you are selling hours, and you do not need minute-level rows to bill hourly. Let the agent collapse its own samples into one hourly interval per resource before shipping: ~24M rows a day, around 280/s — roughly a 60× reduction. Raw samples still land in the archive, so audit loses nothing.

03Core entities

Tenant: Who receives the bill.
Resource: A GPU, or a MIG slice of one.
UsageEvent: A closed interval, not a point sample.
PriceVersion: Effective-dated, so rates can change without rewriting history.
ChargeLedgerEntry: The atom of the bill — append-only.
Invoice / InvoiceLine: A frozen snapshot over a closed period.
Adjustment: How post-issue corrections are expressed.

04API

Two surfaces with two protocols: a high-volume ingest path for agents, and a read path for tenants. Agent identity comes from the mTLS certificate, never from the request body — otherwise any agent can bill on behalf of any other.

ReportUsage(intervals[]) → accepted, rejected[]

Idempotent on a deterministic interval id, so a retry after timeout cannot double-bill.

ReportHeartbeat(agent_id, at) → ok

Exists so the system can distinguish zero usage from missing data. If an agent goes silent and you read silence as 'consumed nothing', you have quietly stopped billing a data centre — the largest source of revenue leakage in real metering systems.

GetEstimatedSpend(tenant, period) → estimated_micro_cents

The field name, the API and the UI all say estimated. Showing a customer one number and charging another is a trust problem, not a rounding detail.

GetInvoice(tenant, period) → Invoice + lines[] + lineage

Every line drills back to the intervals that produced it.

05Data flow

01Agent samples local GPU state every 60 seconds.
02Agent collapses samples into one closed hourly interval per resource.
03Interval ships with a deterministic id; ingest rejects duplicates.
04Rating attaches the price_version_id in force at usage time, not processing time.
05Rated rows append to the charge ledger; nothing is updated in place.
06Derived counters fan out to the spend view; the ledger stays authoritative.
07At period close, the ledger is frozen into an invoice with lineage retained.

06High level design

Start naive on purpose: ingest → rate → append to a ledger table → sum for the invoice. Serve the spend view from that same table initially, and only split it once you have shown why the single path cannot satisfy both consistency contracts. Two things, though, cannot be retrofitted and must be decided now.

Decide now — cannot be retrofitted

Money is integer micro-cents, never floats. Round exactly once, at the invoice line — rounding per event accumulates error across a billion rows.
Nothing is ever updated in place. Corrections are new rows, so lineage survives.

07Deep dives

Lateness is the normal operating condition

Design for late arrival as the default rather than the exception. The property that makes this tractable is additivity: if the aggregate is a sum of independent intervals, a late event is just another addend and order stops mattering.

Additivity only holds if duplicates cannot. Deterministic interval ids plus a uniqueness constraint are what make the sum safe to replay.
Derive watermarks from heartbeats rather than from the data itself, so an absence of events is distinguishable from an absence of usage.
Have an explicit close policy for events arriving after issue: absorb silently inside 3 days, adjust explicitly out to 90, and set a materiality threshold. That threshold is a finance decision, not an engineering one.

Real contracts are not per-unit

Committed spend, tiered rates, and volume discounts break the assumption that each event can be priced independently. This is where the naive design earns its replacement.

Rate on write gives cheap reads but bakes in a price you may need to restate. Rate on read stays flexible but makes every read expensive.
Split by whether the work commutes: per-unit pricing is commutative and belongs on the write path; tiering and commitments are order-dependent and belong in a period-scoped computation.
Month-end close is a state machine, not a cron job — it has states you can be in, and transitions you must be able to explain.

A spend view that is fast and honest

The spend view is derived, may be stale, and must announce that. It is also where skew hurts: a whale tenant with orders of magnitude more resources than the median will hot-spot any naive partitioning by tenant.

Label estimates as estimates everywhere — field name, API, and UI.
Partition counters below tenant granularity for the largest tenants, then sum on read.

Auditability is set up now or never

Never destroy information. Every rated row carries its price_version_id, which is the single field that makes replay deterministic seven years later.

Being correct in the happy path is not the same as being correct. Add controls that do not depend on the pipeline being right: reconcile billed GPU-hours against scheduler allocation records and alarm on divergence.
Billing more GPU-hours than physically exist in a region is a bug you want to find before your customer does.

Accuracy is won at the edge

Most revenue leakage originates on the host, not in the pipeline. Agent-side buffering, clock discipline, and independent liveness reporting decide how accurate the whole system can possibly be.

Trade-offs worth naming out loud

Saying what you rejected, and why, is most of what separates a senior answer from a correct one.

Intervals with deterministic ids

instead of Point samples, duration reconstructed downstream

Samples force you to rebuild duration later, which breaks quietly the moment an agent has a gap — and quiet is the problem, because you under-bill without noticing. Intervals are additive and reorder-safe.

Append-only ledger, recomputable aggregates

instead of Mutable per-tenant counters

Counters are smaller and give real-time numbers, but a duplicate or lost update is undetectable and unrepairable. For money, auditability beats storage cost.

Price at usage time

instead of Price at processing time

An event arriving three days late must be charged at the rate in force when the GPU actually ran. Pricing on arrival makes replay non-deterministic and invoices indefensible.

Explicit adjustments

instead of Editing the issued invoice

Editing destroys the audit trail and breaks anything downstream that already consumed the invoice. Adjustments are how finance systems actually work.

What earns each level

Mid-level

A working pipeline. Recognises an invoice cannot be a live query, reaches event-time semantics with a nudge, and reasons toward freezing an issued invoice when prompted.

Senior

Gets to the additive ledger unprompted and explains why commutativity is the property that matters. Distinguishes heartbeat watermarks from arrival watermarks.

Staff+

Opens with the two-consistency-contracts framing. Names the close policy as a finance decision with a materiality threshold, and identifies where pricing stops commuting.

If you remember nothing else

01Money system → correctness and replay beat freshness. Say it in minute one.
02One pipeline, two consistency contracts: CP ledger, AP spend view.
03Intervals, not samples. Additive ledger, not mutable aggregates.
04Watermarks from heartbeats, so missing is never read as zero.
05Every row carries price_version_id → deterministic replay.
06Have an explicit policy for post-issue arrivals rather than hand-waving it.
07Integer micro-cents. Round once, at the invoice line.
08Reconcile against an independent source; do not trust your own pipeline.

← Back to the question Study System design →