Decide whether a ranking change shipped well
ML designMediumSingle report · 1 reportLast seen Aug 18, 2026
The question
Offline metrics improved but the online engagement metric is flat. Enumerate the plausible explanations, then describe the smallest experiment that distinguishes between them.
No write-up yet
We have the question but not yet a full breakdown. If you were asked this, the fastest way to improve the page is to say what the interviewer pushed on.
Add what you remember →1 person reported this question. One more independent report moves it to corroborated.
I was asked this tooAlso reported at OpenAI.