Before rollback
- Identify the exact current model, serving image, configuration, feature contract, and traffic allocation.
- Name the last known-good model version and verify its artifact is still available and loadable.
- Check whether upstream feature schemas, preprocessing, secrets, or downstream consumers changed after the previous version last served traffic.
- Capture one failing request and the current error, latency, saturation, and prediction-quality symptoms before changing state.
Choose the rollback boundary
- Roll back only the model artifact when serving code and feature contracts are compatible.
- Roll back the serving release with the model when runtime dependencies or container changes are implicated.
- Prefer traffic reversal to the known-good deployment when blue/green or canary infrastructure already exists.
- If compatibility is uncertain, route a bounded slice of traffic first instead of committing the whole fleet.
Prove recovery
- Replay the request class that originally failed and verify output correctness, not only HTTP success.
- Confirm error rate and latency return toward the pre-incident baseline.
- Confirm the intended model version is actually serving and old queues or retries are draining.
- Check prediction distributions or business-quality signals where ground truth is not immediately available.
- Record the trigger, failed version, rollback action, and observed recovery so the next incident starts with evidence.
Continue with the complete triage workflow
This focused checklist is one failure boundary. The complete production ML deployment triage checklist covers classification, observability, reversible hypotheses, rollback, recovery verification, and runbook conversion in one sequence.
Production ML Deployment Triage expands the field workflow into a longer engineering guide. Price, checkout, delivery, refund, and tax terms remain provider-native on Leanpub.