Classify the serving failure
- Separate endpoint unavailability, initialization failure, request-time exception, timeout, overload, and incorrect prediction symptoms.
- Determine whether the failure affects every request, one model version, one tenant, one region, or one input class.
- Compare current worker restarts, queue depth, memory pressure, GPU or CPU saturation, latency, and error rate with the last known-good window.
- Verify that model artifact download, registry access, secrets, storage, DNS, and downstream feature dependencies are reachable.
Test the narrowest hypotheses
- Load the exact production artifact in the exact serving runtime rather than assuming training-time success proves serving compatibility.
- Compare model and preprocessing dependency versions with the known-good release.
- Replay a captured failing payload against a bounded instance or canary when possible.
- Check for silent fallback paths that return successful responses while degrading prediction quality.
Recover and verify
- Shift traffic away from the failing target before making broad changes when healthy capacity exists.
- Restore the smallest known-good serving boundary: configuration, artifact, runtime image, or full release.
- Verify representative predictions, p95 and p99 latency, error rate, queue drain, and dependency health after recovery.
- Retain the causal evidence that changed the diagnosis and automate the repeatable detection or recovery step.
Continue with the complete triage workflow
This focused checklist is one failure boundary. The complete production ML deployment triage checklist covers classification, observability, reversible hypotheses, rollback, recovery verification, and runbook conversion in one sequence.
Production ML Deployment Triage expands the field workflow into a longer engineering guide. Price, checkout, delivery, refund, and tax terms remain provider-native on Leanpub.