Concept guide · Agents, tools, and MCP
An agent succeeds in testing but fails in production. How do you debug it?
A slower explanation for when the short interview answer is not enough. Start with the intuition, then connect it to the technical idea and the way an interviewer expects you to reason about it.
ML area
Generative AI → Agent production debugging
Interview format
Advanced · Design
Question
GENAI-088
Start with the idea
In simple terms
When an agent works in testing but fails in production, compare the full traces: inputs, model and prompt versions, context, tools, permissions, latency, retries, and state, not only the final text.
Make it concrete
A test may use a fast mock search result, while production returns a large conflicting response that pushes the important instruction out of context and triggers a loop.
Build the answer
Structure your answer
- 1Capture a comparable, privacy-safe trace with state and tool results.
- 2Separate planning, execution, recovery, and environment differences.
- 3Turn the incident into a deterministic regression or adversarial test.
Go one level deeper
The technical explanation
Reproduce the smallest failing run from a redacted production trace. Classify whether planning, context, retrieval, tool execution, permissions, recovery, or model variance caused the failure. Add the trace to regression and failure-injection suites, then fix one layer at a time so a prompt change does not mask an operational bug.
Say this in the interview
Compare production traces, inputs, tool results, context size, latency, model versions, permissions, and user goals with the test environment. Reproduce the smallest failing trace, identify whether planning, retrieval, execution, or recovery failed, then add the case to regression and safety evaluation.
Keep preparing