Troubleshoot a Foundry hybrid agent
Symptoms you are likely to hit, what causes each, the fix, and where to find the container logs.
Before you startA deployed agent and read access to the project's Application Insights workspace.View as Markdown
Symptoms and fixes
| Symptom | Cause | Fix |
|---|---|---|
Runs die at about 15 minutes: stream cut, background stuck in_progress, no envelope | No session idle timeout on the version (default 900 s) | Set SESSION_IDLE_TIMEOUT_SECONDS=3600 and redeploy |
internal server error on every request; container looks healthy | Protocol declared as responses 1.0.0 | Declare 2.0.0 |
Version rejected: reserved for platform use | An AGENT_* or FOUNDRY_* name in environment_variables | Move it to the Dockerfile ENV |
image_pull_failed, UnauthorizedAcrPull, InvalidAcrPullCredentials | Account and project managed identities lack classic AcrPull | Grant AcrPull on the registry to both |
401 Principal does not have access to API/Operation | The model grant went to the blueprint, or to the project | Grant Foundry User to the instance identity at account scope |
Long run dies with EACCES then Prompt is too long | The CLI's config dir is under the unwritable /home/session | Keep HOME and CLAUDE_CONFIG_DIR under /work in build_options |
JSON message exceeded maximum buffer size | SDK default 1 MiB buffer | Keep the template's 64 MiB max_buffer_size |
Every request refused: MAX_TURNS and MAX_USD_PER_RUN must be set | Ceilings missing from the version | Set both and redeploy |
| Every request refused: request field collides with Starting step fields | A request field named title, delivered, model or agent_version | Rename it and rebuild |
| Run fails mid-way on a model that "doesn't exist" | An unpinned tier fell back to a missing deployment | Pin all three ANTHROPIC_DEFAULT_*_MODEL |
ZoneInfoNotFoundError in the container | No tzdata in the image | Install tzdata in the Dockerfile |
| Failed publish but AMC shows a green run | Turn ended normally, so the stream ended response.completed | Raise RunFailed after the markers in _run_stream |
Run costs several times normal, over a hundred Bash calls, skills_bypassed set | The prompt led with a command and the skill never loaded | Restore the "FIRST, invoke the skill" opening |
| Run dies with no envelope after fanning out | Task enabled for a skill that doesn't delegate | Put Task back in disallowed_tools |
| MCP calls start failing 401 near the end of a long run | The per-turn token expired (about 60 to 90 minutes) | Lower RUN_TIMEOUT_SECONDS or put a refresh proxy in front of the server |
| HITL agent ends green having asked nobody | The brain didn't write hitl_request.json | Enforce a mandatory pause in _postprocess |
Steps fragment into many (2), (3) repeats | Phase idle close changed | Keep PHASE_IDLE_MESSAGES = 2 |
| Cost in AMC far too low | input_tokens reported without cached tokens | Report cache-inclusive input_tokens with cached_input_tokens as the subset |
| Cost in AMC doubled | Run totals or cost_usd on the trailing markers while steps carry tokens | Trailing markers carry the residual only |
| A probe fails with a bare socket error at start | Cold container still starting | Connect with a generous timeout (the probe uses 180 s) |
Where the logs are
Container stdout lands in the Log Analytics workspace behind the project's Application Insights.
- `AppTraces`, filtered on
AppRoleName == '<agent-name>', oneAppRoleInstanceper container. Every step is also logged asstep: <title>. - `AppExceptions`:
Detailsholds the stack,Propertiesthe response id, agent version and caller. - Lines logged at startup, before the hosting server configures the exporter, reach stdout only. That's why a readiness refusal is logged again per request.
- A
ConnectionErrorto169.254.169.254 /metadata/instanceis the App Insights resource detector probing a sandbox that has no metadata service. It's noise.
bash
az monitor log-analytics query -w <workspace-customer-id> \
--analytics-query "AppTraces | where AppRoleName == 'invoice-summarizer-agent' | order by TimeGenerated desc | take 200"Query the workspace rather than the App Insights traces blade, which has gone empty mid-investigation. KQL reserves first and latest as column names.
Process-loss recovery
A hosted run dies with the process serving it (a deploy, a replica recycle) and sticks at in_progress. Foundry's resilient_background fix is workflow-only and a hybrid agent is a single agent, so it doesn't apply. If you genuinely need process-loss recovery, model the unit as a one-node workflow; see Microsoft Agent Framework (Python).
Related
Last updated 2026-10-21