<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Naresh Kumar]]></title><description><![CDATA[Naresh Kumar]]></description><link>https://nareshh74.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Thu, 12 Nov 2026 09:36:58 GMT</lastBuildDate><atom:link href="https://nareshh74.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[What 84,800 local spans taught me about my coding agents and my local AI services]]></title><description><![CDATA[Token trackers tell you what your agents cost. They do not tell you why a child agent hung for ten minutes.]]></description><link>https://nareshh74.hashnode.dev/what-84-800-local-spans-taught-me-about-my-coding-agents-and-my-local-ai-services</link><guid isPermaLink="true">https://nareshh74.hashnode.dev/what-84-800-local-spans-taught-me-about-my-coding-agents-and-my-local-ai-services</guid><category><![CDATA[AI]]></category><category><![CDATA[Open Source]]></category><category><![CDATA[observability]]></category><category><![CDATA[Developer Tools]]></category><dc:creator><![CDATA[naresh kumar]]></dc:creator><pubDate>Mon, 09 Nov 2026 15:09:50 GMT</pubDate><content:encoded><![CDATA[<p>When I run one coding agent in one terminal, I can scroll up to find out what failed. My setup stopped working that way a few months ago.</p>
<p>On my workstation I run two kinds of software at the same time. The first kind is coding agents: Claude Code with subagents, OpenAI Codex CLI, GitHub Copilot CLI, Aider, and Cline. The second kind is the local services those agents call: a LiteLLM router for model routing and fallbacks, Ollama for local models, SearXNG for search, Firecrawl for scraping, and a local memory service.</p>
<p>Failures in that setup rarely print a stack trace. A parent agent stops answering. A nightly job produces nothing. A scrape-and-summarize run dies in the middle. I could not tell whether the fault was in the agent, in the router, or in a service.</p>
<p>Tools like ccusage and CodeBurn answer how many tokens I spent. agentsview lets me read transcripts. I needed something different: one place to see which child session hung, which service was slow, and whether last night's jobs ran.</p>
<p>So I wrote <a href="https://github.com/nareshh74/agent-tap">agent-tap</a>. It reads the logs, databases, and health endpoints that my agents and services already produce, and it stores them in one SQLite file. A dashboard at <code>http://127.0.0.1:6007</code> shows the result.</p>
<h3>Key takeaways (TL;DR)</h3>
<p>If you want to monitor and debug Claude Code, OpenAI Codex CLI, GitHub Copilot CLI, Aider, Cline, and local AI services in one local SQLite database:</p>
<ul>
<li><strong>One local SQLite database for coding agents and local AI services.</strong> agent-tap reads existing logs, SQLite state, and HTTP health endpoints from Claude Code, OpenAI Codex CLI, GitHub Copilot CLI, Aider, Cline, LiteLLM, Ollama, vLLM, SearXNG, and Firecrawl, then serves a dashboard at <code>http://127.0.0.1:6007</code>.</li>
<li><strong>Parent-to-child subagent tracing.</strong> It links parent agent sessions to child subagents across processes so you can see which delegated run timed out or stalled on an interactive prompt.</li>
<li><strong>Normalized error classes.</strong> It groups silent background failures such as <code>mcp_oauth_400_refresh_loop</code>, <code>gateway_first_byte_timeout_120s</code>, <code>litellm_midstream_fallback_error</code>, and <code>acp_prompt_timeout_600s</code> into queryable buckets.</li>
<li><strong>No proxy, no cloud account, and no prompt text on disk by default.</strong> It polls files in place with saved offsets, drops prompt and response text before writing spans, and masks API keys and bearer tokens.</li>
</ul>
<h3>What the first run showed</h3>
<p>The local SQLite database indexed 21,032 traces and 84,866 spans across 21,030 sessions. I looked at the error and correlation tables and found five problems that I had not noticed before.</p>
<ol>
<li><strong>An MCP token refresh failed about 2,030 times.</strong> An MCP server returned HTTP 400 on every OAuth refresh (<code>mcp_oauth_400_refresh_loop</code>), and the harness retried in the background without showing an error. Grouping spans by <code>error.class</code> made the loop visible in one query.</li>
<li><strong>The local router closed connections after 120 seconds.</strong> Long prompts took more than 120 seconds to reach the first token, so the router hit its <code>first_byte_timeout</code> and logged <code>gateway_http_499</code> drops. I found it by joining harness spans with gateway spans.</li>
<li><strong>LiteLLM 1.85.0 crashed in the middle of a stream.</strong> Delegated runs died with truncated tool calls (<code>litellm_midstream_fallback_error</code>). The parent-to-child link took me from the failed parent to the exact line in the gateway log.</li>
<li><strong>Child agents hung for 600 seconds.</strong> A child session waited on an interactive prompt with no terminal attached, and the parent looked frozen until the 600 second ACP timeout (<code>acp_prompt_timeout_600s</code>).</li>
<li><strong>Codex background jobs failed with HTTP 401.</strong> In a terminal, an expired credential fails loudly. In a job without a terminal, the 401 turned into an empty review. A query for Codex spans with status <code>ERROR</code> listed every affected job.</li>
</ol>
<h3>Agents and services in one model</h3>
<p>An agent run fails in the agent, or in a service the agent called. To see both, agent-tap treats every target as a service with a kind.</p>
<ul>
<li>A <code>harness</code> is a coding agent such as Claude Code, Codex, Copilot CLI, Aider, or Cline. It appears in Harness Activity with sessions, turns, tool calls, skill calls, and parent-to-child links.</li>
<li>A <code>gateway</code> is a model router or proxy such as LiteLLM or an OpenAI-compatible local gateway.</li>
<li>A <code>service</code> is anything else, such as Ollama, SearXNG, or Firecrawl. A service can be an HTTP endpoint, a Docker container, a process, or a log file.</li>
</ul>
<p>Gateways and services appear in Running Services &amp; Gateways with their status, request count, error count, and p50 and p90 latency. All targets share the error classes, the trace drawer, and the schedule view. To add a service of your own, run <code>agent-tap onboard --name my-api --kind service --health-url http://127.0.0.1:9000/health</code>.</p>
<h3>How it works</h3>
<p>agent-tap does not wrap your agents or sit in front of them. It reads files and databases that already exist and remembers its position in each one, so every poll reads only new data. The storage is one SQLite file in WAL mode. The tables have the same shape as Arize Phoenix, and you can query them with <code>agent-tap query</code>.</p>
<p>Prompt text is off by default. Before agent-tap writes a span, it drops prompt and response text and tool arguments. It keeps sizes, token counts, tool names, durations, session links, and error classes. If you turn prompt storage on, a redaction pass still masks bearer tokens, <code>sk-</code> keys, GitHub tokens, and JSON Web Tokens.</p>
<p>If an adapter misreads your logs, run <code>agent-tap bug-report</code>. It prints a diagnostic bundle with secrets masked and your home directory replaced by <code>~</code>, and <code>--open</code> starts a pre-filled GitHub issue.</p>
<h3>How agent-tap compares to ccusage, CodeBurn, agentsview, Arize Phoenix, and Langfuse</h3>
<table>
<thead>
<tr>
<th>Feature</th>
<th>agent-tap</th>
<th>ccusage</th>
<th>CodeBurn</th>
<th>agentsview</th>
<th>Arize Phoenix</th>
<th>Langfuse</th>
</tr>
</thead>
<tbody><tr>
<td>Primary job</td>
<td>Local runtime debugging, subagent traces, and service health</td>
<td>Token and cost reports from local logs</td>
<td>Token cost estimates and optimization advice</td>
<td>Transcript browsing and session search</td>
<td>LLM app tracing and evaluation</td>
<td>LLM app tracing, prompt management, and metrics</td>
</tr>
<tr>
<td>Normalized error classes from agent logs</td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>Only if your code emits them</td>
<td>Only if your code emits them</td>
</tr>
<tr>
<td>Parent-to-child subagent correlation</td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>Only with manual trace propagation</td>
<td>Only with manual trace propagation</td>
</tr>
<tr>
<td>Local AI service and gateway health (LiteLLM, Ollama, vLLM, SearXNG, Firecrawl)</td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>No</td>
</tr>
<tr>
<td>Scheduled task and cron monitoring</td>
<td>Yes</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>No</td>
<td>No</td>
</tr>
<tr>
<td>Storage and setup</td>
<td>One local SQLite file, no SDK or proxy</td>
<td>Reads local logs on demand</td>
<td>Local SQLite</td>
<td>Local SQLite</td>
<td>Server or container plus OTLP or SDK</td>
<td>Cloud or self-hosted server plus SDK</td>
</tr>
</tbody></table>
<p>agent-tap does not price tokens in dollars, so I run it next to ccusage or CodeBurn. I use Arize Phoenix or Langfuse when I instrument an application I am building, and agent-tap to monitor the coding agents and local services running on my workstation.</p>
<h3>Try it</h3>
<p>agent-tap is MIT licensed and needs Python 3.11 or later. It runs on Linux, macOS, Windows, and WSL2.</p>
<pre><code class="language-bash">pipx install agent-tap
uvx agent-tap doctor
pipx install git+https://github.com/nareshh74/agent-tap
agent-tap init
agent-tap run --once
agent-tap serve
</code></pre>
<p>Run <code>agent-tap doctor</code> first if you only want to see what it finds. The code and documentation are at <a href="https://github.com/nareshh74/agent-tap">https://github.com/nareshh74/agent-tap</a>. If you run several agents or local services, I would like to know which one your logs break first.</p>
<h3>Frequently asked questions</h3>
<h3>How do I find out why a Claude Code or Codex subagent hung?</h3>
<p>Run <code>agent-tap run --once</code> and open <code>http://127.0.0.1:6007</code>, or run <code>agent-tap query errors --json</code>. agent-tap links parent sessions to child subagent sessions across Claude Code, OpenAI Codex CLI, and GitHub Copilot CLI, and tags stalled child runs with error classes such as <code>acp_prompt_timeout_600s</code> or <code>gateway_first_byte_timeout_120s</code>.</p>
<h3>How does agent-tap differ from ccusage and CodeBurn?</h3>
<p>ccusage and CodeBurn calculate token usage and dollar cost from local logs. agent-tap focuses on operational debugging: normalized error classes, parent-to-child subagent trees, scheduled job runs, and health checks for local AI services. They read the same agent directories without conflicting, so you can run them side by side.</p>
<h3>Does agent-tap send prompts or code to the cloud?</h3>
<p>No. agent-tap makes no outbound network requests except local health probes to the localhost endpoints you configure. By default, <code>store_prompt_text</code> is <code>false</code>, so prompt text, response text, and tool arguments are dropped before any span is written to SQLite.</p>
<h3>How do I monitor local AI services like LiteLLM, Ollama, SearXNG, and Firecrawl?</h3>
<p>Run <code>agent-tap init</code> to auto-detect common local services on their default ports, or register any HTTP endpoint, Docker container, process, or log file with <code>agent-tap onboard --name litellm --kind gateway --health-url http://127.0.0.1:4000/health/liveliness --docker-container litellm</code>. agent-tap records status, error counts, and p50 and p90 latency in the Running Services &amp; Gateways table.</p>
<p>Naresh Kumar</p>
]]></content:encoded></item></channel></rss>