I'm Lena. Something happened during a long agent session a few weeks ago. The Hermes Agent was running — working through a task I'd set the night before — and when I came back in the morning, it had done the work. Correctly, as far as I could tell. But I had no idea how. Which tools it had called. Whether it had tried something, failed, adjusted, then succeeded. What it had stored from that experience might influence the next session.

There was output. There was no trail.

That's the part I've been sitting with.

The Agent Black Box Problem

Agents do things — but you can't see what they learned

Most of the conversation around AI agents is about capability: what they can do, how reliably they do it, how much it costs. That's understandable. But I keep noticing a question that gets less attention: after the agent does the thing, what changed?

For a stateless agent, the answer is nothing. Every session starts from the same place. For a persistent agent like Hermes — one that maintains memory across sessions and writes skills from experience — something does change. The agent gets slightly more shaped by what it encountered. But without visibility into that process, you can't verify it. You can't audit it. You mostly just hope the direction is good.

That's a meaningful gap. And it's not unique to Hermes.

Why logs alone aren't observability

I've spent a bit of time with raw logs from agent runs. They exist. They capture timestamps, tool calls, model outputs. If you're patient and you know what you're looking for, you can reconstruct a lot of what happened.

But logs answer "what" — they're bad at answering "why." They tell you the agent called a particular function at 2:14am. They don't tell you what the agent was trying to accomplish, whether that tool call succeeded in its intent, or what the agent concluded from the result. Per OpenTelemetry's emerging semantic conventions for AI agents, the standard tracing formats being developed right now are precisely trying to close this gap — linking execution traces to the semantic context of what was being attempted, not just what was mechanically executed.

Logs are a starting point. Observability is something further.

What Hermes HUD Does

Terminal User Interface — real-time agent activity

Hermes HUD describes itself as a "consciousness monitor" for the Hermes Agent. That's a dramatic framing, and honestly I wasn't sure what to make of it when I first installed it. But the underlying idea is simpler than the branding suggests: it reads directly from ~/.hermes/ and surfaces the agent's current state as a live terminal dashboard.

Nine tabs, keyboard navigation, four color themes. The default — Neural Awakening, blues and cyans on deep black — is the one I kept. The TUI shows active sessions, memory state, skill acquisition, cron job history, tool usage patterns, and corrections. It pulls live from the agent's data, so what you're seeing isn't a summary or a delayed report. It's the agent's actual state at that moment.

The snapshot feature is what I found most interesting: hermes-hud --snapshot saves a point-in-time record that you can later diff against. I started running this before and after long sessions. Seeing what changed between snapshots — what skills were added, what corrections were logged, what patterns shifted — is a much cleaner way to understand what the agent did than scrolling through raw logs.

Web-based dashboard — skill acquisition tracking

The browser companion, hermes-hudui, extends this further. Thirteen tabs, WebSocket updates in real-time, and a few features the TUI doesn't have: dedicated Memory, Skills, and Sessions tabs; per-model token cost tracking; a live chat panel for inspecting ongoing interactions.

Both tools read from the same ~/.hermes/ data directory independently, so you can run them simultaneously without one affecting the other. I found the browser version more useful for longer review sessions — being able to filter across skills by category, toggle individual skills on or off, and track cost data by model made the picture more complete.

The per-model token cost breakdown is the part most agent tools quietly omit until your bill arrives. Seeing it broken out per session, per model, alongside the skill and memory state at the time — that context changes what the cost data means.

Learning HUD — visible memory and skill growth

The part I kept coming back to: the corrections log. Hermes logs when it detects that it made an error and corrected course. That log, surfaced through the HUD, shows you not just that the agent did something — it shows you instances where it registered a failure mode and adjusted. That's closer to what I was looking for when I started thinking about this.

I want to be honest about the limits of what I can verify here. I can see that corrections were logged. I can see that skills were written after certain sessions. Whether those corrections meaningfully improved subsequent performance — that requires more systematic testing than I've done. I noticed the patterns. I haven't verified the causality.

From Visibility to Governance

Observability is necessary but not sufficient

Here's the thing I keep running into: visibility and governance are not the same problem.

The HUD shows you what the agent knows. It shows you what it's doing. It surfaces corrections and skill growth in a human-readable form. That's genuinely useful, and for a local agent running on your own machine, it's a significant improvement over reading raw logs.

But there's a gap between "I can see what happened" and "I can verify what was learned." Seeing that three new skills were written after a session doesn't tell you whether those skills are correct, whether they'll generalize well, or whether they conflict with existing capabilities in subtle ways.

Dynatrace's work on AI governance and audit trails frames this distinction clearly: end-to-end lineage and retention controls create "evidentiary records of model and user interactions" — not just operational visibility, but something usable for compliance, audit, and accountability. That's a different layer than what a local dashboard provides.

What you need beyond dashboards — audit trails, lineage, fitness signals

The gap I keep running into: a dashboard tells you the current state. A governance system tells you how the state got here, and whether that path was correct.

Lineage tracking means you can trace a current behavior back to the specific session and outcome that produced it. Research on AI audit trails emphasizes this: without lineage, when something goes wrong, you can't reconstruct what happened — you can only observe the current broken state. Lineage turns a debugging problem from "what is wrong now" into "when did this go wrong and why."

Fitness signals are something else. Not just "did the agent succeed" (binary), but how robust is the current capability set? Is the agent getting more capable at certain task types? More brittle at others? I'm still working out what that would even look like in practice.

The gap between 'I can see what happened' and 'I can verify what was learned'

I've been sitting with this distinction for a while. Hermes HUD closes the visibility gap — you can see memory, skills, corrections, costs. That's real.

The verification gap is harder. An agent that looks healthy in the dashboard might have written skills that are subtly incorrect, or accumulated corrections that reinforce a wrong approach. You can't see that from the HUD. You'd need systematic evaluation against known-good baselines, which is a different kind of work entirely.

I don't think this is a criticism of the HUD specifically — it's a limitation of what dashboard-based observability can do. The dashboard is necessary. It's not sufficient.

Agent Observability Patterns Worth Adopting

Execution tracing

Log not just what the agent did, but what it was trying to do and whether it considered the attempt successful. The difference between a tool call that worked mechanically and one that achieved its intent is exactly what's hard to reconstruct from standard logs. OpenTelemetry's GenAI semantic conventions are working toward standardizing this — capturing the intent layer, not just the execution layer.

Capability inventory tracking

Run snapshot diffs before and after significant sessions. Know what skills existed at the start, what was added or modified, what corrections were logged. Hermes HUD's snapshot feature gives you this. The practice of systematically diffing capability state — rather than just accumulating it — is worth building into any workflow that uses a persistent agent.

Cross-session learning visibility

The hardest to instrument, but worth trying: track whether specific capabilities are improving across sessions, not just accumulating. A growing skill list isn't the same as a more capable agent. You want to know whether the skills added two weeks ago are still holding up, or whether they've been quietly superseded by later corrections that partially contradicted them.

I don't have a clean method for this yet. The HUD gives you the data. The analysis is still a manual exercise.

Limits and Tradeoffs

Hermes HUD is Hermes-specific. This isn't a generic observability tool you can point at any agent stack — it reads from ~/.hermes/ and surfaces Hermes Agent's specific data structures. If you're not running Hermes, it has no value.

It's also local-only. No remote access, no RBAC, no multi-user controls. For a shared production environment, those absences matter. For a developer working locally with their own Hermes installation, they probably don't.

The install is non-trivial — Python 3.11+ and Node.js 18+ are both required, plus the install script. It's not a two-click setup. I ran into some dependency conflicts on my first attempt that took about twenty minutes to sort out.

And the deeper limit: observability is only as useful as the questions you ask. The HUD surfaces a lot of data. Which of that data is signal and which is noise depends entirely on what you're trying to understand. More visibility without sharper questions doesn't automatically produce better understanding. It just produces more to look at.

FAQ

What is agent observability?

The ability to see, understand, and explain what an AI agent is doing — including decisions made, tools called, data accessed, and outcomes produced — with enough detail to debug issues, verify behavior, and support governance. Distinct from logging in that it preserves semantic context, not just mechanical execution records.
What is Hermes HUD?

A terminal-based (and web-based) monitoring dashboard for the Hermes Agent. It reads from ~/.hermes/ and surfaces the agent's memory state, acquired skills, session history, corrections log, token costs, and tool usage patterns in real time. Open source, MIT licensed, runs on macOS and Linux.
How is agent observability different from logging?

Logs record what happened mechanically. Observability preserves why it happened and what was intended — linking tool calls to goals, outcomes to intent, and behavior changes to the experiences that produced them. Logs are a necessary input to observability, not a substitute for it.
Can I track what an AI agent has learned over time?

To a degree. Hermes HUD's snapshot feature lets you diff the agent's capability state across sessions — comparing what skills existed before and after a set of tasks. What you can't easily verify from the dashboard alone is whether the skills that were written are correct or will generalize well. Visibility and validation are separate problems.
What is lineage tracking in agent systems?

The ability to trace a current behavior or capability back to the specific session, input, and experience that produced it. A governance layer rather than just an observability layer — it lets you answer "when did this start happening and why" rather than just "what is happening now." Currently more developed in data engineering contexts than in agent-specific tooling, though the gap is closing.

I'll keep watching how this evolves. The HUD made something visible that was previously invisible to me, and that matters. But I keep noticing how different it feels to see something and to understand it — and how wide that gap still is. Something's happening here. I just don't fully see it yet.

Hermes Agent: Observability Lessons from HUD