Hermes Agent: what contributing to an open source AI agent teaches us

Anyone can claim to know AI agents. Few vendors can prove it in front of strangers who owe them nothing.

Open source makes that possible. We contribute to Hermes Agent, the open source AI agent from Nous Research. This article explains what the project is, what we have contributed, and above all what the exercise teaches us about running an AI agent in production, for ourselves and for the businesses we work with.

Hermes Agent in brief

Hermes Agent is an AI agent released under the MIT licence. Unlike a chat assistant, it does not live in a browser tab: it runs on a server, keeps memory between sessions, and acts. It schedules recurring jobs, delegates work to subagents, runs commands in isolated environments (Docker or SSH, among others) and answers from the messaging tools a team already uses: Telegram, Slack, Discord and more.

It is not tied to any model provider. You pick the model that suits you, and you can change it without rebuilding the installation. For a business, that is decisive. An agent whose hosting and model you do not control is an agent whose data you do not control.

What we contributed

Our contributions target the least visible and most critical part of an agent: what happens when it runs for a long time with nobody watching.

The one now in the project's main branch fixes how the agent behaves after a restart. When the service restarts, during an update for instance, the agent notifies its team on their messaging channel. It used to send that notice to the wrong place whenever the original conversation was a thread: the "back online" message landed in the main channel, out of context. The fix keeps the exact destination across the restart, scheduled tasks included, and it ships with its regression tests. The maintainers merged it by cherry-picking our commit onto the main branch, with authorship kept.

We have proposed other improvements of the same kind: passing arguments to scheduled job scripts, refreshing system services before an update, loading plugins at startup, routing commands from GitHub pull request labels, interface translations. Not all of them are accepted, and that is normal: the maintainers of a project this size sort through a constant flow of proposals, and a good idea may be picked up in another form. The pull requests, the discussions and the merged code are all public, and we walk through them in a meeting with anyone who wants to see them.

What maintainer review teaches us

An agent is judged over time, not in a demo

Getting an agent to answer once is easy. Keeping it running for weeks, through restarts, updates and network outages, without losing the thread or a scheduled task, is a different job. The defects we fixed never show up in a demo. They show up on a Sunday, when nobody is looking.

That is the gap most AI agent projects fall into. A proof of concept works in a meeting room; the questions that matter come later. What happens to a task that was running when the server restarted? Where does a notification go when the conversation it belongs to has moved? Who finds out when a scheduled job silently stops? An agent you rely on needs answers to these before it gets access to anything important.

A fix without tests is not a fix

A proposal that does not prove what it fixes is not reviewed. The fix that was merged contains noticeably more lines of tests than lines of code. It is the same discipline we apply to our own software, and that we describe in our article on AI-assisted development and human review.

Tests matter even more for agents than for ordinary software. An agent's behaviour depends on timing, on external services and on a model whose output varies. Regression tests pin down the parts that must never change, such as where a message is delivered or whether a task survives a restart, so that everything around them can evolve safely.

An outside reviewer sees what you no longer see

Maintainers know nothing of our context or our constraints. They read the code as it is written, not as we meant to write it. That is exactly the role of the second review in our own process: having the work read by someone, human or model, who does not share the author's blind spots.

Public review also has a useful side effect: it is a record. Every comment, every requested change and every decision stays attached to the code. When a business asks how a vendor works, there is something better than a promise to point at.

What this means for your project

An AI agent in a business is not one more chatbot. It has access to tools, to data, sometimes to production systems. It must therefore be built, hosted and monitored like any other critical software:

• least privilege: the agent gets the permissions its task needs, and nothing more;
• human approval for sensitive actions such as sending money, deleting data or emailing customers;
• logging of what it did, when and why, so an incident can be traced;
• monitoring of the boring things: restarts, failed scheduled jobs, running costs;
• a plan for model changes, because providers update and retire models on their own timetable.

Choosing an open source agent such as Hermes Agent or building on a managed platform is a separate decision, with its own trade-offs between control, effort and cost. Our article on the cost of a custom AI agent lays out the budget lines on both paths.

This is what we put into practice, with the same method we use for a business application: human-led scoping, a free visual prototype within 5 working days, then development at a fixed price or on a subscription, with no billing by the hour or the head. If you are considering an agent for your team, the first meeting is there to check that it is the right answer to your problem. It is not always, and we will tell you so.