The agent that watches the agents (part 1)

How it started

We build Kinetik - an AI agent for content creators. Which means our production already houses dozens of agents helping people create content. And recently they got an overseer of their own - yet another agent, one that watches how these assistants (and the whole infrastructure beneath them) are feeling.

It sounds like recursion for recursion's sake: an agent that watches the agents that agents build. In practice, it's turned out to be one of the most useful tools we've put together in a long time. His name is Gilfoyle, he lives in Slack. He doesn't drink coffee, doesn't take vacation, is never late to standup. The perfect colleague - if you can look past his insufferable personality.

The pain

The context is simple: our company is small, the product is growing fast, and prod has to stay rock solid. Burning precious hours on the manual grind of alert triage is a luxury we can't afford. We'd rather write code and ship features than work as an on-call firefighter.

Normally, digging into any incident is a full-blown crusade. You go to Grafana for metrics, then into GCP logs for detail, then into APM traces to line one up against the other... In the tricky cases, these excavations eat hours.

Observability is usually covered by three disjointed tools. Metrics scream that something broke; logs and traces whisper why. But they usually live in different places, so someone has to sit in the middle and stitch them together by hand.

And we wanted more than fast reactions. We wanted to catch problems, bottlenecks, and performance regressions before they turn into incidents at all. For that you need one place that can pull every metric, log, and trace together, correlate the whole mess, and hand back a ready-made diagnosis.

The idea was hiding in plain sight

If modern LLMs can write decent code, why can't they debug infrastructure? Especially since we were already running part of our investigations with AI's help. But it was a local tool - you couldn't quickly ping it in chat or dictate a panicked voice memo straight from your phone on the way to the computer.

We honestly tried to find something off-the-shelf and open-source. We kicked the tires on a few options - no dice, they didn't cover our specific wishlist. So we built our own.

And if we're building our own - we decided to make not just a utility, but a virtual colleague with a personality. We thought about which engineer we'd most want on the team, and the answer came instantly: the brilliant, dry, cynical systems architect from Silicon Valley. And so our Gilfoyle was born.

We taught him to talk in that signature sarcastic tone. Some folks on the team found it jarring at first, but hey - it's part of his identity. And his verdicts land dead on target.

The architectural paradox: why Gilfoyle lives on his own

A fair question comes up here: "Guys, you build a platform for AI agents. Why not just run Gilfoyle inside your main infrastructure, as one of the system agents?"

The answer lies in the classic problem of the circular dependency. If our production goes down, our smart SRE agent goes down with it. And who's going to investigate the death of the infrastructure then? A detective can't solve his own murder.

So we applied the principle of isolation: we "chipped off," so to speak, a piece of tech from our main product and deployed it on a completely separate, isolated platform as an internal tool for the cloud team. If our prod turns into radioactive ash, Gilfoyle stays alive, calm, and ready to tell you exactly who broke everything.

Anatomy

Under the hood, he's lean: the open-source OpenClaw framework, wired up with a set of specialized skills for SRE analysis. We use OpenClaw heavily on the project: it's great at turning a faceless language model into an autonomous "colleague" that knows how to use tools.

Gilfoyle's heart is his skills - and they're older than he is. We'd been sharpening them by trial and error long before the bot existed, back when we only had local agents:

1. Reading logs without polluting the context. A subagent gets its instructions - where to go, which logs to pull, which patterns to look for - so the main agent never clogs the session context with junk logs.
2. Metrics analysis. The agent hits the Grafana API directly and pulls whatever metrics it needs.
3. Cloud and cluster state. The agent has a read-only service account that lets it reach into GCP and GKE directly and get full read visibility across our entire infra.
4. Deep debugging of user flows. When an incident is about what a specific user ran into, he can retrace the whole flow - working only from anonymized identifiers, never anyone's personal data.
5. The cherry on top - the project repo itself. Since so much of our work is done by agents, the repo is full of agent md files describing the structure of every service, the user flows, the infra, and more. Any agent connected to our repo is, by design, spared from having to grind through hundreds of thousands of lines of code to understand how things work - always-up-to-date instructions and skills do the heavy lifting.

So Gilfoyle works only from what's actually there - no guesses, no hallucinations, only dry facts, real metrics, and real logs. And once we saw that nine times out of ten his diagnosis was dead-on, we went a step further and let him create branches and pull requests with code fixes. With mandatory human approval, of course.

Why we're not afraid to let him into prod

Giving an AI agent access to your infrastructure and codebase is every lead's biggest fear. But we built hard safety boundaries:

1. Read-only on infrastructure and data. GCP, the cluster, the data stores, the logs - Gilfoyle only reads them. He changes nothing there.
2. On code, the most he can write is a proposal. He can open a branch and a pull request with a suggested fix - and that's the ceiling. He can't push to main; a human reviews and merges, or doesn't.
3. Full audit trail. Every action he takes is logged, and the tokens he uses to reach the cloud and GitHub are his own. The audit trail always tells us: "This request was made by Gilfoyle."

Essentially, he has fewer permissions than an intern on day one. But more patience and focus than the whole team combined.

For all the noise about AI in engineering, remarkably few teams actually let an agent touch production - everyone's scared to give one real access to sensitive data. But if you set up the constraints and the principle of least privilege properly, an agent becomes a massive force multiplier. It makes the concept of observability actually work.

The second popular fear: "My logs will leak to OpenAI/Anthropic's servers!" But come on - your data already sits on Google's, Amazon's, or Microsoft's servers. Being afraid to let an agent into your infrastructure even in read-only mode is destructive - you're depriving yourself of a smart helper over an irrational anxiety.

The punchline, in the field

The beautiful part kicks in when an alert fires.

The old scenario looked like this: an alert comes in about rising API response times. An engineer opens the logs, tries to figure out which endpoint is lagging, digs into the database to look at slow queries. That's 10 to 30 minutes of manual work.

Now we just tag Gilfoyle in Slack. A minute later, he hands back a structured answer. For example:

Gilfoyle: "Latency spike detected on the /api/agent/chat/sessions/{projectId} endpoint. Logs show a DB connection error (Postgres ConnectionTimeout). The Hikari connection pool is exhausted by a heavy query added in the last PR, #114. I've created a branch fix/db-pool-leak and pushed a fix there that tightens the query and bumps the pool limit. Feel free to review. And yes - don't forget to thank the author of that PR."

Sometimes he finds the root cause faster than I can open the GCP console on my laptop. The path from hypothesis to a precise diagnosis has shrunk from half an hour to about a minute.

What's still not right

Perfect systems don't exist, and we keep polishing Gilfoyle:

1. Alert-storm protection. We haven't set up deduplication yet. If Gilfoyle were running in automatic mode, he'd try to work through every single one of a hundred simultaneous alerts and blow through the API limits. That's exactly why, for now, we tag him onto a problem manually.
2. Walls of text. He still lapses into graphomania now and then - you ask "what broke?" and get back a three-act essay. We've mostly trained it out (short answers by default, the long version only on request), but he relapses on a bad day.
3. Mistakes. Sometimes he chases a false lead. But if you pull him back down to earth in time with something like "Gilfoyle, the database has nothing to do with it, look at the cache," he instantly rebuilds his hypothesis, admits the mistake, and finds the right root cause. Admitting mistakes, by the way, is something not every living developer manages.

What's next

We're already inching toward proactive routine work: every morning Gilfoyle dumps a health analysis of our main services on us, and every evening - a list of PRs and a summary of the day's changes. Next up: analyzing anomalies that haven't triggered alerts yet, hunting for hidden errors in the logs, and digging into user-behavior anomalies. He'll become the project's éminence grise - quietly keeping watch in the background - but that's a story for the next episode.

And we have a very down-to-earth reason to push this direction.

Picture the classic 3 a.m. incident. A critical alert fires. Then comes the chain of delays: the system has to page PagerDuty, the phone has to wake the on-call engineer, who has to rub his eyes, get to the computer, log in through every VPN, open the terminals... In reality, 10 to 20 minutes go by. For systems with a 99.99% SLA, that's an eternity.

Gilfoyle doesn't wait for any of that. The investigation starts the millisecond the trigger fires - no page, no VPN, nobody rubbing their eyes. By the time someone actually reads the alert, the work is already done.

We didn't hire ourselves a colleague who never sleeps - we built one. And this one will happily say "I told you so" - he remembers every bad commit you've ever made.

Watch in the next episode