Hackathon (3rd place prize): AI agents that patrol our Grafana
Context
Our Grafana fires a steady stream of alerts. Most get skimmed, acknowledged or ignored, and whatever someone learns investigating one is gone by the next time it fires. In a hackathon week, four of us built Ranger: AI agents that patrol Grafana, investigate what fires, and write up each result with a root cause, evidence and a recommendation. A person reviews each one in a web UI and accepts or dismisses it. When they accept, an agent files the ticket, opens the pull request, or writes the alert rule change for someone to apply. One agent looks for failure signals nothing alerts on, like crash-looping containers or API error rates, and proposes one rule per gap that covers every service.
Problem
Agents that query monitoring run against the same log and metrics servers everyone's alerts depend on, so a careless run can break alerting for everyone else. They also had to stay cheap, never change Grafana on their own, and make sense to someone who wasn't on the team. And four people building in parallel needed a shared contract on day one, or we'd spend the week blocking each other.
What I did
On the first day I built the API, the SQLite knowledgebase schema and migrations, the Docker Compose setup and CI, so the UI and agent work could start right away and we could run and rebuild the app locally without any issue. Then I built a load guard. It watches the log and metrics servers' own health while agents run. It slows agents' queries when the servers are busy, holds new runs when they're saturated, and stops runs when other callers start failing. I also built an alert history service that pulls 30 days of fires from Grafana with no AI involved. It ranks the noisiest rules, and it tells the alerts agent which rules it has already investigated, so a repeat fire just gets counted.
Result
The guard got a real test mid-week. Before proposing a rule, the gap-finding agent replays it against past data to check it wouldn't be noisy, and long replays over high-cardinality metrics overloaded our shared metrics server. I sized replays by what they cost and taught the guard to watch the server's memory. It now trips far less often on ordinary load and still catches the pressure that matters. By the demo the whole loop worked: an agent spotted a problem in Grafana, a person accepted the fix, and an agent filed the ticket and opened pull requests across 3 repositories simultaneously.
What I'd change
I'd have the guard watch the servers' raw metrics from the start, not just their query queues. My first version tripped on ordinary queries and tried too hard to catch 'ramp up patterns', ending up acting like a hair trigger that blocked local work an annoying amount. It also still managed to miss the memory pressure that mattered. I'd also spend more time on the interaction patterns with agent issues, but this project will enter actual usage among several teams, so I may get that chance.