1,356 turns in fifteen minutes

Share

Give every agent loop a turn budget before you give it a loop guard. The budget bounds the bug you have not found yet; the guard only bounds the bug you thought of. I build and operate agent fleets for a living, and that ordering is the most load-bearing rule I apply when a fleet goes from one agent to several. Here is the evidence behind it, and the two controls I now install before a second bot enters any room.

One human sentence went into a room with 4 bots in it. Fifteen minutes later those bots had generated 1,356 turns between them, the agent gateway was returning 429s, and Rocket.Chat was failing its liveness probe into a restart loop with the database probe timing out behind it.

Nothing crashed, and no single service was misconfigured. That is the part to internalize: there was no bug to fix anywhere in the estate, only a set of individually reasonable settings that composed into a machine for generating traffic. Multi-agent failures live in the composition, not the components — which means code review of any one service will never find them, and only a budget at the boundary bounds them.

Why four bots is not four times the traffic

Work the fan-out and the shape becomes obvious. In a room configured to answer everything, each bot sees every message — including the replies its 3 siblings just posted. So 1 message produces 4 replies, each of which produces 4 more — a 3x fan-out per lap, and growth that is exponential rather than linear. My room went from idle to saturated in under 5 minutes.

The per-bot counts landed at 310 to 380 turns each, which is the signature of the runaway rather than a busy afternoon. And 100% of the bridges logged the same reason for answering: the room was set to always-answer.

Here is the definition that was wrong, and it is worth stating precisely, because it reads as correct right up until you have more than 1 agent. "Always-answer room" has to mean answers humans unconditionally — not answers whoever speaks. Seeing a peer's reply and billing a turn on it are separate decisions, and only the first was ever what I wanted.

What does a loop guard that does nothing look like?

It looks exactly like a working one. My guard had 3 clauses: reject the message if the sender id is me, reject if the sender username is me, and reject if the platform marked the message as coming from a bot.

The third clause was dead.

Not wrong. Dead. Rocket.Chat does not set the bot flag on a message posted through chat.sendMessage by a token-authenticated bot user. The field is simply absent, so the check evaluated falsy and passed the message straight through.

Now read the first 2 clauses again. Both of them only recognize me. So my guard identified itself perfectly and held no concept of a sibling. Each peer's reply arrived looking like an ordinary human message in a room that answers everything.

A guard that tests a field the platform does not set is indistinguishable from no guard at all, and mine proved it across 1,356 turns. Assert your guard against a real message captured from the real producer, not against the message shape you believe the API returns.

How can a regex bug stay harmless for months?

Every name in my fleet is a prefix of another name, and my mention-matching used a word boundary. In Python, \b matches at a hyphen, because a hyphen is a non-word character — the re module documentation is explicit about it.

So a message addressed to one agent also woke the agent whose name was its prefix. The file's own docstring claimed the opposite, which is a good reminder that a comment is a claim and not a test.

# Wrong: \b matches at the hyphen, so "@hermes-dev" also matches "hermes"
pattern = rf"@{re.escape(name)}\b"

# Right: reject a following word character OR hyphen
pattern = rf"@{re.escape(name)}(?![\w-])"

This bug sat harmless for 6 months. It turned load-bearing the instant peers could address each other, which is the general shape worth remembering: a latent defect activates when a neighboring feature ships, not when you touch its code. I had not edited that file in 90 days.

The diagnosis took one line of shell

Count who is being answered, and compare that to who is human. That ratio is the entire investigation.

grep 'INFO answered' bridge.log | awk '{print $5}' | sort | uniq -c | sort -rn

Sibling names outnumbering mine is not subtle. It took under 60 seconds to read, and it beat every theory I had about rate limits and gateway capacity — both of which were real symptoms and neither of which was the cause.

LayerWhat it showedCause or symptom?
Chat serverLiveness probe failing, restart loopSymptom
DatabaseProbe timing outSymptom
Agent gateway429 rate limitingSymptom
Bridge logPeers outnumbering humans in answered-message countsCause

Fix the addressing, then add the backstop

Give each bridge the full roster of its peers and have it drop its own name from that list. A peer then triggers a turn only by explicit mention, so a deliberate handoff between agents still works and a broadcast does not cascade.

Then add the backstop that holds regardless of whether the addressing logic is right: a per-room turn budget, 12 turns per 300 seconds, claimed after the addressing check and before the run starts. Ordering matters here. Claim it after addressing, and an ignored message costs no budget.

Make the breaker announce itself once per window and then go quiet. A breaker that posts a warning on each trip becomes its own storm, which I say having briefly built one.

Keep a kill switch that does not require a code change. Mine is scaling the bridge deployment to 0 replicas: the chat server's webhook posts then fail harmlessly, no turns start, and because the deployment is declared in Terraform the next apply restores it. The fix and the restart are 1 action.

What I would tell you to check tonight

The key takeaway: your loop guard is probably asserting against a message shape you have never actually captured. Go pull a real message your own bot posted, print it, and confirm the field your guard reads is present in it.

Then answer the harder question — what bounds your loop if that guard is wrong? If the answer is nothing, you do not have a safety property, you have a correct-looking function. This is the same reasoning I used when deciding what an agent is allowed to do without a human, and it applies with more force once agents can hear each other. My fleet has been stable for 20 hours and counting since the roster and the budget went in, verified by watching peers log that they ignored each other.

Think of it like a room full of people who each answer any sentence they hear, including each other's. If you are running more than 1 agent in a shared room, I would genuinely like to know how you bounded it — the failure modes here are new enough that I do not think any of us have the full catalogue. I wrote about the earlier stage of this, when a bot became my repo's top committer, and the answer keeps being budgets rather than cleverness. As always, I'm here to help!