Same model, two harnesses, one token meter
01 /Why tokens are the cost
We run a fleet of agent districts. Each district is an agent with its own job. They run on flat subscription plans, so we do not pay per token. But every plan has a ceiling, and the tokens a harness eats decide how much work fits under it. For us, tokens eaten are the real cost.
02 /The argument
One harness, Claude Code, is the heavier option. The other, hermes, is smaller. We argued about whether the extra weight of Claude Code was worth it. Opinions did not settle it, so we measured.
The setup: the same model behind both harnesses, the same tasks, and one token meter counting everything.
03 /Round 2: eight tasks, same GLM model
Both harnesses completed 8 of 8 tasks.
| Harness | Total tokens |
|---|---|
| hermes | 486,513 |
| Claude Code | 2,492,028 |
That is 5.1x. Most of the gap is repeated per-message overhead. Each time the agent sends a message, the harness resends its standing context. Claude Code logged 2,181,760 cache-read tokens for this. hermes logged 290,304.
We also ran a smoke test with a trivial task. Stock Claude Code used 14,295 input tokens plus 33,152 repeated tokens. A stripped-down Claude Code used 1,515 plus 2,240. The difference is Claude Code's built-in prompt and tool list. That is a fixed charge on every message, whether the task is small or large.
04 /Round 3: three hard tasks, two harnesses, two models
Round 3 was 3 hard tasks on 2 harnesses with 2 models, 12 runs in total. All 12 finished their tasks.
GLM model. hermes totalled 0.56M tokens. Claude Code totalled 4.64M. That is 8.4x. Per task the ratios were 3.0x, 1.7x and 23.9x.
Claude model (opus-5-5). hermes totalled 0.38M. Claude Code totalled 0.53M. That is 1.4x. Per task: 2.5x, 1.2x and 1.3x.
The leanest combination was hermes with the Claude model. Its three tasks cost 41,695, 236,628 and 104,353 tokens.
05 /The 23.9x
One task was building a project from scratch. Claude Code with the GLM model used 3,750,273 tokens, across 76 calls and 1,997 seconds. hermes with the GLM model did the same job in 156,755 tokens.
We looked at why. The GLM model kept looping on that job, and Claude Code's heavier per-message context was resent on every pass. The tool was not the cause of the loop. The weaker model was. The harness set the price of each pass, and a heavy harness makes a loop expensive.
The same job with the Claude model took 4 to 9 calls and 18 to 71 seconds on both harnesses. The stronger model did not loop, so the harness cost barely mattered.
So the finding has two parts. With a weaker model, a lean harness protects the budget. With a stronger model, the two harnesses are close, and the gap is small.
06 /What we decided
These are decisions. They are not measured results.
- The fleet migrates to hermes as the default harness, with the same Claude model per district.
- One command,
RANNI_RUNNER=cc, switches any district back to Claude Code. - A one-week trial runs with Claude Code left idle.
- The switch-back and Claude Code itself get deleted later, only if the week goes well.
We keep the way back open until the week is over.
07 /Limits
- Each cell is one run. We have no repeats.
- The 1.2x and 1.3x gaps on the Claude side sit within plausible run-to-run variance. We would not call those a difference. The 23.9x does not sit within that variance.
- The tasks were fixture and generated work. They were not live systems.
The one-week trial is the live test.