Engineering note
Reducing agent token use: a controlled benchmark
On the same model and the same five agent tasks, our in-house workbench LLM Studio used 0.25× the tokens of Nous Research’s Hermes Agent in its latest run, with all 25 runs succeeding. The main lever was not a better model but a smaller prompt: each request sees one tool, not the whole catalogue.
The setup
- Tasks. Five small agent tasks — read a file, search files, edit a file, a multi-step task and counting lines in a terminal — each repeated five times: 25 runs per measurement.
- Verification. Every task carries a nonce, so the agent must actually run the tool and return the right value. An answer from memory does not count.
- Model.
deepseek-v4-flashon both sides, through the same provider. - Baseline. Hermes Agent was measured once, on 25 July 2026, and kept as a frozen reference so that the reference itself would not drift between versions.
What counts as a token
Total tokens = every prompt token sent to the provider (cache reads included) + every token the model returned (reasoning included) + any auxiliary or compression calls. Early in the project the two tools recorded usage on different bases; the definitions were normalised to this one before any comparison was made. Comparing vendors without this step compares bookkeeping, not systems.
Results
Average total tokens per task
By task (average tokens per run)
- Read a file9,6342,1540.22×
- Search files9,6632,5930.27×
- Edit a file15,0863,5330.23×
- Multi-step task10,2122,8600.28×
- Count lines (terminal)9,5362,4990.26×
| Run | Date | Succeeded | API calls | Total tokens | Per task | vs Hermes |
|---|---|---|---|---|---|---|
| Hermes Agent (baseline) | 2026-07-25 | 25/25 | 55 | 270,656 | 10,826 | 1.00× |
| LLM Studio V2j | 2026-07-28 | 24/25* | 52 | 86,045 | 3,442 | 0.32× |
| LLM Studio V2m | 2026-07-29 | 25/25 | 50 | 62,804 | 2,512 | 0.23× |
| LLM Studio V2o (latest) | 2026-07-29 | 25/25 | 53 | 68,197 | 2,728 | 0.25× |
*The single failure in V2j was a provider-side HTTP 400. Its root cause was found and fixed (below); every later run scored 25/25. LLM Studio used fewer tokens than Hermes on all five tasks.
How we got there
- 01
First version: 1.42×
Our first measured version used more tokens than Hermes. A separate tool-routing step called the model 25 extra times — 28,079 tokens — just to choose tools.
- 02
Narrower exposure: 0.85–0.93×
Removing that step and offering fewer tools brought us under Hermes, but only modestly. Prompt tokens were still between about 223,000 and 247,000 against Hermes’s 266,000, because the model still received a tool catalogue on every call.
- 03
One tool per request: 0.32×
Requests are now routed by intent — verb, then job class, then a single matching tool — and the catalogue is not sent at all on routed turns. Prompt tokens fell to about 81,000. On a file-read task, the first call dropped from 2,743 to 1,308 tokens.
- 04
Later phases: 0.23–0.27×
Further routing and presentation work brought the best runs to 0.23×; the latest measured run is 0.25×.
The cost is in the prompt
In the Hermes baseline, 265,671 of 270,656 tokens — about 98% — were prompt tokens sent to the model; the model returned 4,985. For agent workloads like these, shortening answers barely matters. What matters is what the model is shown on every call: tool schemas, history and instructions. That is why the work above is context engineering, not prompt wording.
Fewer tools, tighter control
- Presentation lock. On a routed turn, a call to a tool that was not offered is rejected at dispatch. A model that remembers a tool name from earlier history cannot reach around the routing.
- Strict providers. Some providers return HTTP 400 when the history mentions a tool that is not in the current tool list — the cause of the V2j failure. A minimal stub schema for the requested name now goes into the next call, which removes the error at its root.
- Every call accounted for. Auxiliary and compression calls are counted in the total; none run unmeasured.
Verify it yourself
The evidence pack contains the raw per-run records for both sides, SHA-256 hashes and a script that recomputes every total from the raw data, with nothing hard-coded. For each of the 100 runs it checks that prompt plus returned tokens equal the recorded total; for Hermes, that the total matches Hermes’s own usage record and API call count; for LLM Studio, that it equals the sum of the per-call trace. The pack is available on request — ask for it.
Limits, stated openly
- Five small file and terminal tasks on one model. This is a large reduction on these workloads, not a claim about every workload or every model.
- The Hermes baseline was measured once and reused; its harness run included some uncommitted fixes.
- The ratio measures tokens, not money.
What this means for enterprise AI
- Measure tokens per request on your own tasks before choosing a framework or a model.
- Normalise the token definition before comparing vendors.
- Show the model only what the request needs; route first, then call.
- Use verifiable tasks — nonces, executed tools — so savings never come from skipped work.
See the LLM Studio case study and our enterprise AI integration work.
Have a project in mind?
Send a short description. If you need a mutual NDA first, we will send ours before you share details.