Case study · Research & development

LLM Studio

A local-first, multi-agent AI workbench for macOS. It is where we develop and measure the techniques we bring to enterprise AI work: provider routing, retrieval, agent control and token economy.

At a glance

Status: Research & development, used daily in-house
Platform: macOS desktop application with a local API core
Started: July 2026
Stack: Python, FastAPI, SQLite, React, TypeScript, Tauri (Rust), Swift

Real screens

LLM Studio — Chat — Chat: model selection, deep thinking, tools (terminal, image, deep research, web, files) and the local switch.

LLM Studio — Crew — Crew: a multi-agent round table with automatic, contextual, balanced and fixed-order settings.

LLM Studio — Manager — Manager: goal-driven runs with a workspace, separate manager and code-agent models and an explicit permission level.

LLM Studio — Fusion — Fusion: two models assess a topic independently, read each other and converge; the round count is a cap, not a target.

LLM Studio — Model Hub — Model Hub: local, CLI, OAuth and API providers in one place, with health status (account name masked).

LLM Studio — Knowledge stacks — Knowledge stacks: files, folders and Obsidian vaults compiled into searchable retrieval workspaces.

Why it exists

Many agent frameworks let auxiliary model calls run unchecked and hand the model the whole tool catalogue on every request. We wanted to measure where tokens actually go, and build a system in which every model call is accounted for and every tool choice is deliberate.

Architecture

Layers

Desktop shellTauri window, React interface
Local APIFastAPI with streaming responses
CorePersonas, crews, retrieval, policies
ProvidersLocal and cloud models behind one interface

The core owns all logic and runs without the interface: the same functions are available from the command line and as tools over the Model Context Protocol (MCP).

Capabilities

Persona runtime with deterministic system-prompt compilation and authority policies.
Crew engine: blind sequential and round-table multi-agent modes, with an evaluator and a quality threshold.
Hybrid retrieval: BM25 and vector search fused with reciprocal rank fusion, cited context, named workspaces.
Multi-provider layer with health checks, timeouts and per-call token accounting.
MCP registry with policy, resilience and tool-retrieval layers.
Approval and change-safety gates before any action with side effects.
A native macOS accessibility agent written in Swift, with an audit log.

Measured result

tokens of Hermes Agent on the same 25 tasks: 0.25×
tasks succeeded in the latest run: 25/25
fewer tokens across three runs: 68–77%

Same model on both sides (deepseek-v4-flash), five agentic tasks × five repeats, July 2026. Scope is limited to these workloads; the ratio measures tokens, not dollar cost.

The main lever: each request is routed by intent to a single matching tool, so the model sees one tool schema instead of the full catalogue. On a file-read task, the first call dropped from 2,743 to 1,308 tokens.

Average total tokens per task

Hermes Agent10,826

LLM Studio2,728 0.25×

By task (average tokens per run)

Read a fileHermes Agent9,634LLM Studio2,1540.22×
Search filesHermes Agent9,663LLM Studio2,5930.27×
Edit a fileHermes Agent15,086LLM Studio3,5330.23×
Multi-step taskHermes Agent10,212LLM Studio2,8600.28×
Count lines (terminal)Hermes Agent9,536LLM Studio2,4990.26×

Total tokens = prompt tokens sent (cache included) + tokens returned (reasoning included). Same model on both sides, 5 tasks × 5 runs, 25/25 succeeded. Results vary by system.

Scale and quality

commits since July 2026: 1,240+
automated tests in 818 test files (Python and TypeScript): 12,000+
lines of Python, plus TypeScript, Rust and Swift: ~450k

Measured from the repository: commits and tests on 1 November 2026, lines on 31 October 2026; the Python line count includes tests.

The benchmark evidence pack — raw per-run data, SHA-256 hashes and a script that recomputes both sides — is available on request.

Have a project in mind?

Send a short description. If you need a mutual NDA first, we will send ours before you share details.

Start a project projects@zerentechnologies.com or message us on WhatsApp