Benchmarks · agency-ops
The agency-ops 35B punches at the frontier.
A niche-tuned 35B beats the base model and a 744B general model on agency tasks — and on Apple Silicon it ties the frontier. Here’s exactly how it was measured — and why the result holds up.
Other niches publish held-out validation loss; they have not been run through this frontier head-to-head yet.
Head-to-head quality
50 held-out items · proven 0% n-gram overlap with training data. blind panel of 2 independent judges (Gemini 3.5 Flash + DeepSeek v3.1), absolute 1–10.
Bars are the mean blind-panel score (0–10 scale). Our agency-ops 35B lands within a tenth of GPT-5.5 and Fable 5, and more than two points above GLM-5.2 — a general model 20× its size.
What you get, by how you run it
Same weights, different serving engines. We publish the number for each path — including the ones below our best — so what you can reproduce matches what we claim.
| Run it via | Hardware | Quality |
|---|---|---|
| MLX 8-bit · agency-ops | Apple Silicon · M-series, 24GB+ unified | 9.05 · ties frontier |
| Q8_0 GGUF / hosted API · agency-ops | NVIDIA GPU, CPU, Mac, or our API | 8.35 |
| Q4_K_M GGUF · agency-ops | NVIDIA GPU, CPU, or Mac | 7.97 · earlier 20-item run |
| Q4_K_M GGUF / hosted API · other 9 niches | NVIDIA GPU, CPU, Mac, or our API | No catalog-wide score · not yet measured |
9.05 is agency-ops via MLX 8-bit; 8.35 is agency-ops via Q8_0 GGUF/hosted API. Agency-ops measured 7.97 as Q4_K_M on an earlier 20-item run; the nine Q4_K_M catalog niches have no catalog-wide blind score yet.
How we measured it
The whole point is a result that survives scrutiny. Four rules make it honest:
Held-out test set
50 questions the model never saw in training — proven 0% n-gram overlap with the train and validation splits. We don't teach to the test.
Blind, independent panel
Every answer is graded 1–10 by two judges from different labs (Gemini 3.5 Flash + DeepSeek v3.1). Neither is a contestant, and the judge never sees which model wrote the answer.
Same prompt for everyone
Identical system prompt, temperature, and token budget for every model — ours and the frontier APIs alike. No home-field advantage.
Reproducible
The harness, the golden set, and the judge prompts all live in the repo. Run the eval yourself and you get the same board.
Why agency-ops beats a 744B
A general model spreads its capacity across everything — code, poetry, trivia, translation. A niche model spends all of it on one trade. On that trade, focus wins: agency-ops beats GLM-5.2 (744B) by more than two points and ties the frontier, while staying small enough to run on a laptop and cheap enough to serve at a fraction of the cost. That’s the measured agency-ops result; the other niches are not represented in this head-to-head.
As of July 2026. Benchmarks are re-run as models are retrained; numbers update with each release.