Jev vs. the Frontier
RESEARCH NOTE · SEPTEMBER 2026
A small model matched a frontier model on automated decisions, at one-fiftieth of the cost.
HOW IT WORKS
A fair fight: same task, same rules, locked in advance
- IN / OUT
- CODE
- AI MODEL
- CHECKPOINT
- IN Banking77 customer messages, 77 categories
- SAME MESSAGES, TWO MODELSAI MODEL Jev small, specializedAI MODEL Claude Sonnet 5 frontier
- CHECKPOINT Confident? pre-registered threshold
Not confident: routed to a person
- CODE Score against a locked test set
- OUT 91.4% vs 91.0% $0.10 vs $5.25 per 1,000
Rules written before the run. Whole experiment: $16. Simplified. The full story is on the project site.
The question
Can a small, specialized model do everyday business sorting work as well as a frontier model? If it can, a lot of companies are overpaying for AI.
The test
I set up a realistic triage task on Banking77, a public set of customer banking messages sorted into 77 categories. The rule was the one a real operation would use: answers the model is confident about are accepted automatically, and everything else goes to a person.
To keep the result honest, the rules were written down before the test ran, and the test set was locked so nothing could be tuned against it. The whole experiment cost $16.
The result
On the decisions it made automatically, Jev was right 91.4% of the time. Claude Sonnet 5 was right 91.0% of the time.
Jev did it for $0.10 per thousand messages. Sonnet cost $5.25. Same accuracy, one-fiftieth of the price.
Why it matters
Most AI cost is decided by habit: send everything to the biggest model. Measured carefully, many tasks don’t need it. That’s the routing problem at the heart of the AI Operations Management textbook, shown with real numbers.
What’s next
Phase 2 runs the same test on real emails.