CHOOSEDevtools & code
Jev Model Code Accuracy Test
Aly@aliyilmazco
This project tested the Jev model's ability to accurately identify specific lines of code that answer questions, comparing its performance and cost against other models like Claude Opus and Haiku, as well as the grep tool.
SignalIt doesn't write text: it scores every line and picks the most likely one.
I tested @typesafeai's new Jev model against Claude on real code.
150 questions · 21 files · Go, Python, TypeScript
Right answers out of 150:
• Jev: 146
• Opus 5: 148
• Haiku 4.5: 125
• grep: 16
On the 108 questions that don't use any words from the code:
• Jev: 106
• Opus 5: 106
• Haiku 4.5: 89
• grep: 0
Cost per 1,000 questions:
• Jev: $0.46
• Haiku 4.5: $4.76
• Opus 5: $30.21
On the hard questions, Jev ties Opus 5 at 1/65 of the cost. It doesn't write text: it scores every line and picks the most likely one. That makes it an interesting fit for the "find the right code" step in coding agents.
—
How I tested it
• The task: given one file and a question, find the exact line that answers it.
• Every model got the same file, with numbered lines.
• One try per question. Only the first guess counts.
• 21 questions are traps: the answer isn't in the file, but something similar is. The right reply is "not in file". Opus 21/21, Jev 19/21.
• grep is the free baseline. It matches the question's words to each line's words.
• Cost = tokens reported by the API × list price.
• Latency is left out. Jev is in early access and ranged from 0.3s to 12s.
• Questions were drafted with Claude sub-agents and checked against the code before any model ran.
Scoring is strict: pointing at the line just above the answer counts as wrong. With a ±2-line tolerance, on the 129 questions that have an answer: Opus 129, Jev 128, Haiku 122. So the real gap is cost, not accuracy.
Next: letting Jev find the right file across a whole repo, then measuring cost per finished task inside an agent.