CHOOSEAgents & automation
Jev Performance Benchmarking
ickas@ickasdev
This project benchmarks the performance of Jev, an AI model that takes a state and typed questions to return probabilities over defined options, comparing its 'pure' and 'hybrid' configurations against heuristic and random strategies in a Battleship game context.
SignalThe model picks among those. 46.0 shots at ~54K tokens per game.
I benchmarked @typesafeai's Jev at Battleship. The most useful result is the one where it lost to 50 lines of code.
Five strategies on the same 60 seeded fleet layouts, 60 games per strategy. 6,000 model calls, $0.77 total.
Mean shots to sink the fleet, lower is better:
→ Jev hybrid 46.0
→ Density solver 48.3
→ Hunt/Target heuristic 51.9
→ Jev pure 85.5
→ Random 95.3
Both Jev rows are the same model, jev-1.13.0. The difference between them is what the code hands it.
Jev takes a state and typed questions (I used Choice) and returns probabilities over options you define.
Jev pure gets the raw board plus all ~90 untried cells as options. Code supplies only the rules. It scored 85.5 shots against 95.3 for random, p = 1.3e-10, so it does read the board. A ~50-line hunt/target heuristic beat it by 33.6 shots and won 59 of 60 boards. ~297K tokens per game.
TypeSafe's own docs explain why. Every option carries the same label, "untried cell", so the option list has no signal. Probabilities come back rounded to two decimals, so across ~90 options most of the distribution rounds to zero.
Jev hybrid: code ranks the top 16 cells by placement density and describes each one in plain words. The model picks among those. 46.0 shots at ~54K tokens per game. It beats hunt/target with p = 0.00015.
The density solver that builds that shortlist scores 48.3 on its own. Against it, 46.0 vs 48.3 is within noise: the confidence interval spans zero, p = 0.13, and the model won 31 of 60 boards. The honest claim is that Jev matches the code baseline.
If you're building with Jev: give it a short list of options, each one described in words. Keep the arithmetic in code. Put the code baseline on the same table as the model. From the writeup: "The configuration where the model looks good is the same configuration where the code did the hard part."
One methodology finding: the fleet layout family decides the ranking. Density beats hunt/target by 8.3 shots on uniform-random layouts and loses to it by 3.1 on adversarial ones, because density assumes a uniform placement prior. Benchmarking only on random layouts tests the baseline on the distribution it expects.