Automated Codebase Conformance Checking

Sep 20

This project uses Jev to automatically check code changes against predefined implementation plans and Architecture Decision Records (ADRs), identifying discrepancies and ensuring adherence to specifications.

SignalEach rule declares which files it needs, code filters first.

Everyone on X is fafo'ing with @typesafeai's Jev right now, and if you just vibe with it you'll get nothing useful. Here's why. Jev doesn't reason. It judges data you hand it against options you define. It's only as good as your spec — and most repos don't have one. Ours does. Our product factory is strict: implementation plans and ADRs are crafted deliberately and fully human-reviewed before anything gets built. That turns them into machine-checkable contracts. So we pointed Jev at them. What we got today: → Scored 686 PR review threads across 192 merged PRs for $0.04. Found 93 review findings that were never answered before merge — 49 of them serious. Thread and unanswered counts verified exactly against the GitHub API (686/686, 93/93), so the numbers aren't the model's opinion. → Checked all 210 pairs of our 21 ADRs for contradictions. 14 seconds, $0.019, zero real conflicts. Cheap enough to run on every PR that touches a decision doc. → Validated a 5,184-line PR against our ADRs without ever sending it the whole diff. Each rule declares which files it needs, code filters first. Diff size stops mattering. → Per-PR plan + ADR conformance: $0.00016. Whole day of experiments: under $0.25. The real unlock wasn't prompting. We had an agent read the full plan unit + the PR and write the questions itself — and decide which checks were just git diff and which actually needed judgement. My hand-written generic question scored 0.47. The agent-derived ones: 0.90–0.99, and 0.04–0.13 on a control PR they should fail. Every "all clean" result got re-run with deliberately injected violations to prove the check wasn't just dead. Controls fired 16/16 and 4/4. Why this matters to us: it automates the boring half of an architect's review — re-reading the plan and the ADRs to check a diff actually honours them. That's the part that gets skipped at 6pm on a Friday. And at these prices you stop sampling and check everything. Caveats, loudly: this is day one. Small n on some checks. Controls ≠ a real eval suite. Plenty failed too — you can't judge behaviour from a diff, and every composite score came back as mush. Lots more evals before this gates anything. But super interesting to play around with. ooh btw product factory sounds better then software factory imo)

Original post