Vectorless RAG with Jev

Sep 19

This project introduces a novel approach to Retrieval Augmented Generation (RAG) that replaces traditional embedding methods with a structured, table-of-contents-based navigation system, significantly improving speed and reducing costs by offloading non-generative tasks to Jev.

In March this year I started working on an idea I believe changes how we do RAG. Vectorless RAG. The idea has been around for a while; I designed my own process, which borrows a lot from the llms.txt convention: a document becomes a table of contents with real page ranges, and retrieval navigates it the way a reader would, instead of embedding chunks and hoping. I think vectorless is the best way to do this. The catch was always latency. It depends entirely on the model, and on the language you build the pipeline in. I built mine in Go for concurrent document processing, and the language was never the problem. The model round-trips were. I tried to cut them for months and couldn't. Two days ago @typesafeai released Jev. I integrated Jev into Vectorless and rebuilt the pipeline around one rule: a generative model only where text must actually be written. Everything else, every "is this a contents page", "does this section begin here", "is the answer on this page", goes to Jev as a batch of yes/no questions, hundreds per request. Jev was the magic wand I had been waiting for. Here is what happened, benchmarked on FinanceBench 10-K filings, 68 to 549 pages each. The PepsiCo filing is 549 pages. Vectorless with Jev now builds its full table of contents, every section placed on its page, in 8.1 seconds, for $0.0006. The same stage on the chat model took between 100 seconds and 14 minutes per filing, and nine of the 21 filings didn't finish inside a 5-minute limit at all. How each number came to be: How each benchmark was run 𝗧𝗮𝗯𝗹𝗲 𝗼𝗳 𝗰𝗼𝗻𝘁𝗲𝗻𝘁𝘀 • 21 FinanceBench 10-K filings, 68 to 549 pages. • The chat client was swapped for one that fails on any call. Zero generative calls is proof, not a claim. • Checked against FinanceBench's gold evidence pages: 47 of 47 fall inside a section of the tree. • The section holding a gold page is 36 pages at the median. The old path gave 183, because it lost every section's page on long filings. • Trees compared title by title with the chat model's trees: 0.975 recall, 0.966 precision. 𝗦𝗽𝗲𝗲𝗱 • Whole table-of-contents stage: 9.5 seconds per filing on average, 7 to 24 seconds, $0.0003 to $0.0015. • Same stage on the chat model: 103 to 840 seconds. 9 of 21 filings never finished inside 5 minutes. • With an adaptive limiter: 21 filings in 122 seconds wall instead of 199, zero rate limits. 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 • 40 FinanceBench questions, each with a gold evidence page. • Jev ranks sections, then page heads, then full pages. Four requests, $0.003, 9 to 20 seconds per question. • Right section chosen: 37 of 40. Every gold page in the evidence: 34 of 40. • The chat model is asked once, at the end, over the pages Jev chose. It never navigates. I have not yet benchmarked this against a chunk-and-embed pipeline on the same questions. I will, and I'll post it either way. Engine: github.com/hallelx2/vectorless-engine Every benchmark, with the numbers I'm less proud of: github.com/hallelx2/vectorless-engine/tree Website: vectorless.store Leave a star as you check this out.

Original post Code