LLM Reply Performance Improvement
SMTPfast@fastsmtp
Replaced a slow LLM with Jev, reducing median reply time from 7.3s to 605ms and eliminating unreadable prose errors.
Our admin review account button awaited an LLM: 7.3s median, 12.7s worst 🦥
About 1 in 16 replies was prose our parser couldn't read so you waited 7s for an error
Swapped in @typesafeai's Jev which returns typed answers: 605ms median, 38ms spread, invalid output not possible ⚡️ x.com/fastsmtp/status/2101263368535372030/