#evals

2 posts

25 Aug 2026 · AI-assisted dev · 13 min

Regex can't count pages

My ebook pipeline's top-ranked pick for Atomic Habits was a 115-page abridged edition with perfect metadata. So I put an LLM judge in the pipeline, benchmarked eleven models on ten fixed candidates with five runs each, and watched the only real data loss of the week come from deterministic code instead.

04 Jul 2026 · AI-assisted dev · 11 min

A regex skipped a fifth of my films for being foreign. A $0 model read every one.

filmoteka's parser silently dropped 22% of real releases, every one multilingual. I benchmarked four GLM tiers against it on 115 real titles, five runs each. Every tier recovered all of them; only the paid ones did it without ever grabbing the wrong film.