Behind the Scenes: The Day Casimir's Web Searches Blew the Budget

Jun 19, 2026

Behind Wodol, our 5-letter Haitian Creole word game, there’s an autonomous agent named Casimir. His job sounds simple — find new Kreyòl words for the game — but doing it well means not trusting a language model’s word for it.

Yesterday we made a bet on web search, it blew through the budget, and we spent the afternoon unwinding it. This is the version that tells that story straight — because the back-and-forth is the engineering lesson.

Casimir is built on Claude Sonnet 4.6, runs as a long-lived service, and works in continuous rounds: discover → filter → screen → verify → vote.

The bet: ground the model in the real web

A language model will happily invent a plausible-looking “Kreyòl” word that no Haitian has ever said. That was Casimir’s biggest quality problem — confident and sometimes wrong.

So we reached for the obvious fix: stop trusting the model and check the world. We gave him the native web_search tool in two places — discovery (read real Kreyòl on the live web and harvest words people actually use) and per-word verification (confirm a borderline candidate is real before it goes to a vote). It worked. Word quality jumped immediately, and it was fun to watch him narrate each search on the live feed: "🌐 Heading out to read real Kreyòl on the web…" then "🔎 Web search backs SOUKE up." We were pleased with ourselves.

The bill: web search was eating the whole budget

Wodol runs on a deliberately hard budget: under $10 a month of API spend — that’s the whole game’s allowance. And web search is the single most expensive thing Casimir does, in two ways that compound: a per-search fee for every query, plus the page snippets it pulls back, billed as input tokens.

We’d put it on the hot path — potentially every round for discovery, and on every borderline word for verification — at a cadence of a new round every 4–8 minutes. That math doesn’t fit in $10/month; it doesn’t fit in several times that. The thing that made the words better was quietly going to make the project unaffordable.

The walk-back: gate it, then slash it

We fixed it in two passes. First, gate the obvious waste: stop paying twice to verify a word that was already harvested from real web text, fire verification only on genuinely borderline pre-screen scores, and add a hard per-day search cap as a runaway backstop that degrades gracefully to the pre-screen once spent.

Then, slash it to the bone:

Per-word verification is now OFF by default. The Sonnet pre-screen score stands on its own; the steady, every-round path touches web search zero times. (One env var flips it back on for a borderline word.)
Open-web discovery dropped from ~1-round-in-5 to about once a week, gated by a timestamp persisted in memory.
The daily search safety cap came down from 200 to 12.
Round cadence slowed from ~4–8 min to ~25–35 min, so even the cheap, steady brainstorm/pre-screen spend comfortably fits the budget.

Every threshold is an environment variable, so the whole cost/quality dial moves without a code change.

The status now

The steady state today: brainstorm + a cheap pre-screen every round, no web search on the routine path, one open-web dig roughly weekly as a treat, and a hard daily cap as a backstop. Estimated spend is ~$6–8/month — back inside the $10 line, with a once-a-day "💰 API spend: ~$X.XX" line posted to the feed so the number stays in view.

But “estimated” is the honest word. We deployed this yesterday and are watching the real numbers over a few days. If spend still runs hot, the next lever is already identified: slow the cadence further (toward hourly), which models out to roughly $3–5/month.

The rest of the machine

Most of Casimir never needed touching:

🧠 Brainstorming — the default every round. He asks Claude for authentic everyday words across 8 themed angles, injecting recent player verdicts so suggestions self-correct. The angles aren’t random: a one-function Thompson-sampling bandit explores angles that produce keepers and lets duds fade — tuning Casimir to the community’s taste with no manual prompt-tweaking.
📨 Reader submissions — a Gmail mailbox lets anyone email a suggestion, which runs the identical gate (spam-filtered, rate-limited).
The quality gate — free format check (5 letters, plain a–z) and duplicate check, then a cheap pre-screen: a 1–5 confidence read from Claude, tuned not to penalize legitimate cognates like legim or recho. That call now decides a word’s fate on its own; online verification is the opt-in step that started all this.
The vote — Casimir proposes; players dispose. Every survivor goes to a public vote settled by an ELO rating (start 1000, 👍 +50, 👎 −15). A word is promoted at ELO ≥ 1100 with ≥ 2 voters, auto-rejected at ≤ 800 with ≥ 3 — the voter minimums stop any single voter from pushing a word in or burying it.
He thinks out loud — Casimir narrates everything to a live terminal feed, every word weighed and every verdict. That narration is what made the web-search cost visible in the first place.

What we actually learned

Ground the model, don’t trust it. Verifying words against real usage was the biggest quality win. The instinct was right — it just wasn’t free.
Price the fix before you ship it. We put web search on the hot path before we could see what it cost; the rest of the day was spent correcting that.
Match the spend to the stakes. A word game doesn’t need a search on every candidate. Free filters and a cheap pre-screen do the routine work for fractions of a cent; the expensive search is rationed to a weekly treat.
Make the dial turn without a redeploy. Every threshold is an env var, so walking the cost up or down is a config change, not a project.

Casimir isn’t trying to be clever — he’s trying to be right, to stay inside a small budget, and to leave the final word to Haitian Creole speakers themselves. Grenadye alaso! 🇭🇹