Testing 25 AI models for Spomen’s family interviews

I’m building Spomen, a voice-first app for recording family stories. You talk, an AI listens, works out what is missing from the memory and asks one useful follow-up: “What year was that?” or “Who else was there?” It then turns the conversation into a clean chapter.

The part doing the listening is the analyzer. After every answer, it checks how complete the story is, extracts the people, places and dates, and writes the next question as strict JSON. It runs on every turn while a 78-year-old waits for a reply. The model therefore has to do three things well:

• Be fast. You feel the delay during the conversation.
• Work across languages. My dad, our first real user, tells his stories in Bulgarian.
• Handle messy speech. Real conversations include corrections such as “no, wait, it was sixty-seven, not sixty-five.”

Finding the right model took a few days and about 25 models across five providers.

A benchmark, not a vibe check

A clean sentence such as “I was born in 1952 in Vienna” does not reveal much. Every model can handle it. I wrote deliberately messy transcripts in seven languages instead. Each one included:

• a spoken correction: “nineteen sixty-five, no wait, sixty-seven”
• one person referred to by two names: “my brother Tomas, uh, Tomash”
• uncertain ages, unfinished sentences and trailing “ums”

A model passed only if it kept the corrected value, merged the name and nickname into one person, and answered in the storyteller’s language. Bulgarian was a hard requirement. If it could not follow my dad’s story, it could not ship.

The test harness sent the same prompt to every model and recorded latency, token cost and raw JSON. This made it possible to compare the actual answers instead of relying on a public leaderboard.

Speed by language

This table decided the first round. It shows median latency in milliseconds across seven languages:

Model	EN	BG	DE	ES	FR	RU	ZH
Groq Llama 3.3 70B	565	645	559	459	552	638	545
Gemini 2.5 Flash	1898	1527	1329	1455	1497	1434	1384
gpt-4.1-nano	1311	1319	1312	1101	970	1192	963
gpt-4o-mini	1866	2033	2019	1816	1849	1831	2415
Claude Haiku 4.5	3113	3688	3333	2568	3015	2957	2705

Groq was in a different class. It was more than twice as fast as the next option and around five times faster than Claude, without slowing down in a particular language. When the call runs after every answer, the difference between 570ms and 1.5 seconds is the difference between “the app is listening” and “the app is buffering.”

Gemini 3.1 Flash-Lite is not in the table because it varied from 1.2 to 7 seconds across runs. That was too unpredictable for a live conversation.

Price

All of the models were cheap. Compared with speech-to-text and the final story edit, the analyzer cost very little:

Model	Cost / call	Cost / 1,000 calls	Avg latency
gpt-5-nano	$0.00013	$0.13	~2,100ms
gpt-4.1-nano	$0.00014	$0.14	~1,170ms
Gemini 3.1 Flash-Lite	$0.00017	$0.17	spiky
gpt-4o-mini	$0.00021	$0.21	~1,980ms
Groq Llama 3.3 70B	$0.00060	$0.60	~570ms
Gemini 2.5 Flash	$0.00078	$0.78	~1,500ms
Claude Haiku 4.5	$0.00340	$3.40	~3,050ms

The cheapest option was about 25 times cheaper than the most expensive, but the difference across a whole recording was still only a fraction of a cent. Price did not decide the result.

Quality by language

The marks below show how each model handled the messy transcripts. ✅ means it removed the mistake, merged the names and stayed in the right language. 🟡 means correct but rough. ⚠️ means it kept the wrong value or split one person into two.

Model	EN	BG	DE	ES	FR	RU	ZH
Groq Llama 3.3 70B †	⚠️	✅	⚠️	⚠️	⚠️	⚠️	⚠️
Gemini 2.5 Flash	🟡	✅	✅	✅	✅	✅	✅
gpt-4.1-nano	⚠️	⚠️	✅	⚠️	✅	✅	✅
gpt-4o-mini	⚠️	✅	🟡	⚠️	⚠️	✅	⚠️
Claude Haiku 4.5	✅	✅	✅	✅	✅	✅	✅

† Groq’s row uses the basic test prompt without the later deduplication instruction. With the production prompt, it handled the corrections cleanly.

Claude Haiku was the most reliable extractor in every language. The cheap, fast models were less consistent: some kept the wrong year in three of the seven languages. For this job, a cheap model that loses family details is not useful.

Four things the benchmark taught me

1. Reasoning can be a problem on a fast, structured task. Our first analyzer used gpt-5-nano. It took around 16 seconds per turn because it spent thousands of thinking tokens on what was mostly classification. gemini-3.5-flash had a similar problem: it took five to eight seconds and sometimes returned broken JSON until I set reasoning_effort: 'none'. It then fell to about 1.1 seconds and produced clean output. Before rejecting a reasoning-capable model for a simple task, turn the extra reasoning off.

There was one more trap. gpt-5.1 did not accept the minimal setting used by the other models. It expected none, so every call failed with a 400 error until I found the mismatch.

2. Clean benchmarks hide the failures that matter. Llama 4 Scout looked like the winner on an earlier, tidier test. It was fast, cheap and merged names correctly. On the messy transcripts—with several corrections and two different people called George—it split “Harold/Harry” into two people and kept both the wrong and corrected values. It scored two out of five. The clean test had hidden the exact problem real speech would expose.

3. The benchmark found a bug in my production prompt. Given “we lived in Pernik, no, in Kremikovtsi,” the shipped analyzer saved both cities. The prompt already said “never the mistaken one,” but the models followed that rule for a single name or year and missed it with places and stacked corrections. I rewrote the instruction to mention places, streets and counts, then added a final check. The problem disappeared.

The lesson was useful beyond model selection: a difficult benchmark can test the product’s instructions too.

4. Put limits on agents. I ran many of these tests through coding agents. One agent received “if it errors, retry and debug” with no time limit. A JSON failure sent it into a 35-minute loop of probe scripts before I stopped it. Every run now has a hard timeout and every call can be aborted. An agent needs a time budget.

What I shipped in May

I chose Groq Llama 3.3 70B for production. It was much faster than the other models, its quality gaps could be fixed with a better prompt, and it used the same provider as speech-to-text. Claude Haiku 4.5 was the fallback when extraction quality mattered more than speed. Gemini 2.5 Flash was the balanced option if I changed providers.

The test harness was just as useful as the winner. It listed the current models from each provider, recorded speed and cost, and scored every language on deliberately messy input. It had already found one production bug and stopped one bad model switch. The next comparison could run with one command instead of taking another three days.

If an LLM sits on a live path, build the difficult benchmark first. Use messy input, disable unnecessary reasoning and measure the wait where the user feels it.

July update: production changed the question

The May benchmark used short, messy fixtures. That was useful for multilingual extraction, but it did not reproduce the main production problem: after every answer, the analyzer received the whole interview again.

A 30-day snapshot from storykept_ai_calls showed what people were waiting for:

Production analyzer metric	Before the redesign
Calls in the snapshot	113
Median latency	4.0s
p95 latency	10.3s
Slowest call	26.7s
Average input	5,185 tokens
Average output	172 tokens
Total model cost	$0.10

Cost was still unimportant. Waiting was not. A four-second median was manageable, but a ten-second p95 broke the rhythm of an interview.

The second benchmark therefore used the exact production prompt and graded the complete job: question quality, story-type classification, entity retention, median latency, worst-case latency and cost. An Opus judge scored the follow-up questions without knowing which model wrote them.

The final run used four passes across eight fixtures and all nine story types. Claude Haiku 4.5 won both main measures: a 2,201ms median and 4.14/5 question quality. It kept every required person, place and date in all eight fixtures, classified seven of eight correctly, and kept its slowest result to 7.3 seconds. Another model had a tempting median but reached 16 seconds in the worst case. In a conversation, the longest waits matter almost as much as the typical one.

This does not contradict the May result. The prompts, fixtures, judging and production constraints were different. The first benchmark found the fastest acceptable extractor. The second found the better interviewer for the code people were actually using.

The model switch was only half the fix

Changing providers did not solve the larger problem of resending a growing interview. We added a small saved analyzer state containing:

• a faithful summary of the story so far
• six completeness grades: setting, people, setup, action, outcome and significance
• the turn count
• the people, places, dates and themes already mentioned

After eight turns, the next call receives this state and the last three exchanges instead of the complete transcript. The full list of people and places, previous questions, storyteller coverage and family roster still travels separately, so the model does not forget an early name when the original line leaves the window.

We compared the windowed and full-history versions on English and Bulgarian interviews of 16 to 26 turns. They produced the same six completeness scores, the same completion decisions, identical next questions and full entity retention. A blind quality comparison ended in a tie. Token use fell by around 10% at those lengths, with larger savings as the story grew. Only then did we enable the shorter context by default, with a kill switch still available.

The static instructions are also prompt-cached. The live chain became:

Claude Haiku 4.5
        ↓ provider failure
OpenAI gpt-4o-mini

Groq left the production chain in July because of the new provider decision, not because the May timings were wrong. The fallback also crosses providers so an Anthropic outage does not stop the interview.

The lesson after two benchmarks

An LLM benchmark is not a permanent ranking. It is a test of what the product needs at that moment.

In May, we cared about multilingual correction handling and a sub-second response. By July, real sessions showed that question quality, classification, retained context and worst-case latency mattered too. The benchmark had to change with the product.

The useful conclusion is not “Haiku is the best model.” It is “Haiku is the best fit for this production contract, and the contract is measured well enough that I can change my mind again.”

---

Spomen records family stories by voice. It is private by default, built to outlast its users and tested in Bulgarian because that is the language my dad tells his stories in. spomen.app.

---

AI helped me write this post as an experiment—and because a busy family man has more ideas than uninterrupted writing time. The experience, opinions and final editorial decisions are mine.