Skip to content
quineFXResumeContact
Prev02 / 03Next

Marchivum: Carlo Schmid

A historical figure visitors can talk to, built for Marchivum in Mannheim

Client
Marchivum, Mannheim
Year
2025
Role
Speech assistant pipeline and MetaHuman
Tools
Whisper, Hybrid retrieval (FAISS + BM25), Mistral, XTTS v2, Audio2Face, Unreal Engine + MetaHuman

01 Context

A speech assistant for Marchivum, the city archive in Mannheim. Visitors ask Carlo Schmid a question out loud. He answers in his own cloned voice, from the archive's documents, with a face animated by the audio. Every model runs locally, on one machine, so nothing from the archive leaves the building.

02 How it works

Seven steps, from the visitor's question to Carlo's face on screen. Whisper turns speech into text; retrieval finds the right archive documents; Mistral answers in character; XTTS v2 speaks it in his cloned voice, a sentence at a time; Audio2Face moves his MetaHuman face in Unreal Engine 5.

A historical figure you can talk to

~1.2 s
average response
256 / 64
token chunks / overlap
0
cloud calls
  1. 01Visitor speaksmicrophone
  2. 02Whisperspeech to text
  3. 03Hybrid retrievalFAISS 70% + BM25 30% over OCR'd archive documents
  4. 04Mistrallocal LLM, in character
  5. 05XTTS v2cloned voice, streamed sentence by sentence
  6. 06Audio2Facelip-sync and expression from the audio
  7. 07MetaHumanrendered live in Unreal Engine 5

03 How it came together

The first working version, in February 2025, had the whole loop: Whisper, retrieval over the archive, XTTS v2 and Audio2Face. By September 2025 it had a walkthrough and promo films.

Next in Client work · 03 / 03Making Freedom Fly