The Investigation
A voice-first, multi-agent mystery game — designed and built independently
The Insight
Every murder mystery game makes the same mistake: the characters already know who did it.
When you talk to a suspect in Clue, or click through dialogue in an adventure game, you're talking to a script. The character's responses were written before you arrived. They can't be surprised by what you know. They can't lie to you in ways that feel genuinely spontaneous. They can't adjust their story based on what the detective in the next room just told them. The mystery is unsolvable in any interesting sense — you're finding breadcrumbs the writer left, not actually conducting an investigation.
I wanted to build something different. What if each character in the game was a persistent AI agent with their own memory, their own secrets, and their own awareness of what the detective had — and hadn't — discovered? What if the mystery could be different every time, with a genuinely locked solution that no one reveals until the player earns it?
That's the invention: a multi-agent narrative architecture where each character holds their own truth, and truth only surfaces through real investigation.
The Invention
Architecture diagram
The Investigation is set in Ravenshollow Manor, a Victorian estate on a stormy evening. A priceless ruby has been stolen. A guard is unconscious. You arrive as the detective at 10:45 PM.
Three characters — Lady Beatrice, Guard Jenkins, Janitor Cole — each run as a separate persistent AI agent with their own memory blocks: their backstory, behavioral patterns, what they know about tonight, what they're hiding, and critically, a shared consistency state that tells them what the detective has already uncovered. A fourth agent, the Director, orchestrates everything — managing scene transitions, providing atmospheric narration, and maintaining the ground truth of the mystery.
Every game generates a new solution at the start: a different culprit, different motive, different method, different clue distribution. The solution is locked immediately and never changes. Characters aren't improvising — they're working from an agreed-upon set of facts that only the Director knows in full.
The player investigates by talking. Moving between rooms. Writing in a detective's notebook. The Director reads every notebook entry and silently assesses whether the player has verified something true — not to guide them, but to track whether the Accuse option should unlock. There are no collectible clues, no checklists, no UI confirmation when you find something. Discovery lives in the player's understanding, the way it does in real detective fiction.
The game is designed to be played by voice. The primary experience is an iOS app — voice input through speech-to-text, character responses spoken back through AI text-to-speech, each character with their own distinct voice. The interface has four states the player feels as much as sees: Muted, Listening, Thinking, Speaking.
The Build
Architecture overview:
Player voice input (iOS)
↓
Speech recognition (Apple Speech framework)
↓
Backend API (Next.js / Firebase)
↓
Director Agent (Letta) — orchestrates scene, manages consistency block
↓
Character Agents (Letta) — Lady Beatrice, Guard Jenkins, Janitor Cole
Each with: persona, behavioral patterns, memory of tonight,
investigation progress, shared consistency state (read-only),
overhear protocol, non-verbal reaction rules
↓
Text-to-speech synthesis (Google TTS / Gemini)
↓
Audio playback → player
The consistency block is the architectural heart of the system. It's a shared memory block that the Director writes to and all character agents read from. It holds the locked winning condition (culprit, motive, method), confirmed facts revealed to the player, each character's alibi status, physical evidence examined, and the Director's structured assessment of the player's notebook entries. Characters don't receive updates directly — they read the block. This makes the world feel like it has memory without requiring agent-to-agent coordination on every interaction.
The Director is the most complex agent. It operates under a strict 100-word limit per response — "film noir, not Victorian novel." It orchestrates scene transitions via a manage_scene_transition tool, signals which character the player needs via signal_character_needed, narrates non-verbal reactions when characters overhear conversations, and translates notebook entries into structured consistency block updates. It is, as the PRD describes it, "the world, not a participant."
The endgame is designed as a three-beat reveal sequence: the player's accusation, the Director's final scene-setting and handoff, and then the accused character breaking the fourth wall to speak directly to the player — not to the detective, but to the person who has been hunting them. Each confession sounds unmistakably like its character. Jenkins's relief still carries his nervous energy.
iOS app stack: Swift / SwiftUI, MVVM with manager-based architecture, AVFoundation for audio, Apple Speech framework for recognition, Firebase Auth + Firestore for sessions, SwiftData for local persistence, real-time polling architecture mirroring the web app. The input module was designed as a discrete component with four communicative states — the texture of waiting is part of the experience.
Web app stack: Next.js, React, TailwindCSS, LangGraph, Firebase, Letta SDK.
Current status: Core gameplay loop complete end-to-end — voice input, character conversations, scene navigation, detective's notebook, accusation system, three-beat endgame, session resume. iOS and web app share the same backend. App Store preparation in progress.
The characters
Behind everything we do is a team of people who truly care. We bring our full selves to the table—open-minded, collaborative, and ready to make something great.
The Validation
Every decision we make is shaped by a clear sense of purpose.
This project isn't backed by a press release. It's backed by the fact that it works — that you can speak to Lady Beatrice in an empty manor at midnight and she will answer you in her own voice, remember what you told her ten minutes ago, and lie to your face with enough internal consistency to make the lie matter.
The architecture solves a real problem that shipped game studios haven't solved at this level of accessibility. The voice-first design removes every layer of friction between the player and the story. The procedurally generated mystery means the second playthrough is genuinely different — not reskinned, not reordered.
The project also served as a direct proof-of-concept for agentic architecture patterns I'd been developing professionally — multi-agent orchestration, shared memory state, human-in-the-loop design (the notebook), and voice-first interaction design. It demonstrates those patterns in a context that anyone can experience, not just evaluate on a slide.
The Honesty
Every decision we make is shaped by a clear sense of purpose.
Multi-agent systems at this complexity level have real latency. Each character response involves multiple round-trips — player input, speech recognition, backend routing, Letta agent processing, TTS synthesis, audio delivery. Getting that chain under three seconds consistently required architectural decisions that weren't always obvious: polling design, streaming, consistency block structure to reduce redundant context loading.
The Director's 100-word limit was a hard constraint I imposed after watching early versions become Victorian novelists. Brevity creates tension. Tension creates atmosphere. That lesson shaped how I think about system prompt design more broadly.
The notebook-to-consistency-block translation is the most fragile piece. Natural language is messy. Players write in ways that are clear to them but ambiguous to a structured assessment system. The gap between "player understands something true" and "system correctly detects that understanding" is a design problem that isn't fully solved — it's the most interesting unsolved problem in the project.
What this project proves, to me: When I'm not inventing AI experiences for companies, I invent them for myself. The Investigation exists because I couldn't stop thinking about it until I built it.