What if you had a Jarvis-like AI assistant sitting in your Windows Terminal, ready to answer questions, process information, and help with tasks? No fancy GUI. No cloud dependencies. Just you, your terminal, and Claude thinking in real time.
I built exactly that. It's called Q—a push-to-talk voice assistant that combines OpenAI's Whisper, the Anthropic Claude API, edge-tts for speech synthesis, and a personal knowledge vault stored in Obsidian. Here's how it works and how you can build something similar.
What Q Does
Q is a conversational AI assistant running locally on a Windows laptop. Here's the workflow:
- Press a hotkey to activate push-to-talk mode
- Speak your question or request
- Whisper transcribes your audio to text locally (no cloud transcription)
- Claude processes your request, optionally reading from a personal knowledge vault
- The response is synthesized to speech using edge-tts
- Q speaks the answer back to you in a natural female voice
It feels like talking to an AI assistant. It is—just with zero latency, complete privacy, and direct access to your personal context and knowledge base.
The Architecture: Four Key Components
1. Whisper: Local Speech-to-Text
OpenAI's Whisper runs locally on your machine. It transcribes your voice without sending audio to the cloud.
Installation is straightforward:
pip install openai-whisperWhisper supports multiple languages, handles accents well, and runs fast on modern CPUs. For a laptop with decent specs, you're looking at ~2-5 seconds to transcribe a 30-second recording.
In Q, Whisper is invoked whenever you activate push-to-talk. The audio is temporarily saved, transcribed, and then the text is sent to Claude.
2. Claude API: The Brain
Claude Opus is the engine. Q sends your transcribed question plus context from your personal vault to Claude, which returns a thoughtful, grounded response.
A basic request looks like this:
const response = await fetch("https://api.anthropic.com/v1/messages", {
method: "POST",
headers: {
"x-api-key": process.env.ANTHROPIC_API_KEY,
"content-type": "application/json",
},
body: JSON.stringify({
model: "claude-opus-4-1",
max_tokens: 1024,
system: systemPrompt,
messages: [{ role: "user", content: transcribedText }],
}),
});The key is the system prompt. Q uses a custom prompt that establishes Claude's personality, context, and knowledge boundaries. It tells Claude about your projects, priorities, and the kind of answers you're looking for.
3. Edge-TTS: Voice Synthesis
Microsoft's edge-tts is lightweight and produces natural-sounding speech without needing external APIs or large model files.
pip install edge-ttsYou can choose from dozens of voices. Q uses the female voice en-US-AriaNeural—it sounds human, has good cadence, and handles technical content without stumbling.
The synthesis step is simple: take Claude's response text and convert it to an MP3 stream, then play it back.
4. Obsidian Vault: Knowledge Base
Q doesn't operate in a vacuum. It has access to your personal knowledge vault—an Obsidian vault stored on your laptop.
The vault structure might look like:
Vault/
├── CLAUDE.md (boot file with system context)
├── VAULT-INDEX.md (index of all notes)
├── Projects/
│ ├── Cogny.md
│ ├── Radio.md
│ ├── TryAnswr.md
│ └── Websites.md
└── Reference/
├── Technical.md
└── Business.mdWhen you ask Q a question, it can optionally read relevant vault files and incorporate that context into Claude's response. This turns Q into a personalized assistant that knows your projects, ongoing work, and key information.
A simple implementation reads the vault index and uses Claude to determine which files are relevant to the question, then pulls those into the conversation context.
The Flow: From Voice to Answer
Here's how Q actually works end-to-end:
Step 1: Activate Push-to-Talk
You configure a hotkey (or use Windows Voice Access with a custom wake phrase). Pressing it signals Q to start recording audio.
Step 2: Transcribe with Whisper
Your speech is captured as a WAV file and immediately fed to Whisper:
import whisper
model = whisper.load_model("base")
result = model.transcribe("audio.wav")
transcribed_text = result["text"]Whisper returns the transcribed text with high confidence. You see it printed in the terminal.
Step 3: Load Vault Context (Optional)
Q reads your VAULT-INDEX.md and asks Claude which files are relevant:
"Given this question: [user question], which of these vault
files would be most helpful? Return only the filenames."Claude responds with the relevant files, and Q loads their content into the system context.
Step 4: Send to Claude
Q constructs a message with:
- System prompt (your instructions to Claude)
- Vault context (relevant notes from your personal files)
- Your transcribed question
Claude processes it and returns a response, which Q displays in the terminal in real time.
Step 5: Synthesize and Speak
Q feeds Claude's response to edge-tts:
import edge_tts
async with edge_tts.Communicate(
text=response,
voice="en-US-AriaNeural",
rate="+0%"
) as communicate:
await communicate.save("response.mp3")The MP3 is played back immediately. You hear the answer.
Key Design Decisions
Local-First Architecture
Whisper and edge-tts run locally. Only your transcribed text is sent to the Claude API (and only if you choose to). This means:
- Privacy: Your voice is never uploaded to Whisper's cloud infrastructure.
- Latency: Local processing is fast.
- Reliability: Works offline (except for the Claude API call itself).
Push-to-Talk, Not Always Listening
Q doesn't run a wake-word detector or always-on microphone. You activate it explicitly. This prevents accidental recordings and keeps Q out of the way until you need it.
Personal Knowledge Base
By integrating with an Obsidian vault, Q becomes a personalized research assistant. It doesn't hallucinate about your projects—it reads actual notes you've written. This transforms it from a general chatbot into a domain-specific assistant tailored to your work.
Common Challenges and Solutions
Challenge: Vault Context Window Management
If you ask Q a question and it tries to load 10 vault files, you'll quickly exceed token limits. The solution: use Claude to intelligently select only the 2-3 most relevant files.
Challenge: Audio Quality
Background noise degrades transcription. A USB headset or desk microphone helps. Whisper is robust, but it performs better with clean input.
Challenge: Handling Tool Use in Vault Writes
There's a known issue in Q where Claude's tool_use requests cause a 400 error when trying to write back to the vault. The workaround: keep the vault read-only for now, or manually add findings to Obsidian after Q completes its response.
Use Cases
What would you use Q for?
- Quick research: Ask Q about a project, and it pulls relevant context from your vault and synthesizes an answer.
- Hands-free brainstorming: Talk through ideas while your hands are busy. Q listens, thinks, and responds.
- Personal documentation assistant: Ask Q to summarize your notes or explain a concept from your vault.
- Learning tool: Q can tutor you on subjects if you populate the vault with educational material.
- Task automation: Q could accept voice commands to trigger scripts or workflows (read-only for now, but possible future extension).
The Future: From Q to Production
Q is a proof-of-concept—and a powerful one. To take it further, you could:
- Add persistent memory: Store conversation history so Q remembers context across sessions.
- Enable vault writes: Resolve the tool_use issue so Q can update your vault with new findings.
- Expand to mobile: Run a lighter version on a phone or tablet.
- Build a team version: Multi-user vault for shared AI assistants across an organization.
- Integrate with external APIs: Q could check email, manage tasks, pull live data, etc.
The Bigger Picture: AI-Powered Workflow
Q is more than a parlor trick. It's a window into how AI can augment knowledge work.
Imagine every business having a similar assistant tailored to their domain, products, and processes. Sales teams could ask Q about customer history and competitor intel. Engineers could ask it architectural questions grounded in documentation. Support teams could ask it to synthesize customer feedback and suggest solutions.
That's the future of AI in business: not flashy chatbots, but deeply integrated assistants that know your context, protect your privacy, and answer intelligently.
Q is just the beginning.
Building Custom AI Assistants for Your Business?
Archway specializes in tailored AI implementations—from voice assistants to agency agents to AI-powered lead generation systems. Let's talk about what's possible for your team.
Schedule a Consultation