~/projects/crumbot

SOFTWARE2025

Crumbot

A work-in-progress local voice assistant for Windows that listens for a wakeword, transcribes speech, and uses a local LLM with tool calling to answer questions or control the computer.

Source
TIMELINE
Jun 2025 – Jul 2025
TEAM
Solo
CONTEXT
Personal project
Pipeline diagram: wakeword, transcription, a tool-calling agent with app, browser, display and PowerShell skills, then speech output.
3built-in skills

Overview

Crumbot is a composite voice assistant for Windows, with Android support planned. It is triggered by a custom wakeword, transcribes speech locally, and routes the resulting text through a local LLM that can call tools to fulfill commands or hold a short back-and-forth conversation.

Features

  • Custom-trained wakeword detection via openWakeWord (ONNX/TFLite models)
  • Local speech-to-text transcription with faster-whisper (large-v3-turbo)
  • Command fulfillment and dialogue through a local Ollama model (qwen3:1.7b)
  • Tool calling via a LangGraph ReAct agent
  • Text-to-speech responses via Kokoro
  • Built-in skills for launching apps, controlling the browser, and adjusting the display
  • PowerShell script execution for tasks like opening Discord or Minecraft
  • Configuration via a TOML file for wakeword model, STT model, LLM model, and TTS voice/speed

How it works

The pipeline starts with an onnx/tflite wakeword model listening continuously on the microphone. Once triggered, audio is captured and transcribed with faster-whisper. The transcript is passed to a LangGraph ReAct agent backed by a locally served Ollama model, which can invoke Python “skill” functions (apps, browser, display) or run PowerShell scripts to fulfill the request. The agent’s text response is then synthesized to speech with Kokoro and played back to the user.