Developer docs / AI Fabric tutorials

Full-duplex voice chat with Moshi

  • GPU
  • Web endpoint
  • Image DSL

Take it for a spin

The whole program in one file, ready to run.

moshi_voice_chat.py
import bladekit

app = bladekit.app("moshi-voice-chat")

# Moshi (kyutai) is a full-duplex speech-to-speech model: it listens and speaks
# at the same time over a single 24 kHz audio stream, so there is no separate
# STT -> LLM -> TTS pipeline and no turn-taking latency. The `moshi` package
# ships a ready-made server that serves a browser UI at "/" and a websocket at
# "/api/chat" carrying Opus-framed audio in both directions.
#
# The bf16 PyTorch weights (kyutai/moshiko-pytorch-bf16) are a ~7B temporal
# transformer plus the Mimi neural audio codec, about 15 GB of VRAM, which
# fits an 18 GB RTX-A4500 with headroom for the streaming KV cache.
image = (
    bladekit.Image.debian_slim(python_version="3.12")
    .apt_install("ffmpeg")
    # sphn provides the Opus (de)framing the websocket protocol uses.
    .pip_install("moshi==0.2.4", "torch", "sphn", "huggingface_hub")
)

MODEL_REPO = "kyutai/moshiko-pytorch-bf16"


# @app.web_server: the function runs ONCE at container boot. It must start a
# long-lived HTTP/websocket server on the port AI Fabric assigns
# ($FABRIC_SERVER_PORT) and return immediately. subprocess.Popen is
# non-blocking, so AI Fabric can then probe the port until Moshi answers (weight
# download + load is slow, hence the generous startup_timeout) and only then
# marks the replica ready and starts forwarding traffic.
#
# min_replicas=1 keeps one warm replica: a ~7B model load far exceeds the
# cold-start wake window, so scaling to zero would make the first request time
# out. hf-token lets huggingface_hub pull the weights fast.
@app.web_server(startup_timeout=900, min_replicas=1, max_replicas=1)
@app.function(
    gpu="RTX-A4500",
    image=image,
    secrets=[bladekit.Secret.from_name("hf-token")],
    timeout=3600,
)
def serve():
    import os
    import subprocess

    port = os.environ["FABRIC_SERVER_PORT"]
    subprocess.Popen(
        [
            "python", "-m", "moshi.server",
            "--host", "0.0.0.0",
            "--port", port,
            "--hf-repo", MODEL_REPO,
        ],
    )

Run it

  1. Moshi pulls in sentencepiece, which only ships prebuilt wheels for Python 3.12, so pin the image to 3.12 and match it on the client (AI Fabric requires the client Python to match the image): uv venv -p 3.12 .venv312 && uv pip install --python .venv312/bin/python bladekit, then .venv312/bin/blade deploy moshi_voice_chat.py. AI Fabric builds the image, pulls the bf16 weights, and prints the public URL, https://moshi-voice-chat.apps.run.blade.sh.
  2. On boot the replica runs serve once, which Popens python -m moshi.server on $FABRIC_SERVER_PORT. The replica is only marked ready AFTER the server answers on the loopback port, so the ~7B model has finished loading onto the A4500 before any request is forwarded (startup_timeout=900 covers the download + load).
  3. Probe that the server is up and the model loaded. Endpoints are private, so send your AI Fabric token: curl -H 'Authorization: Bearer <FABRIC_TOKEN>' https://moshi-voice-chat.apps.run.blade.sh/ returns Moshi's web UI HTML (HTTP 200). Moshi owns all routing under /, including the websocket at /api/chat.
  4. Talk to it: open that URL in a browser with the token, allow the microphone, and speak. Moshi streams Opus audio both ways over the /api/chat websocket and replies in real time, interrupting and back-channelling like a real conversation. Because it is one full-duplex model, there is no STT -> LLM -> TTS latency stack.
  5. Swap moshiko (male voice) for moshika (female) via --hf-repo kyutai/moshika-pytorch-bf16. Stop the app when you're done: blade app stop moshi-voice-chat.

Concepts used: Cold start, Endpoint, Replica, GPU

Contact the team

Loading the form…