Developer docs / AI Fabric tutorials

Live transcription over WebRTC

  • Web endpoint
  • Image DSL

Most endpoints answer one request and hang up. But some experiences (a voice assistant, a live caption track, a call transcriber) need a continuous stream of audio flowing both ways with almost no delay. That's what WebRTC is for, and in this tutorial you'll wire your microphone straight into an AI Fabric container and watch the words appear as you speak.

There's a catch that shapes the whole design: WebRTC media travels over UDP, peer-to-peer, which AI Fabric's HTTP ingress can't carry. AI Fabric nodes have public addresses and containers run in the host network, so the audio normally flows browser straight to your container. For networks that won't allow that, every node also runs its own TURN relay as a fallback, still on the same node and never a central hop. Only the tiny signalling handshake (the SDP offer/answer) goes over the normal HTTP endpoint.

Before you start

You need the AI Fabric CLI authenticated (blade setup). Nothing else to set up: the platform injects the relay's URL and shared secret into your container as env vars, and your app mints its own short-lived WebRTC credentials from them. No GPU required: faster-whisper runs on CPU here.

Build the image

Start a file live_transcribe.py. The image is fabric/base plus the WebRTC + speech stack: aiortc (the Python WebRTC engine), av (audio decoding), faster-whisper (fast CPU transcription), and FastAPI to serve the signalling:

live_transcribe.py
import bladekit

image = (
    bladekit.Image.from_catalog("fabric/base")
    .apt_install("ffmpeg")
    .pip_install("aiortc", "av", "faster-whisper", "fastapi", "uvicorn", "numpy")
)

app = bladekit.app("live-transcribe")

Serve an ASGI app

@app.asgi_app runs a FastAPI app under uvicorn and forwards HTTP to it. The platform injects the relay of the node your container landed on: FABRIC_TURN_URL and FABRIC_TURN_SECRET, plus FABRIC_TURNS_URL for the TLS leg on 443. The same daemon answers STUN, so the browser's STUN url is that node too. Read them, then load the Whisper model once at startup:

Python
@app.asgi_app(startup_timeout=600, min_replicas=0, max_replicas=2)
@app.function(
    image=image,
    timeout=3600,
)
def web():
    import asyncio, base64, hashlib, hmac, json, os, time
    import numpy as np
    from aiortc import (RTCConfiguration, RTCIceServer, RTCPeerConnection, RTCSessionDescription)
    from av.audio.resampler import AudioResampler
    from faster_whisper import WhisperModel
    from fastapi import FastAPI, Request
    from fastapi.responses import HTMLResponse

    TURN_URL = os.environ["FABRIC_TURN_URL"]  # e.g. turn:<node-ip>:3478
    TURN_SECRET = os.environ["FABRIC_TURN_SECRET"]
    TURNS_URL = os.environ.get("FABRIC_TURNS_URL")  # TLS leg, tcp/443
    _, TURN_IP, TURN_PORT = TURN_URL.split(":")
    STUN_URL = f"stun:{TURN_IP}:{TURN_PORT}"
    model = WhisperModel("base", device="cpu", compute_type="int8")

Mint short-lived TURN credentials

The relay uses the standard REST credential scheme: the username is an expiry timestamp and the password is base64(HMAC-SHA1(secret, username)). The same helper credentials the container's aiortc and the browser, so neither ever sees the long-lived secret:

Python
    def ice_server() -> RTCIceServer:
        username = str(int(time.time()) + 3600)
        cred = base64.b64encode(
            hmac.new(TURN_SECRET.encode(), username.encode(), hashlib.sha1).digest()
        ).decode()
        urls = [
            f"turn:{TURN_IP}:{TURN_PORT}?transport=udp",
            f"turn:{TURN_IP}:{TURN_PORT}?transport=tcp",
        ]
        return RTCIceServer(urls=urls, username=username, credential=cred)

Handle the WebRTC offer

The browser POSTs an SDP offer to /offer. Create an RTCPeerConnection with the node's relay as fallback, wire up the incoming audio track and the transcript data channel, then answer. The HTTP request returns the SDP answer, and that's the last the ingress sees of this session; the audio flows browser-to-container from here on (or through the node's own relay if the network forces it):

Python
    fapp = FastAPI()
    pcs: set = set()

    @fapp.post("/offer")
    async def offer(req: Request):
        body = await req.json()
        pc = RTCPeerConnection(RTCConfiguration(iceServers=[ice_server()]))
        pcs.add(pc)
        state = {"dc": None}

        @pc.on("datachannel")
        def on_datachannel(channel):
            state["dc"] = channel

        @pc.on("track")
        def on_track(track):
            if track.kind == "audio":
                asyncio.ensure_future(transcribe(track, state))

        await pc.setRemoteDescription(RTCSessionDescription(sdp=body["sdp"], type=body["type"]))
        await pc.setLocalDescription(await pc.createAnswer())
        return {"sdp": pc.localDescription.sdp, "type": pc.localDescription.type}

Transcribe the stream and push it back

Audio frames arrive as Opus; resample them to 16 kHz mono, accumulate a rolling window, and every ~1.5 seconds run Whisper over the trailing few seconds and send the running transcript down the data channel. The browser shows it instantly:

Python
    async def transcribe(track, state):
        resampler = AudioResampler(format="s16", layout="mono", rate=16000)
        window = np.zeros(0, dtype=np.float32)
        SR = 16000
        while True:
            try:
                frame = await track.recv()
            except Exception:
                break
            for rf in resampler.resample(frame):
                arr = rf.to_ndarray().flatten().astype(np.float32) / 32768.0
                window = np.concatenate([window, arr])
            if len(window) >= int(SR * 1.5):
                clip = window[-SR * 8 :]
                segments, _ = model.transcribe(clip, language="en", beam_size=1, vad_filter=True)
                text = "".join(s.text for s in segments).strip()
                if text and state["dc"] and state["dc"].readyState == "open":
                    state["dc"].send(json.dumps({"text": text}))
                window = window[-SR * 8 :]

    return fapp

The browser side (served from /) is plain JS: getUserMedia for the mic, an RTCPeerConnection built from the iceServers that /turn hands out, a transcript data channel, and the same /offer handshake. There's no iceTransportPolicy: ICE races the direct and relayed paths and keeps whichever connects, which is usually direct. The full HTML is in the complete program below.

Deploy and speak

Deploy it (CPU build, no GPU):

Terminal
blade deploy live_transcribe.py

Open the printed https://live-transcribe.apps.run.blade.sh URL, click Start, allow the mic, and talk. The transcript updates live as you speak.

Take it for a spin

Everything above in one file, ready to run.

live_transcribe.py
import bladekit

# Live speech-to-text over WebRTC. The browser streams microphone audio to the
# container over a WebRTC PeerConnection; the container runs faster-whisper and
# streams the transcript back over a WebRTC data channel, all in real time.
#
# WebRTC media is UDP and peer-to-peer, which AI Fabric's HTTP ingress can't carry.
# AI Fabric nodes have public IPs and containers run in the host network, so ICE
# usually connects the browser to the container DIRECTLY: the /turn endpoint
# hands the browser a STUN url (to discover its public address) plus TURN relay
# credentials as an automatic fallback for restrictive networks.
#
# All of that comes from the relay running on the app's OWN node, injected by
# the platform as FABRIC_TURN_URL / FABRIC_TURN_SECRET (plus FABRIC_TURNS_URL, the
# TLS leg on 443 for networks that only pass HTTPS). Nothing is centralised, so
# even the relayed path stays browser -> this node. The app mints its own
# short-lived TURN credentials from the injected secret. Only the WebRTC
# *signalling* (SDP offer/answer) rides the normal HTTP endpoint.
#
# Whisper runs on CPU here (faster-whisper int8), so no GPU is needed.

image = (
    bladekit.Image.from_catalog("fabric/base")
    .apt_install("ffmpeg")
    .pip_install("aiortc", "av", "faster-whisper", "fastapi", "uvicorn", "numpy")
)

app = bladekit.app("live-transcribe")


@app.asgi_app(startup_timeout=600, min_replicas=0, max_replicas=2, public=True)
@app.function(
    image=image,
    timeout=3600,
)  # public=True: the page opens directly in a browser (no AI Fabric token in the URL)
def web():
    import asyncio
    import base64
    import hashlib
    import hmac
    import json
    import os
    import time

    import numpy as np
    from aiortc import (
        RTCConfiguration,
        RTCIceServer,
        RTCPeerConnection,
        RTCSessionDescription,
    )
    from av.audio.resampler import AudioResampler
    from faster_whisper import WhisperModel
    from fastapi import FastAPI, Request
    from fastapi.responses import HTMLResponse

    # The platform injects the relay of the node this container runs on. The
    # same daemon answers STUN, so the browser's STUN url is that node too, and
    # no central service is involved anywhere in the media path.
    TURN_URL = os.environ.get("FABRIC_TURN_URL")  # e.g. turn:<node-ip>:3478
    TURN_SECRET = os.environ.get("FABRIC_TURN_SECRET")
    # TLS leg for networks that only pass HTTPS: the same relay on tcp/443
    # (turns:<node>.relay.<domain>:443?transport=tcp). Same secret, so the
    # credentials minted below work for it unchanged.
    TURNS_URL = os.environ.get("FABRIC_TURNS_URL")
    if TURN_URL and TURN_SECRET:
        _, TURN_IP, TURN_PORT = TURN_URL.split(":")
        STUN_URL = f"stun:{TURN_IP}:{TURN_PORT}"
    else:
        # No relay on this node: ICE still connects directly most of the time
        # (the container has a public address and learns the browser's from the
        # incoming connectivity checks), just without a relay fallback.
        STUN_URL = None

    # base int8 transcribes faster than real time on CPU; bump to "small" for
    # better accuracy if you give the function a GPU.
    model = WhisperModel("base", device="cpu", compute_type="int8")

    def ice_server() -> RTCIceServer | None:
        # REST credentials: username = expiry timestamp, password =
        # base64(HMAC-SHA1(secret, username)). Valid for both browser and aiortc.
        if not (TURN_URL and TURN_SECRET):
            return None
        username = str(int(time.time()) + 3600)
        cred = base64.b64encode(
            hmac.new(TURN_SECRET.encode(), username.encode(), hashlib.sha1).digest()
        ).decode()
        urls = [
            f"turn:{TURN_IP}:{TURN_PORT}?transport=udp",
            f"turn:{TURN_IP}:{TURN_PORT}?transport=tcp",
        ]
        return RTCIceServer(urls=urls, username=username, credential=cred)

    fapp = FastAPI()
    pcs: set = set()

    @fapp.get("/turn")
    async def turn():
        s = ice_server()
        servers = []
        if STUN_URL:
            # STUN first: the browser learns its public address and ICE
            # connects browser <-> container directly when the network
            # allows. The TURN entries below are the automatic fallback.
            servers.append({"urls": [STUN_URL]})
        if s:
            servers.append(
                {"urls": s.urls, "username": s.username, "credential": s.credential}
            )
            if TURNS_URL:
                # Extra turns: entry for networks that only pass TLS on 443;
                # ICE races all entries and picks whatever connects.
                servers.append(
                    {"urls": [TURNS_URL], "username": s.username, "credential": s.credential}
                )
        return {"iceServers": servers}

    @fapp.get("/")
    async def index():
        return HTMLResponse(INDEX_HTML)

    @fapp.post("/offer")
    async def offer(req: Request):
        body = await req.json()
        s = ice_server()
        pc = RTCPeerConnection(RTCConfiguration(iceServers=[s] if s else []))
        pcs.add(pc)
        state = {"dc": None}

        @pc.on("datachannel")
        def on_datachannel(channel):
            state["dc"] = channel

        @pc.on("connectionstatechange")
        async def on_change():
            if pc.connectionState in ("failed", "closed"):
                await pc.close()
                pcs.discard(pc)

        @pc.on("track")
        def on_track(track):
            if track.kind == "audio":
                asyncio.ensure_future(transcribe(track, state))

        await pc.setRemoteDescription(RTCSessionDescription(sdp=body["sdp"], type=body["type"]))
        await pc.setLocalDescription(await pc.createAnswer())
        return {"sdp": pc.localDescription.sdp, "type": pc.localDescription.type}

    async def transcribe(track, state):
        resampler = AudioResampler(format="s16", layout="mono", rate=16000)
        window = np.zeros(0, dtype=np.float32)
        SR = 16000
        while True:
            try:
                frame = await track.recv()
            except Exception:
                break
            for rf in resampler.resample(frame):
                arr = rf.to_ndarray().flatten().astype(np.float32) / 32768.0
                window = np.concatenate([window, arr])
            # Once we have ~1.5s of fresh audio, transcribe the trailing window
            # (up to 8s of context) and push the running transcript.
            if len(window) >= int(SR * 1.5):
                clip = window[-SR * 8 :]
                segments, _ = model.transcribe(clip, language="en", beam_size=1, vad_filter=True)
                text = "".join(s.text for s in segments).strip()
                if text and state["dc"] and state["dc"].readyState == "open":
                    state["dc"].send(json.dumps({"text": text}))
                window = window[-SR * 8 :]

    return fapp


INDEX_HTML = """<!doctype html>
<html><head><meta charset="utf-8"><title>AI Fabric · Live transcription</title>
<style>
 body{font-family:system-ui,sans-serif;max-width:680px;margin:48px auto;padding:0 16px;color:#1a1a2e}
 h1{font-weight:600} button{font-size:15px;padding:10px 18px;border-radius:8px;border:0;
   background:#3b5bdb;color:#fff;cursor:pointer} button:disabled{opacity:.5}
 #status{margin:14px 0;color:#666;font-size:13px}
 #out{margin-top:20px;padding:18px;border:1px solid #e2e6f0;border-radius:10px;min-height:120px;
   font-size:18px;line-height:1.6;white-space:pre-wrap}
</style></head><body>
<h1>🎙️ Live transcription over WebRTC</h1>
<p>Click start, allow the mic, and speak. Audio streams to the container; Whisper
streams text back over a WebRTC data channel, not the HTTP ingress.</p>
<button id="go">Start</button>
<div id="status">idle</div>
<div id="out"></div>
<script>
const $=s=>document.querySelector(s), st=m=>$("#status").textContent=m;
$("#go").onclick=async()=>{
  $("#go").disabled=true; st("getting ICE servers…");
  const {iceServers}=await (await fetch("turn")).json();
  const pc=new RTCPeerConnection({iceServers});
  const dc=pc.createDataChannel("transcript");
  dc.onmessage=e=>{ $("#out").textContent=JSON.parse(e.data).text; };
  pc.onconnectionstatechange=()=>st("connection: "+pc.connectionState);
  const stream=await navigator.mediaDevices.getUserMedia({audio:true});
  stream.getTracks().forEach(t=>pc.addTrack(t,stream));
  await pc.setLocalDescription(await pc.createOffer());
  await new Promise(r=>{ if(pc.iceGatheringState==="complete")return r();
    pc.onicegatheringstatechange=()=>pc.iceGatheringState==="complete"&&r(); });
  const ans=await (await fetch("offer",{method:"POST",headers:{"content-type":"application/json"},
    body:JSON.stringify({sdp:pc.localDescription.sdp,type:pc.localDescription.type})})).json();
  await pc.setRemoteDescription(ans); st("listening, speak!");
};
</script></body></html>
"""

Concepts used: Endpoint, Scale-to-zero, Image build

Contact the team

Loading the form…