Developer docs / AI Fabric tutorials

Embeddings web endpoint (sentence-transformers)

  • GPU
  • Volume
  • Web endpoint
  • Class

Take it for a spin

The whole program in one file, ready to run.

embed_svc.py
import bladekit

app = bladekit.app("embed-svc")

# Cache the model weights across deploys (downloaded once, reused forever).
hf_cache = bladekit.Volume.from_name("embed-hf-cache", create_if_missing=True)

# fabric/torch already ships torch + sentence-transformers loads cleanly on it.
image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("sentence-transformers")
    .env({"HF_HOME": "/hf-cache"})
)

MODEL_ID = "BAAI/bge-small-en-v1.5"  # 384-d, ~130 MB, public on the hub

@app.cls(gpu="RTX-A4500", image=image, timeout=600,
         volumes={"/hf-cache": hf_cache})
@app.concurrent(max_inputs=16)
class Embedder:
    def __init__(self):
        from sentence_transformers import SentenceTransformer
        # Loaded once when the replica starts; reused for every request.
        self.model = SentenceTransformer(MODEL_ID, device="cuda")

    @app.web_endpoint(path="/embed", methods=["POST"], min_replicas=0, max_replicas=4)
    def embed(self, item: dict) -> dict:
        texts = item.get("texts") or [item.get("text", "")]
        vecs = self.model.encode(texts, batch_size=64, show_progress_bar=False)
        return {"model": MODEL_ID, "dim": int(vecs.shape[1]),
                "embeddings": vecs.tolist()}

Run it

  1. Deploy the endpoint: blade deploy embed_svc.py.
  2. POST a batch: curl -X POST <app-url>/embed -H 'Authorization: Bearer <FABRIC_TOKEN>' -d '{"texts": ["hello", "world"]}'. Endpoints are private by default. A single {"text": "..."} also works.
  3. There is no local entrypoint: an @app.cls is a remote service, so calling Embedder() directly would construct the class (and import sentence-transformers, load CUDA) on your laptop, not on the GPU. Drive it through the deployed endpoint instead.
  4. The model loads once per replica (__init__ runs once when the container starts); @app.concurrent(max_inputs=16) lets one GPU serve many embed requests at once. Weights cache on the Volume across deploys.

Concepts used: GPU, Cold start, Replica, Endpoint

Contact the team

Loading the form…