Developer docs / AI Fabric tutorials
Embeddings web endpoint (sentence-transformers)
- GPU
- Volume
- Web endpoint
- Class
Take it for a spin
The whole program in one file, ready to run.
import bladekit
app = bladekit.app("embed-svc")
# Cache the model weights across deploys (downloaded once, reused forever).
hf_cache = bladekit.Volume.from_name("embed-hf-cache", create_if_missing=True)
# fabric/torch already ships torch + sentence-transformers loads cleanly on it.
image = (
bladekit.Image.from_catalog("fabric/torch")
.pip_install("sentence-transformers")
.env({"HF_HOME": "/hf-cache"})
)
MODEL_ID = "BAAI/bge-small-en-v1.5" # 384-d, ~130 MB, public on the hub
@app.cls(gpu="RTX-A4500", image=image, timeout=600,
volumes={"/hf-cache": hf_cache})
@app.concurrent(max_inputs=16)
class Embedder:
def __init__(self):
from sentence_transformers import SentenceTransformer
# Loaded once when the replica starts; reused for every request.
self.model = SentenceTransformer(MODEL_ID, device="cuda")
@app.web_endpoint(path="/embed", methods=["POST"], min_replicas=0, max_replicas=4)
def embed(self, item: dict) -> dict:
texts = item.get("texts") or [item.get("text", "")]
vecs = self.model.encode(texts, batch_size=64, show_progress_bar=False)
return {"model": MODEL_ID, "dim": int(vecs.shape[1]),
"embeddings": vecs.tolist()}Run it
- Deploy the endpoint:
blade deploy embed_svc.py. - POST a batch:
curl -X POST <app-url>/embed -H 'Authorization: Bearer <FABRIC_TOKEN>' -d '{"texts": ["hello", "world"]}'. Endpoints are private by default. A single{"text": "..."}also works. - There is no local entrypoint: an
@app.clsis a remote service, so callingEmbedder()directly would construct the class (and import sentence-transformers, load CUDA) on your laptop, not on the GPU. Drive it through the deployed endpoint instead. - The model loads once per replica (
__init__runs once when the container starts);@app.concurrent(max_inputs=16)lets one GPU serve many embed requests at once. Weights cache on the Volume across deploys.
Concepts used: GPU, Cold start, Replica, Endpoint