Developer docs / AI Fabric tutorials

Semantic near-duplicate detection

  • GPU
  • Batch / map
  • Image DSL
  • Volume

Exact-string deduplication only catches sentences that are byte-for-byte identical. It misses the harder, more common case: paraphrases, re-orderings, and trivial edits that mean the same thing but read differently. To catch those you have to compare meaning, and meaning lives in embeddings: two sentences are near-duplicates when their vectors point in nearly the same direction.

In this tutorial you'll embed ~500 sentences with BGE-small on RTX A4500s, fanned out with .map(), then find every near-duplicate pair by cosine similarity. Both halves (the GPU embedding and the similarity scan) run remotely, so blade run needs nothing installed locally beyond the AI Fabric client.

Before you start

You need the AI Fabric CLI installed and authenticated: run blade setup once. The embedding function runs on a single RTX A4500. We extend fabric/torch (the catalog base: CUDA + PyTorch + Python 3.12) with sentence-transformers, and cache the model weights on a Volume so they download once and every parallel container reuses them.

Create the app, Volume, and image

Create semantic_dedup.py. Declare a Volume for the weight cache, then build the image on top of the catalog base and point HF_HOME at the mounted Volume so downloads land in the cache. Pin the model and the similarity threshold: 0.90 cosine and above counts as a near-duplicate:

semantic_dedup.py
import bladekit

app = bladekit.app("semantic-dedup")

# A Volume persists across runs. The BGE weights download once, then every
# future run (and every parallel container) reuses the cache instead of
# re-downloading from the hub.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)

# fabric/torch is the AI Fabric catalog base (CUDA + PyTorch + Python 3.12). We add
# sentence-transformers on top; no Dockerfile, no manual setup. HF_HOME points
# at the mounted Volume so weights land in the cache.
image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("sentence-transformers==3.0.1")
    .env({"HF_HOME": "/hf-cache"})
)

MODEL_ID = "BAAI/bge-small-en-v1.5"  # 384-dim, ~130 MB, public on the HF hub
THRESHOLD = 0.90  # cosine similarity at/above this = "near-duplicate"

Build a dataset with hidden duplicates

To have something to find, seed the corpus with deliberate clusters of near-duplicates (paraphrases and re-orderings of each other), then pad it out to ~500 with distinct filler sentences. This is exactly the mix that exact-string dedup misses but a semantic embedding catches:

Python
def build_dataset() -> list[str]:
    """~500 sentences, with deliberate clusters of near-duplicates mixed in.

    Replace this with any list of strings, e.g. a HuggingFace dataset slice:
        from datasets import load_dataset
        rows = load_dataset("ag_news", split="train[:500]")["text"]

    The near-dup groups below are paraphrases / re-orderings / trivial edits of
    each other, exactly the kind of thing exact-string dedup misses but a
    semantic embedding catches.
    """
    # Each inner list is a cluster of sentences that *mean* nearly the same thing.
    near_dup_groups = [
        [
            "The cat sat quietly on the warm windowsill.",
            "A cat was sitting calmly on the sunny window ledge.",
            "On the warm windowsill, the cat sat quietly.",
        ],
        [
            "Our GPU fleet embeds documents in just a few seconds.",
            "The GPU cluster turns documents into vectors within seconds.",
            "In seconds, the GPU fleet converts documents to embeddings.",
        ],
        [
            "She booked a flight to Tokyo for the spring festival.",
            "He reserved a plane ticket to Tokyo for the spring festival.",
        ],
        [
            "The recipe calls for two cups of flour and one egg.",
            "You need two cups of flour and a single egg for this recipe.",
        ],
        [
            "Per-minute billing means you only pay for the time you use.",
            "With per-minute billing, you are charged only for what you run.",
        ],
        [
            "The quarterly report showed a sharp rise in revenue.",
            "Revenue climbed sharply according to the quarterly report.",
        ],
    ]

    docs: list[str] = []
    for group in near_dup_groups:
        docs.extend(group)

The rest of build_dataset fills the list up to ~500 with varied, textually distinct filler so the dataset is realistic: mostly unique, with the duplicate clusters hidden inside:

Python
    # Pad out to ~500 with varied, *distinct* filler sentences so the dataset is
    # realistic (mostly unique, a few dup clusters hidden inside).
    subjects = [
        "A serverless GPU platform", "The embedding model", "This dataset",
        "A transformer network", "The vector index", "Our retrieval pipeline",
        "The fleet scheduler", "A batch of documents", "The cold-start path",
        "Per-minute billing", "The tokenizer", "A spot instance",
    ]
    predicates = [
        "scales out across many GPUs in parallel.",
        "turns text into dense vectors in milliseconds.",
        "is cached so the next run starts instantly.",
        "fans work out with a single .map() call.",
        "fits comfortably in 20 GB of VRAM.",
        "runs far faster on a GPU than on a CPU.",
        "returns results in the original input order.",
        "only costs you the minutes it actually runs.",
        "handles a surge of traffic without manual scaling.",
        "keeps the model resident between calls.",
    ]
    i = 0
    target = 500
    while len(docs) < target:
        s = subjects[i % len(subjects)]
        p = predicates[(i // len(subjects)) % len(predicates)]
        # row index keeps each filler sentence textually distinct
        docs.append(f"{s} {p} (note {i})")
        i += 1
    return docs

Embed a chunk into normalized vectors

The embedding function takes one chunk, loads BGE onto the GPU once per container, and encodes with normalize_embeddings=True. Unit-length vectors make the next step cheap: when vectors are normalized, a plain dot product is the cosine similarity. It also times a CPU baseline on the same chunk for an honest speedup number:

Python
@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=900,
    volumes={"/hf-cache": hf_cache},
)
def embed_chunk(texts: list[str]) -> dict:
    """Embed one chunk of sentences on the GPU. Loads the model once per container.

    Returns L2-normalized vectors (so a plain dot product == cosine similarity)
    plus timing for both GPU and a one-shot CPU baseline, so the caller can
    report a real speedup.
    """
    import time

    import torch
    from sentence_transformers import SentenceTransformer

    # Load once per container; subsequent chunks routed to this hot container
    # reuse the already-loaded model (no reload, no re-download).
    gpu_model = SentenceTransformer(MODEL_ID, device="cuda")

    # --- GPU timing (the real workload) ---
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    vecs = gpu_model.encode(
        texts, batch_size=128, show_progress_bar=False,
        normalize_embeddings=True,  # unit vectors -> dot product is cosine sim
    )
    torch.cuda.synchronize()
    gpu_seconds = time.perf_counter() - t0

    # --- CPU baseline on the SAME chunk, for an honest speedup number ---
    cpu_model = SentenceTransformer(MODEL_ID, device="cpu")
    t0 = time.perf_counter()
    cpu_model.encode(
        texts, batch_size=128, show_progress_bar=False,
        normalize_embeddings=True,
    )
    cpu_seconds = time.perf_counter() - t0

    return {
        "vectors": vecs.tolist(),
        "n": len(texts),
        "dim": int(vecs.shape[1]),
        "gpu_seconds": gpu_seconds,
        "cpu_seconds": cpu_seconds,
    }

Find the near-duplicate pairs

Because the vectors are already normalized, the entire cosine-similarity matrix is V @ V.T. Scan the strict upper triangle (that skips self-pairs and avoids reporting both (i, j) and (j, i)) and keep every pair at or above the threshold. This runs as a CPU-only remote function (NumPy ships in the fabric/torch image), so nothing needs to be installed on your laptop:

Python
@app.function(image=image, timeout=300)  # CPU only: pure NumPy, no GPU needed
def find_near_duplicates(
    vectors: list[list[float]], threshold: float
) -> list[tuple[int, int, float]]:
    """Return (i, j, similarity) for every pair with cosine sim >= threshold.

    Vectors are already L2-normalized, so the full cosine-similarity matrix is
    just V @ V.T. We scan the strict upper triangle to avoid self-pairs and
    duplicate (j, i) pairs. This runs REMOTELY (NumPy ships in the fabric/torch
    image) so `blade run` needs nothing installed locally beyond the AI Fabric client.
    """
    import numpy as np

    v = np.asarray(vectors, dtype=np.float32)
    sims = v @ v.T  # (N, N) cosine similarity matrix

    pairs: list[tuple[int, int, float]] = []
    n = sims.shape[0]
    iu, ju = np.triu_indices(n, k=1)  # strict upper triangle
    mask = sims[iu, ju] >= threshold
    for i, j in zip(iu[mask].tolist(), ju[mask].tolist()):
        pairs.append((i, j, float(sims[i, j])))
    # Strongest matches first
    pairs.sort(key=lambda t: t[2], reverse=True)
    return pairs

Wire it together

In the entrypoint, chunk the dataset and fan it out with .map(): one GPU container per chunk, results in input order so the vectors line up with docs exactly:

Python
@app.local_entrypoint()
def main():
    docs = build_dataset()
    chunk_size = 100
    chunks = [docs[i : i + chunk_size] for i in range(0, len(docs), chunk_size)]
    print(
        f"Embedding {len(docs)} sentences in {len(chunks)} chunks "
        f"across the GPU fleet..."
    )

    all_vectors: list[list[float]] = []
    total_gpu = 0.0
    total_cpu = 0.0
    dim = None

    # .map() fans the chunks out: one GPU container per chunk, all in parallel.
    # Results arrive in input order, so the vectors line up with `docs` exactly.
    for res in embed_chunk.map(chunks, return_exceptions=True):
        if isinstance(res, Exception):
            print(f"  chunk failed: {res}")
            continue
        all_vectors.extend(res["vectors"])
        total_gpu += res["gpu_seconds"]
        total_cpu += res["cpu_seconds"]
        dim = res["dim"]

With all the vectors in hand, call find_near_duplicates.remote(...) to run the similarity scan on a CPU worker, then print the strongest pairs and the measured speedup:

Python
    if len(all_vectors) != len(docs):
        print(
            f"Got {len(all_vectors)} vectors for {len(docs)} docs; "
            "some chunks failed, check the run logs above."
        )
        if not all_vectors:
            return

    print(f"\nEmbedded {len(all_vectors)} sentences, dim={dim}")

    # --- the actual task: find semantic near-duplicates (runs remotely) ---
    pairs = find_near_duplicates.remote(all_vectors, THRESHOLD)
    print(
        f"\n--- near-duplicate pairs (cosine >= {THRESHOLD:.2f}): "
        f"{len(pairs)} found ---"
    )
    for i, j, sim in pairs[:20]:  # cap the print; there can be many filler dups
        print(f"  sim={sim:.3f}  [{i:>3}] {docs[i]!r}")
        print(f"             [{j:>3}] {docs[j]!r}")
    if len(pairs) > 20:
        print(f"  ... and {len(pairs) - 20} more pairs above threshold.")

Run it

Run the program with the AI Fabric CLI:

Terminal
blade run semantic_dedup.py

The chunks fan out across the GPU fleet, the BGE weights cache on the Volume (so later runs skip the download), and the cosine scan runs on a CPU worker. You'll see the near-duplicate pairs printed strongest-first (the planted clusters surface with similarities well above 0.90), followed by the measured speedup (~27x here). Swap build_dataset() for any list of strings to dedup your own data.

Take it for a spin

Everything above in one file, ready to run.

semantic_dedup.py
"""Semantic near-duplicate detection on the GPU fleet, fanned out with .map().

Run with:  blade run semantic_dedup.py

What it does
------------
- Builds a small text dataset in-process (~500 sentences), no files to download,
  no setup. Some sentences are *intentional* near-duplicates: paraphrases,
  re-orderings, and trivial edits of one another. The goal is to find those pairs.
- Embeds every sentence with the BGE-small sentence-transformers model on a
  RTX-A4500 GPU. The model (~130 MB) downloads from the public HF hub once,
  is cached on a Volume, and fits comfortably in 20 GB of VRAM.
- Splits the dataset into chunks and uses `.map()` to fan the chunks out across
  the GPU fleet. Many containers embed in parallel, results come back in order.
- Computes cosine similarity between all vectors locally and reports every pair
  above a similarity threshold as a near-duplicate.
- Each GPU worker also embeds its chunk on the CPU once, so the script prints a
  real measured GPU-vs-CPU speedup, not a hand-waved number.

The headline: hundreds of docs embedded in seconds on GPU vs minutes on CPU,
AI Fabric bills per started minute, and a single .map() call gives you the fan-out for free.
"""

import bladekit

app = bladekit.app("semantic-dedup")

# A Volume persists across runs: the BGE weights download once, then every
# future run (and every parallel container) reuses the cache instead of
# re-downloading from the hub.
hf_cache = bladekit.Volume.from_name("hf-cache", create_if_missing=True)

# fabric/torch is the AI Fabric catalog base (CUDA + PyTorch + Python 3.12). We add
# sentence-transformers on top; no Dockerfile, no manual setup. HF_HOME points
# at the mounted Volume so weights land in the cache.
image = (
    bladekit.Image.from_catalog("fabric/torch")
    .pip_install("sentence-transformers==3.0.1")
    .env({"HF_HOME": "/hf-cache"})
)

MODEL_ID = "BAAI/bge-small-en-v1.5"  # 384-dim, ~130 MB, public on the HF hub
THRESHOLD = 0.90  # cosine similarity at/above this = "near-duplicate"


def build_dataset() -> list[str]:
    """~500 sentences, with deliberate clusters of near-duplicates mixed in.

    Replace this with any list of strings, e.g. a HuggingFace dataset slice:
        from datasets import load_dataset
        rows = load_dataset("ag_news", split="train[:500]")["text"]

    The near-dup groups below are paraphrases / re-orderings / trivial edits of
    each other, exactly the kind of thing exact-string dedup misses but a
    semantic embedding catches.
    """
    # Each inner list is a cluster of sentences that *mean* nearly the same thing.
    near_dup_groups = [
        [
            "The cat sat quietly on the warm windowsill.",
            "A cat was sitting calmly on the sunny window ledge.",
            "On the warm windowsill, the cat sat quietly.",
        ],
        [
            "Our GPU fleet embeds documents in just a few seconds.",
            "The GPU cluster turns documents into vectors within seconds.",
            "In seconds, the GPU fleet converts documents to embeddings.",
        ],
        [
            "She booked a flight to Tokyo for the spring festival.",
            "He reserved a plane ticket to Tokyo for the spring festival.",
        ],
        [
            "The recipe calls for two cups of flour and one egg.",
            "You need two cups of flour and a single egg for this recipe.",
        ],
        [
            "Per-minute billing means you only pay for the time you use.",
            "With per-minute billing, you are charged only for what you run.",
        ],
        [
            "The quarterly report showed a sharp rise in revenue.",
            "Revenue climbed sharply according to the quarterly report.",
        ],
    ]

    docs: list[str] = []
    for group in near_dup_groups:
        docs.extend(group)

    # Pad out to ~500 with varied, *distinct* filler sentences so the dataset is
    # realistic (mostly unique, a few dup clusters hidden inside).
    subjects = [
        "A serverless GPU platform", "The embedding model", "This dataset",
        "A transformer network", "The vector index", "Our retrieval pipeline",
        "The fleet scheduler", "A batch of documents", "The cold-start path",
        "Per-minute billing", "The tokenizer", "A spot instance",
    ]
    predicates = [
        "scales out across many GPUs in parallel.",
        "turns text into dense vectors in milliseconds.",
        "is cached so the next run starts instantly.",
        "fans work out with a single .map() call.",
        "fits comfortably in 20 GB of VRAM.",
        "runs far faster on a GPU than on a CPU.",
        "returns results in the original input order.",
        "only costs you the minutes it actually runs.",
        "handles a surge of traffic without manual scaling.",
        "keeps the model resident between calls.",
    ]
    i = 0
    target = 500
    while len(docs) < target:
        s = subjects[i % len(subjects)]
        p = predicates[(i // len(subjects)) % len(predicates)]
        # row index keeps each filler sentence textually distinct
        docs.append(f"{s} {p} (note {i})")
        i += 1
    return docs


@app.function(
    gpu="RTX-A4500",
    image=image,
    timeout=900,
    volumes={"/hf-cache": hf_cache},
)
def embed_chunk(texts: list[str]) -> dict:
    """Embed one chunk of sentences on the GPU. Loads the model once per container.

    Returns L2-normalized vectors (so a plain dot product == cosine similarity)
    plus timing for both GPU and a one-shot CPU baseline, so the caller can
    report a real speedup.
    """
    import time

    import torch
    from sentence_transformers import SentenceTransformer

    # Load once per container; subsequent chunks routed to this hot container
    # reuse the already-loaded model (no reload, no re-download).
    gpu_model = SentenceTransformer(MODEL_ID, device="cuda")

    # --- GPU timing (the real workload) ---
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    vecs = gpu_model.encode(
        texts, batch_size=128, show_progress_bar=False,
        normalize_embeddings=True,  # unit vectors -> dot product is cosine sim
    )
    torch.cuda.synchronize()
    gpu_seconds = time.perf_counter() - t0

    # --- CPU baseline on the SAME chunk, for an honest speedup number ---
    cpu_model = SentenceTransformer(MODEL_ID, device="cpu")
    t0 = time.perf_counter()
    cpu_model.encode(
        texts, batch_size=128, show_progress_bar=False,
        normalize_embeddings=True,
    )
    cpu_seconds = time.perf_counter() - t0

    return {
        "vectors": vecs.tolist(),
        "n": len(texts),
        "dim": int(vecs.shape[1]),
        "gpu_seconds": gpu_seconds,
        "cpu_seconds": cpu_seconds,
    }


@app.function(image=image, timeout=300)  # CPU only: pure NumPy, no GPU needed
def find_near_duplicates(
    vectors: list[list[float]], threshold: float
) -> list[tuple[int, int, float]]:
    """Return (i, j, similarity) for every pair with cosine sim >= threshold.

    Vectors are already L2-normalized, so the full cosine-similarity matrix is
    just V @ V.T. We scan the strict upper triangle to avoid self-pairs and
    duplicate (j, i) pairs. This runs REMOTELY (NumPy ships in the fabric/torch
    image) so `blade run` needs nothing installed locally beyond the AI Fabric client.
    """
    import numpy as np

    v = np.asarray(vectors, dtype=np.float32)
    sims = v @ v.T  # (N, N) cosine similarity matrix

    pairs: list[tuple[int, int, float]] = []
    n = sims.shape[0]
    iu, ju = np.triu_indices(n, k=1)  # strict upper triangle
    mask = sims[iu, ju] >= threshold
    for i, j in zip(iu[mask].tolist(), ju[mask].tolist()):
        pairs.append((i, j, float(sims[i, j])))
    # Strongest matches first
    pairs.sort(key=lambda t: t[2], reverse=True)
    return pairs


@app.local_entrypoint()
def main():
    docs = build_dataset()
    chunk_size = 100
    chunks = [docs[i : i + chunk_size] for i in range(0, len(docs), chunk_size)]
    print(
        f"Embedding {len(docs)} sentences in {len(chunks)} chunks "
        f"across the GPU fleet..."
    )

    all_vectors: list[list[float]] = []
    total_gpu = 0.0
    total_cpu = 0.0
    dim = None

    # .map() fans the chunks out: one GPU container per chunk, all in parallel.
    # Results arrive in input order, so the vectors line up with `docs` exactly.
    for res in embed_chunk.map(chunks, return_exceptions=True):
        if isinstance(res, Exception):
            print(f"  chunk failed: {res}")
            continue
        all_vectors.extend(res["vectors"])
        total_gpu += res["gpu_seconds"]
        total_cpu += res["cpu_seconds"]
        dim = res["dim"]

    if len(all_vectors) != len(docs):
        print(
            f"Got {len(all_vectors)} vectors for {len(docs)} docs; "
            "some chunks failed, check the run logs above."
        )
        if not all_vectors:
            return

    print(f"\nEmbedded {len(all_vectors)} sentences, dim={dim}")

    # --- the actual task: find semantic near-duplicates (runs remotely) ---
    pairs = find_near_duplicates.remote(all_vectors, THRESHOLD)
    print(
        f"\n--- near-duplicate pairs (cosine >= {THRESHOLD:.2f}): "
        f"{len(pairs)} found ---"
    )
    for i, j, sim in pairs[:20]:  # cap the print; there can be many filler dups
        print(f"  sim={sim:.3f}  [{i:>3}] {docs[i]!r}")
        print(f"             [{j:>3}] {docs[j]!r}")
    if len(pairs) > 20:
        print(f"  ... and {len(pairs) - 20} more pairs above threshold.")

    # --- speedup + cost note ---
    speedup = (total_cpu / total_gpu) if total_gpu > 0 else float("nan")
    print("\n--- timing (summed over all chunks) ---")
    print(f"  GPU encode time: {total_gpu:6.2f}s")
    print(f"  CPU encode time: {total_cpu:6.2f}s")
    print(f"  speedup:         {speedup:5.1f}x faster on GPU")
    print(
        "\nAI Fabric bills each container per started minute: this whole job used a few "
        "GPU-minutes across the fleet, and nothing while idle."
    )

Concepts used: GPU, Run, Volume

Contact the team

Loading the form…