Developer docs / AI Fabric tutorials

Parallel Parquet processing

  • Batch / map
  • Image DSL

A large Parquet dataset usually comes as many files, and that shape helps, because each file can be processed completely independently. There's no reason to grind through them one at a time on a single machine. The pattern is to fan the shards out: give each file its own container and let them all run at once.

In this tutorial you'll process a folder of Parquet shards in parallel with AI Fabric's .starmap(). You write an ordinary function that handles one shard (read it, aggregate it, write the result), and AI Fabric launches a separate CPU container for every shard, all running simultaneously. No GPU needed; this is pure data crunching.

Before you start

You need the AI Fabric CLI installed and authenticated: run blade setup once. The job runs on CPU, so the image is built from bladekit.Image.debian_slim with pandas, pyarrow, and polars layered on. Put your Parquet files in a data/ folder and create a results/ folder next to your script.

Create the app and image

Create a file called parquet_parallel.py. Declare the App and a lightweight Debian image with the data libraries. No Dockerfile, no GPU drivers:

parquet_parallel.py
import bladekit

app = bladekit.app("parquet-parallel")

image = (
    bladekit.Image.debian_slim(python_version="3.12")
    .pip_install("pandas", "pyarrow", "polars")
)

Write the per-shard worker

The unit of parallelism is one shard. Write a function that reads a single Parquet file with polars, computes an aggregation, and writes the result to the output directory. Request cpu=4 and memory=8192 so each worker has room to chew through its file:

Python
@app.function(image=image, cpu=4, memory=8192, timeout=600)
def process_shard(shard_path: str, output_dir: str) -> dict:
    import polars as pl
    import os

    df = pl.read_parquet(shard_path)
    # Example: compute some aggregation
    result = df.group_by("category").agg(pl.col("value").mean().alias("mean_value"))
    out_path = os.path.join(output_dir, os.path.basename(shard_path))
    result.write_parquet(out_path)
    return {"shard": shard_path, "rows_in": len(df), "rows_out": len(result)}

Returning a small dict of facts (the shard name and the row counts in and out) keeps the results light so they serialize cheaply back to your laptop. Swap the group_by for whatever transform your data needs; the parallelism doesn't change.

Fan out with .starmap()

The local_entrypoint runs on your machine. Glob the shards into a list, then build (shard_path, output_dir) pairs, one tuple per task. .starmap() is like .map() but unpacks each tuple as separate arguments, which is exactly what process_shard(shard_path, output_dir) expects. return_exceptions=True keeps one bad shard from cancelling the rest of the batch:

Python
@app.local_entrypoint()
def main():
    import glob
    shards = glob.glob("data/*.parquet")
    args = [(s, "results/") for s in shards]  # (shard_path, output_dir) pairs

    for outcome in process_shard.starmap(args, return_exceptions=True):
        if isinstance(outcome, Exception):
            print(f"FAILED: {outcome}")
        else:
            print(f"  {outcome['shard']}: {outcome['rows_in']} → {outcome['rows_out']} rows")

Run it

Run the program with the AI Fabric CLI:

Terminal
blade run parquet_parallel.py

Each shard runs in its own container with 4 CPUs and 8 GB of RAM, all at the same time, and you'll see one line per shard with its row counts. Because the containers ran in parallel, the whole dataset is processed in roughly the time of its slowest single shard, and each shard is billed per started minute of vCPU time.

Take it for a spin

Everything above in one file, ready to run.

parquet_parallel.py
import bladekit

app = bladekit.app("parquet-parallel")

# Inputs and outputs live on a Volume. The container filesystem is stateless,
# so writing to a relative dir would vanish when the container exits.
data_vol = bladekit.Volume.from_name("parquet-data", create_if_missing=True)

image = (
    bladekit.Image.debian_slim(python_version="3.12")
    .pip_install("pandas", "pyarrow", "polars")
)

# ── Step 0: generate sample input shards on the Volume (run once) ──
@app.function(image=image, cpu=2, memory=4096, timeout=300,
              volumes={"/data": data_vol})
def make_shards(n_shards: int = 4, rows_per_shard: int = 10_000) -> list[str]:
    import polars as pl, os, random
    os.makedirs("/data/shards", exist_ok=True)
    os.makedirs("/data/results", exist_ok=True)
    paths = []
    for s in range(n_shards):
        df = pl.DataFrame({
            "category": [random.choice("ABCD") for _ in range(rows_per_shard)],
            "value": [random.random() for _ in range(rows_per_shard)],
        })
        path = f"/data/shards/shard_{s:03d}.parquet"
        df.write_parquet(path)
        paths.append(path)
    return paths

@app.function(image=image, cpu=4, memory=8192, timeout=600,
              volumes={"/data": data_vol})
def process_shard(shard_path: str, output_dir: str) -> dict:
    import polars as pl
    import os

    df = pl.read_parquet(shard_path)
    # Example: compute some aggregation
    result = df.group_by("category").agg(pl.col("value").mean().alias("mean_value"))
    os.makedirs(output_dir, exist_ok=True)
    out_path = os.path.join(output_dir, os.path.basename(shard_path))
    result.write_parquet(out_path)   # writes to the mounted Volume, not /tmp
    return {"shard": shard_path, "rows_in": len(df), "rows_out": len(result)}

@app.local_entrypoint()
def main():
    # Materialise sample shards on the Volume, then fan the work out.
    shards = make_shards.remote()
    args = [(s, "/data/results") for s in shards]  # (shard_path, output_dir) pairs

    for outcome in process_shard.starmap(args, return_exceptions=True):
        if isinstance(outcome, Exception):
            print(f"FAILED: {outcome}")
        else:
            print(f"  {outcome['shard']}: {outcome['rows_in']} → {outcome['rows_out']} rows")

Concepts used: Run, Volume

Contact the team

Loading the form…