Developer docs / AI Fabric SDK / Resilience & errors

Exceptions

The full AI Fabric exception hierarchy — platform/API errors, run-level infrastructure failures, and the wrapper for exceptions raised by your own remote code.

Overview

Every exception the SDK raises is either a remote user exception — your own function code raised something on the remote side — or a platform error — the SDK, the control plane, or the fleet failed to run your code at all. The two are never conflated: RemoteFunctionError (an alias of RemoteError) is the sole carrier for the former; everything else descends from FabricError.

Signature
class FabricError(Exception):
    """Base class for every platform-side error surfaced by the SDK."""

APIError family

Raised when the control plane returns its standard JSON error envelope. code is the stable, machine-readable identifier (SCREAMING_SNAKE_CASE); unknown codes are tolerated and simply surface as the base APIError.

Signature
class APIError(FabricError):
    def __init__(
        self,
        code: str,
        message: str,
        *,
        status_code: Optional[int] = None,
        details: Optional[Mapping[str, Any]] = None,
    ) -> None: ...

code str

Stable machine error code, e.g. NOT_FOUND or GPU_QUOTA_EXCEEDED.

message str

Human-readable message from the API envelope (available as **.api_message**).

status_code Optional[int]Default: None

HTTP status code of the response, when known.

details Optional[Mapping[str, Any]]Default: None

Structured error payload (available as **.details**, defaults to {}).

ClassCode / HTTPRaised when
AuthenticationErrorUNAUTHENTICATED (401)Token missing, invalid, or revoked.
PermissionDeniedErrorPERMISSION_DENIED (403)Token is valid but the operation is forbidden.
NotFoundError*_NOT_FOUND (404)The object does not exist, or belongs to another user.
InvalidArgumentErrorINVALID_ARGUMENT / INVALID_CURSOR / IMAGE_ALIAS_UNKNOWN (400)The request argument is malformed or invalid.
ConflictError409 familyState conflict — BUNDLE_NOT_READY, RUN_NOT_CANCELLABLE, etc.
PayloadTooLargeError413 familyBUNDLE_TOO_LARGE / PAYLOAD_TOO_LARGE.
UnavailableErrorUNAVAILABLE (503)A dependency is down; the caller may retry.
GPUQuotaExceededErrorGPU_QUOTA_EXCEEDEDThe namespace GPU quota would be exceeded — only when the namespace uses quota_mode=enforce. The default quota_mode=queue queues the run instead and emits a GPU_QUOTA_REACHED warning.

RunError family

Raised when a run or call fails for an infrastructure reason — never for a user exception. code mirrors the run's own error code from the wire protocol; the SDK maps known codes to a more specific subclass with an actionable message, and falls back to the base RunError for anything unrecognized.

Signature
class RunError(FabricError):
    code: str = "INTERNAL"

    def __init__(
        self,
        message: str,
        *,
        code: Optional[str] = None,
        run_id: Optional[str] = None,
        details: Optional[Mapping[str, Any]] = None,
    ) -> None: ...

message str

Actionable, human-readable error message.

code Optional[str]Default: None

Overrides the subclass's default **.code** when given.

run_id Optional[str]Default: None

The run/call id this error belongs to, when known.

details Optional[Mapping[str, Any]]Default: None

Structured details, e.g. last_error_code for RetriesExhaustedError.

Class`.code`Raised when
NodeLostErrorNODE_LOSTThe GPU node went silent for more than 60s; usually transient — re-run it.
ContainerFailedErrorCONTAINER_FAILEDThe container never started or vanished. Check the image ref and run logs.
ImagePullFailedErrorIMAGE_PULL_FAILEDThe node could not pull the image; check the alias/registry ref.
RunnerCrashedErrorRUNNER_CRASHED / RUNNER_INTERNALThe in-container runner process crashed.
BundleFetchFailedErrorBUNDLE_FETCH_FAILEDThe runner could not download or verify your code bundle; re-run to re-upload.
VersionMismatchErrorPYTHON_VERSION_MISMATCH / CLOUDPICKLE_VERSION_MISMATCHClient/remote handshake mismatch — remote Python or cloudpickle version does not match the client.
TimeoutError_TIMEOUTA one-shot run (or cron / map worker) exceeded its **timeout=**. Deployed endpoint replicas are exempt: there timeout= caps a single request, never the service's lifetime.
PreemptedErrorPREEMPTEDPreempted by a higher-priority job. map() inputs are auto-requeued without consuming the retry budget; simple runs are terminal — re-run or use map().
InputTimeoutErrorINPUT_TIMEOUTA single map() input exceeded **input_timeout_seconds** (distinct from the run-global TIMEOUT).
RetriesExhaustedErrorRETRIES_EXHAUSTEDAn input exhausted all retry attempts. The last infra error code is in details["last_error_code"]; increase **retries=** or fix the underlying error.
RunLostError (subclasses NodeLostError)NODE_LOSTRaised specifically when a run/call transitions to the lost state (node monitor declared the node silent). Existing except NodeLostError handlers still catch it.

RemoteError / RemoteFunctionError

RemoteFunctionError (an alias of RemoteError) wraps an exception raised by your own function on the remote side. When the original exception type can be unpickled locally, the SDK re-raises it directly with the remote traceback attached — so except ValueError on your own code still works across the network. RemoteError/RemoteFunctionError is only used as a fallback wrapper when the original exception can't be reconstructed locally (e.g. it's defined in a module the client doesn't have).

Signature
class RemoteError(FabricError):
    def __init__(
        self,
        exc_type: str,
        message: str,
        remote_traceback: str,
    ) -> None: ...

# Alias kept for parity with the runner-protocol wording.
RemoteFunctionError = RemoteError

exc_type str

Qualified name of the original remote exception type (available as **.exc_type**).

message str

The original exception's message (available as **.remote_message**).

remote_traceback str

The full remote traceback text (available as **.remote_traceback**), appended to str(exc) for readability.

Connection & cancellation

ClassParentRaised when
ConnectionFailedFabricErrorThe control plane could not be reached at all — DNS failure, connection refused, or transport timeout.
RunCancelledFabricErrorThe run was cancelled, either locally (Ctrl-C) or by the timeout watcher. Carries **.run_id**.
Signature
class RunCancelled(FabricError):
    def __init__(self, run_id: str, *, message: Optional[str] = None) -> None: ...

Sandbox errors

Sandbox methods (exec, put_file, get_file, ls, tunnel) raise a family of APIError subclasses for well-known conflict/limit conditions, plus two FabricError-derived exceptions for lifecycle failures.

ClassParentRaised when
SandboxNotRunningErrorAPIErrorSANDBOX_NOT_RUNNING (409) — the sandbox is creating, terminating, terminated, or failed. details["state"] carries the current state.
SandboxExecLimitErrorAPIErrorSANDBOX_EXEC_LIMIT (429) — too many concurrent execs; the sandbox hit max_concurrent_execs (16 by default).
SandboxFileNotFoundErrorNotFoundErrorSANDBOX_FILE_NOT_FOUND (404) — path does not exist in the sandbox rootfs.
SandboxFileTooLargeErrorPayloadTooLargeErrorSANDBOX_FILE_TOO_LARGE (413) — file exceeds the 5 GiB per-file limit.
SandboxPathInvalidErrorInvalidArgumentErrorSANDBOX_PATH_INVALID (400) — path is non-absolute, contains .., or a symlink would resolve outside the rootfs. Always use absolute paths; never build paths from untrusted input.
SandboxDiskQuotaExceededErrorPayloadTooLargeErrorSANDBOX_DISK_QUOTA_EXCEEDED (413) — write would exceed the sandbox's cumulative disk quota (distinct from the per-file limit above).
SandboxPortInUseErrorConflictErrorSANDBOX_PORT_IN_USE (409) — another tunnel already exposes this port. Re-POSTing the same port is idempotent; this only fires for a *different* tunnel object.
SandboxTerminatedErrorFabricErrorThe sandbox died while an exec was in flight (timeout, terminate(), preemption, node lost, OOM). The exec ends in state terminated with no exit_code; create a new sandbox.
SandboxFailedErrorFabricErrorThe sandbox transitioned to failed before becoming running. .code reuses the run-error registry (IMAGE_PULL_FAILED, CONTAINER_FAILED, GPU_QUOTA_EXCEEDED, INTERNAL, …).
RuntimeUnavailableWarningFabricErrorNot raised as an exception — surfaced as a warning in the Sandbox.create() response when no schedulable node announces the requested runtime. The sandbox stays creating (e.g. a runsc sandbox waits rather than silently falling back to runc).
Signature
class SandboxTerminatedError(FabricError):
    def __init__(self, sandbox_id: str, code: str, message: str) -> None: ...

class SandboxFailedError(FabricError):
    def __init__(self, sandbox_id: str, code: str, message: str) -> None: ...

Image build & volume errors

Signature
class ImageBuildError(FabricError):
    def __init__(
        self,
        build_id: str,
        code: str,
        message: str,
        *,
        definition_hash: Optional[str] = None,
    ) -> None: ...

build_id str

Id of the failed build (available as **.build_id**); pass it to blade image logs <build_id>.

code str

The build's own error code: IMAGE_BUILD_FAILED, IMAGE_BUILD_TIMEOUT, IMAGE_PUSH_FAILED, or INTERNAL.

message str

Build failure message (available as **.build_message**).

definition_hash Optional[str]Default: None

Content hash of the image definition, when available.

Raised when a bladekit.Image build does not reach succeeded. The message always points you at the CLI command to inspect the build logs.

ClassRaised when
VolumeErrorA volume operation is invalid, e.g. calling commit() on an unmounted volume.
VolumeCommitErrorvolume.commit() failed or was refused. .code mirrors the server (VOLUME_COMMIT_FAILED, VOLUME_READ_ONLY, VOLUME_COMMIT_IN_PROGRESS, NODE_LOST, RUNNER_CRASHED, …). Never fatal to the container — the caller may retry.
Signature
class VolumeCommitError(FabricError):
    def __init__(self, code: str, message: str) -> None: ...

Catch patterns

Catch RemoteFunctionError (or your own exception types, which are re-raised as-is when unpicklable) to handle failures in your code, RunError subclasses for retryable infrastructure failures, and the narrower APIError subclasses for request-time problems.

Python
import bladekit

try:
    result = my_function.remote(x)
except bladekit.RemoteFunctionError as exc:
    # Your own function raised something the client couldn't reconstruct.
    print(f"user code failed remotely: {exc}")
except bladekit.NodeLostError:
    # Transient infra failure — safe to retry.
    result = my_function.remote(x)
except bladekit.RetriesExhaustedError as exc:
    print("gave up after retries; last error:", exc.details.get("last_error_code"))
except bladekit.RunError as exc:
    # Any other infra failure: CONTAINER_FAILED, IMAGE_PULL_FAILED, TIMEOUT, ...
    raise
except bladekit.NotFoundError:
    print("the app or function no longer exists")

Reference from the Blade console · Updated .

Contact the team

Loading the form…