Overview
Every exception the SDK raises is either a remote user exception — your own function code raised something on the remote side — or a platform error — the SDK, the control plane, or the fleet failed to run your code at all. The two are never conflated: RemoteFunctionError (an alias of RemoteError) is the sole carrier for the former; everything else descends from FabricError.
class FabricError(Exception):
"""Base class for every platform-side error surfaced by the SDK."""APIError family
Raised when the control plane returns its standard JSON error envelope. code is the stable, machine-readable identifier (SCREAMING_SNAKE_CASE); unknown codes are tolerated and simply surface as the base APIError.
class APIError(FabricError):
def __init__(
self,
code: str,
message: str,
*,
status_code: Optional[int] = None,
details: Optional[Mapping[str, Any]] = None,
) -> None: ...codestrStable machine error code, e.g.
NOT_FOUNDorGPU_QUOTA_EXCEEDED.messagestrHuman-readable message from the API envelope (available as **
.api_message**).status_codeOptional[int]Default:NoneHTTP status code of the response, when known.
detailsOptional[Mapping[str, Any]]Default:NoneStructured error payload (available as **
.details**, defaults to{}).
| Class | Code / HTTP | Raised when |
|---|---|---|
AuthenticationError | UNAUTHENTICATED (401) | Token missing, invalid, or revoked. |
PermissionDeniedError | PERMISSION_DENIED (403) | Token is valid but the operation is forbidden. |
NotFoundError | *_NOT_FOUND (404) | The object does not exist, or belongs to another user. |
InvalidArgumentError | INVALID_ARGUMENT / INVALID_CURSOR / IMAGE_ALIAS_UNKNOWN (400) | The request argument is malformed or invalid. |
ConflictError | 409 family | State conflict — BUNDLE_NOT_READY, RUN_NOT_CANCELLABLE, etc. |
PayloadTooLargeError | 413 family | BUNDLE_TOO_LARGE / PAYLOAD_TOO_LARGE. |
UnavailableError | UNAVAILABLE (503) | A dependency is down; the caller may retry. |
GPUQuotaExceededError | GPU_QUOTA_EXCEEDED | The namespace GPU quota would be exceeded — only when the namespace uses quota_mode=enforce. The default quota_mode=queue queues the run instead and emits a GPU_QUOTA_REACHED warning. |
RunError family
Raised when a run or call fails for an infrastructure reason — never for a user exception. code mirrors the run's own error code from the wire protocol; the SDK maps known codes to a more specific subclass with an actionable message, and falls back to the base RunError for anything unrecognized.
class RunError(FabricError):
code: str = "INTERNAL"
def __init__(
self,
message: str,
*,
code: Optional[str] = None,
run_id: Optional[str] = None,
details: Optional[Mapping[str, Any]] = None,
) -> None: ...messagestrActionable, human-readable error message.
codeOptional[str]Default:NoneOverrides the subclass's default **
.code** when given.run_idOptional[str]Default:NoneThe run/call id this error belongs to, when known.
detailsOptional[Mapping[str, Any]]Default:NoneStructured details, e.g.
last_error_codeforRetriesExhaustedError.
| Class | `.code` | Raised when |
|---|---|---|
NodeLostError | NODE_LOST | The GPU node went silent for more than 60s; usually transient — re-run it. |
ContainerFailedError | CONTAINER_FAILED | The container never started or vanished. Check the image ref and run logs. |
ImagePullFailedError | IMAGE_PULL_FAILED | The node could not pull the image; check the alias/registry ref. |
RunnerCrashedError | RUNNER_CRASHED / RUNNER_INTERNAL | The in-container runner process crashed. |
BundleFetchFailedError | BUNDLE_FETCH_FAILED | The runner could not download or verify your code bundle; re-run to re-upload. |
VersionMismatchError | PYTHON_VERSION_MISMATCH / CLOUDPICKLE_VERSION_MISMATCH | Client/remote handshake mismatch — remote Python or cloudpickle version does not match the client. |
TimeoutError_ | TIMEOUT | A one-shot run (or cron / map worker) exceeded its **timeout=**. Deployed endpoint replicas are exempt: there timeout= caps a single request, never the service's lifetime. |
PreemptedError | PREEMPTED | Preempted by a higher-priority job. map() inputs are auto-requeued without consuming the retry budget; simple runs are terminal — re-run or use map(). |
InputTimeoutError | INPUT_TIMEOUT | A single map() input exceeded **input_timeout_seconds** (distinct from the run-global TIMEOUT). |
RetriesExhaustedError | RETRIES_EXHAUSTED | An input exhausted all retry attempts. The last infra error code is in details["last_error_code"]; increase **retries=** or fix the underlying error. |
RunLostError (subclasses NodeLostError) | NODE_LOST | Raised specifically when a run/call transitions to the lost state (node monitor declared the node silent). Existing except NodeLostError handlers still catch it. |
RemoteError / RemoteFunctionError
RemoteFunctionError (an alias of RemoteError) wraps an exception raised by your own function on the remote side. When the original exception type can be unpickled locally, the SDK re-raises it directly with the remote traceback attached — so except ValueError on your own code still works across the network. RemoteError/RemoteFunctionError is only used as a fallback wrapper when the original exception can't be reconstructed locally (e.g. it's defined in a module the client doesn't have).
class RemoteError(FabricError):
def __init__(
self,
exc_type: str,
message: str,
remote_traceback: str,
) -> None: ...
# Alias kept for parity with the runner-protocol wording.
RemoteFunctionError = RemoteErrorexc_typestrQualified name of the original remote exception type (available as **
.exc_type**).messagestrThe original exception's message (available as **
.remote_message**).remote_tracebackstrThe full remote traceback text (available as **
.remote_traceback**), appended tostr(exc)for readability.
Connection & cancellation
| Class | Parent | Raised when |
|---|---|---|
ConnectionFailed | FabricError | The control plane could not be reached at all — DNS failure, connection refused, or transport timeout. |
RunCancelled | FabricError | The run was cancelled, either locally (Ctrl-C) or by the timeout watcher. Carries **.run_id**. |
class RunCancelled(FabricError):
def __init__(self, run_id: str, *, message: Optional[str] = None) -> None: ...Sandbox errors
Sandbox methods (exec, put_file, get_file, ls, tunnel) raise a family of APIError subclasses for well-known conflict/limit conditions, plus two FabricError-derived exceptions for lifecycle failures.
| Class | Parent | Raised when |
|---|---|---|
SandboxNotRunningError | APIError | SANDBOX_NOT_RUNNING (409) — the sandbox is creating, terminating, terminated, or failed. details["state"] carries the current state. |
SandboxExecLimitError | APIError | SANDBOX_EXEC_LIMIT (429) — too many concurrent execs; the sandbox hit max_concurrent_execs (16 by default). |
SandboxFileNotFoundError | NotFoundError | SANDBOX_FILE_NOT_FOUND (404) — path does not exist in the sandbox rootfs. |
SandboxFileTooLargeError | PayloadTooLargeError | SANDBOX_FILE_TOO_LARGE (413) — file exceeds the 5 GiB per-file limit. |
SandboxPathInvalidError | InvalidArgumentError | SANDBOX_PATH_INVALID (400) — path is non-absolute, contains .., or a symlink would resolve outside the rootfs. Always use absolute paths; never build paths from untrusted input. |
SandboxDiskQuotaExceededError | PayloadTooLargeError | SANDBOX_DISK_QUOTA_EXCEEDED (413) — write would exceed the sandbox's cumulative disk quota (distinct from the per-file limit above). |
SandboxPortInUseError | ConflictError | SANDBOX_PORT_IN_USE (409) — another tunnel already exposes this port. Re-POSTing the same port is idempotent; this only fires for a *different* tunnel object. |
SandboxTerminatedError | FabricError | The sandbox died while an exec was in flight (timeout, terminate(), preemption, node lost, OOM). The exec ends in state terminated with no exit_code; create a new sandbox. |
SandboxFailedError | FabricError | The sandbox transitioned to failed before becoming running. .code reuses the run-error registry (IMAGE_PULL_FAILED, CONTAINER_FAILED, GPU_QUOTA_EXCEEDED, INTERNAL, …). |
RuntimeUnavailableWarning | FabricError | Not raised as an exception — surfaced as a warning in the Sandbox.create() response when no schedulable node announces the requested runtime. The sandbox stays creating (e.g. a runsc sandbox waits rather than silently falling back to runc). |
class SandboxTerminatedError(FabricError):
def __init__(self, sandbox_id: str, code: str, message: str) -> None: ...
class SandboxFailedError(FabricError):
def __init__(self, sandbox_id: str, code: str, message: str) -> None: ...Image build & volume errors
class ImageBuildError(FabricError):
def __init__(
self,
build_id: str,
code: str,
message: str,
*,
definition_hash: Optional[str] = None,
) -> None: ...build_idstrId of the failed build (available as **
.build_id**); pass it toblade image logs <build_id>.codestrThe build's own error code:
IMAGE_BUILD_FAILED,IMAGE_BUILD_TIMEOUT,IMAGE_PUSH_FAILED, orINTERNAL.messagestrBuild failure message (available as **
.build_message**).definition_hashOptional[str]Default:NoneContent hash of the image definition, when available.
Raised when a bladekit.Image build does not reach succeeded. The message always points you at the CLI command to inspect the build logs.
| Class | Raised when |
|---|---|
VolumeError | A volume operation is invalid, e.g. calling commit() on an unmounted volume. |
VolumeCommitError | volume.commit() failed or was refused. .code mirrors the server (VOLUME_COMMIT_FAILED, VOLUME_READ_ONLY, VOLUME_COMMIT_IN_PROGRESS, NODE_LOST, RUNNER_CRASHED, …). Never fatal to the container — the caller may retry. |
class VolumeCommitError(FabricError):
def __init__(self, code: str, message: str) -> None: ...Catch patterns
Catch RemoteFunctionError (or your own exception types, which are re-raised as-is when unpicklable) to handle failures in your code, RunError subclasses for retryable infrastructure failures, and the narrower APIError subclasses for request-time problems.
import bladekit
try:
result = my_function.remote(x)
except bladekit.RemoteFunctionError as exc:
# Your own function raised something the client couldn't reconstruct.
print(f"user code failed remotely: {exc}")
except bladekit.NodeLostError:
# Transient infra failure — safe to retry.
result = my_function.remote(x)
except bladekit.RetriesExhaustedError as exc:
print("gave up after retries; last error:", exc.details.get("last_error_code"))
except bladekit.RunError as exc:
# Any other infra failure: CONTAINER_FAILED, IMAGE_PULL_FAILED, TIMEOUT, ...
raise
except bladekit.NotFoundError:
print("the app or function no longer exists")Reference from the Blade console · Updated .