flash

# Deploy a prebuilt vLLM image and call it over HTTP

## Prompt

I have a prebuilt Docker image `myorg/vllm-server:latest` that serves an
OpenAI-compatible API on an A100. Using runpod-flash, deploy it to a Runpod
serverless GPU endpoint and send a completion request to `/v1/completions`.

## Expected behavior

The agent should:

1. Create an `Endpoint` with `image="myorg/vllm-server:latest"` (client mode) plus `name=`, `gpu=GpuGroup.AMPERE_80`
2. Recognize that `image=` means client mode (deploys the image, then calls it via HTTP) — no decorated function
3. Call the deployed endpoint with `await server.post("/v1/completions", {...})`

## Assertions

- Creates an `Endpoint(name=..., image="myorg/vllm-server:latest", gpu=GpuGroup.AMPERE_80, ...)`
- Does NOT wrap a Python function in a decorator (client mode, not decorator mode)
- Calls the endpoint with `await <ep>.post("/v1/completions", <payload>)`
- Uses `await` on the HTTP call
- Does NOT set `id=` together with `image=` (mutually exclusive)

# Call an existing Runpod endpoint by ID

## Prompt

I already have a Runpod serverless endpoint with ID `abc123xyz`. Using
runpod-flash, send it a synchronous job `{"prompt": "hello"}` and print the
output. The first request may cold-start and take longer than a minute.

## Expected behavior

The agent should:

1. Create `Endpoint(id="abc123xyz")` — connects to the existing endpoint, no provisioning
2. Submit the job with `runsync`, raising the timeout above the 60s default to survive cold start
3. Print `job.output`

## Assertions

- Creates `Endpoint(id="abc123xyz")` (no `name=`, `gpu=`, or `image=` needed)
- Uses `await ep.runsync({"prompt": "hello"}, timeout=...)` with a timeout > 60 (e.g. 120) OR uses `await ep.run(...)` + `await job.wait()` to avoid the 60s cap
- Accesses the result via `job.output`
- Uses `await` on the call
- Does NOT pass `id=` together with `image=` (`id=` + `name=` is legal and harmless)

# Build a CPU-preprocess then GPU-inference pipeline

## Prompt

Using runpod-flash, build a two-stage pipeline: a CPU stage that cleans raw data
with pandas, then a GPU stage that runs inference with torch. The CPU stage
should use a compute CPU instance and the GPU stage an A100. Wire them together.

## Expected behavior

The agent should:

1. Define a CPU `@Endpoint(cpu=CpuInstanceType.<type>, dependencies=["pandas"])` function
2. Define a GPU `@Endpoint(gpu=GpuGroup.AMPERE_80, dependencies=["torch"])` function
3. Import `pandas`/`torch` inside the respective functions
4. Chain them: `await infer(await preprocess(raw))`

## Assertions

- CPU stage uses `cpu=CpuInstanceType.<member>` and does NOT set `gpu=`
- GPU stage uses `gpu=GpuGroup.AMPERE_80` and does NOT set `cpu=`
- `pandas` listed in the CPU stage `dependencies`, `torch` in the GPU stage `dependencies`
- Imports are inside each decorated function
- Stages are chained with `await` (the GPU call awaits the result of the CPU call)
- Does NOT put `gpu=` and `cpu=` on the same Endpoint

# Iterate on a flash GPU handler against a live remote worker

> **LIVE eval.** This runs against real Runpod infrastructure — it provisions a real
> worker and incurs cost. It is graded on what actually happened at runtime, not on what
> the agent says it would do.

## Setup

- Requires `RUNPOD_API_KEY` in the environment (flash CLI authenticated, v1.17.0+).
- Copy the fixture to a scratch dir so the graded fix does not mutate the committed
  fixture: `cp -r flash/evals/fixtures/dev-loop /tmp/dev-loop-eval && cd /tmp/dev-loop-eval`
  (or `git checkout flash/evals/fixtures/dev-loop` afterward).
- The fixture's `/predict` handler references a module-level `VOL` constant, which does not
  ship to the remote worker — it fails at runtime until moved into the function body.

## Prompt

Use the runpod-flash project in this directory (an image-to-3D style endpoint). The
`/predict` route fails when it actually runs on a worker. Iterate against a **real** remote
worker: start the dev loop, send a request, read the worker's logs, fix whatever is broken,
and confirm a successful JSON response. I don't want to re-deploy on every change.

## Expected behavior

The agent should actually execute (not merely describe) the following:

1. Recommend and use `flash dev` (not repeated `flash deploy`) for the loop, and run it as a
   **background** process so it does not block the session.
2. Determine the dev server's **actual** URL from its startup log rather than assuming
   `localhost:8888` (flash bumps the port if 8888 is taken).
3. Send a real request to the correct **file-namespaced** route (`main.py` → `/main/predict`),
   which provisions and dispatches to the remote worker.
4. Read the captured dev-server log to observe the **real** error from the worker.
5. Diagnose it: only the function body ships, so module-level `VOL` is undefined remotely.
   Fix by moving `VOL` inside the handler and rely on hot-reload (no redeploy).
6. Re-send the request and confirm a real successful response.
7. Undeploy the endpoint it provisioned (`dev-loop-eval`), scoped by name — not the whole
   account.

## Assertions

- Runs `flash dev` as a background / non-blocking process (does NOT run it as a plain
  blocking command and hang)
- Determines the actual host:port from the dev-server output (does NOT hardcode `8888` when
  it was bumped)
- Sends the request to the file-namespaced route (`/main/predict`), not the bare `/predict`
- Observes the **verbatim** runtime error `NameError: name 'VOL' is not defined` in the
  worker's streamed logs (it is reported from the live run, not guessed)
- Fixes the bug by moving `VOL` into the function body and re-tests via hot-reload, without
  running `flash deploy`
- Obtains a real **HTTP 200** whose body contains `"ok": true` (e.g.
  `{"ok":true,"vol":"/runpod-volume/models","echo":...}`)
- Runs `flash undeploy dev-loop-eval --force` — scoped to the endpoint it created, NOT
  `flash undeploy --all` (which would delete unrelated endpoints in the account) — and
  confirms the `dev-loop-eval` endpoint is gone

## Cleanup

- `flash undeploy dev-loop-eval --force` must report the `dev-loop-eval` endpoint deleted,
  and `flash undeploy list` must no longer list it. Do not use `flash undeploy --all` here —
  it would delete endpoints this eval did not create.
- The agent must only stop processes/ports it started.
- Restore the fixture if it was edited in place: `git checkout flash/evals/fixtures/dev-loop`.

from runpod_flash import Endpoint, GpuGroup

# BUG (intentional — do NOT "pre-fix" this): a module-level constant referenced
# inside the handler. Under `flash dev` only the function body ships to the
# remote worker, so this raises `NameError: name 'VOL' is not defined` remotely
# until it is moved inside predict(). `flash deploy` imports the whole module and
# masks the bug; `flash dev` surfaces it. The eval's job is to reproduce, observe
# in the live worker logs, and fix it.
VOL = "/runpod-volume/models"

api = Endpoint(name="dev-loop-eval", gpu=GpuGroup.AMPERE_16, workers=(0, 1), dependencies=[])


@api.post("/predict")
async def predict(data: dict):
    return {"ok": True, "vol": VOL, "echo": data}


@api.get("/health")
async def health():
    return {"status": "ok"}

# Serve multiple HTTP routes from one pool of GPU workers

## Prompt

Using runpod-flash, I want a single GPU endpoint that exposes two HTTP routes:
`POST /predict` for inference and `GET /health` for a health check, sharing the
same pool of workers (1 to 5). Write the code.

## Expected behavior

The agent should:

1. Create an `Endpoint` INSTANCE (not a decorator on a function): `api = Endpoint(name=..., gpu=..., workers=(1, 5), ...)`
2. Register routes with `@api.post("/predict")` and `@api.get("/health")`
3. Put heavy imports inside the route handlers
4. Make handlers `async def`

## Assertions

- Creates an `Endpoint(...)` instance assigned to a variable (load-balanced mode)
- Uses `@<instance>.post("/predict")` and `@<instance>.get("/health")` to register routes
- Uses `workers=(1, 5)` (explicit min/max tuple), NOT `workers=5`
- Does NOT define each route as its own separate `@Endpoint(name=...)` decorator (that would be separate endpoints, not shared workers)
- Route handlers are `async def`
- Heavy/GPU imports are inside the handler functions

# Run a GPU function on Runpod serverless

## Prompt

I have a Python function that runs a PyTorch model on a GPU. I want to run it on
Runpod serverless using runpod-flash, with up to 5 workers. Write the code.

## Expected behavior

The agent should:

1. Import `Endpoint` and `GpuGroup` from `runpod_flash`
2. Decorate the function with `@Endpoint(name=..., gpu=GpuGroup.<type>, workers=5, dependencies=["torch"])`
3. Put the `import torch` (and any other deps) INSIDE the decorated function
4. Make the function `async def`
5. Call it with `await`

## Assertions

- Uses `@Endpoint(...)` as a decorator with a `name=` (queue-based mode)
- Sets `gpu=` to a `GpuGroup` member and `workers` to 5
- Lists `torch` in `dependencies=[...]`
- The `import torch` statement is INSIDE the function body, not at module top level
- The function is `async def` and is invoked with `await`
- Does NOT use the deprecated `@remote` decorator
- Does NOT set both `gpu=` and `cpu=`

# Flash — Endpoint API & compute-type reference

## Endpoint Constructor

```python
Endpoint(
    name="endpoint-name",                  # required (unless id= set)
    id=None,                               # connect to existing endpoint
    gpu=GpuGroup.AMPERE_80,               # GpuGroup tier, GpuType model, or list of either (default: GpuGroup.ANY)
    cpu=CpuInstanceType.CPU5C_4_8,        # CPU type (mutually exclusive with gpu)
    workers=5,                             # shorthand for (0, 5)
    workers=(1, 5),                        # explicit (min, max)
    max_concurrency=1,                     # concurrent requests per worker (default 1)
    idle_timeout=60,                       # seconds before scale-down (default: 60)
    dependencies=["torch"],                # pip packages for remote exec
    system_dependencies=["ffmpeg"],        # apt-get packages
    image="org/image:tag",                 # pre-built Docker image (client mode)
    env={"KEY": "val"},                    # environment variables
    volume=NetworkVolume(...),             # persistent storage
    datacenter=DataCenter.US_CA_2,         # DataCenter | list | str (default: None)
    gpu_count=1,                           # GPUs per worker
    template=PodTemplate(containerDiskInGb=100),
    flashboot=True,                        # fast cold starts
    accelerate_downloads=True,             # speed up model/file downloads (default True)
    min_cuda_version=CudaVersion.V12_8,    # minimum CUDA version (default 12.8)
    scaler_type=ServerlessScalerType.QUEUE_DELAY,  # default unset; or REQUEST_COUNT
    scaler_value=4,                        # scaler threshold (default 4)
    execution_timeout_ms=0,                # max execution time (0 = unlimited)
)
```

- `gpu=` and `cpu=` are mutually exclusive
- `gpu=` accepts a `GpuGroup`, a `GpuType`, or a list of either (see GPU Types below)
- `workers=5` means `(0, 5)`. Default is `(0, 1)`
- `max_concurrency` -- requests handled concurrently per worker (default 1). Raise it for I/O-bound LB routes so one worker serves multiple requests
- `idle_timeout` default is **60 seconds**
- `flashboot=True` (default) -- enables fast cold starts via snapshot restore
- `gpu_count` -- GPUs per worker (default 1), use >1 for multi-GPU models
- `datacenter` -- a `DataCenter` enum, list, or string; defaults to `None` (unset)
- `scaler_type` -- defaults to `QUEUE_DELAY` for queue-based endpoints and `REQUEST_COUNT` for load-balanced endpoints; pass `ServerlessScalerType.QUEUE_DELAY` or `REQUEST_COUNT` to override
- `DataCenter`, `CudaVersion`, and `ServerlessScalerType` are importable from `runpod_flash`

> **These defaults are flash-SDK defaults, and differ from the Runpod platform defaults**
> (Console / `runpodctl` / API): `idle_timeout` 60s here vs **5s** on the platform,
> `workers` (0, 1) here vs max **3** on the platform, `execution_timeout` unlimited here vs
> **600s** on the platform. When you read a default, note which layer it belongs to.

### NetworkVolume

```python
NetworkVolume(name="my-vol", size=100)  # size in GB, default 100
```

### PodTemplate

```python
PodTemplate(
    containerDiskInGb=64,    # container disk size (default 64)
    dockerArgs="",           # extra docker arguments
    ports="",                # exposed ports
    startScript="",          # script to run on start
)
```

## EndpointJob

Returned by `ep.run()` and `ep.runsync()` in client mode.

```python
job = await ep.run({"data": [1, 2, 3]})
await job.wait(timeout=120)        # poll until done
print(job.id, job.output, job.error, job.done)
await job.cancel()
```

## GPU Types

`gpu=` accepts a `GpuGroup` (a supply pool by VRAM tier), a `GpuType` (a pinned GPU model), or a list of either. `GpuGroup` picks the cheapest available GPU within a tier; `GpuType` pins a specific model.

### GpuGroup (supply pool)

| Enum | GPU | VRAM |
|------|-----|------|
| `ANY` | any | varies |
| `AMPERE_16` | RTX A4000 / A4500 / RTX 4000 Ada / RTX 2000 Ada | 16GB |
| `AMPERE_24` | RTX A5000 / L4 / RTX 3090 | 24GB |
| `AMPERE_48` | A40 / RTX A6000 | 48GB |
| `AMPERE_80` | A100 (PCIe / SXM4) | 80GB |
| `ADA_24` | RTX 4090 | 24GB |
| `ADA_32_PRO` | RTX 5090 | 32GB |
| `ADA_48_PRO` | RTX 6000 Ada / L40 / L40S | 48GB |
| `ADA_80_PRO` | H100 PCIe (80GB) / H100 HBM3 (80GB) / H100 NVL (94GB) | 80GB+ |
| `HOPPER_141` | H200 | 141GB |
| `BLACKWELL_96` | RTX PRO 6000 Blackwell | 96GB |
| `BLACKWELL_180` | B200 | 180GB |

### GpuType (pinned model)

Pin an exact GPU model. Members include `NVIDIA_GEFORCE_RTX_4090`, `NVIDIA_GEFORCE_RTX_5090`, `NVIDIA_RTX_6000_ADA_GENERATION`, `NVIDIA_H100_80GB_HBM3`, `NVIDIA_A100_80GB_PCIe`, `NVIDIA_A100_SXM4_80GB`, `NVIDIA_H200`, `NVIDIA_B200`, the `NVIDIA_RTX_PRO_6000_BLACKWELL_*` editions (Server / Workstation / Max-Q), and the Ampere/Ada RTX A-series models (`NVIDIA_RTX_A4000`, `A4500`, `A5000`, `A6000`, `NVIDIA_L4`, `NVIDIA_A40`, `NVIDIA_GEFORCE_RTX_3090`, `NVIDIA_RTX_4000_ADA_GENERATION`, `NVIDIA_RTX_2000_ADA_GENERATION`).

```python
from runpod_flash import Endpoint, GpuType

@Endpoint(name="pinned", gpu=GpuType.NVIDIA_GEFORCE_RTX_4090, dependencies=["torch"])
async def report_gpu(data):
    import torch
    return {"gpu": torch.cuda.get_device_name(0)}
```

## CPU Types (CpuInstanceType)

| Enum | vCPU | RAM | Max Disk | Type |
|------|------|-----|----------|------|
| `CPU3G_1_4` | 1 | 4GB | 10GB | General |
| `CPU3G_2_8` | 2 | 8GB | 20GB | General |
| `CPU3G_4_16` | 4 | 16GB | 40GB | General |
| `CPU3G_8_32` | 8 | 32GB | 80GB | General |
| `CPU3C_1_2` | 1 | 2GB | 10GB | Compute |
| `CPU3C_2_4` | 2 | 4GB | 20GB | Compute |
| `CPU3C_4_8` | 4 | 8GB | 40GB | Compute |
| `CPU3C_8_16` | 8 | 16GB | 80GB | Compute |
| `CPU5C_1_2` | 1 | 2GB | 15GB | Compute (5th gen) |
| `CPU5C_2_4` | 2 | 4GB | 30GB | Compute (5th gen) |
| `CPU5C_4_8` | 4 | 8GB | 60GB | Compute (5th gen) |
| `CPU5C_8_16` | 8 | 16GB | 120GB | Compute (5th gen) |

```python
from runpod_flash import Endpoint, CpuInstanceType

@Endpoint(name="cpu-work", cpu=CpuInstanceType.CPU5C_4_8, workers=5, dependencies=["pandas"])
async def process(data):
    import pandas as pd
    return pd.DataFrame(data).describe().to_dict()
```

# Flash — Common Patterns

## Choosing a model

Flash has **no model catalog** — name a HuggingFace repo id in code and it downloads to the
worker at runtime (see *Loading ML models* below). Other sources: a custom image's `MODEL_NAME` env
(vLLM etc.), a URL, or your own weights on a NetworkVolume.

- **Start with the smallest model that proves the pipeline** (`gpt2`, `stabilityai/sd-turbo`,
  a 0.5–1B variant) — it provisions in seconds, so you validate the `@Endpoint` wiring, deps,
  GPU, and I/O fast under `flash dev`, then change *only the id string* to the real model.
- **Match the model to GPU VRAM** (fp16 ≈ params × 2 bytes + overhead):

  | Model (fp16) | ~VRAM | `gpu=` |
  |---|---|---|
  | ≤3B / SD1.5 / sd-turbo | ≤8 GB | `GpuGroup.AMPERE_16` or `GpuGroup.ADA_24` |
  | 7–8B | ~16 GB | `GpuGroup.ADA_24` or `GpuGroup.AMPERE_24` |
  | 13B | ~28 GB | `GpuGroup.ADA_32_PRO` or `GpuGroup.AMPERE_48` |
  | 70B | ~140 GB | `GpuGroup.HOPPER_141` / `GpuGroup.BLACKWELL_180` (or quantize) |

- A ready-made hosted model with **no code** is [Runpod Public Endpoints / Hub](https://docs.runpod.io/hub) — a different product, not Flash.

> **Big model? Naming it still works — but mind the re-download.** Model size is not a
> flash limit: naming a large HF repo streams the weights to the worker at runtime. The
> catch is that a scaled-to-zero worker **re-downloads on every cold start**. For a large
> model you call often, cache it so it isn't re-pulled each time — persist to a
> **NetworkVolume** (see *Loading ML models* below), or on the runpodctl/serverless side use
> the **HF model cache** (`--model-reference`) or bake it into the image. This is the same
> tradeoff as the delivery-methods table in
> [`runpodctl/reference/model-caching.md`](../../runpodctl/reference/model-caching.md): easy
> streaming vs. faster/cheaper cold starts for reused weights — not a contradiction.

## Loading ML models (warm workers)

Model **weights are not part of the 1.5GB build artifact** — that cap is your code + pip
deps (torch is auto-excluded). Weights download on the worker at runtime (HuggingFace,
etc.), so **model size is not a Flash limit**. Two things make this fast and cheap:

- **Load once per worker, not per request** — use a *class* `@Endpoint`: `__init__` loads
  the model into VRAM once when the worker starts; methods handle requests and reuse it.
- **Persist the cache on a NetworkVolume** so a cold worker reuses downloaded weights
  instead of re-pulling them every cold start.

```python
from runpod_flash import Endpoint, GpuType, DataCenter, NetworkVolume

vol = NetworkVolume(name="model-cache", size=100, datacenter=DataCenter.US_GA_2)

@Endpoint(
    name="sd",
    gpu=GpuType.NVIDIA_GEFORCE_RTX_5090,
    workers=(0, 3),
    idle_timeout=300,                                # keep workers warm between calls
    datacenter=DataCenter.US_GA_2,
    volume=vol,
    env={"HF_HUB_CACHE": "/runpod-volume/models"},   # cache weights on the volume
    dependencies=["torch", "diffusers", "transformers", "accelerate"],
)
class SD:
    def __init__(self):                              # runs ONCE per worker
        import torch
        from diffusers import StableDiffusionPipeline
        self.pipe = StableDiffusionPipeline.from_pretrained(
            "runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16
        ).to("cuda")

    async def generate(self, prompt: str) -> dict:   # per request, reuses self.pipe
        image = self.pipe(prompt=prompt).images[0]
        image.save("/runpod-volume/out.png")         # /runpod-volume/ persists; elsewhere is wiped
        return {"saved": "/runpod-volume/out.png"}
```

- **Gated** models: pass `env={"HF_TOKEN": "..."}`.
- `workers=(1, n)` keeps one worker warm (no cold start on the first request); `(0, n)` scales to zero and cold-starts after `idle_timeout`.
- The class form is the cleanest way to load once. In function-form `@Endpoint` the same effect needs the module-global cache trick (see Gotcha #11 in the skill); the class form is preferred for real inference.

## CPU + GPU Pipeline

```python
from runpod_flash import Endpoint, GpuGroup, CpuInstanceType

@Endpoint(name="preprocess", cpu=CpuInstanceType.CPU5C_4_8, workers=5, dependencies=["pandas"])
async def preprocess(raw):
    import pandas as pd
    return pd.DataFrame(raw).to_dict("records")

@Endpoint(name="infer", gpu=GpuGroup.AMPERE_80, workers=5, dependencies=["torch"])
async def infer(clean):
    import torch
    t = torch.tensor([[v for v in r.values()] for r in clean], device="cuda")
    return {"predictions": t.mean(dim=1).tolist()}

async def pipeline(data):
    return await infer(await preprocess(data))
```

## Parallel Execution

```python
import asyncio
results = await asyncio.gather(compute(a), compute(b), compute(c))
```

# Flash — Setup & CLI reference

## Setup

```bash
# install the CLI — requires Python 3.10-3.13 (NOT 3.14+ yet)
uv tool install runpod-flash
pip install runpod-flash
# on Python 3.14+ the install fails — pin an older interpreter for the tool:
uv tool install --python 3.13 runpod-flash

# auth option 1: browser-based login (saves token locally)
flash login
# headless: print URL instead of opening a browser
flash login --no-open
# max seconds to wait for browser auth (default 600)
flash login --timeout 300

# auth option 2: API key via environment variable
export RUNPOD_API_KEY=your_key

# scaffold a new project in ./my-project (writes AGENTS.md + CLAUDE.md)
flash init my-project
# scaffold in the current directory
flash init .
# overwrite existing files (-f)
flash init my-project --force
# update the CLI to the latest version
flash update
# pin a specific version (-V also works)
flash update --version 1.16.0
```

`flash init` writes `AGENTS.md` (+ a `CLAUDE.md` symlink). To add them to an existing project: `python -c "from runpod_flash.rules import install_agent_files; from pathlib import Path; install_agent_files(Path.cwd())"`.

**Auth precedence:** a set `RUNPOD_API_KEY` env var **overrides** the saved `flash login` token, so an exported bad/expired key silently beats a good login — a common trap (see Gotcha #13 in the skill).

## CLI

`flash dev` is the canonical dev-server command (`flash run` still works as a hidden alias).

```bash
# local server at :8888, but functions run on REMOTE GPU/CPU workers;
# hot-reloads on save and streams the worker's logs live to your terminal
flash dev
# same, but pre-provision endpoints (no cold start on first call)
flash dev --auto-provision
# custom port/host; --reload/--no-reload toggles autoreload
flash dev --port 9000 --host 0.0.0.0
# build + deploy (auto-selects env if only one)
flash deploy
# build + deploy to "staging" environment
flash deploy --env staging
# deploy a specific app to an environment
flash deploy --app my-app --env prod
# build + launch local preview in Docker
flash deploy --preview
# build flags below also apply to deploy
flash deploy --no-deps --python-version 3.11
# list deployment environments
flash env list
# create "staging" environment
flash env create staging
# show environment details + resources
flash env get staging
# delete environment + tear down resources
flash env delete staging
# list flash apps in your account
flash app list
# create a flash app
flash app create my-app
# show an app's environments + builds
flash app get my-app
# delete an app and all its resources
flash app delete my-app
# list all active endpoints
flash undeploy list
# remove a specific endpoint
flash undeploy my-endpoint
# remove all endpoints (--interactive/-i to pick, --force/-f to skip prompts)
flash undeploy --all
# remove endpoints whose code no longer exists locally
flash undeploy --cleanup-stale

# build-only (no deploy) — mainly for debugging the artifact; `flash deploy` builds for you
# package the artifact without deploying (1500MB limit; torch auto-excluded)
flash build
# build flags: --no-deps, --exclude pkg1,pkg2, --output name.tar.gz, --python-version 3.11
flash build --no-deps
```

---
name: flash
description: >-
  runpod-flash — code-first serverless: write Python locally, run it on remote
  Runpod GPUs/CPUs with `flash dev` (hot-reload + live worker logs), then
  `flash deploy`. Use for @Endpoint/@remote functions, resource config, and
  debugging flash deployments. For CLI-only infra management use runpodctl or
  runpod-mcp.
user-invocable: true
metadata:
  author: runpod
  version: "1.2.0" # x-release-please-version
license: Apache-2.0
---

# Runpod Flash

Write code locally, iterate with `flash dev` — it runs your functions on remote Runpod GPUs/CPUs with hot-reload and live worker logs — then `flash deploy` to ship. `Endpoint` handles provisioning.

`runpod-flash` releases on its own cadence, so **`flash --help` and `flash <command> --help`
are authoritative for the command surface** — this skill is the mental model, the decision
rules, and the gotchas that help output does not carry. Confirm the installed version with
`pip show runpod-flash` before concluding a subcommand or flag is unavailable.

**Worked examples first for anything multi-step.** Flash appears in verified end-to-end
paths — [03 variant B (whisper endpoint via flash)](../runpod/golden-paths/03-whisper-endpoint/variant-b-flash.md)
and [08 (fine-tune → serve)](../runpod/golden-paths/08-finetune-to-serverless.md); the full
index is [runpod/golden-paths/README.md](../runpod/golden-paths/README.md). Open the matching
path before planning a deploy — it carries the ordering and the cost cleanup this skill only
summarizes.

**Load on demand — this skill keeps the mental model + gotchas inline; details live in [`reference/`](reference/):**

| Need | Read |
|------|------|
| Install, auth, `flash init`, and the full `flash` command list | [reference/setup-and-cli.md](reference/setup-and-cli.md) |
| `Endpoint(...)` constructor params, `NetworkVolume`/`PodTemplate`/`EndpointJob`, GPU & CPU enum tables | [reference/api.md](reference/api.md) |
| Worked patterns — choosing a model, warm-worker model loading, CPU→GPU pipeline, parallel calls | [reference/patterns.md](reference/patterns.md) |

Quick start: `uv tool install runpod-flash` → `flash login` (or `export RUNPOD_API_KEY=...`) → `flash init my-project` → `flash dev`. Details in [reference/setup-and-cli.md](reference/setup-and-cli.md).

## Dev vs Deploy

- `flash dev` — **iterate.** Local server at `:8888`, but your decorated functions
  execute on **remote GPU/CPU workers**. Hot-reloads on save and **streams the worker's
  logs live** to the terminal. No build/upload/deploy wait — use this the whole time you
  develop.
- `flash deploy` — **ship.** Builds an artifact and deploys a stable endpoint. Slow
  (build + upload + provision); only do this once the code works under `flash dev`.

`flash dev` ships **only the function body** to the worker, so a `NameError` for a
module-level name surfaces immediately here. `flash deploy` imports the whole module and
can mask that bug (see Gotcha #1). Develop against `flash dev` and you catch it first.

## Autonomous Dev Loop

`flash dev` is a long-running server. Three rules:
- **Run it in the background** — don't block on it.
- **Capture its output** to a log file.
- **Drive it over HTTP.**

The captured log is the remote worker's live stream (cold start, model load, `print`s,
tracebacks) — read it to debug.

```bash
flash dev > /tmp/flash-dev.log 2>&1 &                          # background; never run it blocking
for i in $(seq 1 60); do grep -q "flash dev  localhost:" /tmp/flash-dev.log && break; sleep 2; done  # bounded ~2min; if it never appears, check the log for errors
URL=$(grep -o "localhost:[0-9]*" /tmp/flash-dev.log | head -1)               # actual port (8888 bumps if taken)
curl -s "$URL/main/predict" -d '{"data": {...}}'               # dispatches to the remote worker
```

- **Read the real URL from the log** — flash auto-bumps the port if 8888 is in use, and
  prints `✓ flash dev  localhost:<port>` plus the route table.
- **Routes are namespaced by file**: `main.py`'s `/predict` is served at `/main/predict`.
- **Two route shapes, two body shapes** (mismatch → `422` naming the missing field in `loc`):
  - **Load-balanced** (`@api.post("/predict")`) → `POST /main/predict`, body is the arg
    at top level: a handler `def predict(data: dict)` wants `{"data": {...}}` (not the bare object).
  - **Queue-based** (bare `@Endpoint` decorator) → `POST /main/runsync` (the local dev
    server only generates `/runsync`; production also exposes `/run`),
    body is **double-wrapped** in `input`: a handler `def synthesize(data: dict)` wants
    `{"input": {"data": {...}}}`. The outer `input` is the queue envelope; the inner key is
    the handler's param name.
- Edit a handler and save — hot-reload re-syncs the body; just re-send the request, no
  redeploy. Add `--auto-provision` to skip the first-call cold start. `kill %1` when done.

## Endpoint: Three Modes

Full constructor params and the GPU/CPU enum tables are in [reference/api.md](reference/api.md).

### Mode 1: Your Code (Queue-Based Decorator)

One function = one endpoint with its own workers.

```python
from runpod_flash import Endpoint, GpuGroup

@Endpoint(name="my-worker", gpu=GpuGroup.AMPERE_80, workers=5, dependencies=["torch"])
async def compute(data):
    import torch  # MUST import inside function (cloudpickle)
    return {"sum": torch.tensor(data, device="cuda").sum().item()}

result = await compute([1, 2, 3])
```

### Mode 2: Your Code (Load-Balanced Routes)

Multiple HTTP routes share one pool of workers.

```python
from runpod_flash import Endpoint, GpuGroup

api = Endpoint(name="my-api", gpu=GpuGroup.ADA_24, workers=(1, 5), dependencies=["torch"])

@api.post("/predict")
async def predict(data: list[float]):
    import torch
    return {"result": torch.tensor(data, device="cuda").sum().item()}

@api.get("/health")
async def health():
    return {"status": "ok"}
```

### Mode 3: External Image (Client)

Deploy a pre-built Docker image and call it via HTTP.

```python
from runpod_flash import Endpoint, GpuGroup, PodTemplate

server = Endpoint(
    name="my-server",
    image="my-org/my-image:latest",
    gpu=GpuGroup.AMPERE_80,
    workers=1,
    env={"HF_TOKEN": "xxx"},
    template=PodTemplate(containerDiskInGb=100),
)

# LB-style
result = await server.post("/v1/completions", {"prompt": "hello"})
models = await server.get("/v1/models")

# QB-style
job = await server.run({"prompt": "hello"})        # optional: webhook="https://..." for completion callback
await job.wait()
print(job.output)
```

Connect to an existing endpoint by ID (no provisioning):

```python
ep = Endpoint(id="abc123")
job = await ep.runsync({"prompt": "hello"})  # runsync wraps this as {"input": {"prompt": "hello"}}
print(job.output)
```

## How Mode Is Determined

| Parameters | Mode |
|-----------|------|
| `name=` only | Decorator (your code) |
| `image=` set | Client (deploys image, then HTTP calls) |
| `id=` set | Client (connects to existing, no provisioning) |

The table above is *how* the mode is picked from params. *When* to reach for `image=`:

### When to use `image=` (custom container) vs your own code

Default to writing Python (decorator / routes) — it runs arbitrary code with
`dependencies=[...]`/`system_dependencies=[...]` and needs no Dockerfile. Even large
HuggingFace models stay in decorator mode (weights stream at runtime — see
[reference/patterns.md → Loading ML models](reference/patterns.md#loading-ml-models-warm-workers)).
Reach for `image=` **only** when you need:

- **a pre-built inference server** — vLLM, TensorRT-LLM (`image="vllm/vllm-openai:latest"`, or `runpod/worker-vllm`, `runpod/worker-comfy`)
- **system-level deps not pip-installable** — a specific CUDA/cuDNN, OS libraries
- **models baked into the image** — to skip the runtime download entirely
- **an existing Runpod Serverless worker** — you already have a working image

Trade-off: `image=` mode **can't run arbitrary Python** (the image owns all logic) and the
image must implement a Runpod Serverless handler. Full list + examples:
https://docs.runpod.io/flash/custom-docker-images

## Gotchas

1. **Only the function body ships to the worker** -- most common error. Put imports *and* any module-level constants/helpers the function uses *inside* the decorated body. `flash deploy` imports the whole module so module globals happen to work; `flash dev` ships just the body, so a module-level name raises `NameError`. A handler that works deployed can break under dev — fix it by moving everything inside.
2. **Forgetting await** -- all decorated functions and client methods need `await`.
3. **Missing dependencies** -- must list in `dependencies=[]`.
4. **gpu/cpu are exclusive** -- pick one per Endpoint.
5. **idle_timeout is seconds** -- default 60s, not minutes.
6. **10MB payload limit** -- pass URLs, not large objects. Return binary (audio/images/files) as base64 in the JSON (`{"audio_b64": ...}`) and decode client-side; for larger outputs write to a NetworkVolume or upload to storage and return a URL.
7. **Client vs decorator** -- `image=`/`id=` = client. Otherwise = decorator.
8. **Auto GPU switching requires workers >= 5** -- pass a list of GPU types (e.g. `gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_80]`) and set `workers=5` or higher. The platform only auto-switches GPU types based on supply when max workers is at least 5.
9. **`runsync` timeout is 60s** -- cold starts can exceed 60s. Use `ep.runsync(data, timeout=120)` for first requests or use `ep.run()` + `job.wait()` instead.
10. **Request body shape (raw/external HTTP callers only)** -- match the request shape to the endpoint type:
    - **LB routes** (`@api.post(...)`): send the handler arg at the top level — `{"data": {...}}`.
    - **QB endpoints** (bare `@Endpoint`, hit via `.../run` or `.../runsync`): the worker calls
      **`handler(**job_input)`**, so the request's `input` keys must match the handler's parameter
      names — `def transcribe(input_data: dict)` wants `{"input": {"input_data": {...}}}`, and
      `def read(input: dict)` wants `{"input": {"input": {...}}}`. A mismatch fails with
      `got an unexpected keyword argument …`. Use `**kwargs` if the handler ignores the payload.
    - **Never send an empty `input`.** A QB request with `{"input": {}}` is rejected by the
      worker SDK as `Job has missing field(s): id or input` — always include at least one key.
    - *Context:* the flash client (`ep.runsync(x)`, `api.post(...)`) hides the spreading, so this
      only bites raw HTTP/external callers (mismatch behavior verified 2026-07-10 via worker logs).
      See *Autonomous Dev Loop*.
11. **Load a model once per worker (not per call)** -- for real inference use a class `@Endpoint` whose `__init__` loads the model once per worker (see [reference/patterns.md → Loading ML models](reference/patterns.md#loading-ml-models-warm-workers)). In function-form, reconcile with #1 by caching in a module global *inside* the body so it works under both `flash dev` and `deploy`:
    ```python
    global _MODEL
    try: _MODEL
    except NameError: _MODEL = load_model()   # runs once per worker, reused across calls
    ```
12. **Native CUDA libs go in `dependencies=[]` too** -- e.g. CTranslate2/faster-whisper needs `nvidia-cublas-cu12` + `nvidia-cudnn-cu12` or it silently falls back to CPU. Add them alongside the Python package.
13. **Silent 401 auth failure** -- a set `RUNPOD_API_KEY` env var overrides the `flash login` token, so a bad/expired key wins. The failure is quiet: provisioning logs `GraphQL request failed: 401`, but `flash dev` still prints its normal ready line ("failed endpoints deploy on-demand"), so it *looks* healthy. When endpoints fail to provision:
    1. Check the provisioning log for `GraphQL request failed: 401`.
    2. Verify the current key independently: `curl -s -o /dev/null -w '%{http_code}' https://rest.runpod.io/v1/endpoints -H "Authorization: Bearer $RUNPOD_API_KEY"` (200 = good, 401 = bad).
    3. Fix it: `unset RUNPOD_API_KEY` to fall back to the `flash login` token, or `export` a valid key.
14. **`system_dependencies=` adds to cold start** -- apt packages (e.g. `["ffmpeg", "espeak-ng"]`) install on the worker before first use, so the initial call is slower (on top of any model download); warm calls are unaffected.
15. **Teardown a deployed app with `flash app delete <app>`** -- `flash undeploy list` may show "no endpoints" for an app that is deployed and serving; `flash app delete` (or `runpodctl serverless delete <id>`) reliably removes it.

## Resources

- Setup & CLI: [reference/setup-and-cli.md](reference/setup-and-cli.md) · API & compute enums: [reference/api.md](reference/api.md) · Patterns: [reference/patterns.md](reference/patterns.md)
- Flash source: https://github.com/runpod/flash
- Runnable examples: https://github.com/runpod/flash-examples — clone and adapt the closest one
- Package (PyPI): https://pypi.org/project/runpod-flash/
- Docs: https://docs.runpod.io/flash/overview
  - Custom Docker images (when + how): https://docs.runpod.io/flash/custom-docker-images
  - Storage / network volumes: https://docs.runpod.io/flash/configuration/storage