cargo-diagnostics

# Batch error sweep — group failures by root cause

Use this when many runs are involved and you don't yet know where to look: a batch reports errors, a play's error rate spiked, or "some records didn't come through". The output of a sweep is a **small table of failure groups with an exemplar run UUID each** — not a list of every failed run.

> SQL syntax, table columns, and query caps: [`../../cargo-orchestration/references/examples/queries.md`](../../cargo-orchestration/references/examples/queries.md). All queries below are read-only and workspace-scoped automatically.

## 1. Size the problem

```bash
# For one batch
cargo-ai orchestration query execute \
  "SELECT status, count() FROM runs WHERE batch_uuid = '<batch-uuid>' GROUP BY status"

# For a play/workflow over time
cargo-ai orchestration query execute \
  "SELECT countIf(status='error') / count() AS error_rate, count() AS total
   FROM runs
   WHERE workflow_uuid = '<workflow-uuid>' AND created_at > now() - INTERVAL 7 DAY"
```

Calibration: error rates under ~5% on connector-heavy workflows are often provider coverage, not defects (see the over-provision rule in [`cost-discipline.md`](../../cargo-gtm/references/cost-discipline.md)). A spike above that, or errors on native nodes, is worth the sweep.

## 2. Find where failures concentrate

```bash
# Which node fails most
cargo-ai orchestration query execute \
  "SELECT node_slug, count() AS failures
   FROM spans
   WHERE execution_status='error' AND execution_started_at > now() - INTERVAL 1 DAY
   GROUP BY node_slug
   ORDER BY failures DESC"
```

Scope with `batch_uuid`/`workflow_uuid` predicates when you have them. Two shapes to distinguish:

- **Concentrated** (one node owns most failures) → a config, credential, or expression defect at that node. Proceed to step 3 with that node.
- **Spread across connector nodes, clustered in time** → third-party rate limiting. Confirm with the "signs you are being rate-limited" checklist in [`troubleshooting.md`](../../cargo-orchestration/references/troubleshooting.md); the fix is retry config + smaller sub-batches, not per-run debugging.

## 3. Pick exemplars and read the actual errors

```bash
cargo-ai orchestration query execute \
  "SELECT uuid, created_at
   FROM runs
   WHERE batch_uuid = '<batch-uuid>' AND status='error'
   ORDER BY created_at ASC
   LIMIT 3"
```

Take 2–3 exemplars per failure group and trace each with [`run-trace.md`](run-trace.md) — the error detail lives in `run get`'s `runContext`, not in the SQL tables. Failures with the same node + same error pattern are one group; resist tracing every run.

## 4. Decide: fix, re-run, or report

| Root cause shape | Action |
| --- | --- |
| Expression/branch defect (same wrong output every time) | Fix the node, re-test on exemplar record IDs, then re-run only the failed records: `run download --statuses error` → fix → `batch create --data '{"kind":"recordIds",...}'` (sequence in [`troubleshooting.md`](../../cargo-orchestration/references/troubleshooting.md), "Run error recovery") |
| Expired connector credentials | Re-authenticate the connector (`connection connector update` or the Cargo app), then re-run failed records |
| Provider rate limiting | Retry config + sequential sub-batches; don't chase individual rows |
| Provider coverage (no email exists, company not found) | Expected loss — drop the rows per the over-provision rule; do **not** re-run them through more providers |
| CLI/platform behavior contradicts the docs | File it — this is exactly what the report channel is for: `cargo-ai workspaceManagement report create --title "..." --description "<commands, errorMessage verbatim, expected vs actual, UUIDs>"` |

Any re-run of paid nodes is a paid action: pilot gate + receipt per [`cost-discipline.md`](../../cargo-gtm/references/cost-discipline.md).

## Presenting a sweep

Per [`../../cargo/references/interaction.md`](../../cargo/references/interaction.md): conclusion first ("one root cause explains 18 of 20 failures"), then the groups table, then the recommended action per group. Example shape:

```
Batch <uuid>: 20 of 250 runs errored. Two groups:

| group | runs | node        | root cause                            | action                    |
|-------|------|-------------|----------------------------------------|---------------------------|
| 1     | 18   | find_email  | connector token expired (401 verbatim) | re-auth, re-run 18 records |
| 2     | 2    | enrich_co   | provider has no data for these domains | drop rows (coverage)      |

Re-running group 1 ≈ <n> credits. Proceed?
```

# Play cost profile — where credits go, and how to cut them

Use this when a play costs more than expected or the user asks to reduce spend. The procedure is: attribute (which workflow → which node → which provider), then apply levers in priority order. Never propose a lever before the attribution — "use a cheaper model" is noise if 90% of the spend is a phone-lookup connector.

Credit attribution needs an **admin** token (`billing` commands); the SQL steps work with any token.

## 1. Attribute spend to workflows

```bash
# Credit spend by workflow this month (SQL — fast, no admin needed)
cargo-ai orchestration query execute \
  "SELECT workflow_uuid, sum(credits_used_count) AS credits
   FROM batches
   WHERE created_at >= toStartOfMonth(now())
   GROUP BY workflow_uuid
   ORDER BY credits DESC"

# Billing source of truth, groupable by other dimensions too
cargo-ai billing usage get-metrics --from <YYYY-MM-DD> --to <YYYY-MM-DD> --group-by workflow_uuid
cargo-ai billing usage get-metrics --from <YYYY-MM-DD> --to <YYYY-MM-DD> --group-by integration_slug
```

Map UUIDs to names with `cargo-ai orchestration play list` / `tool list`. When SQL and billing disagree, billing wins.

## 2. Attribute spend to nodes inside the top workflow

Per-node cost lives on the run detail: each `run.executions[]` item carries `creditsUsedCount` (agent and connector nodes are non-zero, native nodes are zero — see [`troubleshooting.md`](../../cargo-orchestration/references/troubleshooting.md)). Pull 2–3 recent representative runs and average:

```bash
cargo-ai orchestration query execute \
  "SELECT uuid FROM runs
   WHERE workflow_uuid = '<workflow-uuid>' AND status = 'success'
   ORDER BY created_at DESC LIMIT 3"

cargo-ai orchestration run get <run-uuid>   # read executions[].creditsUsedCount per nodeSlug
```

Also check **waste**: credits spent on runs that errored anyway —

```bash
cargo-ai orchestration query execute \
  "SELECT status, sum(credits_used_count) AS credits, count() AS runs
   FROM runs
   WHERE workflow_uuid = '<workflow-uuid>' AND created_at > now() - INTERVAL 30 DAY
   GROUP BY status"
```

A meaningful `error`-row credit sum means expensive nodes run **before** the failure point — reordering is a free win.

## 3. Apply levers, cheapest-to-implement first

Work down this list; the first two usually dominate. The canonical lever table is in [`../../cargo-billing/SKILL.md`](../../cargo-billing/SKILL.md) ("Cost levers"); provider prices are in [`../../cargo-gtm/references/credits-cost-table.md`](../../cargo-gtm/references/credits-cost-table.md).

| Lever | When it applies | Where documented |
| --- | --- | --- |
| **Filter earlier** — move `filter` nodes before expensive connector/agent nodes so ineligible records never reach them | Waste query (step 2) shows credits on errored/filtered-late runs | [`cargo-billing/SKILL.md`](../../cargo-billing/SKILL.md) |
| **Cheaper provider for the same stage** — swap the action, keep the graph | One integration dominates the `integration_slug` grouping | [`credits-cost-table.md`](../../cargo-gtm/references/credits-cost-table.md) + [`alternatives.md`](../../cargo-gtm/references/alternatives.md) — beware cheap-but-low-hit-rate providers; total spend is dominated by misses |
| **Cheaper model / lower `maxSteps` on agent nodes** | Agent nodes dominate per-node cost | [`cargo-billing/SKILL.md`](../../cargo-billing/SKILL.md) |
| **Stop early on failure** (`fallbackOnFailure: false`) | Downstream nodes run after an upstream miss | [`cargo-billing/SKILL.md`](../../cargo-billing/SKILL.md) |
| **Reshape waterfall chains** — reorder by hit-rate/price, add stop-early rules | Multi-provider enrichment stages | [`waterfall-strategy.md`](../../cargo-gtm/references/waterfall-strategy.md) |
| **Cut phone lookup from default chains** | Phone actions present without explicit user request (3–7 credits/record, ~10× email) | [`cost-discipline.md`](../../cargo-gtm/references/cost-discipline.md) §5 |

## 4. Prove the saving

Changing the graph is a workflow edit + re-run: stage via draft release, pilot 1–3 records, present the before/after per-record cost, and only then fan out — the full gate is [`cost-discipline.md`](../../cargo-gtm/references/cost-discipline.md) §1, and the receipt format is §2. A cost optimization that skips the pilot is just a different way to spend credits blind.

## Presenting a cost profile

Per [`../../cargo/references/interaction.md`](../../cargo/references/interaction.md): lead with the attribution and the projected saving, then the lever plan as shaped choices. Example shape:

```
"Enrich EMEA leads" spent 412 credits this month; 71% is the find_phone
connector node, which runs on every record before qualification.

| change                                   | est. per-record | est. monthly |
|------------------------------------------|-----------------|--------------|
| move qualify filter before find_phone    | 4.1 → 1.9       | −55%         |
| also: drop phone to on-request only      | 1.9 → 0.6       | −85%         |

Pilot either variant on 3 records (~2 credits) to confirm before deploying?
```

# Run trace — explain one run end-to-end

Use this when you have (or can find) a single run UUID and need to answer "what actually happened to this record?" — a hard failure, or the more common case: `status: "success"` with wrong or empty output.

> Field-by-field semantics for everything used here live in [`../../cargo-orchestration/references/troubleshooting.md`](../../cargo-orchestration/references/troubleshooting.md) ("Debugging a workflow run"). This runbook is the ordered procedure.

## 0. Find the run

Every step below needs a run UUID. Work down this ladder and stop at the first rung that matches what the user actually gave you — most of the time it's a symptom and a company name, not a UUID.

> **`run list` cannot answer "the last run".** `cargo-ai orchestration run list` **requires** `--workflow-uuid`; there is no unfiltered form, and a play's UUID is not a workflow UUID. Orchestration SQL has no such requirement, so it — not `run list` — is the entry point whenever you don't already know the workflow. Concluding "the run data isn't accessible" because `run list` refused is a wrong answer: `runs` is queryable with no filter at all.

**"Look at the last run" / "what just ran"** — no UUID, no workflow, nothing:

```bash
cargo-ai orchestration query execute \
  "SELECT uuid, workflow_uuid, record_title, status, created_at
   FROM runs
   ORDER BY created_at DESC
   LIMIT 10"
```

**A company, domain, or record the user names** ("the run for acme.com") — `record_title` carries the record's title, or for record-less runs the input payload, so a substring match finds it:

```bash
cargo-ai orchestration query execute \
  "SELECT uuid, workflow_uuid, record_title, status, created_at
   FROM runs
   WHERE record_title ILIKE '%acme.com%'
   ORDER BY created_at DESC
   LIMIT 10"
```

**A play or workflow by name** — resolve to a `workflowUuid` first, then filter. `runs` has no play column, so this hop is mandatory:

```bash
cargo-ai orchestration play list        # → find the play, take play.workflowUuid

cargo-ai orchestration query execute \
  "SELECT uuid, status, created_at, credits_used_count
   FROM runs
   WHERE workflow_uuid = '<play.workflowUuid>'
   ORDER BY created_at DESC
   LIMIT 20"
```

Play anatomy and the rest of the play surface: [`../../cargo-orchestration/references/examples/plays.md`](../../cargo-orchestration/references/examples/plays.md).

**Coming from a batch sweep** — you already have exemplar UUIDs; skip ahead.

### When the discovery query itself errors

| Error | Cause and fix |
| --- | --- |
| `Limit for number of columns to read exceeded. Requested: 51, maximum: 50.` | You ran `SELECT *`. `runs` is wider than the 50-column read cap — name the columns you need, as every query above does. |
| `Unknown expression identifier '<col>'` | That column doesn't exist. `runs` has no `play_uuid`, no `name`, and no trigger-source column; the ones used here (`uuid`, `workflow_uuid`, `release_uuid`, `batch_uuid`, `record_id`, `record_title`, `status`, `created_at`, `credits_used_count`) are confirmed present. |

Where a run was triggered from — the CLI, a scheduled play, or a click in the UI editor — is not supposed to change where it lands: all of them write to `runs` and are readable with `run get`. So a run the user can see in the UI but that none of these queries return is a real bug, not a boundary you should work around or explain away. Say so and file a report (skill § "When diagnosis dead-ends"), quoting the queries you ran.

## 1. Pull the trace

```bash
cargo-ai orchestration run get <run-uuid>
```

Read three fields, in this order:

1. **`run.executions[]`** — the node-by-node path. For each node: `nodeSlug`, `status`, `nextNodeUuid`, `nodeChildIndex`, `creditsUsedCount`. This tells you **where execution went**, including which child a `branch` took (`nodeChildIndex` `0` = matched/yes, `1` = not matched/no).
2. **`runContext`** — per-node output keyed by `nodeSlug`. This is the actual data downstream expressions saw as `{{nodes.<slug>...}}`. It is the source of truth; the `title` on an execution is a truncated summary, never evidence.
3. **`runComputedConfigs`** — what each node was *actually called with* after expression resolution. When a node received garbage, this shows the garbage.

Don't paste the raw response into the conversation — extract the two or three nodes that matter (see "Presenting" below).

## 2. Diagnose by symptom

| Symptom | Where to look | Typical conclusion |
| --- | --- | --- |
| Run `error` | First `executions[]` item with `status: "error"`; its `runContext.<slug>` entry carries the error detail | Failing node identified — match it against the error-pattern table in [`troubleshooting.md`](../../cargo-orchestration/references/troubleshooting.md) ("Run error recovery") |
| Run `success`, output empty | `runContext.<upstreamSlug>` of the node that produced the empty value | Expression path doesn't exist — commonly agent output nested under `.answer` (`{{nodes.qualify.answer.qualified}}`, not `{{nodes.qualify.qualified}}`) |
| Wrong branch taken | The branch node's `nodeChildIndex` + the `runContext` of the node its condition references | Condition resolved falsy because the referenced path is missing/undefined — verify the real shape in `runContext` |
| Connector node "worked" but downstream empty | `runContext.<connectorSlug>` | Partial provider response; the real field names differ from the ones referenced (e.g. `contact.email` vs `email`) |
| One node absurdly slow | Spans timing query below | Rate-limited or retrying connector; see the batch-sizing section of `troubleshooting.md` |

Per-node timing for the slow-node case:

```bash
cargo-ai orchestration query execute \
  "SELECT node_slug, execution_status,
          dateDiff('second', execution_started_at, execution_finished_at) AS duration_s
   FROM spans
   WHERE run_uuid = '<run-uuid>'
   ORDER BY duration_s DESC"
```

## 3. Confirm the fix on the same record

Stage → approve → deploy → re-run the exact record IDs that exposed the bug — the command sequence is in [`troubleshooting.md`](../../cargo-orchestration/references/troubleshooting.md) ("Re-run a single record after fixing"). Re-running paid nodes counts as a paid action: pilot gate + receipt per [`../../cargo-gtm/references/cost-discipline.md`](../../cargo-gtm/references/cost-discipline.md).

## Presenting a trace

Per [`../../cargo/references/interaction.md`](../../cargo/references/interaction.md): conclusion first, then a compact path table — one row per relevant node (`nodeSlug` → status → the one field that matters), then the recommended fix. Example shape:

```
The run "succeeded" but the branch took the no-path: the condition reads
{{nodes.qualify.qualified}}, but the agent's output is nested under .answer.

| node      | status  | evidence                                        |
|-----------|---------|--------------------------------------------------|
| qualify   | success | runContext.qualify.answer.qualified = true       |
| branch_1  | success | nodeChildIndex = 1 (no-path) — condition falsy   |

Fix: change the condition to {{nodes.qualify.answer.qualified}} and re-run
record <id> to confirm (1 record ≈ <n> credits).
```

**For a wrong-branch or wrong-path diagnosis, add the graph with the offending
node marked** — the user has to see the fork to agree the run went down the wrong
side of it:

```bash
cargo-ai orchestration node diagram --run-uuid <run-uuid> --highlight branch_1 --raw
```

Free, runs nothing, and it works for either run shape — an ad-hoc `action execute`
run carries its own `nodes`, a run of a deployed tool or play carries only a
`releaseUuid`, and `--run-uuid` follows whichever it has. Flags and mapping rules:
[`../../cargo-orchestration/references/node-diagram.md`](../../cargo-orchestration/references/node-diagram.md).

{
  "$comment": "Generated by .github/scripts/skills-metadata.mjs — do not hand-edit. Regenerate with: node .github/scripts/skills-metadata.mjs --write .",
  "name": "cargo-diagnostics",
  "version": "1.3.0",
  "documents": [
    {
      "path": "SKILL.md",
      "kind": "entrypoint",
      "title": "Cargo CLI — Diagnostics"
    },
    {
      "path": "references/batch-error-sweep.md",
      "kind": "reference",
      "title": "Batch error sweep — group failures by root cause"
    },
    {
      "path": "references/play-optimize-credits.md",
      "kind": "reference",
      "title": "Play cost profile — where credits go, and how to cut them"
    },
    {
      "path": "references/run-trace.md",
      "kind": "reference",
      "title": "Run trace — explain one run end-to-end"
    }
  ],
  "contentHash": "e1373282664ef5ac915ee5b6b42900446781e26c872100e3ddd5a0f11cc4b3d6"
}

---
name: cargo-diagnostics
description: "Explain what a Cargo run or batch actually did, after the fact — trace one run node by node, draw the graph it executed with the failing step marked, sweep a batch or play for errors grouped by root cause, and attribute credit spend down to the node and the provider. Triggers: \"why did this fail\", \"it succeeded but the output is wrong\", \"half my rows are empty\", \"why is this column blank\", \"what broke in this batch\", \"why did that cost so much\", \"which node is burning credits\", \"it worked yesterday\", \"these results look wrong\", \"it went down the wrong path\", \"this step never ran\", \"show me what the run did\". Skip when: setting up an alert for next time — use cargo-observability; just downloading the data — use cargo-analytics."
version: "1.3.0"
compatibility: Requires @cargo-ai/cli (npm). Sign in or create an account with `cargo-ai login --email` (emailed code, no browser), `--oauth`, or an API token
homepage: https://github.com/getcargohq/cargo-skills
metadata:
  author: getcargo
  openclaw:
    requires:
      bins:
        - cargo-ai
    install:
      - kind: node
        package: "@cargo-ai/cli@latest"
        bins:
          - cargo-ai
    homepage: https://github.com/getcargohq/cargo-skills
---

# Cargo CLI — Diagnostics

Forensic runbooks for workflow behavior: trace one run, sweep a batch for errors, profile a play's credit spend. This skill is the **interpretation layer** — the raw surfaces (`run get`, orchestration SQL, billing metrics) are documented in `cargo-orchestration` and `cargo-billing`; each runbook here tells you which of them to pull, in what order, and what each output shape means.

## Bootstrap

Already signed in (`cargo-ai whoami` returns a workspace)? Skip to the next section.

```bash
npm install -g @cargo-ai/cli            # no global install? prefix every command with `npx @cargo-ai/cli`
cargo-ai login --email [email protected]  # emailed code, no browser; creates the account on first use
                                        # alternatives: --oauth (browser) · --token <api-token> (CI)
cargo-ai whoami                         # confirm the active workspace before any write
```

Every command prints JSON to stdout; failures exit non-zero with `{"errorMessage": "..."}`. Anything that creates a run or a batch is async — pass `--wait-until-finished` or poll the matching `get`. Credit attribution steps (`billing usage get-metrics`, `billing subscription get`) need a token with **admin access**; everything else works with a standard token. When the full skill bundle is installed, [`../cargo/references/prerequisites.md`](../cargo/references/prerequisites.md) adds the CLI version pin, token scopes, and the admin-only surface.

## Which runbook?

```
What are you diagnosing?
│
├── One run / one record ("why did this record fail?",
│   "run succeeded but the output is wrong/empty")
│   └── references/run-trace.md
│
├── Many runs ("the batch has errors", "error rate spiked",
│   "which node keeps failing?")
│   └── references/batch-error-sweep.md
│
└── Cost ("this play is expensive", "where do the credits go?",
    "make this cheaper")
    └── references/play-optimize-credits.md
```

Rule of thumb: start with the **sweep** when you don't yet know which run to look at — it ends by handing you exemplar run UUIDs to feed into the **trace**.

**No run UUID at all** ("look at the last run", "the run for acme.com", "what did my play do in the editor")? That's [`references/run-trace.md`](references/run-trace.md) § 0, which resolves a symptom to a UUID. Note that `orchestration run list` **requires** `--workflow-uuid` and cannot answer it — orchestration SQL over `runs` takes no filter and can. Never conclude that run inputs and outputs are inaccessible because `run list` refused.

**Boundary with `cargo-analytics`:** analytics *measures and exports* ("what's the error rate?", "download the batch results", "export this segment"); this skill *explains* ("why is the error rate up?", "why is this record's output empty?"). A diagnosis often starts from an analytics signal (error count spiked, batch reports `failedRunsCount > 0`) and ends back in analytics — once the cause is fixed and runs re-executed, bulk retrieval goes through `run download-outputs` / `batch download` / `segment download`, all documented in `../cargo-analytics/SKILL.md`. This skill's evidence surfaces (`run get`, orchestration SQL, billing metrics) are for diagnosis, not bulk export.

## References

| Doc | What it covers |
| --- | --- |
| [`references/run-trace.md`](references/run-trace.md) | Find a run from a symptom when you have no UUID (§ 0), then walk it end-to-end: per-node executions, `runContext` outputs, branch routing, per-node credits and timing. |
| [`references/batch-error-sweep.md`](references/batch-error-sweep.md) | Find errored runs across a batch/play/workspace, group failures by root cause, pick exemplars, decide fix vs report. |
| [`references/play-optimize-credits.md`](references/play-optimize-credits.md) | Attribute credit spend to workflows and nodes, then apply the cost levers in priority order. |

## The surfaces every runbook draws on

| Surface | Command | Gives you |
| --- | --- | --- |
| Run detail | `cargo-ai orchestration run get <run-uuid>` | `run.executions[]` (node-by-node trace), `runContext` (per-node output keyed by `nodeSlug`), `runComputedConfigs` (what each node was actually called with) |
| Orchestration SQL | `cargo-ai orchestration query execute "<sql>"` | Aggregates over `runs`, `batches`, `spans`, `records` (ClickHouse; no schema prefix; workspace-scoped) |
| Billing metrics | `cargo-ai billing usage get-metrics --from <date> --to <date>` | Credit totals, filterable and groupable by `workflow_uuid`, `connector_uuid`, `agent_uuid`, `integration_slug`, `model_uuid` |
| Graph picture | `cargo-ai orchestration node diagram --run-uuid <uuid> --highlight <slug> --format ascii --raw` | The graph the run executed, with the failing node marked. Free, runs nothing |

**Draw the graph before explaining a routing bug.** For "it took the wrong branch"
or "this step never ran", the picture is the evidence, and it shows one thing
`run get` does not make obvious: the `on failure` edges. A step that looks skipped
is often one the run *reached* via a `fallbackChildUuid` edge, which means the
provider errored rather than returning nothing — a different diagnosis with a
different fix. Flags and the ASCII legend: [`../cargo-orchestration/references/node-diagram.md`](../cargo-orchestration/references/node-diagram.md).

Full query syntax, table columns, and caps: [`../cargo-orchestration/references/examples/queries.md`](../cargo-orchestration/references/examples/queries.md). Debugging field semantics: [`../cargo-orchestration/references/troubleshooting.md`](../cargo-orchestration/references/troubleshooting.md).

## Presenting findings

Follow [`../cargo/references/interaction.md`](../cargo/references/interaction.md): lead with the conclusion ("18 of 20 failures are one cause: the connector's token expired"), summarize evidence in a short table, never dump raw `run get` JSON or full query results into the conversation. Any fix that re-runs paid nodes goes through the pilot gate: re-run **10–20 records** first, report the observed cost and hit-rate, then ask the user to approve the rest quoting the **record count** and **credit estimate** — a diagnosis is not approval to re-bill the batch that produced it. Full spend rules in [`../cargo-gtm/references/cost-discipline.md`](../cargo-gtm/references/cost-discipline.md).

## When diagnosis dead-ends

If the evidence contradicts documented behavior (a field missing from `run get`, a query cap that doesn't match the docs, an error that makes no sense), file a report — that's the official channel and the team reads every one:

```bash
cargo-ai workspaceManagement report create \
  --title "<one-line summary>" \
  --description "<commands run, errorMessage verbatim, expected vs actual, UUIDs>"
```