expo/skills실행 전 동작 확인
SKILL DETAIL
expo-skill-eval
expo/skills/expo-skill-eval
This skill evaluates Expo skills in this repository end-to-end, covering trigger accuracy, generated code quality, and runtime rendering. It runs the skill on iOS simulator and Android emulator via Expo Go, capturing screenshots (web optional). Use it when you want to evaluate an Expo skill, test that a skill produces working code, benchmark a skill with device screenshots, or verify that a skill's output renders correctly.
설치 수 · 183출처 보기
Installation
npx skills add https://github.com/expo/skills --skill expo-skill-eval
스킬 파일
SKILL.md
최근 동기화 · 2026. 8. 29.
agents/visual-grader.md›
# Visual Grader Instructions
Grading instructions for an eval run that produced device/web screenshots. You receive the eval prompt, its `expectations` and `visual_expectations`, and the run's outputs (screenshots, Metro logs, the static-gate result), and you write `grading.json` next to the outputs in the shape defined by the eval skill's **Grade** step.
Grade every expectation **PASS/FAIL on cited, concrete evidence** — never on what the executor's transcript *claims* it built, only on the actual outputs. Quote or name the evidence for each verdict. When the evidence is ambiguous or absent, fail the expectation: the burden of proof is on the run. The sections below add the screenshot-specific process on top of that.
## Extra inputs
- `outputs/<platform>.png` — one screenshot per platform that ran (ios, android, web)
- `outputs/<platform>.metro.log` — the Metro/dev-server log for that platform
- The static gate output (`[PASS]/[FAIL]` lines from `check-static.sh`)
## Process additions
1. **Read every screenshot with the Read tool.** Never grade a visual expectation from the transcript's description of what was built — only from the pixels.
2. **Check failure signatures first**, on every screenshot, regardless of the listed expectations:
- A red error screen / redbox, or Expo Go's "Something went wrong" page
- The Expo Go home/project-list screen (the app never opened)
- A blank white or black screen (bundle loaded but nothing rendered)
- A system permission dialog or Expo Go first-launch prompt covering the app
If any of these appear, every visual expectation for that platform fails, with the signature as evidence. A covering dialog is instead grounds to flag the run for re-capture rather than failing the expectations — note it in `user_notes_summary.needs_review`.
3. **Cross-check with the Metro log.** Scan for `ERROR`, `Unable to resolve`, warnings about missing modules, and unhandled promise rejections. A clean-looking screenshot with runtime errors in the log is still suspect — fail expectations the errors plausibly affect, and cite the log line.
4. **Grade each visual expectation per platform.** If `runtime.platforms` lists ios and android, an expectation like "a settings toggle is visible" must hold on **both** screenshots to pass. Evidence must name the screenshot file and describe what is actually visible, e.g. `ios.png: tab bar with Home/Explore tabs at bottom, toggle rendered in the first list row`.
5. **Static gate is upstream of visuals.** If `check-static.sh` failed and the device stage was skipped, mark all visual expectations failed with evidence pointing at the failed gate's log, and do not speculate about what would have rendered.
## Judgment calibration
- Platform-appropriate rendering differences (status bar, fonts, safe areas, scroll bar styling) are not failures.
- Be strict about the substance of the expectation: "tab bar with 3 tabs" means exactly three visible tabs, not "some tab bar exists".
- When a screenshot is ambiguous (mid-animation, partially loaded), say so in the evidence and fail the expectation — the burden of proof is on the run, and the harness can re-capture with a longer settle.
## Image-prompt cases (`reference_image` present in the eval case)
When the case has a `reference_image`, the prompt was "build an app matching this target screenshot," so grade **fidelity to the target** on top of the usual expectations:
1. **Read the target with the Read tool**, then read each generated screenshot, and compare them directly. Score `reference_match` (0–10): how faithfully the generated app reproduces the target's layout, component set, grouping, typography, spacing, and color treatment. Evidence must name both files and call out concrete matches and divergences, e.g. `ios.png vs target.png: same grouped list + header, but accent is blue not green and the avatar is square not round`.
2. **Apply `references/design-rubric.md`** for absolute quality, plus the code-quality dimension from the fixture's `git diff`; emit it as the `quality` block.
3. **Failure signatures still cap the score.** A redbox, blank screen, or wrong-screen capture means `reference_match` is 0 for that platform (evidence: the signature) — a crashed app reproduces nothing.
4. **Per platform, take the worst** when the case ran more than one platform (as in the rubric).
5. **Emit both blocks** in `grading.json` — `reference_match` and `quality` — alongside `expectations`; they're additive, so keep the `expectations`/`visual_expectations` grading as before.
```json
"reference_match": {"score": 7, "max": 10, "evidence": "ios.png vs target.png: …"},
"quality": {
"dimensions": [{"name": "Layout & hierarchy", "score": 2, "max": 3, "evidence": "ios.png: …"}],
"subtotal": 17, "max": 24, "summary": "one-line overall read"
}
```
`quality.subtotal` is the sum of dimension scores; `quality.max` is the sum of their maxes (24 for the built-ins).
references/design-rubric.md›
# Design-quality rubric
Applied by the grader when scoring an app's visual quality — primarily for **image-prompt cases** (`reference_image`), and any time a quality grade is requested — on top of the pass/fail expectations, not instead of them. It answers "how *good* is this app?", scored from the captured screenshots and the implementation diff. Combine it with the failure-signature checks in `agents/visual-grader.md`: a redbox, blank screen, or wrong-screen capture caps every visual dimension at 0 for that platform regardless of intent.
## How to score
- **Judge visuals only from the pixels.** Read every screenshot with the Read tool; never infer a visual score from the transcript or from what the code *intended* to render.
- **Grade per platform, then take the worst.** If the case ran ios + android, a dimension's score is the lower of the two — unless the gap is a legitimate platform difference (see *Platform conventions*).
- Each dimension is **0–3**. The seven visual dimensions sum to a max of 21; with the code-quality dimension (also 0–3) the built-in rubric max is **24**.
- If a config/platform screenshot is **missing**, score only the platforms present, note the gap, and do **not** penalize for the absence.
- Cite concrete evidence per dimension, naming the file: e.g. `ios.png: 16px gutters, consistent 12px row spacing, large-title nav bar`.
### Score anchors (apply to every dimension)
- **0** — broken/absent: the dimension fails outright (overlapping or clipped text, unreadable contrast, no hierarchy, content off-screen).
- **1** — below bar: present but with clear problems a designer would reject.
- **2** — solid: clean and conventional, no notable problems; what a competent dev ships.
- **3** — polished: deliberate and refined; App-Store quality for this dimension.
## Visual dimensions
1. **Layout & hierarchy** — clear primary/secondary structure, alignment, logical grouping, balanced composition; nothing clipped or pushed off-screen.
2. **Spacing & density** — consistent padding/margins and vertical rhythm; not cramped, not sparse; respects safe areas, the notch, and the home indicator.
3. **Typography** — readable sizes, a sensible type scale (title vs body vs caption), consistent weights, no truncation/overflow/orphaned text.
4. **Color & contrast** — coherent, limited palette; sufficient text/background contrast for legibility; sensible light/dark handling.
5. **Platform conventions** — feels native to the platform it's shown on:
- **iOS** — large titles / nav bars, grouped/inset lists, SF-style controls, standard tab bars, system spacing.
- **Android** — Material surfaces, app bar, FAB where appropriate, ripple/elevation cues, Material controls.
- **web** — sensible responsive layout, real cursor/hover affordances, no mobile-only chrome stranded on a wide canvas.
A genuine platform difference (status bar, system fonts, control styling, safe areas) is **not** a defect; shipping an iOS-only pattern verbatim on Android **is**.
6. **Polish & states** — pixel alignment, crisp icons/images at the correct density, no stray debug text or placeholder lorem, sensible empty/loading states where the screen implies them.
7. **Visual accessibility** — tap targets look ≥44pt, text isn't clipped at default scale, contrast meets a rough WCAG-AA bar, interactive elements are visually distinguishable from static content.
## Code-quality dimension (0–3, from the diff)
Scored from the fixture's `git diff`, not the screenshots:
- Idiomatic Expo/RN: correct import paths and current APIs (e.g. the `@expo/ui` `Host` wrapper, Expo Router file conventions), no deprecated patterns.
- Reasonable structure: componentized where it helps, no copy-paste duplication, styles organized (not a wall of inline objects), TypeScript types where the template uses them.
- No dead code, leftover `console.log` noise, or unused scaffolding left from the template.
- Anchors: **0** broken/anti-patterns, **1** works but messy, **2** clean and idiomatic, **3** exemplary.
## Extra criteria
If the eval case lists `visual_expectations`, grade each as an **additional 0–3 dimension** with the same anchors, appended to the `dimensions` array and folded into `subtotal`/`max` (`max` grows by 3 per extra criterion). Cite evidence for each just like the built-ins.
references/runtime-matrix.md›
# Runtime applicability matrix
Which `plugins/expo/skills/*` skills can be evaluated at which level. Consult this before writing eval cases — putting `runtime.mode: "expo-go"` on a skill that needs custom native code wastes a device cycle and produces a misleading FAIL.
Modes:
- **expo-go** — generated code runs in Expo Go on simulator/emulator; screenshot grading applies.
- **static-only** — stop after the static gate (`tsc` / lint / `expo export`). Used when the output needs custom native code, a dev build, or produces no UI at all.
- **n/a** — the skill's output is not app code (config, CI YAML, CLI workflows); code evals don't apply, or apply only as file-content assertions without a fixture app.
| Skill | Mode | Platforms | Notes |
|-------|------|-----------|-------|
| expo-router | expo-go | ios, android | Navigation/routing target: routes, links, native stacks, modals, sheets, headers. Native tabs sections may need a dev build — verify per case. |
| expo-native-ui | expo-go | ios, android | Core target for visual evals. RN primitives + styling/controls/media/animations all work in Expo Go. |
| expo-tailwind-setup | expo-go | ios, android, web | NativeWind v5 / react-native-css are JS-level; works in Expo Go. Good web candidate too. |
| expo-data-fetching | expo-go | ios, android | Fetch/React Query/SWR are pure JS. Mock or use stable public endpoints so evals are deterministic. |
| expo-dom | expo-go | ios, android, web | DOM components run in a webview on native (Expo Go, SDK 52+) and as-is on web. Allow extra settle time for the webview to paint. |
| eas-hosting | static-only + HTTP | — | No screenshot: while Metro runs, `curl` the route and save the response as an output file for grading. |
| expo-ui (merged swift-ui + jetpack-compose) | expo-go | ios, android, web | `@expo/ui` ships in the SDK 56 default template (~56.0.17) and its native code is included in Expo Go 56 — verified rendering 2026-06: a SwiftUI Form renders in Expo Go on the iOS simulator. The **universal** components (`Host`, `Row`, `Column`, `Button`, `List`, …) also render on **web**; only the platform-specific `@expo/ui/swift-ui` (SwiftUI) and `@expo/ui/jetpack-compose` (Compose) trees are native-only and render blank on web. Pick platforms by what the eval case uses. |
| expo-module | static-only | — | Custom native modules can't load in Expo Go. Gate on TS compile of the module API; native compilation (prebuild + xcodebuild/gradle) is possible but expensive — out of scope for now. |
| expo-app-clip | static-only | — | App Clip targets require prebuild + Xcode. Assert on config-plugin output instead. |
| expo-brownfield | static-only | — | Brownfield integration has no Expo Go story. Assert on generated integration files. |
| expo-dev-client | n/a | — | Output is build/distribution workflow, not renderable app code. |
| eas-app-stores | n/a | — | Deployment guidance; assert on transcript/config content only. |
| eas-workflows | n/a | — | Output is `.eas/workflows/` YAML; assert on YAML content (and `workflow_validate` via the Expo MCP if available). |
| expo-upgrade | static-only | — | An upgrade eval can run the static gate on the upgraded fixture — a strong signal. Needs a fixture pinned to an *older* SDK (pass e.g. `55` to make-fixture.sh). |
| eas-update-insights | n/a | — | CLI/metrics interpretation; no app code output. |
| eas-observe | expo-go (verify) | ios, android | `expo-observe` instrumentation is JS-level, but confirm the package works in Expo Go before relying on screenshots; fall back to static-only. |
Entries marked "verify" haven't been validated against a real run yet — confirm on first use and update this table.
## Image-prompt (clone-this) candidates
Image prompts (`reference_image` — build an app to match a target screenshot, graded by `reference_match` + `references/design-rubric.md`) are most informative for the **visual** skills — `expo-router`, `expo-ui`, `expo-tailwind-setup`, `expo-dom` — where reproducing a specific UI is the point. Pair them with "Runtime + screenshots" so the harness captures the generated app to compare against the target. They add nothing for `n/a` skills (no app UI).
scripts/check-static.sh›
#!/usr/bin/env bash
# Static gate for a generated Expo app: typecheck, lint, bundle.
#
# Usage: check-static.sh <project-path> [platforms-csv]
# platforms-csv Platforms to bundle-check with `expo export`,
# e.g. "ios,android" (default) or "ios,android,web".
#
# Prints [PASS]/[FAIL] per gate and exits non-zero if any gate fails.
# A passing export catches most import/syntax/missing-module errors
# without needing a device.
set -uo pipefail
PROJECT_PATH="${1:?usage: check-static.sh <project-path> [platforms-csv]}"
PLATFORMS="${2:-ios,android}"
cd "$PROJECT_PATH"
FAILED=0
LOG_DIR=".eval-static"
mkdir -p "$LOG_DIR"
run_gate() {
local name="$1"
shift
if "$@" >"$LOG_DIR/$name.log" 2>&1; then
echo "[PASS] $name"
else
echo "[FAIL] $name (log: $PROJECT_PATH/$LOG_DIR/$name.log)"
tail -20 "$LOG_DIR/$name.log" | sed 's/^/ /'
FAILED=1
fi
}
run_gate tsc bunx tsc --noEmit
# Lint only files changed since the fixture commit - the pristine template has
# pre-existing lint errors that aren't the executor's fault. Falls back to a
# full `expo lint` when the project isn't a git repo.
if git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
CHANGED_FILES=()
while IFS= read -r f; do
[[ -f "$f" ]] && CHANGED_FILES+=("$f")
done < <(
{
git diff --name-only HEAD -- '*.ts' '*.tsx' '*.js' '*.jsx'
git ls-files --others --exclude-standard -- '*.ts' '*.tsx' '*.js' '*.jsx'
} | sort -u
)
if ((${#CHANGED_FILES[@]})); then
run_gate lint env CI=1 bunx eslint "${CHANGED_FILES[@]}"
else
echo "[SKIP] lint (no changed JS/TS files)"
fi
else
run_gate lint env CI=1 bunx expo lint
fi
EXPORT_ARGS=(export --output-dir "$LOG_DIR/export")
IFS=',' read -ra PLATFORM_LIST <<<"$PLATFORMS"
for p in "${PLATFORM_LIST[@]}"; do
EXPORT_ARGS+=(--platform "$p")
done
run_gate export env CI=1 bunx expo "${EXPORT_ARGS[@]}"
exit "$FAILED"
scripts/clean-fixture.sh›
#!/usr/bin/env bash
# Reclaim disk after a fixture's screenshots are captured. Essential for
# dev-build runs, where each `expo run:<platform>` leaves multi-GB native build
# output (iOS Pods + DerivedData, Android Gradle build). Removes the heavy,
# regenerable artifacts and keeps the app source + git history, so the grader's
# `git diff` still works.
#
# Usage: clean-fixture.sh <project-path>
#
# Env: EXPO_SKILL_EVAL_KEEP_DERIVEDDATA=1 skips the iOS DerivedData sweep (set it
# only if you keep a real Xcode project literally named "fixture").
set -uo pipefail
APP="${1:?usage: clean-fixture.sh <project-path>}"
# Safety guard: only ever clean inside an expo-skill-eval workspace.
case "$APP" in
*expo-skill-eval-*) : ;;
*) echo "clean-fixture: refusing to clean '$APP' (not an expo-skill-eval fixture)" >&2; exit 1 ;;
esac
[[ -d "$APP" ]] || { echo "clean-fixture: $APP not found, skipping"; exit 0; }
# Stop any Metro / Expo dev server that may still be running for this fixture.
# The snapshot scripts' EXIT trap normally handles this, but cleaning up here
# catches leftover processes from crashes or interrupted runs.
# Ports 8081 (iOS) and 8082 (Android) are the two ports the eval harness
# reserves — freeing them here is safe because clean-fixture.sh only runs after
# all screenshots for this fixture have been captured.
#
# `-sTCP:LISTEN` is REQUIRED, not a refinement: without it, `lsof -ti tcp:8082`
# also matches the *established* connection the adb daemon holds from the
# `adb reverse tcp:8082` tunnel, and SIGKILLing the adb daemon crashes the
# emulator with a gfxstream gRPC SIGABRT (std::bad_function_call). Keep the flag
# so only the Metro listener is killed.
lsof -ti tcp:8081 -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
lsof -ti tcp:8082 -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
# Per-fixture heavy dirs — all gitignored / regenerable (node_modules, the
# prebuilt native projects incl. iOS Pods and Android Gradle output, caches).
rm -rf \
"$APP/node_modules" \
"$APP/ios" \
"$APP/android" \
"$APP/.expo" \
"$APP/dist" \
"$APP/web-build" 2>/dev/null || true
# iOS DerivedData for fixture builds. create-expo-app names the project
# "fixture", so its build output lives under DerivedData/fixture-<hash>. This is
# a cache (worst case a rebuild), so it's safe to drop between fixtures.
if [[ "${EXPO_SKILL_EVAL_KEEP_DERIVEDDATA:-0}" != "1" ]]; then
DD="$HOME/Library/Developer/Xcode/DerivedData"
[[ -d "$DD" ]] && rm -rf "$DD"/fixture-* 2>/dev/null || true
fi
echo "cleaned fixture: $APP"
scripts/generate_viewer.py›
#!/usr/bin/env python3
"""Generate the self-contained HTML viewer for an expo-skill-eval run.
Usage: generate_viewer.py <workspace-root> [--artifact]
<workspace-root> Run root, e.g. /private/tmp/expo-skill-eval-expo-ui.
Iteration dirs (iteration-*) and trigger-evals/ live under it;
viewer.html is written there and opened in the browser.
--artifact Emit viewer_artifact.html (page-content only, no
<!DOCTYPE>/<html>/<head>/<body>) for the Artifact tool, and
do NOT open a browser. The standalone viewer.html is still
written on a normal (non-artifact) run.
Reads, per iteration, <iter>/evals.json plus each eval's
eval-<id>/<config>/{grading.json,static.json,outputs/<platform>.png}. Screenshots
are embedded as base64 data: URIs so the file is fully self-contained (and
satisfies the Artifact CSP). Self-paced by argv so it can be a checked-in script
rather than regenerated each run.
"""
import base64
import json
import sys
import webbrowser
from pathlib import Path
def b64_img(path):
if not path:
return None
p = Path(path)
if not p.exists() or not p.is_file():
return None
with open(p, "rb") as f:
data = base64.b64encode(f.read()).decode()
ext = p.suffix.lower().lstrip(".")
mime = {"png": "image/png", "jpg": "image/jpeg", "jpeg": "image/jpeg"}.get(ext, "image/png")
return f"data:{mime};base64,{data}"
def score_color(score, max_score):
if not max_score:
return "#888"
pct = score / max_score
if pct >= 0.85:
return "#4ade80"
elif pct >= 0.65:
return "#fbbf24"
else:
return "#f87171"
def load_json(path, default=None):
p = Path(path)
if not p.exists():
return default
try:
with open(p) as f:
return json.load(f)
except Exception:
return default
def render_expectations(expectations):
if not expectations:
return ""
html = '<ul class="exp-list">'
for exp in expectations:
if isinstance(exp, dict):
passed = exp.get("passed", None)
text = exp.get("text", str(exp))
evidence = exp.get("evidence", "")
if passed is True:
badge = '<span class="badge pass">PASS</span>'
elif passed is False:
badge = '<span class="badge fail">FAIL</span>'
else:
badge = '<span class="badge unknown">?</span>'
ev_html = f'<div class="evidence">{evidence}</div>' if evidence else ""
html += f'<li>{badge} {text}{ev_html}</li>'
else:
html += f'<li><span class="badge unknown">?</span> {exp}</li>'
html += "</ul>"
return html
def render_quality(quality):
if not quality:
return ""
dims = quality.get("dimensions", [])
subtotal = quality.get("subtotal", 0)
max_total = quality.get("max", 0)
summary = quality.get("summary", "")
html = '<div class="quality-block">'
html += f'<div class="quality-header">Design Quality: <b>{subtotal}/{max_total}</b></div>'
for d in dims:
name = d.get("name", "")
score = d.get("score", 0)
mx = d.get("max", 3)
evidence = d.get("evidence", "")
pct = (score / mx * 100) if mx else 0
color = score_color(score, mx)
html += f'''<div class="quality-dim">
<div class="dim-label">{name} <span style="color:{color}">{score}/{mx}</span></div>
<div class="dim-bar-wrap"><div class="dim-bar" style="width:{pct:.0f}%;background:{color}"></div></div>
{f'<div class="dim-evidence">{evidence}</div>' if evidence else ""}
</div>'''
if summary:
html += f'<div class="quality-summary">{summary}</div>'
html += "</div>"
return html
def render_config_card(iter_dir, eval_id, config, eval_case, grading, static_result):
screenshots = []
for plat in eval_case.get("runtime", {}).get("platforms", ["ios"]):
img_path = iter_dir / f"eval-{eval_id}" / config / "outputs" / f"{plat}.png"
screenshots.append((plat, b64_img(img_path)))
static_pass = static_result and static_result.get("exit_code") == 0
static_badge = '<span class="badge pass">BUILD OK</span>' if static_pass else '<span class="badge fail">BUILD FAIL</span>'
score = grading.get("score", 0) if grading else None
max_score = grading.get("max_score", 1) if grading else 1
score_html = ""
if score is not None:
color = score_color(score, max_score)
score_html = f'<div class="score" style="color:{color}">{score}/{max_score}</div>'
label = "With Skill" if config == "with_skill" else "Without Skill"
html = '<div class="config-card">'
html += f'<div class="config-header"><span class="config-label {"with" if config == "with_skill" else "without"}">{label}</span>{static_badge}{score_html}</div>'
for plat, img_b64 in screenshots:
if img_b64:
html += f'<div class="screenshot-wrap"><div class="plat-label">{plat}</div>'
html += f'<img src="{img_b64}" class="screenshot" onclick="this.classList.toggle(\'zoomed\')" />'
html += '</div>'
else:
html += f'<div class="screenshot-wrap"><div class="plat-label">{plat}</div><div class="no-screenshot">No screenshot</div></div>'
if grading:
html += render_expectations(grading.get("expectations", []))
ref_match = grading.get("reference_match")
if ref_match:
rm_score = ref_match.get("score", 0)
rm_max = ref_match.get("max", 10)
rm_color = score_color(rm_score, rm_max)
html += f'<div class="ref-match"><b>Reference Match:</b> <span style="color:{rm_color}">{rm_score}/{rm_max}</span>'
if ref_match.get("evidence"):
html += f'<div class="evidence">{ref_match["evidence"]}</div>'
html += '</div>'
html += render_quality(grading.get("quality"))
notes = grading.get("user_notes_summary", {})
if notes and notes.get("notes"):
html += f'<div class="reviewer-notes"><b>Notes:</b> {notes["notes"]}</div>'
html += '</div>'
return html
def render_iteration(iter_dir):
evals = load_json(iter_dir / "evals.json")
if not evals:
return f"<p>No evals.json found in {iter_dir}</p>"
html = ""
total_with = total_without = 0
scored_with = scored_without = 0
qual_with = qual_without = 0
total_evals = 0
for ev in evals:
eid = ev["id"]
ref_img_b64 = b64_img(ev.get("reference_image", ""))
with_grading = load_json(iter_dir / f"eval-{eid}" / "with_skill" / "grading.json")
without_grading = load_json(iter_dir / f"eval-{eid}" / "without_skill" / "grading.json")
with_static = load_json(iter_dir / f"eval-{eid}" / "with_skill" / "static.json")
without_static = load_json(iter_dir / f"eval-{eid}" / "without_skill" / "static.json")
total_evals += 1
if with_grading and with_grading.get("max_score"):
scored_with += with_grading.get("score", 0)
total_with += with_grading.get("max_score", 0)
if without_grading and without_grading.get("max_score"):
scored_without += without_grading.get("score", 0)
total_without += without_grading.get("max_score", 0)
if with_grading and with_grading.get("quality"):
qual_with += with_grading["quality"].get("subtotal", 0)
if without_grading and without_grading.get("quality"):
qual_without += without_grading["quality"].get("subtotal", 0)
html += '<div class="eval-case">'
html += f'<div class="eval-header">Eval #{eid}</div>'
html += f'<div class="eval-prompt">{ev.get("prompt", "")[:200]}</div>'
if ref_img_b64:
html += '<div class="ref-image-wrap"><div class="ref-label">Target Reference</div>'
html += f'<img src="{ref_img_b64}" class="ref-image" onclick="this.classList.toggle(\'zoomed\')" />'
html += '</div>'
html += '<div class="configs-row">'
html += render_config_card(iter_dir, eid, "with_skill", ev, with_grading, with_static)
html += render_config_card(iter_dir, eid, "without_skill", ev, without_grading, without_static)
html += '</div>'
html += '</div>'
with_pct = (scored_with / total_with * 100) if total_with else 0
without_pct = (scored_without / total_without * 100) if total_without else 0
delta = with_pct - without_pct
delta_color = "#4ade80" if delta > 0 else "#f87171" if delta < 0 else "#888"
qual_html = ""
if qual_with or qual_without:
qdelta = qual_with - qual_without
qcolor = "#4ade80" if qdelta > 0 else "#f87171" if qdelta < 0 else "#888"
qual_html = f' <div class="summary-item">Quality Δ: <span style="color:{qcolor}">{qdelta:+d}</span></div>\n'
summary = f'''<div class="summary-bar">
<div class="summary-item">With Skill: <span style="color:{score_color(scored_with, total_with) if total_with else "#888"}">{with_pct:.0f}%</span></div>
<div class="summary-item">Without Skill: <span style="color:{score_color(scored_without, total_without) if total_without else "#888"}">{without_pct:.0f}%</span></div>
<div class="summary-item">Delta: <span style="color:{delta_color}">{delta:+.0f}pp</span></div>
{qual_html} <div class="summary-item">Evals: {total_evals}</div>
</div>'''
return summary + html
def render_trigger_table(results_path):
data = load_json(results_path)
if not data:
return ""
html = '<div class="trigger-section"><h2>Trigger Accuracy</h2>'
recall = data.get("recall", 0)
color = score_color(recall, 1)
passed_count = data.get("passed", data.get("triggered", 0))
html += f'<div class="trigger-summary">Recall: <span style="color:{color}">{100*recall:.0f}%</span> ({passed_count}/{data.get("total")} passed)</div>'
html += '<table class="trigger-table"><tr><th>#</th><th>Prompt</th><th>Should Trigger</th><th>Triggered</th><th>Result</th><th>Time</th></tr>'
for i, r in enumerate(data.get("results", []), 1):
# Support both schema variants: {id, prompt, should_trigger, pass} and {query, triggered, duration}
rid = r.get("id", i)
prompt = r.get("prompt", r.get("query", ""))
should_trigger = r.get("should_trigger", True) # default True for recall-only runs
triggered = r.get("triggered", False)
passed = r.get("pass", triggered if should_trigger else not triggered)
result_badge = '<span class="badge pass">PASS</span>' if passed else '<span class="badge fail">FAIL</span>'
should = "Yes" if should_trigger else "No"
triggered_str = "Yes" if triggered else "No"
elapsed = r.get("elapsed_s", r.get("duration", "?"))
html += f'<tr><td>{rid}</td><td class="prompt-cell">{prompt[:80]}</td><td>{should}</td><td>{triggered_str}</td><td>{result_badge}</td><td>{elapsed}s</td></tr>'
html += '</table></div>'
return html
CSS = """\
:root {
--ground: #090C14;
--surface: #0E1320;
--surface-hi: #131929;
--border: #1A2238;
--text: #D9E2F5;
--text-muted: #6B7A9E;
--text-dim: #3A4560;
--accent: #7B6FD3;
--accent-glow: rgba(123,111,211,0.12);
--pass: #4DB87C;
--pass-bg: rgba(77,184,124,0.10);
--fail: #E05555;
--fail-bg: rgba(224,85,85,0.10);
--score-with: #9B8EE8;
--score-without: #5A6480;
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body { background: var(--ground); color: var(--text); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 14px; line-height: 1.5; padding: 20px 24px; }
h1 { font-size: 15px; font-weight: 500; letter-spacing: 0.06em; text-transform: uppercase; color: var(--text-muted); margin-bottom: 20px; }
h2 { font-size: 11px; font-weight: 600; letter-spacing: 0.08em; text-transform: uppercase; color: var(--text-muted); margin: 24px 0 10px; }
/* Tabs */
.tabs { display: flex; gap: 6px; margin-bottom: 20px; }
.tab { background: var(--surface); border: 1px solid var(--border); color: var(--text-muted); padding: 5px 14px; border-radius: 5px; cursor: pointer; font-size: 12px; font-weight: 500; transition: color .15s, border-color .15s; }
.tab:hover { color: var(--text); border-color: #2A3558; }
.tab.active { background: var(--surface-hi); border-color: var(--accent); color: var(--text); }
/* Summary strip */
.summary-bar { display: flex; gap: 0; background: var(--surface); border: 1px solid var(--border); border-radius: 8px; overflow: hidden; margin-bottom: 20px; }
.summary-item { flex: 1; padding: 12px 16px; border-right: 1px solid var(--border); font-size: 11px; letter-spacing: 0.04em; text-transform: uppercase; color: var(--text-muted); }
.summary-item:last-child { border-right: none; }
.summary-item span { display: block; font-size: 22px; font-weight: 700; font-variant-numeric: tabular-nums; letter-spacing: -0.02em; color: var(--text); margin-top: 2px; }
/* Eval card */
.eval-case { background: var(--surface); border: 1px solid var(--border); border-radius: 10px; margin-bottom: 24px; overflow: hidden; }
.eval-header { padding: 10px 16px; font-size: 11px; font-weight: 600; letter-spacing: 0.07em; text-transform: uppercase; color: var(--text-muted); border-bottom: 1px solid var(--border); }
.eval-prompt { padding: 12px 16px; font-size: 13px; color: var(--text); border-bottom: 1px solid var(--border); line-height: 1.55; }
/* Reference image */
.ref-image-wrap { padding: 16px; background: #07090F; border-bottom: 1px solid var(--border); }
.ref-label { font-size: 10px; font-weight: 600; letter-spacing: 0.09em; text-transform: uppercase; color: var(--text-dim); margin-bottom: 8px; }
.ref-image { max-height: 260px; width: auto; border-radius: 10px; border: 1px solid #1E2840; cursor: zoom-in; display: block; }
.ref-image.zoomed { max-height: none; cursor: zoom-out; }
/* Side-by-side config columns */
.configs-row { display: grid; grid-template-columns: 1fr 1fr; }
.config-card { padding: 16px; border-right: 1px solid var(--border); position: relative; }
.config-card:last-child { border-right: none; }
/* The aesthetic risk: the with_skill card gets a faint violet radial glow from its header edge */
.config-card:first-child::before {
content: "";
position: absolute;
top: 0; left: 0; right: 0; height: 180px;
background: radial-gradient(ellipse 80% 120px at 50% 0, var(--accent-glow), transparent 70%);
pointer-events: none;
}
.config-header { display: flex; align-items: center; gap: 8px; margin-bottom: 14px; flex-wrap: wrap; }
.config-label { font-size: 10px; font-weight: 700; letter-spacing: 0.07em; text-transform: uppercase; padding: 3px 8px; border-radius: 4px; }
.config-label.with { background: rgba(123,111,211,0.15); color: var(--accent); border: 1px solid rgba(123,111,211,0.25); }
.config-label.without { background: var(--surface-hi); color: var(--text-muted); border: 1px solid var(--border); }
.score { font-size: 22px; font-weight: 700; font-variant-numeric: tabular-nums; margin-left: auto; }
.config-card:first-child .score { color: var(--score-with); }
.config-card:last-child .score { color: var(--score-without); }
/* Badges */
.badge { display: inline-flex; align-items: center; font-size: 9px; font-weight: 700; padding: 2px 7px; border-radius: 3px; letter-spacing: 0.05em; text-transform: uppercase; }
.badge.pass { background: var(--pass-bg); color: var(--pass); border: 1px solid rgba(77,184,124,0.2); }
.badge.fail { background: var(--fail-bg); color: var(--fail); border: 1px solid rgba(224,85,85,0.2); }
.badge.unknown { background: var(--surface-hi); color: var(--text-dim); border: 1px solid var(--border); }
/* Screenshots */
.screenshot-wrap { margin-bottom: 14px; }
.plat-label { font-size: 9px; font-weight: 600; letter-spacing: 0.09em; text-transform: uppercase; color: var(--text-dim); margin-bottom: 6px; }
.screenshot { max-width: 100%; max-height: 380px; width: auto; border-radius: 14px; border: 1px solid #1A2540; cursor: zoom-in; display: block; box-shadow: 0 4px 24px rgba(0,0,0,0.5); }
.screenshot.zoomed { max-height: none; cursor: zoom-out; }
.no-screenshot { color: var(--text-dim); font-size: 11px; padding: 16px; background: var(--surface-hi); border-radius: 6px; text-align: center; border: 1px dashed var(--border); }
/* Expectations */
.exp-list { list-style: none; margin-top: 12px; }
.exp-list li { padding: 6px 0; border-bottom: 1px solid var(--border); font-size: 12px; color: var(--text-muted); display: flex; flex-wrap: wrap; gap: 6px; align-items: flex-start; }
.exp-list li:last-child { border-bottom: none; }
.evidence { font-size: 10px; color: var(--text-dim); width: 100%; line-height: 1.5; margin-top: 2px; }
/* Reference match block */
.ref-match { margin-top: 12px; font-size: 12px; color: var(--text-muted); background: var(--surface-hi); padding: 10px 12px; border-radius: 6px; border-left: 2px solid var(--accent); }
.ref-match b { color: var(--text); }
/* Quality rubric */
.quality-block { margin-top: 12px; background: var(--surface-hi); border: 1px solid var(--border); border-radius: 6px; padding: 12px; }
.quality-header { font-size: 11px; font-weight: 600; letter-spacing: 0.05em; text-transform: uppercase; color: var(--text-muted); margin-bottom: 10px; }
.quality-dim { margin-bottom: 10px; }
.quality-dim:last-of-type { margin-bottom: 0; }
.dim-label { font-size: 10px; color: var(--text-muted); margin-bottom: 4px; display: flex; justify-content: space-between; }
.dim-label span { font-variant-numeric: tabular-nums; }
.dim-bar-wrap { height: 3px; background: var(--border); border-radius: 2px; overflow: hidden; }
.dim-bar { height: 100%; border-radius: 2px; }
.dim-evidence { font-size: 10px; color: var(--text-dim); margin-top: 3px; line-height: 1.45; }
.quality-summary { font-size: 11px; color: var(--text-dim); margin-top: 8px; padding-top: 8px; border-top: 1px solid var(--border); }
.reviewer-notes { margin-top: 10px; font-size: 11px; color: var(--text-dim); background: var(--ground); padding: 8px 10px; border-radius: 4px; }
/* Trigger table */
.trigger-section { margin-top: 32px; }
.trigger-summary { font-size: 13px; margin-bottom: 12px; color: var(--text-muted); }
.trigger-table { width: 100%; border-collapse: collapse; font-size: 12px; }
.trigger-table th { background: var(--surface); padding: 9px 10px; text-align: left; font-size: 10px; font-weight: 600; letter-spacing: 0.07em; text-transform: uppercase; color: var(--text-muted); border-bottom: 1px solid var(--border); }
.trigger-table td { padding: 8px 10px; border-bottom: 1px solid var(--border); color: var(--text-muted); vertical-align: top; }
.trigger-table tr:last-child td { border-bottom: none; }
.trigger-table tr:hover td { background: var(--surface-hi); }
.prompt-cell { max-width: 320px; word-break: break-word; color: var(--text); }
@media (prefers-reduced-motion: reduce) { .dim-bar { transition: none; } }
"""
JS = """\
function showTab(i) {
document.querySelectorAll('.panel').forEach((p, idx) => { p.style.display = idx === i ? 'block' : 'none'; });
document.querySelectorAll('.tab').forEach((t, idx) => { t.classList.toggle('active', idx === i); });
localStorage.setItem('expo-eval-tab', i);
}
(function() {
const saved = parseInt(localStorage.getItem('expo-eval-tab') || '0', 10);
if (saved > 0) showTab(saved);
})();
"""
def build_page(ws, title, artifact=False):
iterations = sorted(p for p in ws.glob("iteration-*") if p.is_dir())
if not iterations:
return f"<p>No iteration directories found under {ws}.</p>"
tabs_html = ""
panels_html = ""
for i, it in enumerate(iterations):
active = "active" if i == 0 else ""
tabs_html += f'<button class="tab {active}" onclick="showTab({i})" id="tab-{i}">{it.name}</button>'
content = render_iteration(it)
panels_html += f'<div class="panel" id="panel-{i}" style="display:{"block" if i == 0 else "none"}">{content}</div>'
trigger_html = render_trigger_table(ws / "trigger-evals" / "trigger_results.json")
body_content = f'<h1>{title}</h1>\n<div class="tabs">{tabs_html}</div>\n{panels_html}\n{trigger_html}'
if artifact:
return f'<title>{title}</title>\n<style>\n{CSS}</style>\n{body_content}\n<script>\n{JS}</script>'
return f'''<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>{title}</title>
<style>
{CSS}
</style>
</head>
<body>
{body_content}
<script>
{JS}
</script>
</body>
</html>'''
def main():
args = [a for a in sys.argv[1:] if a != "--artifact"]
artifact = "--artifact" in sys.argv
if not args:
print("usage: generate_viewer.py <workspace-root> [--artifact]", file=sys.stderr)
sys.exit(1)
ws = Path(args[0]).resolve()
skill = ws.name.replace("expo-skill-eval-", "")
title = f"{skill} Skill Eval"
html = build_page(ws, title, artifact=artifact)
if artifact:
out = ws / "viewer_artifact.html"
out.write_text(html)
print(f"Artifact viewer written: {out}")
else:
out = ws / "viewer.html"
out.write_text(html)
print(f"Viewer written: {out}")
webbrowser.open("file://" + str(out))
if __name__ == "__main__":
main()
scripts/latest-sdk.sh›
#!/usr/bin/env bash
# Print the latest Expo SDK major version.
#
# Usage: latest-sdk.sh
#
# The `expo` package's major version IS the SDK version ([email protected] -> SDK 56).
# Uses bun to run `npm view expo dist-tags --json` (so it honors the local npm
# registry / proxy / auth config rather than hardcoding registry.npmjs.org),
# JSON.parse the result, and take the major via the `semver` package (falling
# back to a numeric split if semver can't be resolved). Run via the skill's
# `bash *expo-skill-eval/scripts/*` rule so SDK detection stays prompt-free —
# the npm child runs under bun, not as its own Bash tool call.
set -euo pipefail
exec bun -e '
let out;
try {
out = await Bun.$`npm view expo dist-tags --json`.text();
} catch (e) {
console.error("error: `npm view expo` failed: " + (e?.message ?? e));
process.exit(1);
}
let latest;
try {
latest = JSON.parse(out)?.latest;
} catch {
console.error("error: could not JSON.parse `npm view` output");
process.exit(1);
}
if (!latest) {
console.error("error: no `latest` dist-tag for expo");
process.exit(1);
}
let major;
try {
const semver = await import("semver");
major = (semver.default ?? semver).major(latest);
} catch {
major = Number.parseInt(String(latest).split(".")[0], 10);
}
if (!Number.isInteger(major)) {
console.error(`error: could not parse SDK major from "${latest}"`);
process.exit(1);
}
console.log(major);
'
scripts/make-fixture.sh›
#!/usr/bin/env bash
# Create an Expo app fixture for an eval run.
#
# Usage: make-fixture.sh <project-path> [sdk-version] [variant]
# project-path Where the app should end up (must not exist yet)
# sdk-version Expo SDK major version, e.g. "56". Omit for the latest template.
# variant "clean" (default) runs the template's reset-project script so
# executors start from a blank app instead of the example
# screens. "full" keeps the example tabs (Home/Explore) - use
# it for evals whose prompt assumes an existing app.
#
# One pristine app is created per SDK version + variant with
# `bunx create-expo-app -t default@sdk-<version>` and cached under
# $EXPO_SKILL_EVAL_CACHE (default ~/.cache/expo-skill-eval/fixtures).
# Each call clones the cache with APFS copy-on-write, so only the first
# call per SDK version pays the install cost.
set -euo pipefail
PROJECT_PATH="${1:?usage: make-fixture.sh <project-path> [sdk-version] [variant]}"
SDK_VERSION="${2:-}"
VARIANT="${3:-clean}"
if [[ "$VARIANT" != "clean" && "$VARIANT" != "full" ]]; then
echo "error: variant must be 'clean' or 'full', got '$VARIANT'" >&2
exit 1
fi
CACHE_ROOT="${EXPO_SKILL_EVAL_CACHE:-$HOME/.cache/expo-skill-eval/fixtures}"
if [[ -e "$PROJECT_PATH" ]]; then
echo "error: $PROJECT_PATH already exists" >&2
exit 1
fi
TEMPLATE="default"
CACHE_KEY="latest"
if [[ -n "$SDK_VERSION" ]]; then
TEMPLATE="default@sdk-${SDK_VERSION}"
CACHE_KEY="sdk-${SDK_VERSION}"
fi
if [[ "$VARIANT" == "clean" ]]; then
CACHE_KEY="${CACHE_KEY}-clean"
fi
CACHE_DIR="$CACHE_ROOT/$CACHE_KEY"
if [[ ! -d "$CACHE_DIR" ]]; then
echo "building fixture cache for template '$TEMPLATE'..." >&2
mkdir -p "$CACHE_ROOT"
TMP_DIR="$(mktemp -d)"
trap 'rm -rf "$TMP_DIR"' EXIT
(cd "$TMP_DIR" && bunx create-expo-app --yes --template "$TEMPLATE" fixture) >&2
if [[ "$VARIANT" == "clean" ]]; then
# Strip the example screens so executors start from a blank app.
# Answer "n" to the "move to /example?" prompt - delete them instead.
(cd "$TMP_DIR/fixture" && echo n | bun run reset-project) >&2
fi
# Pin a stable iOS bundle identifier and Android package. The default template
# leaves both unset, so a dev-build run (`expo run:ios`/`expo run:android`)
# would prompt for them - which fails in the snapshot scripts' non-interactive
# mode - and the snapshot scripts need a known id to relaunch the app by.
(cd "$TMP_DIR/fixture" && bun -e '
const f = "app.json";
const j = await Bun.file(f).json();
j.expo ??= {};
j.expo.ios ??= {};
j.expo.android ??= {};
j.expo.ios.bundleIdentifier ??= "com.exposkilleval.fixture";
j.expo.android.package ??= "com.exposkilleval.fixture";
await Bun.write(f, JSON.stringify(j, null, 2) + "\n");
') >&2
# The CLI normally generates expo-env.d.ts on first `expo start`; the static
# gate runs tsc before any start, so create it up front (it provides the
# CSS-module and Metro type declarations from expo/types).
printf '/// <reference types="expo/types" />\n' >"$TMP_DIR/fixture/expo-env.d.ts"
# expo-dev-client is required for dev-build snapshots: it registers the
# expo-development-client URL scheme and enables custom Metro port connections
# (expo run:ios/android --port N opens the app via a deep link that the dev
# client handles). Without it, a custom port deep link may not open the app.
(cd "$TMP_DIR/fixture" && bunx expo install expo-dev-client) >&2
# First `expo lint` self-installs eslint + eslint-config-expo and writes
# eslint.config.js if missing; it may exit non-zero on that bootstrap run.
(cd "$TMP_DIR/fixture" && CI=1 bunx expo lint) >&2 || true
printf '.eval-static/\n' >>"$TMP_DIR/fixture/.gitignore"
mv "$TMP_DIR/fixture" "$CACHE_DIR"
fi
mkdir -p "$(dirname "$PROJECT_PATH")"
# -c uses APFS clonefile (instant, no extra disk); fall back to a plain copy.
cp -Rc "$CACHE_DIR" "$PROJECT_PATH" 2>/dev/null || cp -R "$CACHE_DIR" "$PROJECT_PATH"
# Fresh git history so `git diff` in the app shows exactly what the executor changed.
rm -rf "$PROJECT_PATH/.git"
git -C "$PROJECT_PATH" init -q
git -C "$PROJECT_PATH" add -A
git -C "$PROJECT_PATH" commit -qm "fixture: pristine $TEMPLATE template" --no-verify
echo "$PROJECT_PATH"
scripts/make-workspace.sh›
#!/usr/bin/env bash
# Create the eval workspace directory tree for a run.
#
# Usage: make-workspace.sh <workspace> <iteration> <num-evals> [configs]
# workspace Run root, e.g. /private/tmp/expo-skill-eval-<skill>
# iteration Iteration folder name, e.g. "iteration-1"
# num-evals Number of eval cases; creates eval-0 .. eval-(N-1)
# configs Space- or comma-separated config names
# (default: "with_skill without_skill")
#
# Creates <workspace>/trigger-evals/scratch and, per eval x config,
# <workspace>/<iteration>/eval-<i>/<config>/outputs. Run this once at the
# start of a run instead of ad-hoc `mkdir` so the call is covered by the
# skill's `Bash(bash *expo-skill-eval/scripts/*)` rule (the mkdir calls
# inside run as children of this script and need no permission of their own).
set -euo pipefail
WORKSPACE="${1:?usage: make-workspace.sh <workspace> <iteration> <num-evals> [configs]}"
ITERATION="${2:?usage: make-workspace.sh <workspace> <iteration> <num-evals> [configs]}"
NUM_EVALS="${3:?usage: make-workspace.sh <workspace> <iteration> <num-evals> [configs]}"
CONFIGS="${4:-with_skill without_skill}"
CONFIGS="${CONFIGS//,/ }"
mkdir -p "$WORKSPACE/trigger-evals/scratch"
for ((i = 0; i < NUM_EVALS; i++)); do
for cfg in $CONFIGS; do
mkdir -p "$WORKSPACE/$ITERATION/eval-$i/$cfg/outputs"
done
done
echo "workspace ready: $WORKSPACE/$ITERATION ($NUM_EVALS evals x [$CONFIGS])"
scripts/snapshot-android.sh›
#!/usr/bin/env bash
# Run an Expo app in Expo Go on an Android emulator and capture a screenshot.
#
# Usage: snapshot-android.sh <project-path> <output-png> [port] [settle-seconds]
#
# Env: EXPO_SKILL_EVAL_RUNNER = expo-go (default) | dev-build — dev-build runs the
# app via `expo run:android` (native dev-client compile, slow) instead of Expo Go.
#
# If no device is attached, boots the first available AVD and waits for it
# (the slow path - keep one emulator running across an eval session).
# Starts Metro with `expo start --android` (auto-installs Expo Go, sets up
# adb reverse, opens the app), waits for the bundle, settles, screenshots
# via screencap, and tears Metro down. The Metro log is saved next to the
# screenshot as <name>.metro.log. On bundle failure it still captures a
# best-effort screenshot and exits 1.
set -uo pipefail
PROJECT_PATH="${1:?usage: snapshot-android.sh <project-path> <output-png> [port] [settle-seconds]}"
OUT="${2:?usage: snapshot-android.sh <project-path> <output-png> [port] [settle-seconds]}"
PORT="${3:-8082}"
RUNNER="${EXPO_SKILL_EVAL_RUNNER:-expo-go}" # expo-go | dev-build
# Step-level logging with timestamp. Every major phase emits a marker so a
# crash report can be correlated to "which step was running when it died."
log_step() { echo "[$(date '+%H:%M:%S')] [snapshot-android] STEP: $*" >&2; }
# Track the emulator qemu PID so liveness checks in wait loops can detect a
# mid-flight crash and emit a clear "emulator crashed at step X" message.
EMULATOR_PID=""
emulator_alive() {
[[ -n "$EMULATOR_PID" ]] && kill -0 "$EMULATOR_PID" 2>/dev/null
}
log_step "start runner=$RUNNER port=$PORT project=$PROJECT_PATH"
# GPU mode is fixed to host (Metal on Apple Silicon, hardware-accelerated). The
# env-var knob was removed on purpose. If host self-aborts the emulator on a
# given machine (qemu SIGABRT deep in gfxstream/Metal — possible on Apple Silicon
# under load), change this to "guest" and bump SETTLE to ~25 (software rendering
# paints slowly). Avoid swiftshader_indirect — it hangs at boot on arm64.
GPU_MODE="host"
log_step "gpu_mode=$GPU_MODE"
SETTLE="${4:-8}"
if [[ "$RUNNER" == "dev-build" ]]; then
BUNDLE_TIMEOUT="${EXPO_SKILL_EVAL_BUNDLE_TIMEOUT:-900}" # native compile is slow
else
BUNDLE_TIMEOUT="${EXPO_SKILL_EVAL_BUNDLE_TIMEOUT:-240}"
fi
LOG="${OUT%.png}.metro.log"
mkdir -p "$(dirname "$OUT")"
# Free the port up front in case a previous run (or a crash) left Metro on it,
# so `expo start` binds the port we expect instead of silently incrementing.
# Runs inside this script, so the eval harness never needs ad-hoc lsof/kill.
lsof -ti "tcp:$PORT" -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
find_sdk_tool() {
local tool="$1" subdir="$2"
if command -v "$tool" >/dev/null; then
command -v "$tool"
return
fi
local sdk
for sdk in "${ANDROID_HOME:-}" "${ANDROID_SDK_ROOT:-}" "$HOME/Library/Android/sdk" "$HOME/Android/Sdk"; do
if [[ -n "$sdk" && -x "$sdk/$subdir/$tool" ]]; then
echo "$sdk/$subdir/$tool"
return
fi
done
return 1
}
ADB="$(find_sdk_tool adb platform-tools)" || { echo "error: adb not found (set ANDROID_HOME)" >&2; exit 1; }
# Recycle a wedged emulator before the boot check. An "offline"/unresponsive
# emulator isn't in "device" state, so the check below would otherwise reuse it
# (and hang) or stack a second emulator beside it. Graceful console shutdown
# first; force-kill only the survivors, then reset adb to clear the stale entry —
# so the boot check sees a clean slate and cold-boots a fresh (swiftshader) one.
log_step "checking for wedged/offline emulators"
STALE="$("$ADB" devices | awk 'NR>1 && $1 ~ /^emulator-/ && $2!="device" {print $1}')"
if [[ -n "$STALE" ]]; then
echo "recycling wedged emulator(s): $STALE" >&2
for serial in $STALE; do
"$ADB" -s "$serial" emu kill 2>/dev/null || true # graceful: ask the console to quit
done
sleep 2
# Still wedged after the graceful attempt? Force-kill the qemu process(es) and
# reset the adb server to drop the stale "offline" entry. pkill is broad (all
# qemu emulators), which is fine here: the eval harness runs a single emulator.
if "$ADB" devices | awk 'NR>1 && $1 ~ /^emulator-/ && $2!="device"' | grep -q .; then
pkill -9 -f qemu-system-aarch64 2>/dev/null || true
"$ADB" kill-server 2>/dev/null || true
"$ADB" start-server 2>/dev/null || true
sleep 2
fi
fi
# GPU mismatch: if an emulator is already running with a different GPU mode than
# the one we need, recycle it so the boot path cold-starts with the correct mode.
# gfxstream (host GPU) uses a gRPC channel that can crash on Metro teardown via
# a bad_function_call SIGABRT; guest GPU avoids this by keeping rendering in-guest.
CURRENT_GPU="$(ps aux | grep qemu-system-aarch64 | grep -v grep | grep -o -- '-gpu [a-z_]*' | awk '{print $2}' | head -1)"
log_step "current_gpu=${CURRENT_GPU:-none} required=$GPU_MODE"
if [[ -n "$CURRENT_GPU" && "$CURRENT_GPU" != "$GPU_MODE" ]]; then
echo "GPU mismatch: running=$CURRENT_GPU required=$GPU_MODE — recycling emulator" >&2
"$ADB" emu kill 2>/dev/null || true
sleep 2
pkill -9 -f qemu-system-aarch64 2>/dev/null || true
"$ADB" kill-server 2>/dev/null || true
"$ADB" start-server 2>/dev/null || true
sleep 2
fi
log_step "checking device availability"
if ! "$ADB" devices | awk 'NR>1 && $2=="device"' | grep -q .; then
EMULATOR="$(find_sdk_tool emulator emulator)" || { echo "error: no device attached and emulator binary not found" >&2; exit 1; }
AVD="$("$EMULATOR" -list-avds | head -1)"
if [[ -z "$AVD" ]]; then
echo "error: no AVDs configured" >&2
exit 1
fi
# Reset adb server before cold-booting. A previously crashed emulator leaves
# the server in a bad state (still trying to reconnect to dead ports 5554/5555),
# which causes adb wait-for-device to hang indefinitely on the next boot.
log_step "resetting adb server before cold boot"
"$ADB" kill-server 2>/dev/null || true
"$ADB" start-server 2>/dev/null || true
sleep 1
log_step "booting emulator avd=$AVD gpu=$GPU_MODE"
# nohup + disown so the emulator outlives this script and is reused by later
# eval runs. -no-snapshot forces a clean cold boot (no stale snapshot to replay)
# and writes no multi-GB snapshot (disk is often tight).
nohup "$EMULATOR" -avd "$AVD" -no-boot-anim \
-gpu "$GPU_MODE" -no-snapshot </dev/null >/dev/null 2>&1 &
disown
EMULATOR_PID="$(pgrep -f "emulator.*-avd $AVD" | head -1)"
log_step "emulator booted pid=${EMULATOR_PID:-unknown}"
"$ADB" wait-for-device
log_step "adb device appeared — waiting for full boot"
until [[ "$("$ADB" shell getprop sys.boot_completed 2>/dev/null | tr -d '\r')" == "1" ]]; do
if [[ -n "$EMULATOR_PID" ]] && ! kill -0 "$EMULATOR_PID" 2>/dev/null; then
echo "error: emulator pid=$EMULATOR_PID crashed during boot (check DiagnosticReports for qemu SIGABRT)" >&2
exit 1
fi
sleep 2
done
log_step "emulator fully booted"
else
EMULATOR_PID="$(pgrep -f qemu-system-aarch64 | head -1)"
log_step "reusing running emulator pid=${EMULATOR_PID:-unknown}"
fi
# --android makes the CLI install Expo Go on the device when missing. Pin the
# packager hostname to 127.0.0.1 so every exp:// URL goes through the adb
# reverse tunnel - the default LAN-IP URL is often unreachable from the
# emulator (host firewall) and leaves a stuck experience in Expo Go's task
# stack that hijacks later deep links.
# dev-build: expo run:android prebuilds + native-compiles a dev client, installs
# and launches it, and starts Metro — no Expo Go involved.
if [[ "$RUNNER" == "dev-build" ]]; then
log_step "starting expo run:android port=$PORT"
# Pass --port so Metro binds to $PORT and the health-check curl matches.
(cd "$PROJECT_PATH" && exec env -u CI REACT_NATIVE_PACKAGER_HOSTNAME=127.0.0.1 bunx expo run:android --port "$PORT") </dev/null >"$LOG" 2>&1 &
else
log_step "starting expo start --android port=$PORT"
(cd "$PROJECT_PATH" && exec env -u CI REACT_NATIVE_PACKAGER_HOSTNAME=127.0.0.1 bunx expo start --port "$PORT" --android) </dev/null >"$LOG" 2>&1 &
fi
METRO_PID=$!
# APP_PKG is set later (dev-build only) but must be in scope for cleanup.
APP_PKG=""
cleanup() {
kill "$METRO_PID" 2>/dev/null
lsof -ti "tcp:$PORT" -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
wait "$METRO_PID" 2>/dev/null
}
trap cleanup EXIT
# Resolve the app package name up front so the cleanup trap has it even on
# early exit. app.json is a fixture source file always present before Metro starts.
if [[ "$RUNNER" == "dev-build" ]]; then
APP_PKG="$(node -e "try{var a=require('$PROJECT_PATH/app.json');console.log((a.expo&&a.expo.android&&a.expo.android.package)||'com.exposkilleval.fixture')}catch(e){console.log('com.exposkilleval.fixture')}" 2>/dev/null || echo 'com.exposkilleval.fixture')"
log_step "app package=$APP_PKG"
fi
DEADLINE=$((SECONDS + BUNDLE_TIMEOUT))
STATUS=0
log_step "waiting for Metro on port=$PORT"
until curl -sf "http://localhost:$PORT/status" >/dev/null; do
if ! kill -0 "$METRO_PID" 2>/dev/null; then
echo "error: Metro (pid=$METRO_PID) exited before coming up. Log tail:" >&2
tail -40 "$LOG" >&2
exit 1
fi
if emulator_alive; then : ; elif [[ -n "$EMULATOR_PID" ]]; then
echo "error: emulator pid=$EMULATOR_PID crashed while waiting for Metro" >&2
tail -40 "$LOG" >&2
exit 1
fi
if ((SECONDS > DEADLINE)); then
echo "error: timed out after ${BUNDLE_TIMEOUT}s waiting for Metro. Log tail:" >&2
tail -40 "$LOG" >&2
exit 1
fi
sleep 2
done
log_step "Metro is up"
# Expo Go only: wait for it to install (the CLI may still be installing it).
if [[ "$RUNNER" != "dev-build" ]]; then
log_step "waiting for Expo Go install on device"
until "$ADB" shell pm list packages 2>/dev/null | grep -q host.exp.exponent; do
if ((SECONDS > DEADLINE)); then
echo "error: Expo Go was not installed on the device. Log tail:" >&2
tail -40 "$LOG" >&2
exit 1
fi
sleep 2
done
fi
# Both runners: route the bundle over the adb reverse tunnel.
log_step "setting up adb reverse tunnel port=$PORT"
"$ADB" reverse "tcp:$PORT" "tcp:$PORT"
if [[ "$RUNNER" == "dev-build" ]]; then
# expo run:android launches the dev client BEFORE Metro is ready, so it shows
# "Cannot connect to Expo CLI". Now that Metro + adb-reverse are both up,
# force-restart via explicit MainActivity to avoid the "Open in X?" dialog
# that an implicit VIEW intent triggers. expo run:android --port embeds the
# Metro URL in the build, so the dev client auto-connects on relaunch.
if [[ -n "$APP_PKG" ]]; then
log_step "launching dev-build app pkg=$APP_PKG"
"$ADB" shell am force-stop "$APP_PKG" 2>/dev/null || true
sleep 2
"$ADB" shell am start -W -n "${APP_PKG}/.MainActivity" >/dev/null 2>&1 || true
fi
else
log_step "launching Expo Go exp://127.0.0.1:$PORT"
# Force-stop so a stale experience from a previous run can't be resumed, then
# give the system a moment to finish task cleanup - launching immediately
# after force-stop races it and the new activity gets torn down too.
"$ADB" shell am force-stop host.exp.exponent
sleep 2
"$ADB" shell am start -W -a android.intent.action.VIEW -d "exp://127.0.0.1:$PORT" >/dev/null
fi
log_step "waiting for JS bundle"
until grep -q 'Bundled' "$LOG"; do
if ! kill -0 "$METRO_PID" 2>/dev/null; then
echo "error: Metro (pid=$METRO_PID) exited before bundling. Log tail:" >&2
tail -40 "$LOG" >&2
STATUS=1
break
fi
if emulator_alive; then : ; elif [[ -n "$EMULATOR_PID" ]]; then
echo "error: emulator pid=$EMULATOR_PID crashed while waiting for JS bundle" >&2
tail -40 "$LOG" >&2
STATUS=1
break
fi
if ((SECONDS > DEADLINE)); then
echo "error: timed out after ${BUNDLE_TIMEOUT}s waiting for bundle. Log tail:" >&2
tail -40 "$LOG" >&2
STATUS=1
break
fi
sleep 2
done
log_step "bundle done — settling ${SETTLE}s"
sleep "$SETTLE"
log_step "taking screenshot"
"$ADB" exec-out screencap -p >"$OUT" || STATUS=1
# Raw emulator captures are ~2800px tall and overflow the eval viewer's
# window. Downscale to a review/grading-friendly size.
if [[ -s "$OUT" ]]; then
sips -Z "${EXPO_SKILL_EVAL_MAX_DIM:-600}" "$OUT" >/dev/null 2>&1 || true
fi
log_step "done status=$STATUS screenshot=$OUT"
echo "screenshot: $OUT"
echo "metro log: $LOG"
exit "$STATUS"
scripts/snapshot-ios.sh›
#!/usr/bin/env bash
# Run an Expo app in Expo Go on an iOS simulator and capture a screenshot.
#
# Usage: snapshot-ios.sh <project-path> <output-png> [port] [settle-seconds]
#
# Env: EXPO_SKILL_EVAL_RUNNER = expo-go (default) | dev-build — dev-build runs the
# app via `expo run:ios` (native dev-client compile, slow) instead of Expo Go.
#
# Boots the newest available iPhone simulator if none is booted, starts
# Metro with `expo start --ios` (auto-installs Expo Go on the simulator and
# opens the app), waits for the bundle to load, settles, screenshots, and
# tears Metro down. The Metro log is saved next to the screenshot as
# <name>.metro.log. On bundle failure it still captures a best-effort
# screenshot (the error screen is grading evidence) and exits 1.
set -uo pipefail
PROJECT_PATH="${1:?usage: snapshot-ios.sh <project-path> <output-png> [port] [settle-seconds]}"
OUT="${2:?usage: snapshot-ios.sh <project-path> <output-png> [port] [settle-seconds]}"
PORT="${3:-8081}"
SETTLE="${4:-8}"
RUNNER="${EXPO_SKILL_EVAL_RUNNER:-expo-go}" # expo-go | dev-build
if [[ "$RUNNER" == "dev-build" ]]; then
BUNDLE_TIMEOUT="${EXPO_SKILL_EVAL_BUNDLE_TIMEOUT:-900}" # native compile is slow
else
BUNDLE_TIMEOUT="${EXPO_SKILL_EVAL_BUNDLE_TIMEOUT:-240}"
fi
LOG="${OUT%.png}.metro.log"
mkdir -p "$(dirname "$OUT")"
# Free the port up front in case a previous run (or a crash) left Metro on it,
# so `expo start` binds the port we expect instead of silently incrementing.
# Runs inside this script, so the eval harness never needs ad-hoc lsof/kill.
lsof -ti "tcp:$PORT" -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
if ! xcrun simctl list devices booted | grep -q '(Booted)'; then
UDID="$(xcrun simctl list devices available | grep 'iPhone' | tail -1 | grep -oE '[0-9A-F-]{36}')"
if [[ -z "$UDID" ]]; then
echo "error: no available iPhone simulator found" >&2
exit 1
fi
echo "booting simulator $UDID..." >&2
xcrun simctl boot "$UDID"
open -a Simulator
xcrun simctl bootstatus "$UDID"
fi
# expo-go: --ios installs Expo Go when missing; its own open is unreliable if a
# previous experience is foregrounded, so we relaunch deterministically below.
# dev-build: expo run:ios prebuilds + native-compiles a dev client, installs and
# launches it on the simulator, and starts Metro — no Expo Go involved.
if [[ "$RUNNER" == "dev-build" ]]; then
# Pass --port so Metro binds to $PORT and the health-check curl matches.
(cd "$PROJECT_PATH" && exec env -u CI bunx expo run:ios --port "$PORT") </dev/null >"$LOG" 2>&1 &
else
(cd "$PROJECT_PATH" && exec env -u CI REACT_NATIVE_PACKAGER_HOSTNAME=127.0.0.1 bunx expo start --port "$PORT" --ios) </dev/null >"$LOG" 2>&1 &
fi
METRO_PID=$!
cleanup() {
kill "$METRO_PID" 2>/dev/null
lsof -ti "tcp:$PORT" -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
wait "$METRO_PID" 2>/dev/null
}
trap cleanup EXIT
DEADLINE=$((SECONDS + BUNDLE_TIMEOUT))
STATUS=0
until curl -sf "http://localhost:$PORT/status" >/dev/null; do
if ! kill -0 "$METRO_PID" 2>/dev/null || ((SECONDS > DEADLINE)); then
echo "error: Metro did not come up. Log tail:" >&2
tail -40 "$LOG" >&2
exit 1
fi
sleep 2
done
# Expo Go only: wait for Expo Go to install, then relaunch the experience
# deterministically. dev-build skips this — expo run:ios launches the dev client.
if [[ "$RUNNER" != "dev-build" ]]; then
until xcrun simctl get_app_container booted host.exp.Exponent >/dev/null 2>&1; do
if ((SECONDS > DEADLINE)); then
echo "error: Expo Go was not installed on the simulator. Log tail:" >&2
tail -40 "$LOG" >&2
exit 1
fi
sleep 2
done
# Terminate any running Expo Go so a stale experience can't stay foregrounded,
# then open the project URL explicitly.
xcrun simctl terminate booted host.exp.Exponent 2>/dev/null || true
sleep 2
xcrun simctl openurl booted "exp://127.0.0.1:$PORT"
fi
until grep -q 'Bundled' "$LOG"; do
if ! kill -0 "$METRO_PID" 2>/dev/null; then
echo "error: Metro exited before bundling. Log tail:" >&2
tail -40 "$LOG" >&2
STATUS=1
break
fi
if ((SECONDS > DEADLINE)); then
echo "error: timed out after ${BUNDLE_TIMEOUT}s waiting for bundle. Log tail:" >&2
tail -40 "$LOG" >&2
STATUS=1
break
fi
sleep 2
done
# dev-build: after bundling, relaunch the app so it is reliably in the
# foreground. Use xcrun simctl launch (not openurl) to avoid the "Open in X?"
# confirmation dialog that xcrun simctl openurl triggers on the iOS Simulator.
# expo run:ios embeds the Metro URL in the build, so the dev client auto-connects.
if [[ "$RUNNER" == "dev-build" ]]; then
BUNDLE_ID="$(node -e "try{var a=require('$PROJECT_PATH/app.json');console.log((a.expo&&a.expo.ios&&a.expo.ios.bundleIdentifier)||'com.exposkilleval.fixture')}catch(e){console.log('com.exposkilleval.fixture')}" 2>/dev/null || echo 'com.exposkilleval.fixture')"
xcrun simctl terminate booted "$BUNDLE_ID" 2>/dev/null || true
sleep 2
xcrun simctl launch --terminate-running-process booted "$BUNDLE_ID" >/dev/null 2>&1 || true
sleep 3
fi
sleep "$SETTLE"
xcrun simctl io booted screenshot "$OUT" || STATUS=1
# Raw simulator captures are 3x retina (~2600px tall) and overflow the eval
# viewer's window. Downscale to a review/grading-friendly size.
if [[ -f "$OUT" ]]; then
sips -Z "${EXPO_SKILL_EVAL_MAX_DIM:-600}" "$OUT" >/dev/null 2>&1 || true
fi
echo "screenshot: $OUT"
echo "metro log: $LOG"
exit "$STATUS"
scripts/snapshot-web.sh›
#!/usr/bin/env bash
# Run an Expo app's web target and capture a browser screenshot. (Phase 4 -
# only for eval cases whose runtime.platforms includes "web".)
#
# Usage: snapshot-web.sh <project-path> <output-png> [port] [settle-seconds]
#
# Starts Metro with `expo start --web`, waits for the dev server to respond,
# then screenshots with Playwright's CLI at an iPhone-ish viewport. The first
# run downloads Chromium via `bunx playwright install chromium`.
set -uo pipefail
PROJECT_PATH="${1:?usage: snapshot-web.sh <project-path> <output-png> [port] [settle-seconds]}"
OUT="${2:?usage: snapshot-web.sh <project-path> <output-png> [port] [settle-seconds]}"
PORT="${3:-8081}"
SETTLE="${4:-8}"
SERVER_TIMEOUT="${EXPO_SKILL_EVAL_BUNDLE_TIMEOUT:-240}"
LOG="${OUT%.png}.metro.log"
mkdir -p "$(dirname "$OUT")"
# Free the port up front in case a previous run (or a crash) left Metro on it,
# so `expo start` binds the port we expect instead of silently incrementing.
# Runs inside this script, so the eval harness never needs ad-hoc lsof/kill.
lsof -ti "tcp:$PORT" -sTCP:LISTEN 2>/dev/null | xargs kill -9 2>/dev/null || true
bunx playwright install chromium >/dev/null 2>&1 || true
(cd "$PROJECT_PATH" && exec env -u CI bunx expo start --port "$PORT" --web) </dev/null >"$LOG" 2>&1 &
METRO_PID=$!
cleanup() {
kill "$METRO_PID" 2>/dev/null
pkill -f "expo start --port $PORT" 2>/dev/null
wait "$METRO_PID" 2>/dev/null
}
trap cleanup EXIT
DEADLINE=$((SECONDS + SERVER_TIMEOUT))
until curl -sf "http://localhost:$PORT" >/dev/null; do
if ! kill -0 "$METRO_PID" 2>/dev/null || ((SECONDS > DEADLINE)); then
echo "error: web dev server did not come up. Log tail:" >&2
tail -40 "$LOG" >&2
exit 1
fi
sleep 2
done
STATUS=0
bunx playwright screenshot \
--browser chromium \
--viewport-size "390,844" \
--wait-for-timeout "$((SETTLE * 1000))" \
"http://localhost:$PORT" "$OUT" || STATUS=1
# Keep parity with the device snapshot scripts: cap the largest dimension so
# screenshots fit the eval viewer's window.
if [[ -f "$OUT" ]]; then
sips -Z "${EXPO_SKILL_EVAL_MAX_DIM:-600}" "$OUT" >/dev/null 2>&1 || true
fi
echo "screenshot: $OUT"
echo "metro log: $LOG"
exit "$STATUS"
SKILL.md›
---
name: expo-skill-eval
description: Evaluate Expo skills in this repo end-to-end - trigger accuracy, generated code quality, and runtime screenshots on iOS simulator and Android emulator via Expo Go (web optional). Use when the user wants to eval an Expo skill, test that a skill produces working code, benchmark a skill with device screenshots, or verify a skill's output renders correctly.
version: 1.0.0
license: MIT
allowed-tools: "Read(~/.cache/expo-skill-eval/**), Read(/tmp/expo-skill-eval-*/**), Read(/private/tmp/expo-skill-eval-*/**), Write(/tmp/expo-skill-eval-*/**), Write(/private/tmp/expo-skill-eval-*/**), Edit(/tmp/expo-skill-eval-*/**), Edit(/private/tmp/expo-skill-eval-*/**), Bash(python3 /tmp/expo-skill-eval-*), Bash(python3 /private/tmp/expo-skill-eval-*), Bash(python3 *expo-skill-eval/scripts/*), Bash(tee /tmp/expo-skill-eval-*), Bash(tee /private/tmp/expo-skill-eval-*), Bash(bash *expo-skill-eval/scripts/*)"
---
# Expo Skill Eval
Evaluates skills in `plugins/expo/skills/` for trigger accuracy, generated code quality, and/or runtime rendering in Expo Go.
Requirements: macOS with Xcode (iOS simulators), Android SDK with at least one AVD, and `bun`. No other device tooling is assumed.
Workspace root: `/private/tmp/expo-skill-eval-<skill-name>/iteration-N/` (e.g. `/private/tmp/expo-skill-eval-expo-ui/iteration-4/`).
## Before starting — clarify scope
**Confirm all of the following up front, before any pipeline work — don't skip any** (only skip a given item if the request already states that choice). Batch them into `AskUserQuestion` calls of ≤4 questions each, in this order:
1. **Which skill** to eval (if not clear from the request).
2. **Prompts** — which prompts drive the eval. Built-in prompts (from the skill's eval cases) are **pre-selected all**; drop any, add a custom text prompt, or **build from an uploaded screenshot** (a target UI the skill must reproduce). See **Prompts** below.
3. **What to verify** — one multi-select of three options: Runtime + screenshots / Trigger accuracy / Code checks (no device). See *What to verify* below.
4. **Expo SDK** — latest (default, auto-detected) or a pinned version.
5. **Runner** — Expo Go (default) or development build.
6. **Platforms** — iOS / Android / web (always offer all three).
7. **Permission flag** for `claude -p` — skip-permissions (default) or accept-edits.
8. **Viewer delivery** — local only (default) or publish a shareable Artifact.
9. **If trigger accuracy is selected** — confirm the published `expo` plugin is disabled (or not installed).
Each is detailed below. Items 4–6 (SDK, runner, platforms) fit naturally in one `AskUserQuestion` call.
**If the skill to eval is not clear from the request**, list available skills from `plugins/expo/skills/` and ask which one to evaluate.
**How the skill under test is loaded — two mechanisms, one per phase** (don't pick one globally): executor runs reference it by **file path** (`SKILL_PATH = plugins/expo/skills/<skill>/SKILL.md`, read explicitly), while the trigger eval loads it as a **plugin** (`--plugin-dir plugins/expo`, so the model can auto-select it from its description). Both point at the *local, in-repo* version — that's what you're evaluating. You do **not** need any special flag to *launch* the harness session itself (the harness finds the skill by repo path); the mechanisms apply to the `claude -p` subprocesses it spawns. See steps 1 and 3 for why each phase differs. **One pre-run check (required when the trigger eval is in scope):** if the *published* `expo` plugin is installed/enabled, disable it (via `/plugin`) **before launching** the harness and re-enable after. A single disable is a global-config change that both this session and the spawned `claude -p` subprocesses inherit. Why it's required for the trigger eval: that phase loads the local skill via `--plugin-dir`, and a second installed `expo` collides with it — the model may trigger the *published* `expo:expo-ui`, and since detection only sees the tool-call name you'd silently score the published description instead of your local edits (the collision could also just error). The **executor / runtime / static** phases are *not* affected — they read the skill under test by its local `SKILL_PATH` with no `--plugin-dir` — so a run with no trigger eval can skip the disable. Disabling `expo` does **not** disable `expo-skill-eval` (a standalone project skill, not part of the `expo` plugin), so the harness stays available.
**Surface this to the user as an explicit up-front confirmation** — the same way you confirm which skill to eval. When the trigger eval is in scope, ask the user to confirm the published `expo` plugin is disabled (or not installed) *before* you start step 1; if it's still enabled, pause and have them disable it via `/plugin`. Don't run the trigger eval until they confirm — the harness can't reliably detect installed plugins on its own (reading the global plugin config or `claude plugin list` would prompt), so this is a manual confirmation, not an auto-check.
**Pick the prompts — built-in, custom, or a target screenshot.** The prompts are the *inputs* that drive the executor (with-skill and without-skill); they are separate from what you *verify*. Confirm them with `AskUserQuestion` (skip if the request already names a prompt):
- **Built-in prompts** — representative prompts you generate by reading the skill under test (its `SKILL.md` + `references/`) and `references/runtime-matrix.md`, covering the skill's standard use cases. (If the skill already ships eval cases under `evals/evals.json`, fold their `prompt` fields in too — but most skills don't, so you usually derive them.) **Pre-select all** so the default run exercises the skill's standard cases; let the user deselect any.
- **Custom text prompt** — a one-off prompt the user types. Don't spend a dedicated option slot on this: `AskUserQuestion` auto-adds a **"Type something"** / Other entry, and anything typed there becomes a custom text case.
- **Build from an uploaded screenshot** — the user gives the path to a **target screenshot** (a UI to reproduce). The executor is told to open it — `claude -p` reads PNGs with its Read tool — and build an app matching it; the case records the path as `reference_image`, and grading compares the generated app to that target (step 6). This is the strongest visual test for a UI skill: "build *this*."
**Respect `AskUserQuestion`'s 4-option-per-question cap with this priority** (the bug to avoid: the upload option silently dropped once the four slots fill up):
1. **Always reserve a slot for "Build from an uploaded screenshot."** It's the whole point of the visual eval and must never be the option that gets dropped.
2. **Don't add an explicit "Custom text prompt" option** — the auto "Type something" / Other entry already covers it.
3. Fill the remaining ≤3 slots with the built-in/representative prompts, **pre-selected**. If there are more than 3, collapse them into one pre-selected **"All built-in prompts (default)"** option and offer subset-picking in a short follow-up, so the upload option still fits.
Present it as a **multi-select**. When "Build from an uploaded screenshot" is picked, ask for the target image path in a follow-up. Each selected prompt (built-in, typed, or image) becomes one eval case (run with-skill and without-skill).
**Always confirm what to verify** unless the request makes it unambiguous. Present these options and let the user pick one or more (defaults in bold based on the skill's `references/runtime-matrix.md` entry):
| Option | What it does | When to suggest as default |
|--------|-------------|---------------------------|
| **Runtime + screenshots** | Full pipeline: fixture → executor → static gate → run the app on iOS/Android and screenshot it. The runner (Expo Go or dev build) is a separate question — don't name it here. | **Default** for any skill that renders an app screen (the `expo-go`/`dev-build` rows in `references/runtime-matrix.md`). Requires a booted simulator/emulator. |
| **Trigger accuracy** | Run realistic prompts via `claude -p`, check whether the skill is read. Measures recall (should-trigger queries only). | Always useful as a standalone check. |
| **Code checks (no device)** | `tsc --noEmit` + diff-aware lint + `expo export`, plus the grader checks the generated code against any custom expectations you provide. No device. | **Default** for `static-only` and `n/a` skills, and whenever you want to verify code patterns (correct import path, a `Host` wrapper, …) without running the app. |
**Present these as ONE multi-select question — *"What do you want to verify?"*** These are *grading dimensions* (how to judge what gets built) — distinct from the **Prompts** phase (what to build). The user may pick any combination. When a prompt is an **uploaded screenshot** (see **Prompts**), include **"Runtime + screenshots"** so the harness captures the generated app and the grader can score it against the target.
Read `references/runtime-matrix.md` to find the skill's default mode before suggesting. If the request already specifies a mode (e.g. "just check if it triggers", "run it on device"), skip the question and proceed.
**Pick the Expo SDK version — once, up front.** Detect the latest with `bash /abs/path/expo-skill-eval/scripts/latest-sdk.sh` (it prints the major, e.g. `56`; internally it uses `bun` to run `npm view expo dist-tags --json` and read the major via `JSON.parse`/`semver`, and it's covered by the bash-scripts rule — so don't run the registry query inline yourself, which would prompt). Then confirm with `AskUserQuestion`: default to that latest SDK, or let the user pin an older one (e.g. to reproduce a version-specific issue). Use the chosen version everywhere the fixture is built — pass it as the `<sdk>` arg to `make-fixture.sh` and write it into each eval case's `runtime.sdk`. If the request already names a version ("eval on SDK 54"), skip detection and use it.
**Default to the latest** — it stays compatible with the Expo Go that `expo start` installs on the device. Pinning an SDK *older* than the device's installed Expo Go makes `expo start` try to prompt "Install the recommended Expo Go version?"; with no TTY (the snapshot scripts read stdin from `/dev/null`) it dies with `Input is required, but 'npx expo' is in non-interactive mode` and **every snapshot fails**. So only pin an older SDK when you also pre-install a matching Expo Go on the simulator/emulator — otherwise stick with latest.
**Pick the runner — Expo Go (default) or a development build.** Ask with `AskUserQuestion` (skip if the request already says which):
- **Expo Go (default)** — the snapshot scripts run the app with `expo start --ios` / `expo start --android` as-is. Fast (no native compile), and it runs anything Expo Go bundles (including `@expo/ui` on SDK 56+). Cannot run custom native code (expo-modules, config plugins, native deps not in Expo Go).
- **Development build** — the snapshot scripts run `expo run:ios` / `expo run:android` instead, compiling a native dev client per fixture. Use this for skills whose output needs custom native code (the cases that would otherwise be `static-only`). Much slower — `expo run` prebuilds and natively compiles each fixture (minutes, especially the first), and needs the full iOS/Android build toolchain — so only choose it when the skill actually requires native code. **Disk-heavy:** each fixture's native build is multi-GB. The snapshot phase runs `clean-fixture.sh` after each fixture to keep peak usage to ~one build, but still prefer fewer eval cases and a **single platform** for dev-build runs, and keep a few GB free. `clean-fixture.sh` removes the per-fixture build *output* (`node_modules`, `ios`, `android`, `.expo`, `dist`, and the fixture's iOS DerivedData) and keeps the app source + git. The lever for dev-build disk is **fewer eval cases + one platform** — it only reclaims per-fixture build output and never touches shared dependency caches, so nothing gets re-downloaded.
Pass the choice to the snapshot scripts via the `EXPO_SKILL_EVAL_RUNNER` env var (`expo-go` default, or `dev-build`), and reflect it in each eval case's `runtime.mode` (`expo-go` or `dev-build`). See step 5.
**Pick the platforms — always ask, regardless of skill.** Offer iOS / Android / web (multi-select) with `AskUserQuestion`; default to iOS + Android, but always present web as an option — don't pre-filter by skill. **Web is a valid choice for most skills**: `@expo/ui`'s *universal* components (`Host`, `Row`, `Column`, `Button`, `List`, …) render on web, as do `expo-dom`, NativeWind/Tailwind, API routes, and plain React Native. The only thing that won't show on web is a *platform-specific* native tree (`@expo/ui/swift-ui` or `@expo/ui/jetpack-compose`), which renders blank there — and that blank is itself a useful signal, so it's still the user's call. Web runs via `snapshot-web.sh` (`expo start --web` + Playwright/Chromium) **regardless of the runner** (`expo run` is native-only; there's no web dev build), and it's the least-exercised path. Write the chosen set into each eval case's `runtime.platforms` and have `run_snapshots.py` loop them.
**Confirm how `claude -p` subprocesses run — once, before starting.** Ask with `AskUserQuestion` whether they may run with `--dangerously-skip-permissions`, then apply the same answer to every subprocess this run (never re-prompt mid-run):
- **Skip permissions (recommended)** — pass `--dangerously-skip-permissions`. Each subprocess runs unattended inside a throwaway fixture under `/private/tmp/expo-skill-eval-*` and can write files and run setup commands without prompting.
- **Accept edits only** — pass `--permission-mode acceptEdits` instead. Bash/installs are auto-denied (no TTY), so some evals may produce partial output.
A bare `claude -p` with neither flag can't write files at all. If the request already states a preference ("skip permissions", "don't use the dangerous flag"), skip the question.
**Confirm how to deliver the results viewer — once, up front.** Publishing to claude.ai is outward-facing, so never do it mid-run by surprise; ask in the same up-front `AskUserQuestion` (alongside the permission flag):
- **Local only (default)** — `generate_viewer.py` writes `viewer.html` and opens it in the local browser. Nothing leaves the machine.
- **Publish a shareable Artifact** — additionally render the viewer to a claude.ai Artifact (a default-private web page the user can share with teammates) at the very end. Only do this if the user opts in here.
If the request already says whether to share/publish, skip the question. See the **Viewer** section for the publish mechanics.
## Eval case schema
You generate the run's eval cases — one per chosen prompt — and write them to `<workspace>/iteration-N/evals.json` (the viewer reads them from there). Each case extends the standard skill-creator eval-case shape with a `runtime` block and visual expectations:
```json
{
"id": 1,
"prompt": "Build me a settings screen with a dark mode toggle and a list of options",
"expected_output": "Working Expo Router screen",
"expectations": [
"Uses Expo Router file-based routing",
"TypeScript compiles with no errors"
],
"runtime": {
"mode": "expo-go",
"platforms": ["ios", "android"],
"sdk": "56"
},
"visual_expectations": [
"No red error screen or Expo Go error overlay on any platform",
"A settings screen with a visible toggle control is rendered"
]
}
```
- `runtime.mode`: how the eval runs after the static gate —
- `"expo-go"`: run in Expo Go (`expo start --<platform>`) and screenshot. Fast, JS-only. **Default.**
- `"dev-build"`: build a native dev client (`expo run:<platform>`) and screenshot. For skills whose output uses custom native code; much slower (native compile per fixture).
- `"static-only"`: stop after the static gate — for skills that produce no UI, or when you don't want to run a device at all (CI).
Consult `references/runtime-matrix.md` for which repo skills support which mode. (`dev-build` lets you actually run skills that previously had to be `static-only` for needing native code.)
- `runtime.platforms`: subset of `ios`, `android`, `web` — chosen up front (always offered, not gated on the skill; see **Before starting**). Defaults to `["ios", "android"]`.
- `runtime.sdk`: Expo SDK major for the fixture app — set it to the version chosen up front (see **Before starting — clarify scope**). Omit to use the latest template.
- `reference_image` (optional — **image prompt**): absolute path to a **target screenshot** the skill must reproduce. When set, the executor is told to open it (via its Read tool) and build a matching app, and the grader scores how closely the generated app reproduces it (step 6) on top of the usual expectations. Set in the **Prompts** phase via "build from an uploaded screenshot."
An image-prompt case is a normal case with `reference_image` set; enable "Runtime + screenshots" so the harness captures the result to compare against the target:
```json
{
"prompt": "Build an app whose UI matches the attached reference screenshot.",
"reference_image": "/abs/path/to/target.png",
"runtime": { "mode": "expo-go", "platforms": ["ios"], "sdk": "56" },
"visual_expectations": ["Matches the reference's layout, components, and color treatment"]
}
```
## Pipeline per eval case
**Orchestration model — on the main thread you run `python3 <orchestrator>` and almost nothing else.** Every phase is driven by a small Python orchestrator you `Write` into the workspace and run with `python3 /private/tmp/expo-skill-eval-<skill>/<phase>.py` (covered by the `python3` rule). The orchestrators are the **only** place the `scripts/*.sh` files are invoked — always via `subprocess.run(["bash", "<scripts>/<name>.sh", …])`, which runs as a child of `python3` and needs no rule of its own — and the only place parallelism, logging, and directory creation live. So on the main thread you only ever: **Write** orchestrators, **run** them with `python3`, **inspect** outputs with the `Read`/`Glob`/`Grep` tools, and **spawn the grader subagent**. Never put a command inside a chained/backgrounded/piped shell construct, and never run ad-hoc `mkdir`/`ls`/`cat`/`tail`/`echo` — that is what prompts. (A single standalone `bash …/scripts/<name>.sh …` is fine for one-off manual debugging, e.g. re-running one flaky snapshot, but the pipeline itself goes through the orchestrators.) **Run each orchestrator in the foreground** — let the tool call block until it finishes; the orchestrators already parallelize *within* a phase, so you don't need to overlap phases. Do **not** shell-background a phase with `… & echo "$!"` / `wait` (the `&`, `echo`, and `wait` segments have no rule and prompt). If you genuinely must run a phase while continuing other work, use the **Bash tool's `run_in_background` parameter** on a plain `python3 <orchestrator> 2>&1 | tee <ws>/…log` call — never hand-rolled shell `&`. **Expect exactly one permission prompt at the very start:** the first `Write` into the workspace. `allowed-tools` can suppress `Bash`/`Read` but not `Write`/`Edit`, so choose **"allow all edits in this directory for the session"** on that first prompt — it covers every orchestrator, `evals.json`, and viewer file for the whole run.
### 0. Workspace setup
Create the run's directory tree once, with the workspace script — **never with ad-hoc `mkdir`** (a raw `mkdir` prompts: there is no `mkdir` rule, and a `"$WORKSPACE/…"` variable can't match a path glob anyway):
```bash
bash /abs/path/expo-skill-eval/scripts/make-workspace.sh /private/tmp/expo-skill-eval-<skill> iteration-N <num-evals>
```
This creates `trigger-evals/scratch` and `iteration-N/eval-<i>/{with_skill,without_skill}/outputs` for every eval. It is covered by `Bash(bash *expo-skill-eval/scripts/*)`, and the `mkdir`s inside run as children of the script (no rule of their own). After this, every other directory is made by the scripts/orchestrators that need it (`make-fixture.sh`, the executor orchestrator's `os.makedirs`, the snapshot scripts) or by the `Write` tool auto-creating parents — so you never need another `mkdir`.
### 1. Trigger eval (should-trigger only)
Write a `run_trigger_eval_real.py` script under the workspace's `trigger-evals/` directory. Use **only `"should_trigger": true` queries** — the expo plugin is a family of complementary skills, so multiple skills triggering on the same prompt is not a failure. Measure recall only: realistic prompts that should use the skill, scored by trigger rate.
The script should run `claude -p <query>` per query (with `--output-format=stream-json --verbose --include-partial-messages`, `CLAUDECODE` stripped from the env, and the permission flag confirmed up front in **Before starting — clarify scope**) and detect whether the target skill was triggered by watching for its `Skill` or `Read` tool call in the stream. Note: `--include-partial-messages` requires both `--output-format=stream-json` and `--verbose` — omitting either causes an immediate CLI error.
**Load the skill under test — pass `--plugin-dir` to every trigger subprocess.** The trigger eval measures whether the skill's *description* makes the model reach for it, so the subprocess must have the **local** skill (the version with your edits) loaded. A `claude -p` subprocess does **not** inherit the parent session's `--plugin-dir`, so add it explicitly: `--plugin-dir <plugin-root>`, where `<plugin-root>` is the **absolute** path to the plugin directory that owns the skill — the `plugins/expo` ancestor containing `.claude-plugin/plugin.json` (e.g. `--plugin-dir /Users/.../skills/plugins/expo`). It must be absolute: the subprocess runs from the throwaway `scratch/` cwd, so a relative `plugins/expo` won't resolve — and a missing plugin dir silently loads nothing, which masquerades as a 0% trigger rate. Then watch for the skill triggering under its plugin-qualified name (`<plugin>:<skill>`, e.g. `expo:expo-ui`). Two caveats: (1) if the **published** `expo` plugin is also installed globally, disable it (via `/plugin`) for the run and re-enable after — otherwise two copies of `expo` collide in the subprocess and the model may trigger the *published* `expo:expo-ui`, silently scoring its description instead of your local edits (trigger detection only sees the tool-call name, so it can't tell the copies apart; dev checkouts usually don't have it installed). (2) Never make a synthetic duplicate of the skill — a real loaded copy always wins, so the synthetic harness scores 0%. (Executors are unaffected by an installed plugin: they read the local `SKILL_PATH` directly and pass no `--plugin-dir`.)
Run each query's subprocess from an empty throwaway cwd (e.g. `trigger-evals/scratch/`), not the repo root. A should-trigger prompt like "build me a settings screen" can make the subprocess write files, and with `--dangerously-skip-permissions` those writes would otherwise land in the skills repo. Trigger detection only needs the skill's `Skill`/`Read` call to appear in the stream — it doesn't need a fixture — so any incidental writes are throwaway.
Set a per-query subprocess timeout of at least **300 seconds**. A 180s limit is too short — some queries cause the model to start generating code before triggering the skill, which pushes total runtime past 3 minutes.
Run trigger evals once per skill, not per code eval case.
### 2. Fixture
Each executor run gets a fresh Expo app, created by `scripts/make-fixture.sh <app-path> <sdk> [clean|full]`:
```bash
scripts/make-fixture.sh <workspace>/iteration-N/eval-X/<config>/app <sdk> # blank app (default)
scripts/make-fixture.sh <workspace>/iteration-N/eval-X/<config>/app <sdk> full # keep example tabs
```
The script creates the app with `bunx create-expo-app -t default@sdk-<version>` (or the latest template when no version is given) once per SDK version + variant, caches it under `~/.cache/expo-skill-eval/fixtures/`, and clones the cache with APFS copy-on-write — so the first run per variant pays the install cost and every later run is near-instant. The default `clean` variant runs the template's `reset-project` script, so executors start from a blank app and every screen in the output is theirs — a much cleaner grading signal. Use `full` only when the eval prompt assumes an existing app (e.g. "I have an app with two tabs..."). The script also resets git inside the clone, so `git diff` in the app shows exactly what the executor changed (useful evidence for the grader).
**Build fixtures sequentially, then fan out executors — never create fixtures concurrently.** `make-fixture.sh` shares a cache under `~/.cache/expo-skill-eval/fixtures/` keyed by SDK+variant. If two runs both find the cache cold and call `bunx create-expo-app` at the same time, bun's link step collides and one fails with `EEXIST` / "could not determine executable to run for package create-expo-app". So in the executor orchestrator (step 3), create **all** fixtures one at a time first — a plain Python loop calling `subprocess.run(["bash", "<scripts>/make-fixture.sh", app, sdk, variant])` (where `sdk` is the version chosen up front) — *then* fan out the `claude -p` executors with a `ThreadPoolExecutor`. Sequential creation is cheap: only the first fixture per SDK+variant pays the install cost; the rest are ~1s APFS clones. (And never fan fixtures out with ad-hoc shell like `make-fixture.sh A & make-fixture.sh B & wait` — the `&`/`wait` segments prompt; the sequential Python loop avoids both the race and the prompt.)
### 3. Generate (executor subagents)
Run executors as `claude -p` subprocess calls from a Python script, **not** via the `Agent` tool. The `Agent` tool spawns subagents with their own permission context — file edits inside the fixture app will prompt the user. A `claude -p` subprocess is a separate process outside the permission system entirely (the same pattern the trigger eval harness uses).
Write a Python script to `/private/tmp/expo-skill-eval-<skill>/run_executors.py`. **First create every run's fixture in a sequential loop** — `subprocess.run(["bash", "<scripts>/make-fixture.sh", app, sdk, variant], …)` one at a time (concurrent creation races the shared bun cache — see step 2). **Then** run the with-skill and without-skill `claude -p` calls in parallel via a `ThreadPoolExecutor`. Both phases live inside Python (covered by the `python3` rule), so nothing runs as ad-hoc shell on the main thread. Each executor prompt must include:
- The skill path (with-skill runs only) and the eval prompt.
- **Image-prompt cases (`reference_image` set):** the absolute path to the target screenshot plus an instruction like "Open the reference screenshot at `<path>` with your Read tool and build an app whose UI matches it as closely as you can — layout, components, spacing, and colors." (`claude -p` renders PNGs read this way, so the executor can actually see the target.)
- The fixture app path: "Make your changes inside `<app-path>`. The project already exists and has dependencies installed. Use absolute paths for all file operations."
- "Before writing any files, inspect the project layout — run `ls`, read `package.json` and `app.json` — to find the correct routes directory. Recent SDK default templates place Expo Router routes in `src/app/`; older ones use `app/` at the project root — inspect to confirm which this fixture uses."
- "Do NOT start the dev server, boot simulators, or take screenshots — the harness does that after you finish."
- Where to save a short summary of what was built.
Flags for the `claude -p` subprocess:
- Strip `CLAUDECODE` from the environment (`env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}`) — otherwise `claude -p` hangs silently when nested inside a running Claude Code session.
- A permission flag, confirmed with the user up front (see **Before starting — clarify scope**): either `--dangerously-skip-permissions` or `--permission-mode acceptEdits`. Bake the chosen flag into the generated script. A bare `claude -p` with neither flag can't write files — it has no TTY to approve the edit and emits code as text instead.
- **Do NOT pass `--plugin-dir` to executors** (unlike the trigger eval). The with-skill run already reads the skill by its absolute `SKILL_PATH`, so it tests the local content directly; and the without-skill run must have **no** skill available at all — loading the plugin would let the skill auto-trigger and contaminate the baseline. Keeping executors path-based also cleanly separates the two questions: the executor measures *content quality* (is the skill useful once read?), the trigger eval measures *triggering* (does the description get it picked?).
Capture stdout/stderr per run to a log file next to the fixture for grading evidence. Set timeout to 900s per executor — with-skill runs read multiple reference files before coding and regularly take 5–10 minutes.
### 4. Static gate
Write `run_static.py` and run it with `python3`. For each eval/config app it calls `subprocess.run(["bash", "<scripts>/check-static.sh", app, "ios,android"], capture_output=True, …)` across a `ThreadPoolExecutor` (static gates are independent — run them concurrently *inside Python*, never with shell `&`/`wait`), and writes each result to `eval-<i>/<config>/static.json` (exit code + captured output) for the grader.
`check-static.sh` runs `tsc --noEmit`, `expo lint`, and `expo export` for the listed platforms. A passing export catches most import/syntax/missing-module failures without touching a device; a failing export short-circuits step 5 with a clean FAIL — record it and have the snapshot orchestrator skip that app.
### 5. Run + screenshot (serial across evals)
Write `run_snapshots.py` and run it with `python3`. Simulators and emulators are shared resources, so this orchestrator runs **serially** (no thread pool): for each app that passed the static gate, and each platform, it `os.makedirs` the `outputs/` dir and calls `subprocess.run(["bash", "<scripts>/snapshot-<platform>.sh", app, f"{outputs}/<platform>.png", port], env={**os.environ, "EXPO_SKILL_EVAL_RUNNER": runner}, …)`. Pass the port as a positional argument: use `8081` for iOS and `8082` for Android — `expo run:ios/android --port N` is supported and using separate ports lets you run both platforms without port collisions if you ever parallelize. Screenshots land in the run's `outputs/` directory so the viewer renders them inline.
**Reclaim disk after each fixture — essential for `dev-build` runs.** Once all selected platforms for an app are captured (and *before* the next fixture builds), call `subprocess.run(["bash", "<scripts>/clean-fixture.sh", app])`. Each `expo run:<platform>` leaves multi-GB native build output (iOS Pods + DerivedData, Android Gradle build); without this, evals × configs × iterations pile up and fill the disk mid-run (the instability you'll see is the disk filling). `clean-fixture.sh` removes the heavy regenerable dirs (`node_modules`, `ios`, `android`, `.expo`, `dist`) and the fixture's iOS DerivedData, keeping the app source + git so the grader's `git diff` still works. With serial snapshots + per-fixture cleanup, peak disk stays at ~one fixture's build instead of all of them. (Harmless for `expo-go` runs too — they just have little to reclaim.)
`runner` is the up-front choice (`expo-go` default, or `dev-build`). The snapshot scripts honor `EXPO_SKILL_EVAL_RUNNER`: `expo-go` launches with `expo start --<platform>` (and the Expo Go install/deep-link dance); `dev-build` launches with `expo run:<platform> --port <port>`, which compiles+installs a native dev client and skips the Expo Go steps. The scripts already default the `dev-build` timeout to 900s, but bump `EXPO_SKILL_EVAL_BUNDLE_TIMEOUT` higher if the first native compile needs it. `make-fixture.sh` pre-installs `expo-dev-client` in every fixture so the dev-client URL scheme is registered before `expo run` tries to deep-link the app open.
**Snapshot scripts always capture the initial route `/`.** They open the app via a deep link and take one screenshot — they cannot tap or navigate. Design eval prompts so the feature under test renders at the root route. If the executor places the main UI behind a navigation action (e.g. an "Open Settings" button on the index), the snapshot will miss the feature entirely and all visual expectations will fail.
Each `snapshot-<platform>.sh` **frees its Metro port on startup** (kills any stale process left on it by a crashed prior run) and tears Metro down on exit — so you never need to run `lsof`/`kill`/`pkill` yourself to clear ports (that would prompt, and it's already handled). It then starts Metro, waits for the "Bundled" line in the Metro log, settles, captures a screenshot, and tears Metro down. iOS boots the newest available iPhone simulator if none is booted; Android boots the first AVD if no device is attached (the slow path — boot once and reuse across the whole iteration). Android first **recycles a wedged/`offline` emulator** (graceful `adb emu kill`, then force-kill + adb reset) so a half-dead instance can't poison the run, and boots with **hardware GPU** (`-gpu host`, Metal-accelerated on Apple Silicon). If `host` self-aborts the emulator on a given machine (qemu `SIGABRT` deep in gfxstream/Metal — possible on Apple Silicon under load), edit `GPU_MODE` in `snapshot-android.sh` to a software mode (`guest` renders reliably but slowly — bump the settle; avoid `swiftshader_indirect`, which **hangs at boot** on arm64). `snapshot-web.sh` runs only when `platforms` includes web. Each script writes a Metro log next to the screenshot (`<name>.metro.log`) — include it in the grader's inputs. If a script exits non-zero it still attempts a best-effort screenshot (an error screen is evidence too). **dev-build relaunch:** after Metro is up, the scripts relaunch the app via `xcrun simctl launch` (iOS) and `adb shell am start -n <pkg>/.MainActivity` (Android) — both avoid the "Open in X?" system dialog that a URL-scheme deep link triggers on first launch.
After all screenshots for the iteration are captured, always generate the viewer — pass the workspace root to the checked-in script:
```bash
python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill>
```
It writes `viewer.html` into the workspace root (one level above `iteration-N/`) and opens it in the browser itself (via `webbrowser.open`) — so no separate `open` command (and no `Bash(open:*)` rule) is needed. See the **Viewer** section below.
### 6. Grade
Spawn a grader subagent in the foreground. Its prompt must include:
- The eval prompt, expectations list, and visual_expectations from the eval case.
- The instructions in `agents/visual-grader.md` (screenshot grading, redbox detection).
- The screenshot files, Metro logs, and the step-4 `static.json` as inputs.
- **Image-prompt cases** (case has `reference_image`): also include the **target screenshot** (`reference_image`), `references/design-rubric.md`, and the fixture's `git diff`. Tell the grader to compare the generated screenshot(s) to the target and emit the `reference_match` + `quality` blocks below.
The grader writes `grading.json` next to the outputs with this shape:
```json
{
"score": 8.5,
"max_score": 9,
"expectations": [
{"text": "...", "passed": true, "evidence": "..."}
],
"reference_match": {
"score": 7, "max": 10,
"evidence": "ios.png vs target.png: same two-section grouped list + toggle; accent color differs (blue vs target's green); row spacing tighter than target"
},
"quality": {
"dimensions": [
{"name": "Layout & hierarchy", "score": 2, "max": 3, "evidence": "ios.png: …"}
],
"subtotal": 17,
"max": 24,
"summary": "…"
},
"user_notes_summary": {"needs_review": false, "notes": ""}
}
```
Visual expectations go into the same `expectations` array with evidence naming the screenshot file and describing what is visible. The `reference_match` block (how closely the generated app reproduces the target screenshot) and the `quality` block (design-rubric scores from `references/design-rubric.md`) are emitted **only for image-prompt cases** — or when a quality grade is explicitly requested. Omit both for plain text-prompt runs.
## Rollout phases
Build out and debug the pipeline in this order — each phase is independently useful:
1. **Static**: steps 1–4 only (`runtime.mode: "static-only"` for everything). No devices needed; CI-friendly.
2. **iOS**: add `snapshot-ios.sh` to the loop. `simctl` is the most scriptable target.
3. **Android**: add `snapshot-android.sh`. Emulator boot is the slowest part — keep one emulator running for the whole session.
4. **Web**: add `snapshot-web.sh` for skills that target web (uses Playwright via `bunx`; first run downloads Chromium).
## Practical notes
- **Temp locations**: all eval workspaces go under `/private/tmp/expo-skill-eval-<skill-name>/iteration-N/`. Everything in this run — `Read`, `Write`, `Edit`, and `Bash` — is covered by the `allowed-tools` frontmatter, so a correctly-loaded skill runs prompt-free.
- **Permission rule forms (why this skill stays prompt-free)**: the rule *syntax* matters and the two tool families behave differently:
- **`Bash(...)` rules — path-scoped to the skill's own code (no broad interpreters).** `Bash(python3 /private/tmp/expo-skill-eval-*)` (plus the `/tmp` alias) runs the Python orchestrators you generate under the workspace; `Bash(python3 *expo-skill-eval/scripts/*)` runs the checked-in `scripts/generate_viewer.py`; `Bash(tee /private/tmp/expo-skill-eval-*)` (+ `/tmp`) lets `python3 … 2>&1 | tee <workspace>/…log` write a log without prompting; `Bash(bash *expo-skill-eval/scripts/*)` runs only this skill's `scripts/*.sh`. Because every path is pinned, the escape hatches stay denied: `python3 -c …`, `bash -c …`, `tee /etc/…`, and running code anywhere else do **not** match (verified empirically — a scoped rule allows `bash <dir>/run.sh` but blocks `bash -c …` and any other path). Commands the scripts call internally — `bunx`, `xcrun simctl`, `adb`, `git`, `mkdir`, `expo` — are children of the script, not Bash tool calls, so they need no rule. Do **not** run ad-hoc `mkdir`/`ls`/`find`/`cat`/`grep` from the main thread (they have no rule and prompt — and a raw `mkdir "$WORKSPACE/…"` can't match a path glob because the path is an unexpanded variable): create the directory tree with `make-workspace.sh` (step 0), let orchestrators create their own dirs (`os.makedirs`), and **inspect results with the `Read`/`Glob`/`Grep` tools** (no Bash rule needed).
- **Bash rule matching (tested, non-obvious):** a Bash rule is a gitignore-style glob over the command string. `*` matches any run of characters **including `/` and spaces** and works **mid-pattern** — so `Bash(python3 /private/tmp/expo-skill-eval-*)` matches `python3 /private/tmp/expo-skill-eval-x/run.py 2>&1`, and `Bash(bash *expo-skill-eval/scripts/*)` matches `bash /any/abs/path/expo-skill-eval/scripts/foo.sh args`. Two gotchas that burned earlier attempts: `**` is matched **literally** (never use it in a Bash rule), and the `:*` suffix only works right after the command token (`Bash(python3:*)`) — **not** after a partial path (`Bash(python3 /path-:*)` does not match). Compound commands split on `|`, `&&`, `||`, `;`, `&` and each segment needs its own matching rule.
- **`Read` rules suppress prompts; `Write`/`Edit` rules do *not*.** This is a Claude Code asymmetry (not a pattern bug, and not reload — in a session where the `Bash`/`Read` rules from this same frontmatter are clearly working, `Write` still prompts): file creation/editing always goes through Claude Code's edit-approval flow regardless of `allowed-tools`. The frontmatter still scopes `Read`/`Write`/`Edit` to `…/expo-skill-eval-*/**` (both the `/tmp` and `/private/tmp` forms, since macOS doesn't auto-resolve the symlink) as documentation and a guardrail, but those `Write`/`Edit` entries won't silence the prompt on their own. **Practical consequence:** at the start of a run you get **one** Write prompt for the workspace — choose **"Yes, allow all edits in this directory for the session"** and every later orchestrator / `evals.json` / viewer write under that workspace goes through silently. That single directory approval, not a rule, is what makes file-writing prompt-free.
- **Reload after editing frontmatter — a full restart, not `/reload-skills`.** `allowed-tools` is read once when the skill loads at session start; `/reload-skills` reloads the skill *body* but does **not** reliably refresh the permission rules. After editing this file, **quit Claude Code entirely and start a new session**, then re-run the skill — otherwise a stale (cached) ruleset keeps prompting even though the file on disk is correct.
- **Grader subagents** run with their own permission context and will still prompt for file access — that is expected and separate from the main thread's rules.
- **Calling eval scripts — one standalone command, never chained.** Invoke each script as its own Bash call with an absolute path: `bash /abs/path/expo-skill-eval/scripts/snapshot-ios.sh arg1 arg2` (covered by `Bash(bash *expo-skill-eval/scripts/*)`). Do **not** combine it with `&`, `&&`, `||`, `;`, `wait`, `tail`, `head`, or `echo` — compound commands are checked per segment, and those extra segments have no rule, so the whole thing prompts even though the `bash …/scripts/…` part is allowed. (The one allowed pipe is `… 2>&1 | tee <workspace>/…log`, since the scoped `tee` rule covers it.) Need parallelism or output trimming? Put it in a Python orchestrator (covered by `python3 /…/expo-skill-eval-*`), which runs scripts via `subprocess` across a `ThreadPoolExecutor`. Inspect results with the `Read`/`Glob`/`Grep` tools, not `cat`/`ls`/`grep`. **General rule: under this skill's tight scoping, any ad-hoc shell the agent improvises will prompt — the fix is to move it into a script/orchestrator (or use the scoped `tee`), never to broaden a rule.**
- **Inspecting outputs (screenshots, logs, files) — use tools, not shell.** To find files use the **Glob** tool (e.g. `/private/tmp/expo-skill-eval-<skill>/iteration-N/**/ios.png`); to view them use the **Read** tool — Read renders PNGs visually, which is exactly what you need to confirm a screenshot rendered. To search file contents use **Grep**. Never use `find`/`ls`/`cat` for this: they prompt, and `find … -exec …` is deliberately *not* allowed because its `-exec` can run anything (e.g. `-exec rm`). These tools are scoped and prompt-free; reach for them every time you'd otherwise type `find`/`ls`/`cat`.
- **Generated Python scripts**: write orchestration/aggregation scripts under the workspace (e.g. `/private/tmp/expo-skill-eval-<skill>/aggregate.py`) and run them with `python3` (covered by `Bash(python3 /private/tmp/expo-skill-eval-*)`). The viewer is the exception — it's the checked-in `scripts/generate_viewer.py`, run via `Bash(python3 *expo-skill-eval/scripts/*)`. `Write` auto-creates parent dirs but prompts the first time — approve the workspace directory once (see the `Write`/`Edit` note above). Capture output either by having the script write its own log or via `python3 … 2>&1 | tee <workspace>/…log` (covered by the scoped `tee` rule); read logs back with the `Read` tool. Don't use `python3 -c …` for setup (the scoped rule only matches a workspace script *path*, so a bare `-c` prompts).
- **Trigger evals vs installed plugin**: detect the real installed skill name (e.g. `expo:expo-ui`) in the stream — a synthetic-duplicate harness always scores 0% when the real plugin is installed because the model picks the genuine skill over the synthetic copy.
- **Benchmark aggregation**: save each run's `grading.json` + `timing.json` under `eval-<N>/<config>/run-1/`. Write a Python aggregation script under the workspace and run it with `python3`.
- **Expo Go ceiling**: anything requiring custom native code (expo-module, App Clips, brownfield) cannot run in Expo Go. Use `static-only` mode for those — see `references/runtime-matrix.md` before writing eval cases for a skill (note: `@expo/ui` *does* run in Expo Go on SDK 56+).
- **API-route skills**: instead of a screenshot, verify with `curl` against the route while Metro is up; record the response as an output file for grading.
- **Timing data**: capture token counts and duration into `timing.json` immediately after each executor run — it is not recoverable later. To capture token counts, add `--output-format=stream-json --verbose` to the executor `claude -p` call and parse the `message_start` / `message_delta` events from the log. Without these flags the log only contains prose and elapsed seconds are the only recoverable metric.
- **First-launch dialogs**: Expo Go occasionally shows a one-time prompt on a fresh simulator. If a screenshot captures a dialog instead of the app, re-run the snapshot script (it reopens the URL) and re-capture.
## Viewer
After taking screenshots, always generate and open the HTML viewer so the user can see results immediately without being asked. The viewer is the checked-in `scripts/generate_viewer.py` — run it with the workspace root as its argument:
```bash
python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill>
```
It writes a self-contained `/private/tmp/expo-skill-eval-<skill>/viewer.html` and opens it in the browser itself (`webbrowser.open`). What it renders:
- A tab per iteration (`iteration-*` under the workspace root; remembers the last active tab in `localStorage`).
- For each eval case (read from `<iteration>/evals.json`): side-by-side with_skill / without_skill columns, each showing static-gate status, score, the platform screenshots (click to zoom; embedded as base64 `data:` URIs so the file is self-contained), the expectation list with PASS/FAIL badges, and reviewer notes.
- For **image-prompt cases** (a `grading.json` with `reference_match` / `quality`): the **target screenshot** beside the generated ones, the `reference_match` score (generated vs target), the `quality` rubric per config (one bar per dimension with its score/max plus the subtotal), and the **quality delta** (with_skill − without_skill subtotal) in the summary bar alongside the correctness delta.
- A summary bar with with_skill %, without_skill %, and delta.
- A trigger accuracy table when `trigger-evals/trigger_results.json` exists.
- A dark background with color-coded scores (green ≥85%, amber ≥65%, red below).
### Publishing the viewer (only if opted in up front)
The local `viewer.html` is always generated. **Only when the user chose "Publish a shareable Artifact"** in the up-front confirmation, additionally render it to a claude.ai Artifact at the very end — never publish without that opt-in (it's outward-facing and a published page can be cached/indexed). Mechanics:
- The `Artifact` tool wraps the file in its own `<!doctype html>…<head></head><body>` skeleton, so the file you hand it must be **page content only** — inline `<style>`/`<script>`, base64 `data:` images, and a `<title>`, but **no** `<!DOCTYPE>/<html>/<head>/<body>` tags of its own (a full standalone document gets double-wrapped and renders wrong).
- The script emits an Artifact-friendly variant when you add `--artifact`: `python3 /abs/path/expo-skill-eval/scripts/generate_viewer.py /private/tmp/expo-skill-eval-<skill> --artifact` writes `viewer_artifact.html` (same content, skeleton stripped, no browser open). Pass that file to the `Artifact` tool (`favicon: "📊"`), not the standalone one.
- The viewer is already self-contained (base64 screenshots, inline CSS/JS), so it satisfies the Artifact CSP (no external hosts).
## References
- `references/runtime-matrix.md` — per-skill runtime applicability (expo-go vs static-only, platform notes).
- `agents/visual-grader.md` — screenshot grading instructions for the grader subagent.