heygen-com/hyperframesVérifier avant exécution
SKILL DETAIL
media-use
heygen-com/hyperframes/media-use
Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.
Installations · 9 188Voir la source
Installation
npx skills add https://github.com/heygen-com/hyperframes --skill media-use
Fichiers du skill
SKILL.md
Dernière synchronisation · 22 sept. 2026
.gitignore›
eval-report.html
audio/assets/sfx/CREDITS.md›
# SFX Credits
All sound effects in this directory are sourced from [Pixabay](https://pixabay.com/sound-effects/) and used under the [Pixabay Content License](https://pixabay.com/service/license-summary/).
The Pixabay license allows free use for commercial and non-commercial purposes without attribution, but attribution is appreciated and given here for transparency.
## Files
The following `.mp3` files are bundled with this skill:
- `chime.mp3`
- `click.mp3` / `click-soft.mp3`
- `error.mp3`
- `glitch-1.mp3` / `glitch-2.mp3` / `glitch-3.mp3`
- `impact-bass-1.mp3` / `impact-bass-2.mp3`
- `key-press.mp3`
- `notification.mp3`
- `ping.mp3`
- `pop.mp3`
- `riser.mp3`
- `sparkle.mp3`
- `typing.mp3`
- `whoosh.mp3` / `whoosh-short.mp3` / `whoosh-cinematic.mp3`
See `manifest.json` for per-file metadata (duration, energy character, recommended use).
## License
All files are distributed under the [Pixabay Content License](https://pixabay.com/service/license-summary/), which permits:
- Commercial and non-commercial use
- Modification and remixing
- Redistribution as part of derivative works (such as videos rendered with HyperFrames)
without any attribution requirement.
audio/assets/sfx/manifest.json›
{
"chime": {
"file": "chime.mp3",
"duration": 2.5,
"description": "Soft melodic chime — gentle positive beat: success/confirmation or a lighthearted transition. Sync to the visual moment."
},
"click-soft": {
"file": "click-soft.mp3",
"duration": 0.37,
"description": "Quiet short click — low-key UI tap / soft selection. Short accent, sync exactly to the on-screen action."
},
"click": {
"file": "click.mp3",
"duration": 0.37,
"description": "Crisp UI click — button press, toggle, selection. Short accent, sync exactly to the on-screen action."
},
"error": {
"file": "error.mp3",
"duration": 1.62,
"description": "Negative / error tone — failure state, a 'wrong' beat, or a glitchy interruption."
},
"glitch-1": {
"file": "glitch-1.mp3",
"duration": 2.64,
"description": "Punchy digital glitch — hard-cut accent or sudden reveal. Trigger on the hit; let the decay bleed into the next shot (J-cut)."
},
"glitch-2": {
"file": "glitch-2.mp3",
"duration": 3.5,
"description": "Harsh, longer glitch — chaotic / jarring transition or a distorted reveal."
},
"glitch-3": {
"file": "glitch-3.mp3",
"duration": 3.1,
"description": "Low-key glitch texture — subtle digital shift, minimal transition that sits under other audio."
},
"impact-bass-1": {
"file": "impact-bass-1.mp3",
"duration": 2.12,
"description": "Bass impact hit — logo/hero snap, headline slam. Trigger on the visual landing; decay carries into the next shot (J-cut)."
},
"impact-bass-2": {
"file": "impact-bass-2.mp3",
"duration": 2.59,
"description": "Bass impact with a short swell — brief anticipation then a deep hit. Place so the peak lands on the reveal."
},
"key-press": {
"file": "key-press.mp3",
"duration": 0.4,
"description": "Single key press — one keystroke / terminal-input beat. Short accent, sync to the typed character."
},
"notification": {
"file": "notification.mp3",
"duration": 2.46,
"description": "Notification chime — alert, message-in, toast/badge appears. Sync to the element entering."
},
"ping": {
"file": "ping.mp3",
"duration": 1.32,
"description": "Sharp electronic ping — punchy accent on a key reveal or data point. Sync to the beat."
},
"pop": {
"file": "pop.mp3",
"duration": 0.72,
"description": "Quick pop — element appear/spawn, chip/tag/badge in. Small precise accent, sync to the pop-in."
},
"riser": {
"file": "riser.mp3",
"duration": 10.03,
"description": "Long cinematic riser (~10s build, peak at the end). Trigger at (climax_time − 10.03s) so it crests exactly on the reveal."
},
"sparkle": {
"file": "sparkle.mp3",
"duration": 1.8,
"description": "Bright sparkle / shimmer — magical reveal or 'shine' highlight on a hero element. Sync to the highlight."
},
"typing": {
"file": "typing.mp3",
"duration": 1.5,
"description": "Typing burst (~1.5s of keys) — keyboard / code typing reveal, text-being-typed beat. Start as the text begins typing."
},
"whoosh-cinematic": {
"file": "whoosh-cinematic.mp3",
"duration": 5.54,
"description": "Cinematic whoosh build (~5.5s) — sweeping scene transition. Align so the swell peaks on the cut."
},
"whoosh-short": {
"file": "whoosh-short.mp3",
"duration": 0.57,
"description": "Short whoosh — quick swipe/slide accent, fast element move, snappy transition. Sync to the motion."
},
"whoosh": {
"file": "whoosh.mp3",
"duration": 0.57,
"description": "Punchy whoosh/impact — fast reveal or hard transition accent. Sync to the motion."
}
}
audio/references/bgm.md›
# Background music (BGM)
One music bed per composition, produced by the shared audio engine (`scripts/audio.mjs` → `scripts/lib/bgm.mjs`). Two routes, chosen by the engine's one switch — whether a HeyGen credential is present:
- **HeyGen retrieval — the default when credentialed.** Search HeyGen's music catalog by mood, download the top track. No generation; same `~/.heygen` / `$HEYGEN_API_KEY` credential as TTS.
- **Local generation (Lyria → MusicGen) — the fallback when there is no credential** (or when asked for explicitly). Generate a WAV from a mood prompt. There is **no `npx hyperframes bgm` command**; the engine spawns `scripts/lyria-recipe.py` or an inline MusicGen script directly.
> **Run the Preflight first — no credential is not a green light to silently generate locally.** Before generating, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen's music library, or continue offline with local generation). This applies to a one-off "generate a BGM" request just as much as inside a full workflow.
## Driving it from the request
`audio_request.json` → `bgm: { mode?, query?, prompt? }`:
- **`mode`** — `retrieve | generate | none`. Omit for **auto** (retrieve when credentialed, else generate). An **explicit** `retrieve` is strict: no credential ⇒ skip, never a detached generate (so a caller with no `wait-bgm` step, e.g. product-launch, can't get a pending job it won't await).
- **`query`** — the mood, used for retrieval and as a fallback prompt seed (e.g. a storyboard's `music:` field, falling back to `message` → `arc` → `"calm cinematic underscore"`).
- **`prompt`** — an explicit full prompt for generation; omit and the engine infers one (see Mood inference). Optional `blob` / `archetype` / `arc` feed that inference.
## HeyGen retrieval (default)
`searchSounds(query, "music", { limit: 5 })` → `GET /audio/sounds?query=<mood>&type=music&limit=5`. Take the top result (ranked by `score`), download its presigned `audio_url` → `assets/bgm/track.mp3`. Synchronous. No match → skip (BGM is optional; never fail the render over it). Cue written to `audio_meta.json`:
```jsonc
{
"path": "assets/bgm/track.mp3",
"volume": 0.12,
"mode": "retrieve",
"query": "calm cinematic underscore",
"duration_s": 42.0,
}
```
`volume` comes from the engine's `bgmDefaultVolume()`: `BGM_BED_VOLUME` (currently `0.12` ≈ -18 dB — a bed under the voice) under narration, `BGM_SILENT_VOLUME` (currently `0.9`) for a silent film (no voice). Tune those constants in `scripts/lib/bgm.mjs`, not call sites. An explicit `volume` in `audio_meta.json` always overrides this default. `bgm_pending` is `false` — the file is on disk when the engine returns.
For short launch videos, do not assume the beginning of the retrieved file is the best edit point. Check the opening against later five-second sections. If the track starts with a quiet build but a later section has a stronger, clean musical entrance, trim from that section and apply a short fade-in and longer fade-out. Repeat this check whenever the composition duration changes; the final music file must cover the full cut without a silent tail.
## Local generation (fallback) — Lyria → MusicGen
Spawned **detached** so voice work isn't blocked; `audio_meta.bgm_pending: true` and `bgm_pid` / `bgm_log` are set until it finishes. **Run `scripts/wait-bgm.mjs` before assembling** — it polls the output file / process / log, detects crashes, and writes `bgm_status.json` (`status: ready | failed | timeout | disabled`). A failed/absent track is simply omitted; it never blocks voice/SFX.
| Order | Provider | Env / deps | Speed | Quality |
| ----- | ------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------------- | --------------------------- |
| 1 | Google Lyria RealTime | `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `google-genai` (auto-installed on demand) | Real-time stream (≈ requested duration) | Production-grade |
| 2 | MusicGen (`facebook/musicgen-small`) | Python `transformers + torch + soundfile + numpy` (~300 MB first run; auto-installed) | Slow on CPU; fast on Apple MPS / CUDA | Decent; prompt-only control |
Output → `assets/bgm/track.wav`, target = total voice duration. MusicGen generates **one** seed clip (≤28–30s, under the decoder's positional limit) then crossfade-loops it up to the target (or trims down if shorter), avoiding per-segment seams. Backend selection is by what can actually **run**: Lyria only when `import google.genai` succeeds, else MusicGen; if neither can be made to run, BGM is skipped (voice + SFX still render).
## Mood inference (the generate prompt)
`inferBgmPrompt()` in `scripts/lib/bgm.mjs`: an explicit `prompt` wins; otherwise industry-keyword **base** → narrative-**archetype** shape → emotional-**arc** tiebreaker.
| Match in `blob` / `query` | Base prompt | BPM |
| ------------------------------------------------------ | --------------------------------------------------------------------------- | --- |
| `crypto / nft / web3 / defi / token / blockchain` | atmospheric electronic, deep bass, futuristic synths, restrained percussion | 100 |
| `finance / fintech / bank / payment / invest / wealth` | calm cinematic, soft strings, subtle piano, restrained percussion | 92 |
| `creative / agency / design / studio / art / brand` | playful electronic, warm pads, light percussion | 115 |
| _(default: SaaS / tech / platform)_ | uplifting corporate tech, bright modern piano with synth pads | 108 |
Archetype then reshapes the arc — PAS → "MINOR to MAJOR" build; BAB / future-pacing → aspirational rising; feature-cascade → +10 BPM driving; demo-loop → −8 BPM minimal. The emotional arc breaks remaining ties (tension→relief, excitement, trust/reassurance).
## Lyria knobs (direct recipe use)
The engine bakes BPM / scale into the **prompt text** (via the inference above) and passes only `--output` / `--duration` / `--prompt` to the recipe. If you invoke `scripts/lyria-recipe.py` directly you can also set: `--bpm` (90–110 calm, 110–130 energetic), `--brightness` (0–1, ≥0.7 promotional), `--density` (0–1, higher = fuller), `--scale` (`MAJOR` / `MINOR` / `PENTATONIC` / …), `--negative-prompt` (styles to exclude). MusicGen ignores all of these — put the mood in the prompt.
## Failure modes
| Failure | Behavior |
| --------------------------------------------- | ---------------------------------------------------------------------------------------- |
| No music match (retrieve) | `bgm: null`, anomaly logged. Render proceeds without BGM. |
| Explicit `retrieve`, no credential | Skipped (no silent generate fallback). Use `mode: generate` or omit `mode` for auto. |
| Neither Lyria nor MusicGen can run (generate) | `bgm` disabled with a `pip install …` hint. Voice + SFX still render. |
| Generate still rendering at assemble time | `bgm_pending: true`; `wait-bgm.mjs` waits/checks and writes `bgm_status.json` first. |
| Generate crashed | `wait-bgm.mjs` → `bgm_status.json { status: "failed" }`; the `<audio>` track is omitted. |
BGM failure never blocks a render.
audio/references/captions/authoring.md›
# Captions
<!-- registry-items: allow=max-width,data-composition-src,hyperframes-registry,blend-mode,caption-style,font-family -->
**The live search is the source of truth for what the registry has.** The table(s) below are a hand-maintained sample and under-cover by design: run `npx hyperframes catalog --query "<what you want>" --json` — it needs nothing installed — before concluding the registry lacks something. Item names here are checked against `registry/registry.json` by `bun run lint:skills`.
Before authoring: confirm the transcript came from the right Whisper model. CLI default `small.en` silently translates non-English audio — see [`../transcribe.md`](../transcribe.md) → "Language Rule" and [`transcript-handling.md`](transcript-handling.md) for the mandatory quality check.
Analyze spoken content to determine caption style. If user specifies a style, use that. Otherwise, detect tone from the transcript.
## Transcript Source
```json
[
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
]
```
`id` (`w0`, `w1`, …) is the stable reference for per-word overrides and is added by `hyperframes transcribe`. It's optional for backwards compatibility with hand-authored transcripts. See [`../transcribe.md`](../transcribe.md) → "Output Shape" for how this is produced, and [`transcript-handling.md`](transcript-handling.md) for cleanup before consumption.
## Style Detection (When No Style Specified)
Read the full transcript before choosing. Four dimensions:
**1. Visual feel** — corporate→clean; energetic→bold; storytelling→elegant; technical→precise; social→playful.
**2. Color palette** — dark+bright for energy; muted for professional; high contrast for clarity; one accent color.
**3. Font mood** — heavy/condensed for impact; clean sans for modern; rounded for friendly; serif for elegance.
**4. Animation character** — scale-pop for punchy; gentle fade for calm; word-by-word for emphasis; typewriter for technical.
## Per-Word Styling
Scan for words deserving distinct treatment:
- **Brand/product names** — larger size, unique color
- **ALL CAPS** — scale boost, flash, accent color
- **Numbers/statistics** — bold weight, accent color
- **Emotional keywords** — exaggerated animation (overshoot, bounce)
- **Call-to-action** — highlight, underline, color pop
- **Marker highlight** — for beyond-color emphasis (highlight sweep, circle, burst, scribble, sketchout), see `hyperframes-animation/rules/css-marker-patterns.md`.
## Script-to-Style Mapping
| Tone | Font mood | Animation | Color | Size |
| ------------ | ------------------------ | ---------------------------------- | --------------------------- | ------- |
| Hype/launch | Heavy condensed, 800-900 | Scale-pop, back.out(1.7), 0.1-0.2s | Bright on dark | 72-96px |
| Corporate | Clean sans, 600-700 | Fade+slide, power3.out, 0.3s | White/neutral, muted accent | 56-72px |
| Tutorial | Mono/clean sans, 500-600 | Typewriter/fade, 0.4-0.5s | High contrast, minimal | 48-64px |
| Storytelling | Serif/elegant, 400-500 | Slow fade, power2.out, 0.5-0.6s | Warm muted tones | 44-56px |
| Social | Rounded sans, 700-800 | Bounce, elastic.out, word-by-word | Playful, colored pills | 56-80px |
## Word Grouping
- **High energy:** 2-3 words. Quick turnover.
- **Conversational:** 3-5 words. Natural phrases.
- **Measured/calm:** 4-6 words. Longer groups.
Break on sentence boundaries, 150ms+ pauses, or max word count.
## Positioning
- **Landscape (1920x1080):** Bottom 80-120px, centered
- **Portrait (1080x1920):** Lower middle ~600-700px from bottom, centered
- Never cover the subject's face
- `position: absolute` — never relative
- One caption group visible at a time
## Text Overflow Prevention
Use `window.__hyperframes.fitTextFontSize()`:
```js
var result = window.__hyperframes.fitTextFontSize(group.text.toUpperCase(), {
fontFamily: "Outfit",
fontWeight: 900,
maxWidth: 1600,
});
el.style.fontSize = result.fontSize + "px";
```
Options: `maxWidth` (1600 landscape, 900 portrait), `baseFontSize` (78), `minFontSize` (42), `fontWeight`, `fontFamily`, `step` (2).
CSS safety nets: `max-width` on container, `overflow: visible` (**not** `hidden` — hidden clips scaled emphasis words and glow effects), `position: absolute`, explicit `height`. When per-word styling uses `scale > 1.0`, compute `maxWidth = safeWidth / maxScale` to leave headroom.
**Container pattern:** Full-width absolute container, centered. Do **not** use `left: 50%; transform: translateX(-50%)` — causes clipping at composition edges.
## Caption Exit Guarantee
Every group **must** have a hard kill after exit animation:
```js
tl.to(groupEl, { opacity: 0, scale: 0.95, duration: 0.12, ease: "power2.in" }, group.end - 0.12);
// `tl.set` is an instant flip, not a tween — safe to set `visibility` here (core's "no animating
// visibility" rule applies to tweens, which can't smoothly interpolate non-numeric values anyway).
tl.set(groupEl, { opacity: 0, visibility: "hidden" }, group.end);
```
Self-lint after building timeline — place **before** `window.__timelines[id] = tl` so it runs at composition init:
```js
GROUPS.forEach(function (group, gi) {
var el = document.getElementById("cg-" + gi);
if (!el) return;
tl.seek(group.end + 0.01);
var computed = window.getComputedStyle(el);
if (computed.opacity !== "0" && computed.visibility !== "hidden") {
console.warn(
"[caption-lint] group " + gi + " still visible at t=" + (group.end + 0.01).toFixed(2) + "s",
);
}
});
tl.seek(0);
```
## Pre-Built Caption Components
Before building caption styles from scratch, check the registry — 15 ready-to-use caption components cover the most common styles. Install with `npx hyperframes add <name>` and wire as a sub-composition via `data-composition-src` (see `hyperframes-registry`).
```bash
npx hyperframes catalog --tag caption-style # list all caption components
npx hyperframes add caption-highlight # install a specific one
```
| Style | Component | Best for |
| ------------------------- | ---------------------------- | ---------------------------- |
| TikTok-style highlight | `caption-highlight` | Social, high-energy |
| Karaoke pill | `caption-pill-karaoke` | Music, lyric videos |
| Cinematic editorial | `caption-editorial-emphasis` | Documentary, storytelling |
| Glitch / cyber | `caption-glitch-rgb` | Tech, gaming |
| Full-screen slam | `caption-kinetic-slam` | Hype, announcements |
| Neon glow | `caption-neon-glow` | Night, club, neon aesthetics |
| Neon accent (multi-color) | `caption-neon-accent` | Colorful, playful |
| Wipe reveal | `caption-clip-wipe` | Clean, modern |
| Gradient fill | `caption-gradient-fill` | Vibrant, eye-catching |
| Matrix decode | `caption-matrix-decode` | Sci-fi, tech reveals |
| Emoji pop | `caption-emoji-pop` | Social, casual |
| Parallax layers | `caption-parallax-layers` | Depth, cinematic |
| Particle burst | `caption-particle-burst` | Celebration, impact keywords |
| Lava texture | `caption-texture` | Bold, dramatic |
| Weight shift | `caption-weight-shift` | Elegant, typographic |
Related: `caption-blend-difference` (tagged `text` / `blend-mode`, not `caption-style`, so it won't appear under the filter above) auto-inverts text against any background via `mix-blend-mode: difference` — useful when the background is busy or unpredictable.
Browse all with previews: [hyperframes.heygen.com/catalog](https://hyperframes.heygen.com/catalog)
Caption components ship with transparent backgrounds — they're pure overlays. If the underlying video is bright or busy, add a contrast layer (e.g. a semi-transparent dark div) in the host composition beneath the caption sub-composition, not inside the component itself.
## Further References
- [`motion.md`](motion.md) — karaoke, marker effects, audio-reactive modulation, scatter exits.
- [`transcript-handling.md`](transcript-handling.md) — input formats, quality checks, cleaning, external API fallback.
- `hyperframes-animation/rules/css-marker-patterns.md` — marker highlighting (deterministic, fully seekable).
## Constraints
- Deterministic. No `Math.random()`, no `Date.now()`.
- Sync to transcript timestamps.
- One group visible at a time.
- Every group must have a hard `tl.set` kill at `group.end`.
- Fonts: the compiler auto-embeds only its **built-in mapped set** (Inter, Roboto, Montserrat, …) — for those, just declare `font-family` in CSS. Any **other** font (a brand/custom font like `TT Norms Pro`, or a non-Latin CJK/Devanagari family) is **not** auto-supplied: it needs an `@font-face` pointing at a real `.woff2` shipped with the project, or the text silently falls back to a generic font in the render. Don't assume a `font-family` you can see locally will render — the render machine is a clean headless Chrome with no installed fonts.
audio/references/captions/motion.md›
# Dynamic Caption Techniques
You are here because SKILL.md told you to read this file before writing animation code. Pick your technique combination from the table below based on the energy level you detected from the transcript, then implement using standard GSAP patterns.
## Technique Selection by Energy
| Energy level | Highlight | Exit | Cycle pattern |
| ------------ | ------------------------------------- | ------------------- | ----------------------------------------- |
| High | Karaoke with accent glow + scale pop | Scatter or drop | Alternate highlight styles every 2 groups |
| Medium-high | Karaoke with color pop | Scatter or collapse | Alternate every 3 groups |
| Medium | Karaoke (subtle, white only) | Fade + slide | Alternate every 3 groups |
| Medium-low | Karaoke (minimal scale change) | Fade | Single style, vary ease per group |
| Low | Karaoke (warm tones, slow transition) | Collapse | Alternate every 4 groups |
**All energy levels use karaoke highlight as the baseline.** The difference is intensity — high energy gets accent color + glow + 15% scale pop on active words, low energy gets a gentle white shift with 3% scale.
**Emphasis words always break the pattern.** When a word is flagged as emphasis (emotional keyword, ALL CAPS, brand name), give it a stronger animation than surrounding words (larger scale, accent color, overshoot ease). This creates contrast.
**Marker highlight modes add a visual layer on top of karaoke.** For emphasis words that need more than color/scale, add a marker-style effect: highlight sweep, circle, burst, scribble, or sketchout. See `hyperframes-animation/rules/css-marker-patterns.md` for implementation details. Match mode to energy: burst for hype, circle for key terms, highlight for standard, scribble for subtle.
## Audio-Reactive Captions (Mandatory for Music)
**If the source audio is music (vocals over instrumentation, beats, any musical content), you MUST extract audio data and add audio-reactive animations.** This is not optional — music without audio reactivity looks disconnected. Even low-energy ballads get subtle bass pulse and treble glow.
No special wiring is needed. The group loop already iterates over every caption group to build entrance, karaoke, and exit tweens. At that point, read the audio data for each group's time range and use it to modulate the group's animation intensity with regular GSAP tweens.
```js
// Load audio data inline (same pattern as TRANSCRIPT)
var AUDIO = JSON.parse(audioDataJson); // { fps, totalFrames, frames: [{ bands: [...] }] }
GROUPS.forEach(function (group, gi) {
var groupEl = document.getElementById("cg-" + gi);
if (!groupEl) return;
// Read peak energy for this group's time range
var startFrame = Math.floor(group.start * AUDIO.fps);
var endFrame = Math.min(Math.floor(group.end * AUDIO.fps), AUDIO.totalFrames - 1);
var peakBass = 0;
var peakTreble = 0;
for (var f = startFrame; f <= endFrame; f++) {
var frame = AUDIO.frames[f];
if (!frame) continue;
peakBass = Math.max(peakBass, frame.bands[0] || 0, frame.bands[1] || 0);
peakTreble = Math.max(peakTreble, frame.bands[6] || 0, frame.bands[7] || 0);
}
// Modulate entrance — louder groups enter bigger and glowier
tl.to(
groupEl,
{
scale: 1 + peakBass * 0.06,
textShadow:
"0 0 " + Math.round(peakTreble * 12) + "px rgba(255,255,255," + peakTreble * 0.4 + ")",
duration: 0.3,
ease: "power2.out",
},
group.start,
);
// Reset at exit so audio-driven values don't persist
tl.set(groupEl, { scale: 1, textShadow: "none" }, group.end - 0.15);
});
```
This shapes the animation at build time, not playback time — no per-frame callbacks, no `tl.call()` loops, no async fetch timing issues. Loud groups come in with more weight and glow; quiet groups come in soft. The audio data modulates _how much_, the content determines _what_.
Keep audio reactivity subtle — 3-6% scale variation and soft glow. Heavy pulsing makes text unreadable.
To generate the audio data file:
```bash
python3 skills/hyperframes-creative/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json
```
## Combining Techniques
Don't use the same highlight animation on every group — cycle through styles using the group index. Don't combine multiple competing animations on the same word at the same timestamp. Vary techniques across groups to match the content's pace changes.
**Marker highlight effects** layer well with karaoke — use karaoke for the word-by-word reveal, then add a marker effect on emphasis words only. For example: karaoke highlights each word in white, but brand names get a yellow highlight sweep and stats get a red circle. Cycle marker modes across groups for visual variety.
## Runtime Tools
Caption motion uses standard HyperFrames runtime APIs. Use the canonical sources:
- **GSAP timeline + tween syntax** — `hyperframes-animation/adapters/gsap.md` (eases, position parameter, performance)
- **`window.__hyperframes.fitTextFontSize` / `pretext`** — `hyperframes-core/references/determinism-rules.md` → Layout Contract (overflow prevention, per-frame text measurement)
- **Audio data extraction** — generate via `python3 skills/hyperframes-creative/scripts/extract-audio-data.py audio.mp3 --fps 30 --bands 8 -o audio-data.json`, then load inline as shown in "Audio-Reactive Captions" above
audio/references/captions/transcript-handling.md›
# Transcript Guide
For the `transcribe` CLI invocation, the `.en`-translates-non-English rule, and whisper model selection, see [`../transcribe.md`](../transcribe.md). This file covers what to do with the resulting transcript when authoring captions: input formats, mandatory quality checks, cleaning code, external-API fallbacks.
## Supported Input Formats
The CLI auto-detects and normalizes these formats:
| Format | Extension | Source | Word-level? |
| --------------------- | --------- | --------------------------------------------------------------------------- | ----------------- |
| whisper.cpp JSON | `.json` | `hyperframes init --video`, `hyperframes transcribe` | Yes |
| OpenAI Whisper API | `.json` | `openai.audio.transcriptions.create({ timestamp_granularities: ["word"] })` | Yes |
| SRT subtitles | `.srt` | Video editors, subtitle tools, YouTube | No (phrase-level) |
| VTT subtitles | `.vtt` | Web players, YouTube, transcription services | No (phrase-level) |
| Normalized word array | `.json` | Pre-processed by any tool | Yes |
**Word-level timestamps produce better captions.** SRT/VTT give phrase-level timing, which works but can't do per-word animation effects.
## Transcript Quality Check (Mandatory)
After every transcription, **read the transcript and check for quality issues before proceeding.** Bad transcripts produce nonsensical captions. Never skip this step.
### What to look for
| Signal | Example | Cause |
| ---------------------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
| Music note tokens (`♪`, `�`) | `{ "text": "♪" }` or `{ "text": "�" }` | Whisper detected music, not speech |
| Garbled / nonsense words | "Do a chin", "Get so gay", "huh" | Model misheard lyrics or background noise |
| Long gaps with no words | 20+ seconds of only `♪` tokens | Instrumental section — expected, but high ratio means speech is being missed |
| Repeated filler | Many "huh", "uh", "oh" entries | Model is hallucinating on music |
| Very short word spans | Words with `end - start < 0.05` | Unreliable timestamp alignment |
### Automatic retry rules
**If more than 20% of entries are `♪`/`�` tokens, or the transcript contains obvious nonsense words, the transcription failed.** Do not proceed with the bad transcript. Instead:
1. **Retry with `medium.en`** if the original used `small.en` or smaller:
```bash
npx hyperframes transcribe audio.mp3 --model medium.en
```
2. **If `medium.en` also fails** (still >20% music tokens or garbled), tell the user the audio is too noisy for local transcription and suggest:
- Providing lyrics manually as an SRT/VTT file
- Using an external API (OpenAI or Groq Whisper — see below)
3. **Always clean the transcript** before building captions — filter out `♪`/`�` tokens and entries where `text` is a single non-word character. Only real words should reach the caption composition.
### Cleaning a transcript
After transcription (even with a good model), strip non-word entries:
```js
var raw = JSON.parse(transcriptJson);
var words = raw.filter(function (w) {
if (!w.text || w.text.trim().length === 0) return false;
if (/^[♪�\u266a\u266b\u266c\u266d\u266e\u266f]+$/.test(w.text)) return false;
if (/^(huh|uh|um|ah|oh)$/i.test(w.text) && w.end - w.start < 0.1) return false;
return true;
});
```
For model-selection guidance by content type, see [`../transcribe.md`](../transcribe.md) → "Picking a model by content type".
## Using External Transcription APIs
For the best accuracy, use an external API and import the result:
**OpenAI Whisper API** (recommended for quality):
```bash
# Generate with word timestamps, then import
curl https://api.openai.com/v1/audio/transcriptions \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-F [email protected] -F model=whisper-1 \
-F response_format=verbose_json \
-F "timestamp_granularities[]=word" \
-o transcript-openai.json
npx hyperframes transcribe transcript-openai.json
```
**Groq Whisper API** (fast, free tier available):
```bash
curl https://api.groq.com/openai/v1/audio/transcriptions \
-H "Authorization: Bearer $GROQ_API_KEY" \
-F [email protected] -F model=whisper-large-v3 \
-F response_format=verbose_json \
-F "timestamp_granularities[]=word" \
-o transcript-groq.json
npx hyperframes transcribe transcript-groq.json
```
## If No Transcript Exists
1. Check the project root for `transcript.json`, `.srt`, or `.vtt` files.
2. If none found, run [`../transcribe.md`](../transcribe.md) — pick the starting model from "Picking a model by content type" there.
3. Run the quality check above. If it fails, retry with a larger model or fall back to manual lyrics / external API.
audio/references/remove-background.md›
# Background Removal
Make a transparent overlay (typical: a talking head over an arbitrary scene). Uses `u2net_human_seg` (MIT).
```bash
npx hyperframes remove-background subject.mp4 -o transparent.webm # default: VP9 + alpha
npx hyperframes remove-background subject.mp4 -o transparent.mov # ProRes 4444 (editing)
npx hyperframes remove-background portrait.jpg -o cutout.png # single-image cutout
npx hyperframes remove-background subject.mp4 -o subject.webm \
--background-output plate.webm # both layers, one pass
npx hyperframes remove-background subject.mp4 -o transparent.webm --device cpu
npx hyperframes remove-background --info # detected providers
```
## Output Format
- **`.webm` (VP9 alpha)** — default. Plug straight into `<video>` for Chrome-native transparent playback (~1 MB / 4s @ 1080p).
- **`.mov` (ProRes 4444)** — round-trip in editors (Premiere / Resolve / DaVinci). ~50 MB / 4s.
- **`.png`** — single-image cutout.
## Quality (`--quality`)
Controls VP9 encoder CRF only — segmentation quality is fixed. Higher quality keeps the cutout's RGB closer to the source MP4 (important when overlaying the cutout on its own source).
| Preset | CRF | When |
| ---------- | --- | --------------------------------------------- |
| `fast` | 30 | Iterating, smaller files, looser color match |
| `balanced` | 18 | **Default**; visually identical for most uses |
| `best` | 12 | Master / final delivery, tightest color match |
## Device (`--device`)
`auto` (default) picks CoreML on Apple Silicon, CUDA when available, otherwise CPU. Force with `--device cpu | coreml | cuda`. CUDA requires `HYPERFRAMES_CUDA=1` plus a GPU-enabled `onnxruntime-node` build. Use `--info` to inspect detected providers without rendering.
## Compositing patterns — pick the right one
The cutout WebM is a **re-encoded copy** of the source MP4's RGB. What sits behind it matters.
| Pattern | Behind the cutout | Result |
| -------------------------------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Cutout over a different scene** (most common) | Static image, gradient, unrelated video | Looks great. Single RGB source for the subject. |
| **Cutout over its own source mp4** (text-behind-subject) | Same mp4 the cutout came from | At `balanced` doubling is barely visible; at `fast` you'll see color shift / edge halo. Use `best` for masters. |
| **Cutout over a different take of the same person** | Footage of the same subject | **Two overlapping people. Don't do this.** |
## Text-behind-subject pattern (two non-obvious rules)
Putting a headline behind a presenter cutout:
```html
<video
src="presenter.mp4"
id="bg"
data-start="0"
data-duration="6"
data-track-index="0"
muted
playsinline
></video>
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
<div class="cutout-wrap" style="position:absolute; inset:0; z-index:3; opacity:0">
<video
src="presenter.webm"
data-start="0"
data-duration="6"
data-track-index="1"
muted
playsinline
></video>
</div>
```
```js
// Flip the wrapper's opacity at the cut, NOT the video's
tl.set(".cutout-wrap", { opacity: 1 }, 3.3);
```
Two rules that are easy to miss:
1. **Wrap the cutout `<video>` in a non-timed `<div>` and animate the wrapper's opacity, not the video element's.** The framework forces `opacity: 1` on active clips (any element with `data-start` / `data-duration`), so animating the video's opacity directly is silently overridden. The wrapper has no `data-*` attributes, so it's owned by your CSS / GSAP.
2. **Both videos use `data-start="0"` and `data-media-start="0"`** so the framework decodes them in sync from t=0. Late-mounting the cutout (`data-start=3.3`) introduces a seek + warm-up that lands a frame off the base mp4 — visible as one frame of misalignment at the cut.
## Layer separation (`--background-output`)
Emits a **second** transparent video alongside the cutout: same source RGB, alpha is `255 - mask` instead of `mask`. The cutout has the subject opaque; the plate has the surroundings opaque (with a transparent hole where the subject was). Use it when text / graphics need to live **between** the two layers.
| File | Alpha is… | Use it for |
| -------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------- |
| `-o subject.webm` | mask — subject opaque, background transparent | Foreground layer (top) |
| `--background-output plate.webm` | inverse mask — surroundings opaque, subject transparent | Bottom layer; place text / graphics between this and the subject |
Both share the same `--quality` and run from a single inference pass — only encode cost roughly doubles. Only valid for video inputs with `.webm` / `.mov` outputs.
**Hole-cut, not inpainted.** The subject region in `plate.webm` is fully transparent — composite something opaque under it to fill the hole.
**Single test for whether `--background-output` is the right tool:** _will anything ever be visible through the subject's silhouette where the subject used to be?_ If no, you don't need the plate — `subject.webm` alone over a different background is enough.
### Use case → right tool
| Use case | Right tool |
| ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Text/graphics between the cutout and the plate (this command's reason for existing) | **Hole-cut** (`--background-output`) |
| Subject onto an unrelated scene | Just `subject.webm`; ignore the plate |
| Show the room _without_ the person, alone over no other content | **Clean plate** — needs an inpainter (LaMa, ProPainter, E2FGVI). Not this command. |
| Replace the subject with a different subject | **Clean plate** — same as above |
### Canonical 3-layer template (plate + content + cutout)
Ship just the two transparent layers and let arbitrary content live between them — no original mp4 needed:
```html
<!-- z=1 plate: surroundings opaque, subject silhouette transparent -->
<video
src="plate.webm"
data-start="0"
data-duration="6"
data-track-index="0"
muted
playsinline
></video>
<!-- z=2 your content lives between the layers -->
<h1 id="headline" style="z-index:2; ...">MAKE IT IN HYPERFRAMES</h1>
<!-- z=3 cutout floats the subject back on top -->
<div class="cutout-wrap" style="position:absolute; inset:0; z-index:3">
<video
src="subject.webm"
data-start="0"
data-duration="6"
data-track-index="1"
muted
playsinline
></video>
</div>
```
Functionally equivalent to the text-behind-subject pattern above, but doesn't require shipping the original mp4 — the plate replaces it. Use this when delivering just the two transparent layers as a reusable asset.
## When `remove-background` is NOT the right tool
If a user asks for "the room **without** the person, displayed standalone" (no subject anywhere, no compositing on top), `--background-output` is wrong — its plate has a transparent hole, not a filled-in clean plate. They need an **inpainter**: LaMa, ProPainter, or E2FGVI. Tell them this command can't do it.
audio/references/requirements.md›
# Requirements & Caches
## Credential & key priority
Run `npx hyperframes auth status` to see what's configured and which engines a workflow will use (see the skill's **Preflight** section). Keys resolve in this order — **first match wins**:
| Provider | Resolution order (first non-empty wins) | Local deps when used |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| **HeyGen** (TTS + BGM/SFX retrieval) | `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; `$HEYGEN_CONFIG_DIR` overrides the dir; written by `hyperframes auth login`) | none (REST) |
| **ElevenLabs** (TTS fallback) | `$ELEVENLABS_API_KEY` | `pip install elevenlabs` |
| **Lyria** (BGM fallback) | `$GEMINI_API_KEY` → `$GOOGLE_API_KEY` | `pip install google-genai` |
| **Kokoro** (TTS, no key) | always — final voice fallback | `pip install kokoro-onnx soundfile` |
| **MusicGen** (BGM, no key) | always — final music fallback | `pip install transformers torch soundfile numpy` |
`hyperframes auth login` (browser OAuth) is the recommended setup: one sign-in, every project, no per-repo `.env`. An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both are tagged with `X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance for HeyGen TTS (10 min/month); API keys follow the normal API billing path. With no HeyGen credential, voice/BGM run fully locally (Kokoro / MusicGen) — `hyperframes auth status` and `hyperframes doctor` both report whether those local deps are installed.
## Model caches & system dependencies
Each command downloads its own model on first run and caches it under `~/.cache/hyperframes/`:
- **TTS (HeyGen)** — no local deps; needs a HeyGen credential + `ffmpeg` on PATH (to transcode the mp3 response to `.wav`). Credential resolves like the CLI: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → `~/.heygen/credentials` (shared with heygen-cli; run `npx hyperframes auth login`). An OAuth login is sent as `Authorization: Bearer`; an API key as `X-Api-Key`; both include `X-HeyGen-Source: cli` so the backend can apply CLI OAuth free usage.
- **TTS (ElevenLabs)** — same as HeyGen: API key + `ffmpeg`.
- **TTS (Kokoro)** — Kokoro-82M (~311 MB) + voices (~27 MB) in `tts/`. Requires Python 3.8+ with `kokoro-onnx` and `soundfile` (`pip install kokoro-onnx soundfile`). Non-English text also needs `espeak-ng` system-wide.
- **BGM (Lyria)** — needs `$GEMINI_API_KEY` or `$GOOGLE_API_KEY` + `pip install google-genai`. No local model cache.
- **BGM (MusicGen)** — `pip install transformers torch soundfile`. `facebook/musicgen-small` (~300 MB) cached under `~/.cache/huggingface/` on first run.
- **Transcribe** — Whisper model size depending on choice (75 MB – 3.1 GB) in `whisper/`, downloaded from HuggingFace on first use. `whisper.cpp` itself is NOT bundled: the CLI resolves it from PATH, installs via Homebrew (macOS), or builds it from source with git+cmake on first use (`$HYPERFRAMES_WHISPER_PATH` overrides).
- **Remove-background** — `u2net_human_seg` (~168 MB ONNX) in `background-removal/models/`. Peak inference RAM ~1.5 GB.
Run `npx hyperframes doctor` if a command fails because of a missing dependency.
audio/references/sfx.md›
# Sound effects (SFX)
Named sound effects, produced by the shared audio engine (`scripts/audio.mjs` → `scripts/lib/sfx.mjs`). **Provider-gated** by the engine's one switch — whether a HeyGen credential is present, decided once (not per cue):
- **HeyGen credential present → retrieve every cue** from HeyGen's audio library (`/v3/audio/sounds`, `type=sound_effects`, `min_score=0.4`). Search-and-download, **not** generation. The bundled library is NOT consulted.
- **No credential → the bundled 21-file library** (`assets/sfx/` + `manifest.json`): match each cue name, copy the matched file into the project. Offline, deterministic, free.
There is no `npx hyperframes sfx` command. SFX is never generated — it is retrieved (online) or taken from the bundled library (offline).
## Cues — request → meta
Each line names the effects it wants: `lines[].sfx: ["whoosh", "ui click"]`. The engine flattens these into cues, resolves them per the switch, dedupes identical `(id, name)` pairs (the same effect named twice downloads/copies once), and writes `audio_meta.sfx[]`:
```jsonc
{
"id": "3", // joins the cue to the caller's model (frame / scene / segment)
"name": "whoosh",
"file": "assets/sfx/whoosh.mp3", // downloaded or copied, relative to project root
"source": "heygen" | "local", // which route resolved it
"offset_s": 0, // delay from the line's start
"duration_s": 0.57,
"volume": 0.35 // SFX sit UNDER voice + BGM
}
```
A cue that matches nothing is **skipped** (recorded as an anomaly); SFX never blocks a render.
## HeyGen retrieval (credentialed)
`searchSounds(name, "sound_effects", { limit: 3, minScore: 0.4 })` → top hit → `assets/sfx/<slug>.mp3`. Results are ranked by `score` (each carries a presigned `audio_url`, `duration`, `description`). The floor is **0.4** because good SFX hits score ~0.5–0.67 — below the API's default `0.7`, which would silently drop most named cues (only whoosh/swoosh-family clears 0.7). `duration_s` comes from the result (else 1.0). Name effects concretely (`glass shatter`, not `dramatic sound`); a vague query returns a poor match.
## Bundled library (no credential)
21 curated files in `assets/sfx/`, indexed by `manifest.json` — `{ file, duration, description }` per key (e.g. `whoosh`, `pop`, `click`, `chime`, `riser`, `impact-bass-1`, `glitch-1`, `typing`, …). A cue name resolves by **manifest key, file basename, or slug**, so `whoosh`, `whoosh.mp3`, or `"ui click"` (→ slug) all match. Matched files are copied into the project's `assets/sfx/`; `duration_s` comes from the manifest, so timing is known **offline** — e.g. `riser` is 10.03s, so trigger it at `climax − 10.03s`. The manifest's `description` field carries placement hints per effect; read `assets/sfx/manifest.json` for the full set and usage.
## Rules
- **Volume ~0.35.** SFX must sit under narration and BGM, not fight them.
- **No match → skip, don't fail.** A missing effect logs an anomaly and moves on; never a render blocker.
- **Retrieval (credentialed) or bundled library (offline) — never generation.** You search HeyGen by text, or match a name against the 21-file manifest.
- **One asset per distinct name.** Reuse across lines is deduped to a single download/copy, many cues.
- **The switch is global, not per cue.** With a credential, retrieval handles even the long tail (effects not in the 21); without one, only the 21 bundled names resolve.
audio/references/transcribe.md›
# Transcription
Create normalized word-level timestamps. **Always specify `--model` explicitly** — the CLI default is `small.en`, which silently translates non-English audio into English.
```bash
npx hyperframes transcribe audio.mp3 --model small.en # known English
npx hyperframes transcribe video.mp4 --model small --language es # known Spanish
npx hyperframes transcribe audio.mp3 --model small # unknown language (auto-detect)
npx hyperframes transcribe subtitles.srt # import existing
npx hyperframes transcribe subtitles.vtt
npx hyperframes transcribe openai-response.json
```
## Language Rule (Non-Negotiable)
`.en` models (`tiny.en` / `base.en` / `small.en` / `medium.en`) **translate** non-English audio into English. This silently destroys the original language.
1. **Known English** → `--model small.en` (or `medium.en` for music / noisy audio)
2. **Known non-English** → `--model small --language <iso-code>` (no `.en` suffix)
3. **Unknown language** → `--model small` (whisper auto-detects)
**CLI default is `small.en`** — do not rely on it; always pass `--model` to make the choice explicit. `--language` also filters out non-target-language segments from mixed-language audio.
## Model Sizes
| Model | Size | Speed | When |
| ---------- | ------ | -------- | ------------------------------------- |
| `tiny` | 75 MB | Fastest | Quick previews, smoke tests |
| `base` | 142 MB | Fast | Short clips, clear audio |
| `small` | 466 MB | Moderate | Default for most multilingual content |
| `medium` | 1.5 GB | Slow | Music with vocals, noisy audio |
| `large-v3` | 3.1 GB | Slowest | Production quality |
### Picking a model by content type
1. Speech over silence / light background → `small.en`
2. Speech over music, or music with vocals → start with `medium.en`
3. Produced music track (vocals + full instrumentation) → start with `medium.en`; expect to need manual lyrics or an external API ([`captions/transcript-handling.md`](captions/transcript-handling.md) → "Using External Transcription APIs")
4. Multilingual → `medium` or `large-v3` (no `.en` suffix), pair with `--language`
## Output Shape
Compositions consume a flat array of word objects. The `id` (`w0`, `w1`, …) is added during normalization for stable references in caption overrides; optional for backwards compatibility.
```json
[
{ "id": "w0", "text": "Hello", "start": 0.0, "end": 0.5 },
{ "id": "w1", "text": "world.", "start": 0.6, "end": 1.2 }
]
```
For mandatory caption-quality checks, retry rules, and the OpenAI/Groq Whisper API import path, see `captions/transcript-handling.md`.
audio/references/tts-to-captions.md›
# TTS → Captions
When no recorded voiceover exists, generate one and obtain word-level caption timing. Two paths depending on which TTS provider is in use:
## Path A — HeyGen (single call, no Whisper)
HeyGen returns word timestamps in the same response as the audio. Use the
bundled REST helper (the `hyperframes tts` command is Kokoro-only):
```bash
node skills/media-use/audio/scripts/heygen-tts.mjs \
script.txt --output narration.wav --words narration.words.json
```
`narration.words.json` is already in the `[{ id, text, start, end }]` shape the captions pipeline consumes — no separate transcribe pass.
## Path B — ElevenLabs / Kokoro (TTS → Whisper)
These providers don't return word data. Generate the audio, then transcribe:
```bash
npx hyperframes tts script.txt --voice af_heart --output narration.wav
npx hyperframes transcribe narration.wav --model small.en # voice af_heart is American English
```
Whisper extracts precise word boundaries from the generated audio, so caption timing matches delivery without hand-tuning. Match `--model` to the voice's language (use `small.en` for `a`/`b` prefixes, `small --language <code>` otherwise). Then consume `transcript.json` via the caption references in `captions/`.
audio/references/tts.md›
# Text To Speech
`npx hyperframes tts` synthesizes locally with Kokoro. It does not accept a
`--provider` or `--words` flag. For HeyGen audio plus word timestamps, use the
bundled `heygen-tts.mjs` script below.
> **Run the Preflight first — no credential is not a green light to silently use the local voice.** Before generating a voiceover, complete the sign-in **Preflight** (see `../SKILL.md` → Preflight): run `npx hyperframes auth status`, recommend signing in, and **STOP for the user's choice** (sign in for HeyGen voices, or continue offline with local Kokoro). This applies to a one-off "generate a voiceover" request just as much as inside a full workflow.
## Narrating a HyperFrames docs video
Anything that will sit on hyperframes.heygen.com uses one narrator, so the site
does not sound like two products.
| | |
| -------------- | ---------------------------------------------- |
| Voice | **River** — "Relaxed, Neutral, Informative" |
| Provider | ElevenLabs |
| `voice_id` | `SAz9YHcvj6GT2YYXdXww` |
| Model | `eleven_multilingual_v2` |
| Pace | 145–155 words per minute, with room to breathe |
| Music under it | about −31 LUFS, never masking the voice |
```bash
curl -s -X POST "https://api.elevenlabs.io/v1/text-to-speech/SAz9YHcvj6GT2YYXdXww" \
-H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \
-d '{"text":"...","model_id":"eleven_multilingual_v2"}' -o take.mp3
```
This is the voice every user-journey film on the docs site already uses. Falling
back to local Kokoro because a key was not to hand produces a film that sounds
wrong beside the others — three docs videos were built that way and had to be
re-voiced. If you cannot reach ElevenLabs, say so and stop rather than
substituting a different voice.
Use another voice only for a documented reason, and write the reason down.
## Available routes
| Order | Provider | Env trigger | Voice IDs | Word timestamps | Audio format |
| ----- | ----------------- | ------------------------------------------- | ------------------------------------------- | ----------------------------------------- | -------------------- |
| 1 | HeyGen (Starfish) | `$HEYGEN_API_KEY` / `~/.heygen/credentials` | UUIDs from `GET /v3/voices?engine=starfish` | **Yes** (`word_timestamps[]` in response) | mp3 → wav via ffmpeg |
| 2 | ElevenLabs | `$ELEVENLABS_API_KEY` | UUIDs from elevenlabs.io dashboard | No | mp3 → wav via ffmpeg |
| 3 | Kokoro-82M | always (local fallback) | `am_michael`, `af_heart`, … (54 voices) | No | wav direct |
```bash
# Local Kokoro CLI
npx hyperframes tts "Welcome to HyperFrames" -o narration.wav
```
## Self-contained HeyGen (no CLI) — `scripts/heygen-tts.mjs`
The published `hyperframes tts` CLI synthesizes locally with Kokoro only. When you
want HeyGen specifically — best quality **plus** word timestamps in one call — use
the skill's bundled script, which calls the HeyGen v3 REST API directly and needs
no CLI provider plumbing:
The script resolves a HeyGen credential the same way the CLI does — first source
wins: `$HEYGEN_API_KEY` → `$HYPERFRAMES_API_KEY` → a project `.env` (auto-loaded,
walks up ≤5 dirs) → `~/.heygen/credentials` (shared with heygen-cli;
`$HEYGEN_CONFIG_DIR` overrides the dir). An OAuth login is sent as
`Authorization: Bearer`; an API key as `X-Api-Key`; both include
`X-HeyGen-Source: cli`. OAuth CLI users can consume the web-plan free allowance
(10 min/month) before paid usage; API keys follow normal API billing. If the
only credential is an expired OAuth token it stops with a hint to run
`npx hyperframes auth refresh`.
```bash
# Only needed if you haven't run `npx hyperframes auth login`:
export HEYGEN_API_KEY=... # or put it in a project .env
# Synthesize + capture word timestamps in one call (skips a Whisper pass)
node skills/media-use/audio/scripts/heygen-tts.mjs \
"Welcome to HyperFrames." -o narration.wav --words narration.words.json
node skills/media-use/audio/scripts/heygen-tts.mjs ./script.txt -o narration.wav
node skills/media-use/audio/scripts/heygen-tts.mjs --list # public starfish voices
```
- **Voice:** `--voice <id>` must be a **starfish** voice_id (`--list`, or `GET /v3/voices?engine=starfish`). v2-catalog ids are rejected with HTTP 400. Omit `--voice` (English) and it defaults to **Marcia** (`05f19352e8f74b0392a8f411eba40de1`, a fixed default so the choice is deterministic). Non-English with no `--voice` falls back to the first matching catalog voice.
- **Output:** `.wav` → transcoded to 44.1k mono via ffmpeg; `.mp3` → raw bytes (no ffmpeg needed).
- **Words:** `--words <path>` writes the flat `[{id,text,start,end}]` shape below, drop-in for the captions pipeline. HeyGen's `<start>`/`<end>` boundary sentinels are filtered out and ids are re-contiguous.
- **Non-English:** `--lang <code>` (anything but `en`) is sent as the request `language`.
## When to use which provider
| Goal | Use |
| --------------------------------------------------------- | --------------------------------------------------- |
| Best voice quality + word timestamps in one call | **HeyGen** |
| Drop-in cloud TTS, big voice catalog | **ElevenLabs** |
| Offline, no API key, fast iteration | **Kokoro** |
| Non-English multilingual with deterministic phonemization | **Kokoro** (`ef_dora`, `jf_alpha`, `zf_xiaobei`, …) |
## ffmpeg requirement
HeyGen + ElevenLabs return mp3. The bundled HeyGen helper transcodes to wav
when `--output` ends in `.wav` (the default and what downstream `ffprobe` +
Whisper expect). If you'd rather skip the transcode, pass `-o file.mp3`.
Without `ffmpeg` on PATH, wav output from cloud providers fails; the local
Kokoro CLI writes wav directly.
## Voice selection (Kokoro)
Default `af_heart`. Curated picks:
| Content type | Voice |
| ----------------- | ---------------------- |
| Product demo | `af_heart`, `af_nova` |
| Tutorial / how-to | `am_adam`, `bf_emma` |
| Marketing / promo | `af_sky`, `am_michael` |
| Documentation | `bf_emma`, `bm_george` |
| Casual / social | `af_heart`, `af_sky` |
Run `npx hyperframes tts --list` for the bundled set.
## Multilingual (Kokoro voice prefix → language)
The first letter of a Kokoro voice ID picks the phonemizer language; `--lang` overrides auto-detection.
| Prefix | Language |
| ------ | -------------------- |
| `a` | American English |
| `b` | British English |
| `e` | Spanish |
| `f` | French |
| `h` | Hindi |
| `i` | Italian |
| `j` | Japanese |
| `p` | Brazilian Portuguese |
| `z` | Mandarin |
```bash
npx hyperframes tts "La reunión empieza a las nueve" --voice ef_dora
npx hyperframes tts "Today is a nice day" --voice af_heart
```
Valid `--lang` codes (only needed to override the voice's auto-detected language): `en-us`, `en-gb`, `es`, `fr-fr`, `hi`, `it`, `pt-br`, `ja`, `zh`.
Non-English phonemization requires `espeak-ng` system-wide (`brew install espeak-ng` / `apt-get install espeak-ng`).
## Speed
- `0.7-0.8` — tutorial, complex content, accessibility
- `1.0` — natural pace (default)
- `1.1-1.2` — intros, transitions, upbeat content
- `1.5+` — rarely appropriate, test carefully
The `hyperframes tts` command honors `--speed` for Kokoro. Provider-specific
helpers document their own pacing controls.
## Long scripts
Past a few paragraphs, write the text to a `.txt` file and pass the path. Inputs over ~5 minutes of speech may benefit from splitting into segments.
## HeyGen word-timestamp shape
When `--words <path>` is passed to a HeyGen call, the file is written in the same flat shape `transcribe` produces — drop-in compatible with the captions pipeline:
```json
[
{ "id": "w0", "text": "Hi", "start": 0.0, "end": 0.21 },
{ "id": "w1", "text": "there", "start": 0.22, "end": 0.55 }
]
```
For ElevenLabs / Kokoro, run `npx hyperframes transcribe narration.wav --model small.en` to get the same shape.
audio/scripts/audio.mjs›
#!/usr/bin/env node
// audio.mjs — the shared HyperFrames audio engine. ONE implementation of TTS +
// BGM + SFX for every video workflow (product-launch, general-video, pr-to-video,
// …). Workflows do NOT vendor a copy: they write a neutral `audio_request.json`
// (a tiny per-workflow adapter maps their storyboard/scenes into it) and call:
//
// node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json
//
// The three capabilities degrade on ONE switch — whether HeyGen is configured
// (credential present, NOT the CLI). This mirrors the table in ../SKILL.md:
//
// TTS : HeyGen REST → ElevenLabs → Kokoro (CLI)
// BGM : HeyGen retrieve → (no credential) Lyria/MusicGen generate
// SFX : HeyGen retrieve → (no credential) bundled 19-file library
//
// ── audio_request.json (input) ────────────────────────────────────────────────
// {
// "provider": "auto", // auto|heygen|elevenlabs|kokoro (override: --provider)
// "lang": "en", "speed": 1.0,
// "lines": [ // one TTS unit each; id joins back to the caller's model
// { "id": "01", "text": "...", "sfx": ["whoosh", "ui click"] }
// ],
// "bgm": { "mode": "retrieve", // retrieve|generate|none (override: --bgm-mode / --no-bgm)
// "query": "calm cinematic underscore", // mood for retrieval
// "prompt": null, // full prompt for generation (else inferred)
// "blob": "...", "archetype": "...", "arc": "..." } // optional mood-inference hints
// }
//
// ── audio_meta.json (output, id-keyed) ───────────────────────────────────────
// { tts_provider, voice_id,
// bgm: { path, volume, mode, query?, duration_s? } | null,
// bgm_pending, bgm_provider, bgm_pid, bgm_log, bgm_mode, bgm_target_duration_s, …,
// voices: [ { id, path, duration_s, words: [{id,text,start,end}] } ],
// sfx: [ { id, name, file, source, offset_s, duration_s, volume } ],
// total_duration_s }
//
// --only tts,bgm,sfx runs a subset and MERGES into an existing --out (so a
// workflow can do TTS+BGM early, then SFX later once cues exist). When BGM uses
// the generate path it is spawned detached (bgm_pending:true) — run wait-bgm.mjs
// before assembling.
import { existsSync, mkdirSync, readFileSync } from "node:fs";
import { dirname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { heygenAuthHeaders, heygenCredential, loadEnvFromDir } from "./lib/heygen.mjs";
import {
ffprobeDuration,
pickProvider,
resolveVoiceId,
synthesizeOne,
transcribeWav,
withWordIds,
} from "./lib/tts.mjs";
import { generateBgmDetached, inferBgmPrompt, retrieveBgm } from "./lib/bgm.mjs";
import { resolveSfx } from "./lib/sfx.mjs";
import { mapWithConcurrency } from "./lib/concurrency.mjs";
import { openAudioMeta } from "./lib/audio-meta.mjs";
const HERE = dirname(fileURLToPath(import.meta.url));
const argv = process.argv.slice(2);
const flag = (name, def) => {
const i = argv.indexOf(`--${name}`);
return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
};
const has = (name) => argv.includes(`--${name}`);
const die = (m) => {
console.error(`✗ audio engine: ${m}`);
process.exit(1);
};
const r3 = (x) => Number(x.toFixed(3));
// Two independent reports of an unbounded Promise.all over TTS lines
// overwhelming a machine: one OOM'd 12/13 concurrent Kokoro TTS +
// whisper-transcribe lines on a resource-constrained laptop, the other saw
// 7/8 lines fail on first run (concurrent cold-start model loads) and pass on
// retry once the model was cached. Kokoro/Whisper each load their own local
// model per subprocess, so firing every line at once multiplies that cost by
// the line count. mapWithConcurrency caps how many run at once — still
// parallel, just bounded.
const ttsConcurrency = Math.max(1, Number(process.env.HYPERFRAMES_TTS_CONCURRENCY) || 4);
const hyperframesDir = resolve(flag("hyperframes", "."));
const requestPath = resolve(flag("request", join(hyperframesDir, "audio_request.json")));
const outPath = resolve(flag("out", join(hyperframesDir, "audio_meta.json")));
const sfxLibDir = resolve(flag("sfx-lib", join(HERE, "..", "assets", "sfx")));
const lyriaRecipe = resolve(flag("lyria-recipe", join(HERE, "lyria-recipe.py")));
const onlyArg = flag("only", "tts,bgm,sfx");
const only = new Set(
onlyArg
.split(",")
.map((s) => s.trim())
.filter(Boolean),
);
const providerOverride = flag("provider", null);
const bgmModeOverride = flag("bgm-mode", null);
const noBgm = has("no-bgm");
const voiceOverride = flag("voice", null);
const speedOverride = flag("speed", null);
const langOverride = flag("lang", null);
const seedSeconds = Number(flag("seed-seconds", "28")) || 28;
if (!existsSync(requestPath)) die(`audio_request.json not found at ${requestPath}`);
let request;
try {
request = JSON.parse(readFileSync(requestPath, "utf8"));
} catch (e) {
die(`audio_request.json parse: ${e.message}`);
}
const lines = Array.isArray(request.lines) ? request.lines : [];
const lang = langOverride || request.lang || "en";
const speed = Number(speedOverride ?? request.speed ?? 1.0) || 1.0;
// ── env + HeyGen availability (the single switch) ─────────────────────────────
loadEnvFromDir(hyperframesDir);
const heygenOK = heygenCredential() !== null;
const headers = heygenOK ? heygenAuthHeaders() : null;
// ── merge base: preserve sections not selected by --only ──────────────────────
const audioMeta = openAudioMeta(outPath);
const prev = audioMeta.value;
const anomalies = [];
// ── TTS ───────────────────────────────────────────────────────────────────────
let voices = prev.voices ?? [];
let ttsProvider = prev.tts_provider ?? null;
let voiceId = prev.voice_id ?? null;
if (only.has("tts") && lines.length) {
try {
ttsProvider = pickProvider(
providerOverride || (request.provider === "auto" ? null : request.provider),
);
} catch (e) {
die(e.message);
}
voiceId = await resolveVoiceId({
provider: ttsProvider,
userVoice: voiceOverride || request.voice,
lang,
});
console.error(`· tts: ${ttsProvider} · voice ${voiceId} · ${lines.length} line(s)`);
const synthLine = async (line) => {
const id = String(line.id);
const text = String(line.text ?? "").trim();
if (!text) {
anomalies.push(`line ${id}: empty text — skipped`);
return null;
}
const rel = `assets/voice/${id}.wav`;
const abs = join(hyperframesDir, rel);
const { ok, words, error } = await synthesizeOne({
provider: ttsProvider,
text,
voiceId,
lang,
speed,
wavAbs: abs,
hyperframesDir,
});
if (!ok) {
anomalies.push(`line ${id}: TTS failed — omitted${error ? ` (${error})` : ""}`);
return null;
}
let wordArr = words; // heygen: native; else transcribe
if (!wordArr) wordArr = await transcribeWav({ wavRel: rel, lang, hyperframesDir });
const dur = ffprobeDuration(abs);
if (!isFinite(dur) || dur <= 0) {
anomalies.push(`line ${id}: bad voice duration — omitted`);
return null;
}
return { id, path: rel, duration_s: r3(dur), words: withWordIds(wordArr) };
};
const results = await mapWithConcurrency(lines, ttsConcurrency, synthLine);
voices = results.filter(Boolean);
for (const v of voices)
console.error(` voice ${v.id}: ${v.path} (${v.duration_s}s, ${v.words.length} words)`);
}
const hasVoice = voices.length > 0;
const totalDuration = r3(voices.reduce((a, v) => a + (v.duration_s || 0), 0));
// ── BGM ─────────────────────────────────────────────────────────────────────
let bgm = prev.bgm ?? null;
const bgmFields = {
bgm_pending: prev.bgm_pending ?? false,
bgm_provider: prev.bgm_provider ?? null,
bgm_pid: prev.bgm_pid ?? null,
bgm_log: prev.bgm_log ?? null,
bgm_mode: prev.bgm_mode ?? null,
bgm_target_duration_s: prev.bgm_target_duration_s ?? null,
bgm_seed_duration_s: prev.bgm_seed_duration_s ?? null,
bgm_loop_count: prev.bgm_loop_count ?? null,
};
if (only.has("bgm")) {
bgm = null;
Object.keys(bgmFields).forEach((k) => (bgmFields[k] = k === "bgm_pending" ? false : null));
// Mode resolution. An EXPLICIT mode (flag or request.bgm.mode) is strict:
// "retrieve" means retrieve-or-nothing — it never silently starts a detached
// generate (a caller with no wait-bgm step, e.g. product-launch, must not get
// a pending job it can't await). Only the UNSET/auto default picks generate
// when HeyGen is absent.
const explicitMode = bgmModeOverride || request.bgm?.mode || null;
let mode = noBgm ? "none" : explicitMode || (heygenOK ? "retrieve" : "generate");
if (mode === "retrieve" && !heygenOK) {
anomalies.push(
"bgm: retrieve requires a HeyGen credential — skipped (no generate fallback for an explicit retrieve)",
);
mode = "none";
}
if (mode === "none") {
console.error(`· bgm: disabled`);
} else if (mode === "retrieve") {
try {
bgm = await retrieveBgm({ query: request.bgm?.query, headers, hyperframesDir, hasVoice });
if (bgm) {
bgmFields.bgm_provider = "heygen";
bgmFields.bgm_mode = "retrieve";
console.error(` bgm: ${bgm.path} (retrieve "${bgm.query}")`);
} else {
anomalies.push(`bgm: no music match for "${request.bgm?.query ?? ""}" — skipped`);
}
} catch (e) {
anomalies.push(`bgm retrieve failed: ${e.message} — skipped`);
}
} else {
// generate
const prompt = inferBgmPrompt({
userPrompt: request.bgm?.prompt,
blob: request.bgm?.blob || request.bgm?.query,
archetype: request.bgm?.archetype,
arc: request.bgm?.arc,
});
const gen = generateBgmDetached({
prompt,
durationS: totalDuration || 30,
hyperframesDir,
lyriaRecipe: existsSync(lyriaRecipe) ? lyriaRecipe : null,
seedSeconds,
hasVoice,
});
if (gen.disabled) {
anomalies.push(`bgm: ${gen.reason}`);
} else {
bgm = { path: gen.path, volume: gen.volume, mode: gen.mode, duration_s: null };
bgmFields.bgm_pending = true;
bgmFields.bgm_provider = gen.provider;
bgmFields.bgm_pid = gen.pid;
bgmFields.bgm_log = gen.log;
bgmFields.bgm_mode = gen.mode;
bgmFields.bgm_target_duration_s = gen.target_duration_s ?? null;
bgmFields.bgm_seed_duration_s = gen.seed_duration_s ?? null;
bgmFields.bgm_loop_count = gen.loop_count ?? null;
console.error(` bgm: launched ${gen.provider} (detached, pid ${gen.pid}) → ${gen.path}`);
}
}
}
// ── SFX ─────────────────────────────────────────────────────────────────────
let sfx = prev.sfx ?? [];
if (only.has("sfx")) {
const cues = lines.flatMap((l) =>
(Array.isArray(l.sfx) ? l.sfx : [])
.map((name) => ({ id: String(l.id), name: String(name).trim() }))
.filter((c) => c.name),
);
const res = await resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir });
sfx = res.sfx;
anomalies.push(...res.anomalies);
console.error(
`· sfx: ${sfx.length} cue(s) resolved (${heygenOK ? "heygen retrieval" : "bundled library"})`,
);
}
// ── write audio_meta.json ─────────────────────────────────────────────────────
const meta = {
tts_provider: ttsProvider,
voice_id: voiceId,
bgm,
...bgmFields,
voices,
sfx,
total_duration_s: totalDuration,
};
mkdirSync(dirname(outPath), { recursive: true });
audioMeta.write(meta);
console.log(`✓ audio engine → ${outPath}`);
console.log(` heygen: ${heygenOK ? "yes" : "no"} · ran: ${[...only].join(",")}`);
console.log(
` voices: ${voices.length} · bgm: ${bgm ? `${bgmFields.bgm_provider}${bgmFields.bgm_pending ? " (pending)" : ""}` : "none"} · sfx: ${sfx.length}`,
);
console.log(` total voice duration: ${totalDuration}s`);
if (anomalies.length) {
console.log(`\nanomalies (non-fatal):`);
for (const a of anomalies) console.log(` - ${a}`);
}
audio/scripts/audio.test.mjs›
import { strict as assert } from "node:assert";
import { test } from "node:test";
import { mkdtempSync, rmSync, existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
import { resolveSfx } from "./lib/sfx.mjs";
// Proves the relocated engine (skills/media-use/audio/) still resolves its
// bundled SFX library from the moved location — the path most likely to break
// on a subtree move. Offline (heygenOK:false), no network.
const HERE = dirname(fileURLToPath(import.meta.url));
const sfxLibDir = join(HERE, "..", "assets", "sfx"); // same offset the engine uses
test("bundled SFX library resolves from the relocated path", async () => {
assert.ok(existsSync(join(sfxLibDir, "manifest.json")), "moved manifest is present");
const dir = mkdtempSync(join(tmpdir(), "mu-audio-"));
try {
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "1", name: "whoosh" }],
heygenOK: false,
hyperframesDir: dir,
sfxLibDir,
});
assert.equal(sfx.length, 1, `expected 1 resolved cue, got anomalies: ${anomalies.join("; ")}`);
assert.equal(sfx[0].source, "local");
assert.match(sfx[0].file, /assets\/sfx\//);
assert.ok(existsSync(join(dir, sfx[0].file)), "matched SFX copied into the project");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("an unknown cue is reported, not fatal", async () => {
const dir = mkdtempSync(join(tmpdir(), "mu-audio-"));
try {
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "1", name: "definitely-not-a-real-sfx" }],
heygenOK: false,
hyperframesDir: dir,
sfxLibDir,
});
assert.equal(sfx.length, 0);
assert.ok(anomalies.some((a) => /not in bundled library/.test(a)));
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
audio/scripts/heygen-tts.mjs›
#!/usr/bin/env node
// Self-contained HeyGen TTS — single text in → one wav (+ optional words JSON)
// out. A thin CLI over lib/tts.mjs (the same code the audio engine uses), so the
// HeyGen REST call, starfish voice pick, mp3→wav transcode, and word-timestamp
// filtering live in exactly one place. Bypasses the `hyperframes` CLI, which in
// the published build is Kokoro-only.
//
// Usage:
// node heygen-tts.mjs "Text to speak" -o narration.wav [--words narration.words.json]
// node heygen-tts.mjs ./script.txt -o narration.wav --words narration.words.json
// node heygen-tts.mjs "Bonjour" -o fr.wav --lang fr --voice <id>
// node heygen-tts.mjs --list # list starfish voices and exit
//
// Flags: -o/--output (.wav → ffmpeg transcode; .mp3 → raw bytes), --words,
// --voice (starfish id), --speed, --lang, --list.
// Requires: $HEYGEN_API_KEY / OAuth ~/.heygen credentials and ffmpeg for .wav output.
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { heygenAuthHeaders, heygenJSON, loadEnvFromDir } from "./lib/heygen.mjs";
import { ffprobeDuration, resolveVoiceId, synthesizeOne, withWordIds } from "./lib/tts.mjs";
const argv = process.argv.slice(2);
function flag(name, def) {
const i = argv.indexOf(`--${name}`);
if (i < 0) return def;
if (i + 1 >= argv.length) return true;
const v = argv[i + 1];
return v.startsWith("--") ? true : v;
}
const die = (m) => {
console.error(`✗ heygen-tts: ${m}`);
process.exit(1);
};
// First arg that isn't a flag or the -o value is the text / .txt path.
const positional = (() => {
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a.startsWith("--")) {
const next = argv[i + 1];
if (next && !next.startsWith("--")) i++;
continue;
}
if (a === "-o") {
i++;
continue;
}
return a;
}
return null;
})();
const output = resolve(
(typeof flag("output") === "string" && flag("output")) ||
(argv.includes("-o") && argv[argv.indexOf("-o") + 1]) ||
"narration.wav",
);
const wordsPath = typeof flag("words") === "string" ? resolve(flag("words")) : null;
const userVoice = typeof flag("voice") === "string" ? flag("voice") : null;
const speedRaw = typeof flag("speed") === "string" ? Number(flag("speed")) : 1.0;
const speed = isFinite(speedRaw) && speedRaw > 0 ? speedRaw : 1.0;
const lang = typeof flag("lang") === "string" ? flag("lang") : "en";
const listOnly = flag("list") === true;
loadEnvFromDir(process.cwd());
let authHeaders;
try {
authHeaders = heygenAuthHeaders();
} catch (e) {
die(e.message);
}
// ---------- --list ----------
if (listOnly) {
const payload = await heygenJSON(`/voices?engine=starfish&type=public&limit=50`, {
headers: authHeaders,
});
for (const v of payload.data ?? payload.voices ?? []) {
console.log(`${v.voice_id}\t${v.name}\t${v.language ?? ""}`);
}
process.exit(0);
}
// ---------- resolve text + voice ----------
if (!positional) die("no text given. Pass a string or a .txt path, or use --list.");
const text =
positional.endsWith(".txt") && existsSync(resolve(positional))
? readFileSync(resolve(positional), "utf8").trim()
: positional;
if (!text) die("input text is empty");
const voiceId = await resolveVoiceId({ provider: "heygen", userVoice, lang });
if (!userVoice) console.error(`· using voice ${voiceId}`);
// ---------- synthesize (shared engine code) ----------
const { ok, words } = await synthesizeOne({
provider: "heygen",
text,
voiceId,
lang,
speed,
wavAbs: output,
hyperframesDir: process.cwd(),
});
if (!ok) die("synthesis failed (HeyGen request/transcode error)");
let wordCount = 0;
if (wordsPath) {
if (words && words.length) {
mkdirSync(dirname(wordsPath), { recursive: true });
writeFileSync(wordsPath, JSON.stringify(withWordIds(words), null, 2));
wordCount = words.length;
} else {
console.error("⚠ no word_timestamps in response — run `hyperframes transcribe` instead");
}
}
const dur = ffprobeDuration(output);
const durStr = isFinite(dur) ? ` (${dur.toFixed(2)}s)` : "";
console.log(`✓ ${output}${durStr}${wordCount ? ` · ${wordsPath} (${wordCount} words)` : ""}`);
audio/scripts/lib/audio-meta.mjs›
import { closeSync, ftruncateSync, openSync, readFileSync, writeSync } from "node:fs";
// Keep the merge base and output on the same file even if its pathname changes
// while audio generation runs. The CLI owns this handle until write or exit.
export function openAudioMeta(path) {
let fd;
try {
fd = openSync(path, "r+");
} catch (error) {
if (error.code !== "ENOENT") throw error;
}
let value = {};
if (fd !== undefined) {
try {
value = JSON.parse(readFileSync(fd, "utf8"));
} catch (error) {
closeSync(fd);
throw error;
}
}
return {
value,
write(meta) {
const bytes = Buffer.from(JSON.stringify(meta, null, 2));
// Defer new-file creation until generation succeeds. Never overwrite a
// file (or follow a link) that appeared since the missing merge base.
if (fd === undefined) fd = openSync(path, "wx");
try {
let offset = 0;
while (offset < bytes.length) {
offset += writeSync(fd, bytes, offset, bytes.length - offset, offset);
}
ftruncateSync(fd, bytes.length);
} finally {
closeSync(fd);
}
},
};
}
audio/scripts/lib/audio-meta.test.mjs›
import assert from "node:assert/strict";
import { test } from "node:test";
import {
mkdtempSync,
readFileSync,
renameSync,
rmSync,
statSync,
symlinkSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { spawnSync } from "node:child_process";
import { fileURLToPath } from "node:url";
import { openAudioMeta } from "./audio-meta.mjs";
function fixture(t) {
const dir = mkdtempSync(join(tmpdir(), "audio-meta-"));
t.after(() => rmSync(dir, { recursive: true, force: true }));
return dir;
}
test("updates the read file from byte zero, truncates, and retains its mode", (t) => {
const path = join(fixture(t), "meta.json");
writeFileSync(path, JSON.stringify({ voices: ["long existing voice metadata"] }), {
mode: 0o600,
});
const originalMode = statSync(path).mode;
const handle = openAudioMeta(path);
assert.deepEqual(handle.value, { voices: ["long existing voice metadata"] });
handle.write({ voices: ["声"] });
assert.equal(readFileSync(path, "utf8"), JSON.stringify({ voices: ["声"] }, null, 2));
assert.equal(statSync(path).mode, originalMode);
});
test("pathname replacement cannot redirect the existing-file write", (t) => {
const dir = fixture(t);
const path = join(dir, "meta.json");
const moved = join(dir, "original.json");
writeFileSync(path, "{}");
const handle = openAudioMeta(path);
renameSync(path, moved);
writeFileSync(path, "replacement must survive");
handle.write({ voices: [] });
assert.equal(readFileSync(path, "utf8"), "replacement must survive");
assert.deepEqual(JSON.parse(readFileSync(moved, "utf8")), { voices: [] });
});
test("existing symlink outputs retain their original target even after retargeting", (t) => {
const dir = fixture(t);
const path = join(dir, "meta.json");
const target = join(dir, "target.json");
const victim = join(dir, "victim.json");
writeFileSync(target, "{}");
writeFileSync(victim, "untouched");
symlinkSync(target, path);
const handle = openAudioMeta(path);
rmSync(path);
symlinkSync(victim, path);
handle.write({ bgm: null });
assert.equal(readFileSync(victim, "utf8"), "untouched");
assert.deepEqual(JSON.parse(readFileSync(target, "utf8")), { bgm: null });
});
test("missing output is created only when saved", (t) => {
const path = join(fixture(t), "meta.json");
const handle = openAudioMeta(path);
assert.deepEqual(handle.value, {});
assert.throws(() => statSync(path), { code: "ENOENT" });
handle.write({ sfx: [] });
assert.deepEqual(JSON.parse(readFileSync(path, "utf8")), { sfx: [] });
});
for (const replacement of ["file", "symlink"]) {
test(`new output does not overwrite a racing ${replacement}`, (t) => {
const dir = fixture(t);
const path = join(dir, "meta.json");
const victim = join(dir, "victim.json");
const handle = openAudioMeta(path);
writeFileSync(victim, "untouched");
if (replacement === "symlink") symlinkSync(victim, path);
else writeFileSync(path, "untouched");
assert.throws(() => handle.write({}), { code: "EEXIST" });
assert.equal(readFileSync(path, "utf8"), "untouched");
assert.equal(readFileSync(victim, "utf8"), "untouched");
});
}
test("malformed merge base remains an error without changing bytes", (t) => {
const path = join(fixture(t), "meta.json");
writeFileSync(path, "bad json");
assert.throws(() => openAudioMeta(path), SyntaxError);
assert.equal(readFileSync(path, "utf8"), "bad json");
});
test("CLI partial run retains unselected voices, BGM and SFX", (t) => {
const dir = fixture(t);
const path = join(dir, "meta.json");
const request = join(dir, "request.json");
const previous = {
voices: [{ id: "a", duration_s: 2 }],
bgm: { path: "bgm.wav" },
sfx: [{ name: "click" }],
tts_provider: "kokoro",
voice_id: "voice-a",
};
writeFileSync(path, JSON.stringify(previous));
writeFileSync(request, "{}");
const result = spawnSync(
process.execPath,
[
fileURLToPath(new URL("../audio.mjs", import.meta.url)),
"--request",
request,
"--hyperframes",
dir,
"--out",
path,
"--only",
"none",
],
{
encoding: "utf8",
env: {
...process.env,
HEYGEN_CONFIG_DIR: dir,
HEYGEN_API_KEY: "",
HYPERFRAMES_API_KEY: "",
},
},
);
assert.equal(result.status, 0, result.stderr);
const actual = JSON.parse(readFileSync(path, "utf8"));
for (const key of Object.keys(previous)) assert.deepEqual(actual[key], previous[key]);
});
audio/scripts/lib/bgm.mjs›
// bgm.mjs — background music for the media audio engine. Two routes, gated the
// same way as TTS/SFX:
//
// retrieve (default when HeyGen is configured) — search HeyGen's music library
// by mood, download the top track. Synchronous. assets/bgm/track.mp3.
// generate (the alternative; the automatic choice when HeyGen is absent) —
// Lyria (cloud, $GEMINI_API_KEY/$GOOGLE_API_KEY + google-genai) preferred,
// else local MusicGen (facebook/musicgen-small via transformers). Spawned
// DETACHED so the engine can return while audio renders; the caller marks
// bgm_pending and runs wait-bgm.mjs before assembling. assets/bgm/track.wav.
//
// Missing/failed BGM never blocks a render.
import { spawn, spawnSync } from "node:child_process";
import { existsSync, mkdirSync, openSync, closeSync } from "node:fs";
import { join } from "node:path";
import { downloadTo, searchSounds } from "./heygen.mjs";
import { pythonInvocation } from "./python.mjs";
const r3 = (x) => Number(x.toFixed(3));
const lyriaKey = () => process.env.GEMINI_API_KEY || process.env.GOOGLE_API_KEY || "";
// Default BGM level. Under narration music is a bed that must stay under the
// voice — 0.12 linear ≈ -18 dB. A silent film (no voice) has no voice to duck
// beneath, so BGM sits forward at 0.9. Callers may override per composition.
export const BGM_BED_VOLUME = 0.12;
export const BGM_SILENT_VOLUME = 0.9;
export const bgmDefaultVolume = (hasVoice) => (hasVoice ? BGM_BED_VOLUME : BGM_SILENT_VOLUME);
const BGM_PY_DEPS = ["transformers", "torch", "soundfile", "numpy"];
const BGM_PY_PROBE =
"import transformers, soundfile, torch, numpy; from transformers import MusicgenForConditionalGeneration";
const LYRIA_PY_DEPS = ["google-genai", "python-dotenv"];
const LYRIA_PY_PROBE = "import google.genai";
function pyOk(probe) {
const { cmd, args } = pythonInvocation(["-c", probe]);
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
}
// `python -m pip`, not a bare `pip` binary: a Homebrew/system Python often
// exposes only `python3`/`pip3` on PATH, so a plain `pip` spawn silently
// no-ops (ENOENT) and the documented "auto-installed on demand" path never
// actually installs. `-m pip` also guarantees the packages land in the SAME
// interpreter pyOk() probes — a bare `pip`/`pip3` could resolve to a
// different Python installation than `python3` if more than one is on PATH.
function pipInstall(deps) {
const { cmd, args } = pythonInvocation(["-m", "pip", "install", "-q", ...deps]);
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
}
// ── retrieval (HeyGen music library) ──────────────────────────────────────────
export async function retrieveBgm({ query, headers, hyperframesDir, hasVoice }) {
const q = query || "calm cinematic underscore";
const results = await searchSounds(q, "music", headers, { limit: 5 });
if (!results.length) return null;
const top = results[0];
const rel = "assets/bgm/track.mp3";
await downloadTo(top.audio_url, join(hyperframesDir, rel));
return {
path: rel,
volume: bgmDefaultVolume(hasVoice),
query: q,
mode: "retrieve",
duration_s: typeof top.duration === "number" ? r3(top.duration) : null,
};
}
// ── mood inference (for the generate path's prompt) ──────────────────────────
// Industry base → archetype shape → emotional-arc tiebreaker. Exported so a
// workflow adapter can build a rich prompt from its own narrative metadata; the
// engine also calls it when generate has only a plain mood query.
export function inferBgmPrompt({ blob = "", archetype = "", arc = "", userPrompt = "" } = {}) {
if (userPrompt) return userPrompt;
const b = String(blob).toLowerCase();
let base;
let bpm;
if (/\b(crypto|nft|web3|defi|token|blockchain|exchange|wallet|dao)\b/.test(b)) {
base = "atmospheric electronic, deep bass, futuristic synths, restrained percussion";
bpm = 100;
} else if (/\b(finance|fintech|bank|payment|invest|wealth|insurance|treasury)\b/.test(b)) {
base = "calm cinematic, soft strings, subtle piano, restrained percussion";
bpm = 92;
} else if (/\b(creative|agency|design|studio|art|brand|marketing|content)\b/.test(b)) {
base = "playful electronic, warm pads, light percussion";
bpm = 115;
} else {
base = "uplifting corporate tech, bright modern piano with synth pads";
bpm = 108;
}
const at = String(archetype).toLowerCase();
const ar = String(arc).toLowerCase();
if (/\bpas\b|pain.agitate|pain.+solve/.test(at))
return `${base}, starts with subtle tension then builds to resolution, BPM ${bpm}, transitions from MINOR to MAJOR`;
if (/\bbab\b|before.after|future.pac|vision/.test(at))
return `${base}, cinematic and aspirational, steady build with rising energy, BPM ${bpm}, MAJOR`;
if (/cascade|feature.benefit/.test(at))
return `${base}, energetic and driving, consistent momentum, BPM ${Math.min(bpm + 10, 128)}, MAJOR`;
if (/demo.loop|question.+answer/.test(at))
return `${base}, clean and focused, minimal arrangement, BPM ${Math.max(bpm - 8, 88)}`;
if (/frustrat|anxiety|overwhelm|tension/.test(ar) && /relief|excite|triumph/.test(ar))
return `${base}, builds from understated tension to uplifting resolution, BPM ${bpm}, MINOR to MAJOR`;
if (/excit|awe|power|triumph/.test(ar))
return `${base}, energetic and confident, BPM ${bpm}, MAJOR`;
if (/trust|ease|clarity|reassur/.test(ar))
return `${base}, warm and reassuring, BPM ${Math.max(bpm - 5, 85)}`;
return `${base}, BPM ${bpm}, MAJOR`;
}
// ── generation (Lyria → MusicGen, detached) ──────────────────────────────────
// Returns a bgmMeta the caller folds into audio_meta:
// { path, mode, volume, provider, pid, log, target_duration_s, seed_duration_s,
// loop_count, pending:true } on success, or { disabled:true, reason }.
export function generateBgmDetached({
prompt,
durationS,
hyperframesDir,
lyriaRecipe,
seedSeconds = 28,
hasVoice,
}) {
const rel = "assets/bgm/track.wav";
const abs = join(hyperframesDir, rel);
mkdirSync(join(hyperframesDir, "assets", "bgm"), { recursive: true });
const log = join(hyperframesDir, "assets", "bgm", `bgm-${Date.now()}.log`);
const targetS = Math.max(1, durationS);
const baseMeta = { path: rel, mode: null, volume: bgmDefaultVolume(hasVoice), pending: true };
const lyriaConfigured = !!lyriaKey() && !!lyriaRecipe && existsSync(lyriaRecipe);
// Make a backend runnable: prefer Lyria when configured (install google-genai
// on demand), else ensure local MusicGen deps. Installs are synchronous here —
// generation itself is detached, so the engine still returns promptly.
if (lyriaConfigured && !pyOk(LYRIA_PY_PROBE)) pipInstall(LYRIA_PY_DEPS);
const useLyria = lyriaConfigured && pyOk(LYRIA_PY_PROBE);
if (!useLyria && !pyOk(BGM_PY_PROBE)) pipInstall(BGM_PY_DEPS);
const fd = openSync(log, "w");
if (useLyria) {
const { cmd, args } = pythonInvocation([
lyriaRecipe,
"--output",
abs,
"--duration",
String(targetS),
"--prompt",
prompt,
]);
const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
proc.unref();
closeSync(fd);
return {
...baseMeta,
mode: "detached-single",
provider: "lyria",
pid: proc.pid,
log,
target_duration_s: r3(targetS),
};
}
if (pyOk(BGM_PY_PROBE)) {
const seedS = Math.min(Math.max(seedSeconds, 10), 30);
const loops = targetS > seedS ? Math.ceil(targetS / seedS) : 1;
const script = musicgenScript({ prompt, abs, targetS, seedS });
const { cmd, args } = pythonInvocation(["-c", script]);
const proc = spawn(cmd, args, { detached: true, stdio: ["ignore", fd, fd] });
proc.unref();
closeSync(fd);
return {
...baseMeta,
mode: targetS > seedS ? "detached-seed-loop" : "detached-seed-trim",
provider: "musicgen",
pid: proc.pid,
log,
target_duration_s: r3(targetS),
seed_duration_s: seedS,
loop_count: loops,
};
}
closeSync(fd);
return {
disabled: true,
reason: lyriaConfigured
? `Lyria configured but google-genai uninstallable, and local MusicGen unavailable (pip install ${BGM_PY_DEPS.join(" ")})`
: `no Lyria key/recipe and local MusicGen deps unavailable (pip install ${BGM_PY_DEPS.join(" ")})`,
};
}
// Inline MusicGen: generate ONE seed clip (≤30s to stay under the decoder's
// positional limit), then trim it down or crossfade-loop it up to the target.
function musicgenScript({ prompt, abs, targetS, seedS }) {
return `
import math, os, sys, traceback
from pathlib import Path
import numpy as np
import soundfile as sf
from transformers import MusicgenForConditionalGeneration, AutoProcessor
prompt = ${JSON.stringify(prompt)}
out_path = ${JSON.stringify(abs)}
target_s = float(${targetS.toFixed(3)})
seed_s = float(${seedS.toFixed(3)})
token_rate = 50
crossfade_s = 0.3
def apply_fade(arr, sr, fade_in_s=0.08, fade_out_s=0.5):
n_in = min(int(round(fade_in_s * sr)), arr.shape[0] // 2)
n_out = min(int(round(fade_out_s * sr)), arr.shape[0] // 2)
if n_in > 1: arr[:n_in] *= np.linspace(0.0, 1.0, n_in, dtype="float32")
if n_out > 1: arr[-n_out:] *= np.linspace(1.0, 0.0, n_out, dtype="float32")
return arr
def loop_crossfade(seed, target_len, xf):
if seed.shape[0] >= target_len: return seed[:target_len]
xf = min(xf, seed.shape[0] // 2)
if xf < 1:
reps = int(math.ceil(target_len / seed.shape[0]))
return np.tile(seed, reps)[:target_len]
t = np.linspace(0.0, 1.0, xf, dtype="float32")
fade_out = np.cos(t * (math.pi / 2)); fade_in = np.sin(t * (math.pi / 2))
out = seed.copy()
while out.shape[0] < target_len:
tail = out[-xf:] * fade_out; head = seed[:xf] * fade_in
out = np.concatenate([out[:-xf], tail + head, seed[xf:]])
return out[:target_len]
try:
Path(os.path.dirname(out_path)).mkdir(parents=True, exist_ok=True)
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
model.eval()
sr = int(model.config.audio_encoder.sampling_rate)
gen_s = min(seed_s, target_s)
tokens = max(1, int(math.ceil(gen_s * token_rate)))
print(f"[musicgen] seed dur={gen_s:.2f}s tokens={tokens}", flush=True)
inputs = processor(text=[prompt], padding=True, return_tensors="pt")
audio = model.generate(**inputs, max_new_tokens=tokens)
seed = audio[0, 0].detach().cpu().numpy().astype("float32")
peak = float(np.max(np.abs(seed)))
if peak > 1e-6: seed = seed * (0.89 / peak)
want = max(1, int(round(target_s * sr)))
if seed.shape[0] >= want:
final = seed[:want].copy()
else:
final = loop_crossfade(seed, want, int(round(crossfade_s * sr)))
if final.shape[0] < want: final = np.pad(final, (0, want - final.shape[0]))
else: final = final[:want]
final = apply_fade(final, sr)
peak = float(np.max(np.abs(final)))
if peak > 1.0: final = final / peak
sf.write(out_path, final, sr)
print(f"[musicgen] wrote {out_path} samples={final.shape[0]} sr={sr}", flush=True)
except Exception:
traceback.print_exc(); sys.exit(1)
`;
}
audio/scripts/lib/bgm.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { BGM_BED_VOLUME, BGM_SILENT_VOLUME, bgmDefaultVolume } from "./bgm.mjs";
// Regression: narrated pipelines used to ship BGM at 0.8 (≈ -2 dB), ~16 dB
// hotter than a music bed under a voice should be. The default under narration
// must be a proper bed (≈ -18 dB); a silent film keeps the louder default.
const dbfs = (linear) => 20 * Math.log10(linear);
test("BGM under narration is a bed near -18 dB", () => {
assert.equal(bgmDefaultVolume(true), BGM_BED_VOLUME);
assert.equal(BGM_BED_VOLUME, 0.12);
const db = dbfs(BGM_BED_VOLUME);
assert.ok(db < -17 && db > -19, `bed should be ≈ -18 dB, got ${db.toFixed(1)} dB`);
});
test("a silent film (no voice) keeps BGM forward", () => {
assert.equal(bgmDefaultVolume(false), BGM_SILENT_VOLUME);
assert.equal(BGM_SILENT_VOLUME, 0.9);
});
test("the narrated default is well below the voice (≈ 0 dBFS)", () => {
// Voice sits at data-volume="1" (0 dBFS); the bed must be ~16+ dB under it.
const separation = dbfs(1) - dbfs(bgmDefaultVolume(true));
assert.ok(
separation >= 16,
`bed should sit ≥16 dB under the voice, got ${separation.toFixed(1)} dB`,
);
});
audio/scripts/lib/concurrency.mjs›
// mapWithConcurrency — run `fn` over `items` with at most `limit` in flight at
// once. Preserves input order in the result array regardless of completion order.
export async function mapWithConcurrency(items, limit, fn) {
const results = new Array(items.length);
let next = 0;
async function worker() {
while (next < items.length) {
const i = next++;
results[i] = await fn(items[i], i);
}
}
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
return results;
}
audio/scripts/lib/concurrency.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { mapWithConcurrency } from "./concurrency.mjs";
// Regression: audio.mjs used a bare Promise.all(lines.map(synthLine)) to
// synthesize every TTS line at once, spawning one Kokoro/whisper model load
// per line concurrently. Two independent reports of this overwhelming a
// machine (OOM, and cold-start contention causing spurious failures).
// mapWithConcurrency is the extracted cap; test it in isolation since
// audio.mjs itself is a script (runs CLI/exit side effects on import).
test("processes every item and preserves input order regardless of completion order", async () => {
const order = [5, 1, 3, 2, 4];
const results = await mapWithConcurrency(order, 2, async (n) => {
await new Promise((r) => setTimeout(r, n));
return n * 10;
});
assert.deepEqual(results, [50, 10, 30, 20, 40]);
});
test("never runs more than `limit` at once", async () => {
let inFlight = 0;
let maxInFlight = 0;
const items = Array.from({ length: 10 }, (_, i) => i);
await mapWithConcurrency(items, 3, async () => {
inFlight++;
maxInFlight = Math.max(maxInFlight, inFlight);
await new Promise((r) => setTimeout(r, 5));
inFlight--;
});
assert.equal(maxInFlight, 3);
});
test("limit larger than the item count runs everything without hanging", async () => {
const results = await mapWithConcurrency([1, 2], 10, async (n) => n * 2);
assert.deepEqual(results, [2, 4]);
});
test("empty input resolves to an empty array", async () => {
const results = await mapWithConcurrency([], 4, async (n) => n);
assert.deepEqual(results, []);
});
audio/scripts/lib/heygen.mjs›
import { fetchMedia } from "../../../scripts/lib/media-fetch.mjs";
// heygen.mjs — vendored HeyGen REST helpers (auth + transport) for the audio
// pipeline. The credential resolver matches the hyperframes CLI auth: first
// usable source wins — $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY → a nearby .env → ~/.heygen/
// credentials (oauth → Bearer, else api_key → X-Api-Key; $HEYGEN_CONFIG_DIR
// overrides the dir). Vendored so the skill ships standalone. Pure node.
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { dirname, join, resolve } from "node:path";
export const HEYGEN_BASE = "https://api.heygen.com/v3";
export const HEYGEN_CLI_SOURCE_HEADERS = { "X-HeyGen-Source": "cli" };
// Tool-attribution sent on EVERY media-use HeyGen call regardless of auth type, so
// the backend can isolate media-use consumption from other free TTS / avatar video.
// Unconditional — a paying user's media-use call is still media-use — unlike the
// OAuth-only cli-source header above, which also gates the free allowance.
export const HEYGEN_CLIENT_SOURCE_HEADERS = { "X-HeyGen-Client-Source": "media-use" };
// Walk up ≤5 dirs from startDir; load the first .env (shell env always wins).
export function loadEnvFromDir(startDir) {
let dir = resolve(startDir);
for (let i = 0; i < 5; i++) {
const envPath = join(dir, ".env");
if (existsSync(envPath)) {
for (const raw of readFileSync(envPath, "utf8").split("\n")) {
let line = raw.trim();
if (!line || line.startsWith("#")) continue;
if (line.startsWith("export ")) line = line.slice(7).trim();
const eq = line.indexOf("=");
if (eq < 1) continue;
const key = line.slice(0, eq).trim();
let val = line.slice(eq + 1).trim();
if (val.startsWith('"') || val.startsWith("'")) {
const q = val[0];
const end = val.indexOf(q, 1);
val = end > 0 ? val.slice(1, end) : val.slice(1);
}
if (!(key in process.env)) process.env[key] = val;
}
return;
}
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
}
// → { headers } | { expired: true } | null. Never throws.
export function heygenCredential() {
const envKey = process.env.HEYGEN_API_KEY || process.env.HYPERFRAMES_API_KEY;
if (envKey) return { headers: { "X-Api-Key": envKey } };
const file = join(process.env.HEYGEN_CONFIG_DIR || join(homedir(), ".heygen"), "credentials");
if (!existsSync(file)) return null;
const raw = readFileSync(file, "utf8").trim();
if (!raw) return null;
if (!raw.startsWith("{")) return { headers: { "X-Api-Key": raw } };
// A malformed credentials file (partial write / wrong shape) must degrade to
// "no credential", not crash the engine at startup — this function never throws.
let cred;
try {
cred = JSON.parse(raw);
} catch {
return null;
}
const oauth = cred.oauth;
if (oauth?.access_token) {
const expired = oauth.expires_at && new Date(oauth.expires_at).getTime() - 60_000 < Date.now();
if (!expired) return { headers: { Authorization: `Bearer ${oauth.access_token}` } };
if (!cred.api_key) return { expired: true };
}
if (cred.api_key) return { headers: { "X-Api-Key": cred.api_key } };
return null;
}
// → "oauth" | "api_key" | null. Same oauth-vs-api-key check heygenAuthHeaders()
// makes internally, exposed on its own so callers that only need to *tag* the
// auth path (telemetry) don't have to parse headers back apart. Never throws:
// no credential (or an expired one) is just `null`, same as a fresh resolve
// with nothing to tag.
export function heygenAuthMethod() {
const cred = heygenCredential();
if (!cred?.headers) return null;
return "Authorization" in cred.headers ? "oauth" : "api_key";
}
// → auth headers object, or throw with a fix hint.
export function heygenAuthHeaders() {
const cred = heygenCredential();
if (cred?.headers) {
// Only tag OAuth (Bearer) traffic as cli-source — the backend uses it to
// grant the free allowance for OAuth requests and ignores it for API-key
// (X-Api-Key) traffic, where it's dead metadata.
const isOauth = "Authorization" in cred.headers;
return isOauth
? { ...cred.headers, ...HEYGEN_CLI_SOURCE_HEADERS, ...HEYGEN_CLIENT_SOURCE_HEADERS }
: { ...cred.headers, ...HEYGEN_CLIENT_SOURCE_HEADERS };
}
if (cred?.expired)
throw new Error(
"HeyGen OAuth token expired — run `npx hyperframes auth refresh` (or `npx hyperframes auth login`)",
);
throw new Error(
"no HeyGen credentials — set $HEYGEN_API_KEY, or run `npx hyperframes auth login` (writes ~/.heygen/credentials)",
);
}
// Authed JSON request against the v3 API; throws on a non-OK status.
export async function heygenJSON(path, { method = "GET", headers = {}, body } = {}) {
const opts = { method, headers: { ...HEYGEN_CLIENT_SOURCE_HEADERS, ...headers } };
if (body !== undefined) {
opts.headers["Content-Type"] = "application/json";
opts.body = JSON.stringify(body);
}
const res = await fetch(`${HEYGEN_BASE}${path}`, opts);
if (!res.ok) {
const detail = await res.text().catch(() => "");
throw new Error(
`HeyGen ${method} ${path} → HTTP ${res.status}${detail ? `\n${detail.slice(0, 300)}` : ""}`,
);
}
return res.json();
}
// Download a (presigned) URL to destPath; returns byte length.
export async function downloadTo(url, destPath) {
const res = await fetchMedia(url);
if (!res.ok) throw new Error(`download HTTP ${res.status}: ${String(url).slice(0, 80)}`);
const bytes = Buffer.from(await res.arrayBuffer());
mkdirSync(dirname(destPath), { recursive: true });
writeFileSync(destPath, bytes);
return bytes.length;
}
// Retrieval search over HeyGen's audio catalog (NOT generation). type =
// "music" | "sound_effects". Returns the ranked results array (best first); each
// item has a presigned `audio_url` (+ `duration`, `description`, `name`, `score`).
// `query` is required (≥1 char, empty → HTTP 400) and `limit` is capped at 50.
// `minScore`: omit to use the server default (0.7). That default is TOO HIGH for
// sound_effects — good SFX hits score ~0.5–0.67, so callers wanting SFX should
// pass a lower floor (~0.4); music scores high and is fine at the default.
export async function searchSounds(query, type, headers, { limit = 5, minScore } = {}) {
const params = new URLSearchParams({ query, type, limit: String(limit) });
if (minScore != null) params.set("min_score", String(minScore));
const payload = await heygenJSON(`/audio/sounds?${params.toString()}`, { headers });
// `data` comes back as a ranked array (best first). Older responses keyed it by
// numeric index ("0","1",…); normalize both shapes to an array (empty → []).
const data = payload?.data ?? payload;
if (Array.isArray(data)) return data;
if (data && typeof data === "object") return Object.values(data);
throw new Error(
`unexpected /audio/sounds shape — top keys: ${Object.keys(payload ?? {}).join(", ")}`,
);
}
audio/scripts/lib/heygen.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { heygenAuthHeaders, heygenAuthMethod } from "./heygen.mjs";
function withCleanHeygenEnv(fn) {
const previousApiKey = process.env.HEYGEN_API_KEY;
const previousHyperframesApiKey = process.env.HYPERFRAMES_API_KEY;
const previousConfigDir = process.env.HEYGEN_CONFIG_DIR;
try {
delete process.env.HEYGEN_API_KEY;
delete process.env.HYPERFRAMES_API_KEY;
delete process.env.HEYGEN_CONFIG_DIR;
return fn();
} finally {
if (previousApiKey === undefined) delete process.env.HEYGEN_API_KEY;
else process.env.HEYGEN_API_KEY = previousApiKey;
if (previousHyperframesApiKey === undefined) delete process.env.HYPERFRAMES_API_KEY;
else process.env.HYPERFRAMES_API_KEY = previousHyperframesApiKey;
if (previousConfigDir === undefined) delete process.env.HEYGEN_CONFIG_DIR;
else process.env.HEYGEN_CONFIG_DIR = previousConfigDir;
}
}
test("heygenAuthHeaders does not tag API-key requests as CLI traffic, but still carries the media-use tool tag", () => {
withCleanHeygenEnv(() => {
process.env.HEYGEN_API_KEY = "hg_test";
// API-key requests use normal billing; the backend ignores the cli-source
// header for them, so it's not sent. The tool-attribution header IS sent on
// every media-use call (any auth type) so the backend can isolate media-use.
assert.deepEqual(heygenAuthHeaders(), {
"X-Api-Key": "hg_test",
"X-HeyGen-Client-Source": "media-use",
});
});
});
test("heygenAuthHeaders tags OAuth requests as CLI traffic and with the media-use tool tag", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir;
writeFileSync(
join(dir, "credentials"),
JSON.stringify({
oauth: {
access_token: "at_test",
expires_at: "2099-01-01T00:00:00Z",
},
}),
);
assert.deepEqual(heygenAuthHeaders(), {
Authorization: "Bearer at_test",
"X-HeyGen-Source": "cli",
"X-HeyGen-Client-Source": "media-use",
});
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
test("heygenAuthMethod returns api_key for an env API key, without tagging headers", () => {
withCleanHeygenEnv(() => {
process.env.HEYGEN_API_KEY = "hg_test";
assert.equal(heygenAuthMethod(), "api_key");
});
});
test("heygenAuthMethod returns oauth for a live OAuth credential", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir;
writeFileSync(
join(dir, "credentials"),
JSON.stringify({
oauth: {
access_token: "at_test",
expires_at: "2099-01-01T00:00:00Z",
},
}),
);
assert.equal(heygenAuthMethod(), "oauth");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
test("heygenAuthMethod returns null with no credential at all", () => {
withCleanHeygenEnv(() => {
const dir = mkdtempSync(join(tmpdir(), "heygen-cred-"));
try {
process.env.HEYGEN_CONFIG_DIR = dir; // no credentials file written
assert.equal(heygenAuthMethod(), null);
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});
audio/scripts/lib/python.mjs›
// python.mjs — resolve which Python 3 executable to spawn, per platform.
//
// The audio engine (tts.mjs, bgm.mjs) shells out to `python3` for ElevenLabs
// TTS and the local Lyria/MusicGen BGM paths. `python3` is the right name on
// macOS/Linux, but on Windows the python.org installer only creates
// `python.exe` plus the `py` launcher — there is no `python3.exe` (only the
// Microsoft Store build adds one). So a bare `spawn("python3", …)` ENOENTs on a
// standard Windows Python install, silently disabling every Python-backed audio
// feature until the user hand-creates a `python3.exe` shim (reported twice).
//
// Resolve once, per process: probe the platform's candidates in order and take
// the first that actually runs. `py` is the launcher, so it needs a `-3` arg to
// select Python 3 — hence candidates are argv PREFIXES, not bare names.
import { spawnSync } from "node:child_process";
function defaultProbe(cmd, args) {
try {
return spawnSync(cmd, args, { stdio: "ignore" }).status === 0;
} catch {
return false;
}
}
/**
* Pick the argv prefix that launches Python 3 on this platform.
* Returns e.g. `["python3"]`, `["python"]`, or `["py", "-3"]`.
*
* Pure except for `probe` (which runs `<cmd> … --version`); both `platform`
* and `probe` are injectable so every branch is unit-testable without spawning.
* If nothing probes OK, falls back to the canonical name for the platform so
* the eventual spawn fails loudly exactly as it did before — never worse.
*/
export function resolvePythonCommand(platform = process.platform, probe = defaultProbe) {
const candidates =
platform === "win32" ? [["python3"], ["python"], ["py", "-3"]] : [["python3"], ["python"]];
for (const prefix of candidates) {
if (probe(prefix[0], [...prefix.slice(1), "--version"])) return prefix;
}
return candidates[0];
}
let cached = null;
/** Cached `resolvePythonCommand()` — probing spawns, so resolve at most once. */
export function pythonCommand() {
if (!cached) cached = resolvePythonCommand();
return cached;
}
/**
* Build a `{ cmd, args }` for running Python 3 with `extraArgs`, using the
* resolved (or supplied) prefix. Keeps the launcher's `-3` (and any future
* prefix args) ahead of the caller's own arguments.
*/
export function pythonInvocation(extraArgs, prefix = pythonCommand()) {
return { cmd: prefix[0], args: [...prefix.slice(1), ...extraArgs] };
}
/** Test-only: clear the cached resolution so a test can re-probe. */
export function _resetPythonCommandCacheForTests() {
cached = null;
}
audio/scripts/lib/python.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { resolvePythonCommand, pythonInvocation } from "./python.mjs";
// Regression: on Windows a standard python.org install has no `python3.exe`
// (only `python.exe` + the `py` launcher), so `spawn("python3", …)` ENOENTs and
// every Python-backed audio feature silently no-ops. resolvePythonCommand takes
// injectable platform/probe params so all branches are testable without
// spawning a real interpreter.
// probeFor(names): a probe that reports success only for the given argv-0 names.
function probeFor(...names) {
const ok = new Set(names);
return (cmd) => ok.has(cmd);
}
test("non-win32 uses python3 when it runs", () => {
assert.deepEqual(resolvePythonCommand("linux", probeFor("python3")), ["python3"]);
assert.deepEqual(resolvePythonCommand("darwin", probeFor("python3")), ["python3"]);
});
test("win32 prefers python3 when the Microsoft Store build provides it", () => {
assert.deepEqual(resolvePythonCommand("win32", probeFor("python3", "python", "py")), ["python3"]);
});
test("win32 falls back to python.exe when python3 is absent (python.org install)", () => {
// The exact reported scenario: no python3, but `python` exists.
assert.deepEqual(resolvePythonCommand("win32", probeFor("python", "py")), ["python"]);
});
test("win32 falls back to the py launcher with -3 when only py exists", () => {
assert.deepEqual(resolvePythonCommand("win32", probeFor("py")), ["py", "-3"]);
});
test("py launcher is probed as `py -3 --version`, not bare `py`", () => {
const seen = [];
const probe = (cmd, args) => {
seen.push([cmd, ...args]);
return cmd === "py";
};
resolvePythonCommand("win32", probe);
assert.deepEqual(seen.at(-1), ["py", "-3", "--version"]);
});
test("falls back to the canonical name (loud failure, unchanged) when nothing runs", () => {
// No interpreter anywhere — must not throw, and must return python3 so the
// eventual spawn fails exactly as it did before this fix, never worse.
assert.deepEqual(
resolvePythonCommand("win32", () => false),
["python3"],
);
assert.deepEqual(
resolvePythonCommand("linux", () => false),
["python3"],
);
});
test("pythonInvocation prepends the resolved prefix ahead of caller args", () => {
assert.deepEqual(pythonInvocation(["-c", "import x"], ["python"]), {
cmd: "python",
args: ["-c", "import x"],
});
// The py launcher's -3 must stay ahead of the caller's own arguments.
assert.deepEqual(pythonInvocation(["-c", "import x"], ["py", "-3"]), {
cmd: "py",
args: ["-3", "-c", "import x"],
});
});
audio/scripts/lib/sfx.mjs›
// sfx.mjs — sound effects for the media audio engine. Provider-gated (NOT a
// per-cue merge): the decision is made once, by whether HeyGen is configured —
// mirroring how TTS and BGM degrade.
//
// HeyGen credential present → retrieve EVERY cue from HeyGen's audio library
// (/v3/audio/sounds, type=sound_effects, min_score=0.4). The bundled
// library is NOT consulted.
// HeyGen credential absent → resolve cues against the bundled 21-file
// library (assets/sfx/manifest.json), copying matched files into the
// project. Offline, deterministic, free.
//
// A cue that matches nothing is skipped (recorded as an anomaly); SFX never
// blocks a render. Every cue sits at volume ~0.35, under voice + BGM.
import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { downloadTo, searchSounds } from "./heygen.mjs";
const SFX_VOLUME = 0.35;
const slug = (s) =>
s
.toLowerCase()
.replace(/[^a-z0-9]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 40) || "x";
const r3 = (x) => Number(x.toFixed(3));
// cues: [{ id, name }] (id = the line/frame/scene the cue fires in). Returns
// { sfx: [{ id, name, file, source, offset_s, duration_s, volume }], anomalies }.
export async function resolveSfx({ cues, heygenOK, headers, hyperframesDir, sfxLibDir }) {
const sfx = [];
const anomalies = [];
const destDir = join(hyperframesDir, "assets", "sfx");
// Dedupe identical (id,name) cues — the same effect named twice in one line
// downloads/copies once.
const seen = new Set();
const uniq = cues.filter((c) => {
const k = `${c.id}:${c.name}`;
if (seen.has(k)) return false;
seen.add(k);
return true;
});
if (heygenOK) {
for (const { id, name } of uniq) {
try {
// SFX hits score low (~0.5–0.67), below the API's default 0.7 which
// silently drops most named cues — floor to 0.4. (BGM/music score high
// and keep the default.)
const results = await searchSounds(name, "sound_effects", headers, {
limit: 3,
minScore: 0.4,
});
if (!results.length) {
anomalies.push(`sfx "${name}" (id ${id}): no HeyGen match — skipped`);
continue;
}
const top = results[0];
const file = `assets/sfx/${slug(name)}.mp3`;
await downloadTo(top.audio_url, join(hyperframesDir, file));
sfx.push({
id,
name,
file,
source: "heygen",
offset_s: 0,
duration_s: typeof top.duration === "number" ? r3(top.duration) : 1.0,
volume: SFX_VOLUME,
});
} catch (e) {
anomalies.push(`sfx "${name}" (id ${id}): retrieval failed — ${e.message}`);
}
}
return { sfx, anomalies };
}
// ── offline: bundled library ──
const manifestPath = join(sfxLibDir, "manifest.json");
if (!existsSync(manifestPath)) {
if (uniq.length)
anomalies.push(`no HeyGen credential and no SFX library at ${sfxLibDir} — all cues dropped`);
return { sfx, anomalies };
}
let manifest;
try {
manifest = JSON.parse(readFileSync(manifestPath, "utf8"));
} catch (e) {
anomalies.push(`SFX manifest parse failed (${e.message}) — all cues dropped`);
return { sfx, anomalies };
}
// Build lookups: by manifest key, by file basename, and by slug of either, so
// a cue can name "whoosh", "whoosh.mp3", or "ui click" (→ slug match).
const byKey = new Map();
for (const [key, entry] of Object.entries(manifest)) {
if (!entry?.file || !isFinite(entry.duration)) continue;
const rec = { key, file: entry.file, duration: entry.duration };
byKey.set(key, rec);
byKey.set(entry.file, rec);
byKey.set(slug(key), rec);
byKey.set(slug(entry.file.replace(/\.\w+$/, "")), rec);
}
mkdirSync(destDir, { recursive: true });
for (const { id, name } of uniq) {
const hit = byKey.get(name) ?? byKey.get(slug(name));
if (!hit) {
const known = [...new Set([...byKey.values()].map((v) => v.key))].slice(0, 8).join(", ");
anomalies.push(
`sfx "${name}" (id ${id}): not in bundled library — skipped (have: ${known}…)`,
);
continue;
}
const src = join(sfxLibDir, hit.file);
const destRel = `assets/sfx/${hit.file}`;
const dest = join(hyperframesDir, destRel);
// The bundled library may be incomplete: some installs of the skill ship
// manifest.json without the actual mp3s. Pushing an sfx entry that points at
// a file we never copied produces a dangling reference that silently drops
// downstream ("not on disk"). Surface it as a loud anomaly and skip the cue
// instead, so the audio_meta never references a missing file.
if (!existsSync(dest)) {
if (!existsSync(src)) {
anomalies.push(
`sfx "${name}" (id ${id}): bundled file ${hit.file} missing from the offline ` +
`library (${sfxLibDir}) — skipped. Reinstall the media-use skill to ` +
`restore assets/sfx/*.mp3, or configure a HeyGen credential for retrieval.`,
);
continue;
}
copyFileSync(src, dest);
}
sfx.push({
id,
name,
file: destRel,
source: "local",
offset_s: 0,
duration_s: r3(hit.duration),
volume: SFX_VOLUME,
});
}
return { sfx, anomalies };
}
audio/scripts/lib/sfx.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, mkdirSync, writeFileSync, existsSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { resolveSfx } from "./sfx.mjs";
// Offline (no HeyGen) SFX resolution: the bundled library may ship manifest.json
// without the actual mp3s. The old code copied only when the source existed but
// pushed the sfx entry unconditionally — producing a dangling reference that
// silently dropped downstream ("not on disk"). These tests lock in the loud
// behavior: a present file is copied + referenced; a missing file yields an
// anomaly and NO dangling entry.
async function withDirs(fn) {
const root = mkdtempSync(join(tmpdir(), "hf-sfx-"));
const libDir = join(root, "lib");
const projDir = join(root, "proj");
mkdirSync(libDir, { recursive: true });
mkdirSync(projDir, { recursive: true });
try {
// `await` is load-bearing: without it the finally cleanup runs before the
// async test body resolves, deleting the temp dir mid-assertion.
return await fn({ libDir, projDir });
} finally {
rmSync(root, { recursive: true, force: true });
}
}
test("offline: copies and references a present bundled file", async () => {
await withDirs(async ({ libDir, projDir }) => {
writeFileSync(
join(libDir, "manifest.json"),
JSON.stringify({ whoosh: { file: "whoosh.mp3", duration: 0.8 } }),
);
writeFileSync(join(libDir, "whoosh.mp3"), "ID3-fake-bytes");
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "s1", name: "whoosh" }],
heygenOK: false,
hyperframesDir: projDir,
sfxLibDir: libDir,
});
assert.equal(sfx.length, 1);
assert.equal(sfx[0].file, "assets/sfx/whoosh.mp3");
assert.equal(sfx[0].source, "local");
assert.ok(existsSync(join(projDir, "assets/sfx/whoosh.mp3")), "mp3 copied into project");
assert.equal(anomalies.length, 0);
});
});
test("offline: a matched-but-missing bundled file yields an anomaly and NO dangling entry", async () => {
await withDirs(async ({ libDir, projDir }) => {
// Manifest names whoosh.mp3, but the mp3 was never shipped (the reported bug).
writeFileSync(
join(libDir, "manifest.json"),
JSON.stringify({ whoosh: { file: "whoosh.mp3", duration: 0.8 } }),
);
const { sfx, anomalies } = await resolveSfx({
cues: [{ id: "s1", name: "whoosh" }],
heygenOK: false,
hyperframesDir: projDir,
sfxLibDir: libDir,
});
assert.equal(sfx.length, 0, "no dangling entry for a file that was never copied");
assert.equal(anomalies.length, 1);
assert.match(anomalies[0], /missing from the offline library/);
assert.ok(!existsSync(join(projDir, "assets/sfx/whoosh.mp3")), "nothing copied");
});
});
audio/scripts/lib/tts.mjs›
import { fetchMedia } from "../../../scripts/lib/media-fetch.mjs";
// tts.mjs — multi-provider TTS for the media audio engine. The provider chain,
// auto-detected from env, is the one documented in ../SKILL.md:
//
// 1. HeyGen (Starfish) — $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen.
// Direct v3 REST (NOT `hyperframes tts`, which in the published build is
// Kokoro-only and silently ignores a HeyGen key). Returns word_timestamps
// in the same call, so no separate transcribe pass.
// 2. ElevenLabs — $ELEVENLABS_API_KEY + `pip install elevenlabs`. No
// word timings → caller chains transcribeWav().
// 3. Kokoro-82M (local) — always available, via the published `hyperframes tts`
// CLI. No word timings → caller chains transcribeWav().
//
// "HeyGen available" is decided by CREDENTIAL presence (heygenCredential), never
// by the CLI — see the note above.
import { spawn, spawnSync } from "node:child_process";
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { heygenAuthHeaders, heygenCredential, heygenJSON } from "./heygen.mjs";
import { pythonInvocation } from "./python.mjs";
// ── provider detection ────────────────────────────────────────────────────────
export function heygenAvailable() {
return heygenCredential() !== null;
}
export function elevenlabsAvailable() {
if (!process.env.ELEVENLABS_API_KEY) return false;
const { cmd, args } = pythonInvocation(["-c", "import elevenlabs"]);
const r = spawnSync(cmd, args, {
stdio: "ignore",
});
return r.status === 0;
}
// First available provider wins; an explicit choice is honored (and validated).
export function pickProvider(userProvider) {
if (userProvider) {
if (!["heygen", "elevenlabs", "kokoro"].includes(userProvider))
throw new Error(`invalid provider "${userProvider}" (heygen | elevenlabs | kokoro)`);
if (userProvider === "heygen" && !heygenAvailable())
throw new Error(
"provider=heygen but no HeyGen credentials (set $HEYGEN_API_KEY or run `npx hyperframes auth login`)",
);
if (userProvider === "elevenlabs" && !process.env.ELEVENLABS_API_KEY)
throw new Error("provider=elevenlabs but $ELEVENLABS_API_KEY is not set");
return userProvider;
}
return heygenAvailable() ? "heygen" : elevenlabsAvailable() ? "elevenlabs" : "kokoro";
}
// ── voice resolution ──────────────────────────────────────────────────────────
// HeyGen /v3/voices/speech only accepts STARFISH voice_ids; auto-pick the first
// English public starfish voice when none is pinned. ElevenLabs/Kokoro have
// their own defaults.
export async function resolveVoiceId({ provider, userVoice, lang = "en" }) {
if (userVoice) return userVoice;
if (provider === "elevenlabs") return "21m00Tcm4TlvDq8ikWAM"; // Rachel
if (provider === "kokoro") {
if (lang === "en") return "am_michael";
throw new Error("Kokoro non-English needs an explicit --voice (see references/tts.md)");
}
// heygen — pin a fixed English default so the choice is deterministic. The old
// "first English voice the API returns" drifts whenever HeyGen re-sorts the
// public catalog. Marcia (mature, low female). Override with --voice / request.voice.
if (lang === "en") return "05f19352e8f74b0392a8f411eba40de1"; // Marcia · English · female
// Non-English: no fixed default — fall back to the first matching catalog voice.
const payload = await heygenJSON(`/voices?engine=starfish&type=public&limit=50`, {
headers: heygenAuthHeaders(),
});
const voices = payload.data ?? payload.voices ?? [];
const pick = voices.find((v) => v.language === "English") ?? voices[0];
if (!pick) throw new Error("no public starfish voice to default to — pass --voice");
return pick.voice_id;
}
// ── helpers ─────────────────────────────────────────────────────────────────
export function withWordIds(words) {
return (words ?? []).map((w, i) => ({
id: `w${i}`,
text: w.text,
start: w.start,
end: w.end,
}));
}
// `ffmpeg -i <file>` prints a `Duration: HH:MM:SS.ms` line to stderr even
// though it exits non-zero with no output requested. Parsing pulled out as
// a pure function so the ENOENT fallback below can be tested without
// depending on whether ffprobe/ffmpeg are actually installed on the
// machine running the tests.
export function parseFfmpegDurationBanner(stderrText) {
const match = /Duration:\s*(\d+):(\d+):(\d+(?:\.\d+)?)/.exec(stderrText ?? "");
if (!match) return NaN;
const [, hours, minutes, seconds] = match;
return Number(hours) * 3600 + Number(minutes) * 60 + Number(seconds);
}
// Some "essentials"-style ffmpeg distributions (common on Windows) ship
// ffmpeg.exe without ffprobe.exe. ffprobeDuration's caller (audio.mjs)
// otherwise reads a spurious NaN as "the WAV file is corrupt" and drops an
// already-successfully-synthesized TTS line, rather than "the tool for
// measuring it is missing".
function ffmpegDurationFallback(absPath) {
const r = spawnSync("ffmpeg", ["-i", absPath], { encoding: "utf8" });
return parseFfmpegDurationBanner(r.stderr);
}
export function ffprobeDuration(absPath) {
const r = spawnSync(
"ffprobe",
["-v", "error", "-show_entries", "format=duration", "-of", "default=nw=1:nk=1", "--", absPath],
{ encoding: "utf8" },
);
if (r.error?.code === "ENOENT") return ffmpegDurationFallback(absPath);
if (r.status !== 0) return NaN;
return parseFloat(String(r.stdout).trim());
}
export function resolveNpxCliFromNpmExecPath(
npmExecPath = process.env.npm_execpath,
pathExists = existsSync,
) {
if (!npmExecPath) return null;
const fileName = npmExecPath.replace(/\\/g, "/").split("/").pop()?.toLowerCase();
const npxCliPath =
fileName === "npx-cli.js" ? npmExecPath : join(dirname(npmExecPath), "npx-cli.js");
return pathExists(npxCliPath) ? npxCliPath : null;
}
export function resolveNpxCliPath(
npmExecPath = process.env.npm_execpath,
nodeExecPath = process.env.npm_node_execpath || process.execPath,
pathExists = existsSync,
) {
const fromNpm = resolveNpxCliFromNpmExecPath(npmExecPath, pathExists);
if (fromNpm) return fromNpm;
const besideNode = join(dirname(nodeExecPath), "node_modules", "npm", "bin", "npx-cli.js");
return pathExists(besideNode) ? besideNode : null;
}
export function resolveSpawnCommand(
cmd,
args,
opts = {},
platform = process.platform,
env = process.env,
pathExists = existsSync,
) {
if (cmd !== "npx" || platform !== "win32") {
return { cmd, args, opts: { stdio: "ignore", ...opts } };
}
// On Windows, npx resolves to npx.cmd, which Node cannot execute directly.
// Avoid `shell:true` and the .cmd shim entirely by invoking npm's JS CLI with
// node, preserving request-provided values as argv data instead of shell text.
const nodeExecPath = env.npm_node_execpath || process.execPath;
const npxCliPath = resolveNpxCliPath(env.npm_execpath, nodeExecPath, pathExists);
if (!npxCliPath) return null;
return {
cmd: nodeExecPath,
args: [npxCliPath, ...args.map((arg) => String(arg))],
opts: { stdio: "ignore", windowsHide: true, ...opts },
};
}
// `platform`/`spawnFn` params (default process.platform / the real spawn)
// exist so tests can exercise the win32 branch without mocking node:child_process
// (its ESM exports are non-configurable, so mock.method can't patch it).
// One-shot so a whole batch of TTS lines doesn't repeat the same diagnostic.
let _warnedNpxResolution = false;
/** Test-only: reset the one-shot npx-resolution warning latch. */
export function _resetNpxResolutionWarnForTests() {
_warnedNpxResolution = false;
}
export function spawnP(
cmd,
args,
opts = {},
platform = process.platform,
spawnFn = spawn,
env = process.env,
pathExists = existsSync,
) {
const resolved = resolveSpawnCommand(cmd, args, opts, platform, env, pathExists);
if (!resolved) {
// resolveSpawnCommand only returns null for the npx-on-win32 case where
// neither npm's configured CLI nor the beside-node fallback exists. Without
// this, every call silently returns status:-1 and stdio:"ignore" hides why.
if (!_warnedNpxResolution) {
_warnedNpxResolution = true;
const reason = env.npm_execpath
? `npm_execpath (${env.npm_execpath}) and the beside-node npm fallback could not be found`
: "npm_execpath is unset and the beside-node npm fallback could not be found";
console.error(
`[media-use] Cannot run "${cmd}" on Windows: ${reason}. ` +
`Every "${cmd}" call is being skipped. Install npm with Node, or run via ` +
`\`npx\`/\`npm run\` with a valid npm_execpath.`,
);
}
return Promise.resolve({ status: -1 });
}
return new Promise((resolve) => {
const p = spawnFn(resolved.cmd, resolved.args, resolved.opts);
p.on("exit", (code) => resolve({ status: code ?? -1 }));
p.on("error", () => resolve({ status: -1 }));
});
}
// mp3/whatever bytes → wav 44.1k mono at destWav (ffmpeg detects true format).
function transcodeToWav(bytes, destWav) {
const td = mkdtempSync(join(tmpdir(), "hf-tts-"));
const tmp = join(td, "a.mp3");
writeFileSync(tmp, bytes);
mkdirSync(dirname(destWav), { recursive: true });
const ff = spawnSync(
"ffmpeg",
["-y", "-loglevel", "error", "-i", tmp, "-ar", "44100", "-ac", "1", destWav],
{ stdio: "ignore" },
);
rmSync(td, { recursive: true, force: true });
return ff.status === 0 && existsSync(destWav);
}
const ELEVENLABS_PY = `
import os, sys
from elevenlabs.client import ElevenLabs
from elevenlabs import save
client = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])
text = open(sys.argv[1]).read()
audio = client.text_to_speech.convert(
text=text, voice_id=sys.argv[2],
model_id="eleven_multilingual_v2", output_format="mp3_44100_128",
)
save(audio, sys.argv[3])
`;
// ── synthesize one line ───────────────────────────────────────────────────────
// Writes wav at wavAbs. Returns { ok, words, error } — words is the raw
// [{text,start,end}] array for HeyGen (native), or null for ElevenLabs/Kokoro
// (caller must transcribeWav). Never throws; failures return { ok:false, error }
// where `error` states WHY (so the caller can surface it, not a bare "TTS failed").
export async function synthesizeOne({
provider,
text,
voiceId,
lang = "en",
speed = 1.0,
wavAbs,
hyperframesDir,
}) {
if (provider === "heygen") return synthesizeHeygen({ text, voiceId, lang, speed, wavAbs });
if (provider === "elevenlabs") {
// The Python helper writes straight to wavAbs; unlike heygen (transcodeToWav)
// and kokoro (the `hyperframes tts` CLI), it does NOT create the parent dir,
// so on a fresh project (no assets/voice/ yet) the save fails and the line is
// silently dropped as "TTS failed - omitted". Create it first, like the other
// providers do. Guarded so a mkdir failure (EACCES/EROFS) returns
// { ok:false } like the rest of this branch rather than throwing (the
// function's contract is "never throws; failures return { ok:false }").
try {
mkdirSync(dirname(wavAbs), { recursive: true });
} catch {
return { ok: false, words: null };
}
const { cmd, args } = pythonInvocation([
"-c",
ELEVENLABS_PY,
writeTmpText(text),
voiceId,
wavAbs,
]);
const r = await spawnP(cmd, args, {});
return synthResult(r, wavAbs, "elevenlabs (python)");
}
// kokoro — via the published CLI; --output is relative to the project dir.
const wavRel = relTo(hyperframesDir, wavAbs);
const args = ["hyperframes", "tts", writeTmpText(text), "--voice", voiceId, "--output", wavRel];
if (lang !== "en") args.push("--lang", lang);
const r = await spawnP("npx", args, { cwd: hyperframesDir });
return synthResult(r, wavAbs, "kokoro (npx hyperframes tts)");
}
// Shape a spawn result into { ok, words, error }, naming why on failure so the
// caller surfaces it instead of a bare "TTS failed".
export function synthResult(r, wavAbs, label) {
if (r.status === 0 && existsSync(wavAbs)) return { ok: true, words: null };
const why =
r.status !== 0 ? `${label} exited with status ${r.status}` : `${label} produced no wav file`;
return { ok: false, words: null, error: why };
}
// `deps` is injectable for tests; production uses the real network/ffmpeg impls.
// Every failure path returns an `error` string so the caller can surface WHY a
// line was dropped instead of the bare "TTS failed" that hid the real cause
// (e.g. an HTTP 402 plan_upgrade_required thrown by heygenJSON was swallowed).
export async function synthesizeHeygen({ text, voiceId, lang, speed, wavAbs }, deps = {}) {
const requestJSON = deps.heygenJSON ?? heygenJSON;
const authHeaders = deps.heygenAuthHeaders ?? heygenAuthHeaders;
const fetchImpl = deps.fetch ?? fetch;
const transcode = deps.transcodeToWav ?? transcodeToWav;
try {
const body = { text, voice_id: voiceId, speed };
if (lang !== "en") body.language = lang;
const payload = await requestJSON(`/voices/speech`, {
method: "POST",
headers: authHeaders(),
body,
});
const inner = payload.data ?? payload;
if (!inner.audio_url) {
return { ok: false, words: null, error: "HeyGen /voices/speech returned no audio_url" };
}
const res = await fetchMedia(inner.audio_url, { fetchImpl });
if (!res.ok) {
return { ok: false, words: null, error: `audio_url fetch failed: HTTP ${res.status}` };
}
const bytes = Buffer.from(await res.arrayBuffer());
// .wav output → transcode to 44.1k mono; .mp3 → raw bytes (no ffmpeg). The
// engine always asks for .wav; the standalone heygen-tts CLI may ask for .mp3.
if (wavAbs.endsWith(".wav")) {
if (!transcode(bytes, wavAbs)) {
return {
ok: false,
words: null,
error: "wav transcode failed (ffmpeg)",
};
}
} else {
mkdirSync(dirname(wavAbs), { recursive: true });
writeFileSync(wavAbs, bytes);
}
const words = Array.isArray(inner.word_timestamps)
? inner.word_timestamps
.filter((w) => w && typeof w.word === "string" && isFinite(w.start) && isFinite(w.end))
.filter((w) => !/^<.*>$/.test(w.word.trim())) // drop <start>/<end> sentinels
.map((w) => ({ text: w.word, start: w.start, end: w.end }))
: [];
return { ok: true, words };
} catch (e) {
return { ok: false, words: null, error: e?.message ? String(e.message) : String(e) };
}
}
// ElevenLabs/Kokoro have no word timings — run Whisper over the wav. Returns the
// flat [{id,text,start,end}] word array, or null. Each call uses a throwaway
// --dir so parallel scenes don't collide on transcript.json.
export async function transcribeWav({ wavRel, lang = "en", hyperframesDir }) {
const model = lang === "en" ? "small.en" : "small";
const td = mkdtempSync(join(tmpdir(), "hf-trans-"));
const args = ["hyperframes", "transcribe", wavRel, "--model", model, "--dir", td];
if (lang !== "en") args.push("--language", lang);
const r = await spawnP("npx", args, { cwd: hyperframesDir });
let words = null;
if (r.status === 0) {
const src = join(td, "transcript.json");
if (existsSync(src)) {
try {
const arr = JSON.parse(readFileSync(src, "utf8"));
if (Array.isArray(arr) && arr.length) words = arr;
} catch {}
}
}
rmSync(td, { recursive: true, force: true });
return words;
}
// ── tiny local utils ──────────────────────────────────────────────────────────
function writeTmpText(text) {
const td = mkdtempSync(join(tmpdir(), "hf-txt-"));
const p = join(td, "line.txt");
writeFileSync(p, text);
return p;
}
function relTo(base, abs) {
return abs.startsWith(base + "/") ? abs.slice(base.length + 1) : abs;
}
audio/scripts/lib/tts.spawn.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { EventEmitter } from "node:events";
import {
resolveNpxCliFromNpmExecPath,
resolveNpxCliPath,
resolveSpawnCommand,
spawnP,
_resetNpxResolutionWarnForTests,
} from "./tts.mjs";
// Regression: on Windows, npx resolves to npx.cmd, which spawn() cannot exec
// without shell:true — it fails ENOENT, silently swallowed as ok:false by the
// caller. spawnP takes injectable platform/spawnFn params so this doesn't
// need to touch the real process.platform or mock node:child_process (whose
// ESM exports are non-configurable).
function fakeSpawn(captured) {
return (cmd, args, opts) => {
captured.push({ cmd, args, opts });
const p = new EventEmitter();
setImmediate(() => p.emit("exit", 0));
return p;
};
}
const envWithNpxCli = {
npm_execpath: "/opt/node/lib/node_modules/npm/bin/npm-cli.js",
npm_node_execpath: "/opt/node/bin/node",
};
const npxCliPath = "/opt/node/lib/node_modules/npm/bin/npx-cli.js";
const pathExists = (path) => path === npxCliPath;
test("resolveNpxCliFromNpmExecPath finds npx-cli next to npm-cli", () => {
assert.equal(resolveNpxCliFromNpmExecPath(envWithNpxCli.npm_execpath, pathExists), npxCliPath);
});
test("resolveNpxCliPath finds npx-cli beside node when npm_execpath is unset", () => {
const node = "C:/Program Files/nodejs/node.exe";
const expected = "C:/Program Files/nodejs/node_modules/npm/bin/npx-cli.js";
assert.equal(
resolveNpxCliPath(undefined, node, (path) => path === expected),
expected,
);
});
test("resolveSpawnCommand routes npx through node+npx-cli on win32 without shell:true", () => {
const resolved = resolveSpawnCommand(
"npx",
["hyperframes", "tts", "C:\\Users\\Test User\\line.txt", "--voice", "am_michael"],
{},
"win32",
envWithNpxCli,
pathExists,
);
assert.ok(resolved);
assert.equal(resolved.cmd, envWithNpxCli.npm_node_execpath);
assert.deepEqual(resolved.args, [
npxCliPath,
"hyperframes",
"tts",
"C:\\Users\\Test User\\line.txt",
"--voice",
"am_michael",
]);
assert.equal(resolved.opts.shell, undefined);
});
test("resolveSpawnCommand preserves Windows npx shell metacharacters as argv data", () => {
const resolved = resolveSpawnCommand(
"npx",
["hyperframes", "tts", "hello & calc"],
{},
"win32",
envWithNpxCli,
pathExists,
);
assert.ok(resolved);
assert.deepEqual(resolved.args, [npxCliPath, "hyperframes", "tts", "hello & calc"]);
});
test("spawnP uses the resolved node+npx-cli command for npx on win32", async () => {
const captured = [];
await spawnP(
"npx",
["hyperframes", "tts"],
{},
"win32",
fakeSpawn(captured),
envWithNpxCli,
pathExists,
);
assert.equal(captured.length, 1);
assert.equal(captured[0].cmd, envWithNpxCli.npm_node_execpath);
assert.deepEqual(captured[0].args, [npxCliPath, "hyperframes", "tts"]);
assert.equal(captured[0].opts.shell, undefined);
});
test("spawnP does not enable shell for npx on darwin/linux", async () => {
const captured = [];
await spawnP("npx", ["hyperframes", "tts"], {}, "darwin", fakeSpawn(captured));
assert.equal(captured[0].cmd, "npx");
assert.deepEqual(captured[0].args, ["hyperframes", "tts"]);
assert.equal(captured[0].opts.shell, undefined);
});
test("spawnP does not enable shell for non-npx commands even on win32", async () => {
const captured = [];
await spawnP("python3", ["-c", "pass"], {}, "win32", fakeSpawn(captured));
assert.equal(captured[0].cmd, "python3");
assert.deepEqual(captured[0].args, ["-c", "pass"]);
assert.equal(captured[0].opts.shell, undefined);
});
test("spawnP resolves npx beside node when npm_execpath is unset on win32", async () => {
_resetNpxResolutionWarnForTests();
const captured = [];
const node = "C:/Program Files/nodejs/node.exe";
const npxCli = "C:/Program Files/nodejs/node_modules/npm/bin/npx-cli.js";
const result = await spawnP(
"npx",
["hyperframes", "tts"],
{},
"win32",
fakeSpawn(captured),
{ npm_node_execpath: node },
(path) => path === npxCli,
);
assert.equal(result.status, 0);
assert.equal(captured.length, 1);
assert.equal(captured[0].cmd, node);
assert.deepEqual(captured[0].args, [npxCli, "hyperframes", "tts"]);
});
test("spawnP warns once with an accurate diagnostic when neither npx path exists", async () => {
_resetNpxResolutionWarnForTests();
const errors = [];
const originalError = console.error;
console.error = (message) => errors.push(String(message));
try {
const env = { npm_execpath: "C:/missing/npm-cli.js", npm_node_execpath: "C:/node/node.exe" };
const missing = () => false;
assert.equal(
(await spawnP("npx", ["hyperframes", "tts"], {}, "win32", fakeSpawn([]), env, missing))
.status,
-1,
);
assert.equal(
(await spawnP("npx", ["hyperframes", "tts"], {}, "win32", fakeSpawn([]), env, missing))
.status,
-1,
);
} finally {
console.error = originalError;
}
assert.equal(errors.length, 1);
assert.match(errors[0], /npm_execpath \(C:\/missing\/npm-cli\.js\)/);
assert.doesNotMatch(errors[0], /npm_execpath is not set/);
});
audio/scripts/lib/tts.test.mjs›
import { test } from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, writeFileSync, chmodSync, rmSync, existsSync } from "node:fs";
import { join, dirname } from "node:path";
import { tmpdir } from "node:os";
import {
parseFfmpegDurationBanner,
ffprobeDuration,
synthesizeOne,
synthesizeHeygen,
synthResult,
} from "./tts.mjs";
test("parseFfmpegDurationBanner reads ffmpeg's stderr Duration line", () => {
const stderr = [
"ffmpeg version 6.0",
"Input #0, wav, from 'a.wav':",
" Duration: 00:00:03.42, bitrate: 705 kb/s",
"At least one output file must be specified",
].join("\n");
assert.equal(parseFfmpegDurationBanner(stderr), 3.42);
});
test("parseFfmpegDurationBanner handles an hours component", () => {
const stderr = " Duration: 01:02:03.50, start: 0.000000, bitrate: 128 kb/s";
assert.equal(parseFfmpegDurationBanner(stderr), 3723.5);
});
test("parseFfmpegDurationBanner returns NaN when there is no Duration line", () => {
assert.ok(Number.isNaN(parseFfmpegDurationBanner("ffmpeg: command not found")));
assert.ok(Number.isNaN(parseFfmpegDurationBanner("")));
assert.ok(Number.isNaN(parseFfmpegDurationBanner(undefined)));
});
// Regression for the actual bug: ffprobeDuration used to collapse "ffprobe
// binary is missing" (ENOENT — the "essentials"-style Windows ffmpeg build
// with no ffprobe.exe) and "file is genuinely unreadable" into the same NaN,
// giving audio.mjs no way to tell "measure differently" from "give up".
//
// Builds an isolated PATH containing only a fake `ffmpeg` stub (no `ffprobe`
// at all) so ffprobeDuration's spawnSync("ffprobe", ...) call ENOENTs for
// real, then verifies it recovers the duration via the ffmpeg fallback
// instead of returning NaN.
test("ffprobeDuration falls back to ffmpeg when the ffprobe binary itself is missing", () => {
const dir = mkdtempSync(join(tmpdir(), "tts-ffprobe-fallback-"));
const fakeFfmpeg = join(dir, "ffmpeg");
writeFileSync(
fakeFfmpeg,
"#!/bin/sh\necho 'Duration: 00:00:02.50, start: 0.000000, bitrate: 128 kb/s' 1>&2\nexit 1\n",
);
chmodSync(fakeFfmpeg, 0o755);
const originalPath = process.env.PATH;
try {
process.env.PATH = dir; // only the fake ffmpeg resolves; no real ffprobe on this PATH
assert.equal(ffprobeDuration("/does/not/matter.wav"), 2.5);
} finally {
process.env.PATH = originalPath;
rmSync(dir, { recursive: true, force: true });
}
});
test("ffprobeDuration returns NaN when neither ffprobe nor ffmpeg resolve", () => {
const dir = mkdtempSync(join(tmpdir(), "tts-no-binaries-"));
const originalPath = process.env.PATH;
try {
process.env.PATH = dir; // empty directory — nothing resolves
assert.ok(Number.isNaN(ffprobeDuration("/does/not/matter.wav")));
} finally {
process.env.PATH = originalPath;
rmSync(dir, { recursive: true, force: true });
}
});
test("synthesizeOne(elevenlabs) creates the output dir before writing", async () => {
const dir = mkdtempSync(join(tmpdir(), "tts-el-mkdir-"));
const wavAbs = join(dir, "assets", "voice", "line-0.wav"); // nested, not yet created
const savedKey = process.env.ELEVENLABS_API_KEY;
try {
// Unset the key so the Python side fails fast — the mkdir must run before
// the spawn regardless, which is what this guards.
delete process.env.ELEVENLABS_API_KEY;
await synthesizeOne({
provider: "elevenlabs",
text: "hi",
voiceId: "v",
wavAbs,
hyperframesDir: dir,
});
assert.ok(existsSync(dirname(wavAbs)), "output directory should be created");
} finally {
if (savedKey === undefined) delete process.env.ELEVENLABS_API_KEY;
else process.env.ELEVENLABS_API_KEY = savedKey;
rmSync(dir, { recursive: true, force: true });
}
});
test("synthesizeHeygen surfaces a thrown HTTP error (e.g. 402) instead of swallowing it", async () => {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
{
heygenAuthHeaders: () => ({}),
heygenJSON: async () => {
throw new Error("HeyGen POST /voices/speech → HTTP 402\nplan_upgrade_required");
},
},
);
assert.equal(res.ok, false);
assert.match(res.error, /402/);
assert.match(res.error, /plan_upgrade_required/);
});
test("synthesizeHeygen surfaces a failed audio_url fetch with its status", async () => {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
{
heygenAuthHeaders: () => ({}),
heygenJSON: async () => ({ data: { audio_url: "http://audio.example/x" } }),
fetch: async () => ({ ok: false, status: 403 }),
},
);
assert.equal(res.ok, false);
assert.match(res.error, /HTTP 403/);
});
test("synthesizeHeygen reports a missing audio_url", async () => {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: "/tmp/x.wav" },
{ heygenAuthHeaders: () => ({}), heygenJSON: async () => ({}) },
);
assert.equal(res.ok, false);
assert.match(res.error, /no audio_url/);
});
test("synthesizeHeygen reports wav transcode failures", async () => {
const dir = mkdtempSync(join(tmpdir(), "hf-tts-test-"));
try {
const res = await synthesizeHeygen(
{ text: "hi", voiceId: "v1", lang: "en", speed: 1, wavAbs: join(dir, "voice.wav") },
{
heygenAuthHeaders: () => ({}),
heygenJSON: async () => ({ data: { audio_url: "http://audio.example/x" } }),
fetch: async () => ({ ok: true, status: 200, arrayBuffer: async () => new ArrayBuffer(0) }),
transcodeToWav: () => false,
},
);
assert.equal(res.ok, false);
assert.equal(res.error, "wav transcode failed (ffmpeg)");
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
test("synthResult names a non-zero subprocess exit", () => {
const res = synthResult({ status: 2 }, "/tmp/none.wav", "kokoro (npx hyperframes tts)");
assert.equal(res.ok, false);
assert.match(res.error, /kokoro .* exited with status 2/);
});
audio/scripts/lyria-recipe.py›
#!/usr/bin/env python3
"""Generate BGM using Google Lyria RealTime API.
Usage:
python lyria-recipe.py --output <path> --duration <seconds> [tuning flags]
Requires:
$GOOGLE_API_KEY or $GEMINI_API_KEY environment variable (treated as aliases).
pip install google-genai python-dotenv. audio.mjs Step 4b installs these on
demand when a key is set but google.genai is not importable; if that install
fails it falls back to local MusicGen rather than leaving the video with no BGM.
"""
from __future__ import annotations
import argparse
import asyncio
import os
import sys
import wave
from pathlib import Path
# Windows sizes stdio to the ANSI code page (cp1252). These scripts emit UTF-8 on
# every platform; say so rather than depending on the console's code page. Carry
# `errors` across: reconfigure() resets it to "strict", and CPython deliberately gives
# stderr "backslashreplace" so the diagnostic path can never itself raise.
for _stream in (sys.stdout, sys.stderr):
if hasattr(_stream, "reconfigure"):
_stream.reconfigure(encoding="utf-8", errors=_stream.errors)
DEFAULT_PROMPT = "Uplifting corporate tech, bright and modern, gentle piano with synth pads"
SAMPLE_RATE = 48000
CHANNELS = 2
SAMPLE_WIDTH = 2 # 16-bit
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description="Generate BGM via Google Lyria RealTime.")
p.add_argument("--output", required=True, help="Output WAV path.")
p.add_argument("--duration", type=float, required=True, help="Target duration in seconds.")
p.add_argument("--prompt", default=DEFAULT_PROMPT, help="Mood / instrumentation prompt.")
p.add_argument("--negative-prompt", default=None, help="Styles to exclude (optional).")
p.add_argument("--bpm", type=int, default=110)
p.add_argument("--brightness", type=float, default=0.8, help="0-1, higher = brighter mood.")
p.add_argument("--density", type=float, default=0.5, help="0-1, higher = fuller mix.")
p.add_argument(
"--scale",
default="MAJOR",
help="MAJOR / MINOR / PENTATONIC / etc. — see google.genai.types.Scale. Pass empty string for none.",
)
return p.parse_args()
async def generate_bgm(args: argparse.Namespace) -> dict:
from google import genai
from google.genai import types
api_key = os.environ.get("GOOGLE_API_KEY") or os.environ.get("GEMINI_API_KEY") or ""
if not api_key:
raise RuntimeError("Neither GOOGLE_API_KEY nor GEMINI_API_KEY is set.")
client = genai.Client(
api_key=api_key,
http_options={"api_version": "v1alpha"},
)
out_path = Path(args.output)
out_path.parent.mkdir(parents=True, exist_ok=True)
target_bytes = int(args.duration * SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH)
cfg: dict = {"bpm": args.bpm, "temperature": 1.0}
if args.density is not None:
cfg["density"] = args.density
if args.brightness is not None:
cfg["brightness"] = args.brightness
if args.scale:
scale_enum = getattr(types.Scale, args.scale, None)
if scale_enum:
cfg["scale"] = scale_enum
prompts = [types.WeightedPrompt(text=args.prompt, weight=1.0)]
if args.negative_prompt:
prompts.append(types.WeightedPrompt(text=args.negative_prompt, weight=-1.0))
buf = bytearray()
timeout = args.duration + 8
async with client.aio.live.music.connect(
model="models/lyria-realtime-exp",
) as session:
await session.set_weighted_prompts(prompts=prompts)
await session.set_music_generation_config(
config=types.LiveMusicGenerationConfig(**cfg),
)
await session.play()
async def collect():
while len(buf) < target_bytes:
async for msg in session.receive():
sc = msg.server_content
if sc and sc.audio_chunks:
for chunk in sc.audio_chunks:
buf.extend(chunk.data)
if len(buf) >= target_bytes:
return
await asyncio.sleep(1e-6)
try:
await asyncio.wait_for(collect(), timeout=timeout)
except TimeoutError:
print(f"Timeout after {timeout:.0f}s, collected {len(buf)} bytes", file=sys.stderr)
audio = bytes(buf[:target_bytes])
with wave.open(str(out_path), "wb") as wf:
wf.setnchannels(CHANNELS)
wf.setsampwidth(SAMPLE_WIDTH)
wf.setframerate(SAMPLE_RATE)
wf.writeframes(audio)
actual_duration = len(audio) / (SAMPLE_RATE * CHANNELS * SAMPLE_WIDTH)
print(f"BGM: {out_path} ({actual_duration:.2f}s)")
return {"file": str(out_path), "duration_sec": round(actual_duration, 2)}
def main() -> None:
args = parse_args()
try:
asyncio.run(generate_bgm(args))
except RuntimeError as exc:
print(f"BGM generation failed: {exc}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
audio/scripts/wait-bgm.mjs›
#!/usr/bin/env node
// Phase 4c pre-assemble helper — wait for detached BGM, then write status.
//
// audio.mjs may launch Lyria / MusicGen in a detached process so voice work can
// keep moving. Before assemble-index.mjs decides whether to emit the BGM audio
// track, this script gives the background renderer a bounded chance to finish
// and converts log/process state into a small bgm_status.json file.
//
// Always exits 0 for normal pipeline use: missing/failed BGM should not block a
// voice/captions/SFX render. Structural invocation errors still exit 1.
//
// Usage:
// node wait-bgm.mjs --audio-meta ./audio_meta.json --hyperframes . \
// [--timeout-ms 120000] [--interval-ms 2000] [--out ./bgm_status.json]
import { existsSync, readFileSync, statSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
const argv = process.argv.slice(2);
const flag = (name, def) => {
const i = argv.indexOf(`--${name}`);
return i >= 0 && i + 1 < argv.length ? argv[i + 1] : def;
};
function die(msg) {
console.error(`✗ wait-bgm.mjs: ${msg}`);
process.exit(1);
}
const audioMetaPath = resolve(flag("audio-meta", "./audio_meta.json"));
const hyperframesDir = resolve(flag("hyperframes", "."));
const outPath = resolve(flag("out", join(hyperframesDir, "bgm_status.json")));
const timeoutMs = Math.max(0, Number(flag("timeout-ms", "120000")) || 0);
const intervalMs = Math.max(250, Number(flag("interval-ms", "2000")) || 2000);
function sleep(ms) {
return new Promise((resolveSleep) => setTimeout(resolveSleep, ms));
}
function isProcessAlive(pid) {
if (!pid || !Number.isFinite(Number(pid))) return false;
try {
process.kill(Number(pid), 0);
return true;
} catch {
return false;
}
}
function readTail(path, maxChars = 6000) {
if (!path) return "";
try {
const txt = readFileSync(path, "utf8");
return txt.slice(Math.max(0, txt.length - maxChars));
} catch (error) {
if (error.code === "ENOENT" || error.code === "ENOTDIR") return "";
throw error;
}
}
function detectFailure(logTail) {
if (!logTail) return "";
const lines = logTail.split("\n");
// Bare "out of range" over-matched benign BGM-renderer logs (e.g. a "sample rate
// out of range, resampling" notice), mislabelling a healthy track as failed and
// silently dropping the music. Anchor to the actual crash strings instead:
// Python "(list) index out of range" and torch "index … out of bounds".
const idx = lines.findIndex((line) =>
/(Traceback|IndexError|RuntimeError|Exception|Killed|No space left|Cannot allocate|index out of range|out of bounds)/i.test(
line,
),
);
if (idx < 0) return "";
return lines.slice(idx).join("\n").trim();
}
function writeStatus(status) {
const payload = {
generated_at: new Date().toISOString(),
...status,
};
writeFileSync(outPath, JSON.stringify(payload, null, 2) + "\n");
return payload;
}
if (!existsSync(audioMetaPath)) die(`audio_meta.json missing at ${audioMetaPath}`);
const audioMeta = JSON.parse(readFileSync(audioMetaPath, "utf8"));
const bgmPath = audioMeta.bgm?.path || "";
const bgmAbsPath = bgmPath ? join(hyperframesDir, bgmPath) : "";
const logPath = audioMeta.bgm_log || "";
const pid = audioMeta.bgm_pid || null;
const base = {
enabled: Boolean(audioMeta.bgm_pending && bgmPath),
provider: audioMeta.bgm_provider || null,
mode: audioMeta.bgm_mode || null,
path: bgmPath || null,
log: logPath || null,
pid,
target_duration_s: audioMeta.bgm_target_duration_s || null,
seed_duration_s: audioMeta.bgm_seed_duration_s || null,
loop_count: audioMeta.bgm_loop_count || null,
timeout_ms: timeoutMs,
};
if (!base.enabled) {
const status = writeStatus({
...base,
status: "disabled",
ready: false,
waited_ms: 0,
message: "BGM not requested or disabled in audio_meta.json.",
});
console.log(`✓ bgm: ${status.status} (${status.message})`);
process.exit(0);
}
const started = Date.now();
let lastFailure = "";
let lastTail = "";
while (Date.now() - started <= timeoutMs) {
if (existsSync(bgmAbsPath)) {
const size = statSync(bgmAbsPath).size;
writeStatus({
...base,
status: "ready",
ready: true,
waited_ms: Date.now() - started,
size_bytes: size,
message: `BGM ready at ${bgmPath}.`,
});
console.log(`✓ bgm: ready (${bgmPath}, ${size}B)`);
process.exit(0);
}
lastTail = readTail(logPath);
lastFailure = detectFailure(lastTail);
const alive = isProcessAlive(pid);
if (lastFailure || (!alive && logPath && existsSync(logPath))) {
const message = lastFailure
? `BGM renderer failed; see ${logPath}.`
: `BGM renderer exited without writing ${bgmPath}; see ${logPath}.`;
const status = writeStatus({
...base,
status: "failed",
ready: false,
waited_ms: Date.now() - started,
process_alive: alive,
message,
error_tail: lastFailure || lastTail.slice(-2000),
});
console.log(`! bgm: failed (${status.message})`);
process.exit(0);
}
if (timeoutMs === 0) break;
await sleep(Math.min(intervalMs, Math.max(0, timeoutMs - (Date.now() - started))));
}
const status = writeStatus({
...base,
status: "timeout",
ready: false,
waited_ms: Date.now() - started,
process_alive: isProcessAlive(pid),
message: `Timed out waiting for ${bgmPath}; assemble-index will skip BGM if still absent.`,
log_tail: lastTail.slice(-2000),
});
console.log(`! bgm: timeout after ${status.waited_ms}ms (${bgmPath})`);
audio/scripts/wait-bgm.test.mjs›
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { test } from "node:test";
import { runInNewContext } from "node:vm";
const source = readFileSync(new URL("./wait-bgm.mjs", import.meta.url), "utf8");
const start = source.indexOf("function readTail(");
const end = source.indexOf("\nfunction detectFailure(", start);
function load(read) {
return runInNewContext(`${source.slice(start, end)}; readTail`, {
readFileSync: read,
existsSync: () => true,
statSync: () => {
throw new Error("pathname stat must not precede log read");
},
});
}
test("reads a log once without a separate pathname check", () => {
let calls = 0;
const read = load((path, encoding) => {
calls++;
assert.equal(path, "renderer.log");
assert.equal(encoding, "utf8");
return "prefix\nRuntimeError: failed";
});
assert.equal(read("renderer.log", 20), "RuntimeError: failed");
assert.equal(calls, 1);
});
test("preserves UTF-16 character tail boundaries and zero-length tails", () => {
const txt = "🎵日本語".repeat(2000);
const read = load(() => txt);
assert.equal(read("log"), txt.slice(-6000));
assert.equal(read("log", 0), "");
});
for (const code of ["ENOENT", "ENOTDIR"]) {
test(`a log removed during polling is absent (${code})`, () => {
const read = load(() => {
throw Object.assign(new Error(code), { code });
});
assert.equal(read("log"), "");
});
}
test("preserves unexpected read errors", () => {
const error = Object.assign(new Error("denied"), { code: "EACCES" });
assert.throws(
() =>
load(() => {
throw error;
})("log"),
(caught) => caught === error,
);
});
test("empty log path does not access the filesystem", () => {
assert.equal(
load(() => {
throw new Error("unexpected read");
})(""),
"",
);
});
luts/index.json›
{
"notes": "Look entries resolve on-demand from a CDN .cube `url` (downloaded + frozen, like bgm/image); `params` is a deterministic buildCube fallback used offline (--local-only) or if the download/validation fails. No .cube bodies are committed. See README.md to author or host a new look.",
"looks": [
{
"id": "teal-orange-blockbuster",
"description": "Teal shadows and warm orange highlights for blockbuster-style cinematic footage.",
"tags": ["teal", "orange", "blockbuster", "cinematic", "split tone", "movie"],
"intensity": 0.85,
"url": "https://static.heygen.ai/luts/teal-orange-blockbuster.cube",
"params": {
"contrast": 0.18,
"saturation": 0.08,
"vibrance": 0.12,
"splitTone": {
"intensity": 0.62,
"balance": 0.52,
"shadows": [-0.04, 0.05, 0.09],
"highlights": [0.1, 0.04, -0.03]
}
}
},
{
"id": "bleach-bypass",
"description": "High-contrast desaturated bleach bypass look with strong blacks.",
"tags": ["bleach", "bypass", "desaturated", "high contrast", "gritty", "film"],
"intensity": 0.8,
"url": "https://static.heygen.ai/luts/bleach-bypass.cube",
"params": {
"blacks": 0.04,
"shadows": -0.08,
"highlights": 0.08,
"whites": 0.18,
"contrast": 0.55,
"temperature": -0.02,
"saturation": -0.72,
"vibrance": -0.25
}
},
{
"id": "film-fade",
"description": "Soft faded film wash with lifted blacks and warm highlights.",
"tags": ["film", "fade", "faded", "wash", "warm", "vintage"],
"intensity": 0.75,
"url": "https://static.heygen.ai/luts/film-fade.cube",
"params": {
"blacks": 0.35,
"shadows": 0.18,
"highlights": 0.02,
"whites": -0.03,
"contrast": -0.28,
"temperature": 0.16,
"saturation": -0.12,
"vibrance": -0.08
}
}
]
}
luts/README.md›
# LUT library (authoring)
`index.json` is the agent-consumed catalog of color-grade looks. Each entry resolves
on demand — no `.cube` bodies are committed to the repo.
Each look has:
- `id`, `description`, `tags`, `intensity` — matching + application metadata.
- `url` (optional) — a hosted `.cube` downloaded, validated, and frozen at resolve
time, exactly like bgm/image assets.
- `params` (optional) — a deterministic `buildCube` spec used offline (`--local-only`)
or as a fallback if the `url` download/validation fails.
An entry needs at least one of `url` or `params`; prefer both (CDN url with a params
fallback) so resolution is never blocked on the network.
## Hosting a new look (operators)
1. Generate the `.cube` (e.g. `resolve -t lut --params '{...}'` or a graded export).
2. Upload it to the public CDN origin bucket:
```
aws s3 cp <id>.cube s3://heygen-public/luts/<id>.cube
```
It is then served at `https://static.heygen.ai/luts/<id>.cube` (CloudFront).
3. Add an entry to `index.json` with that `url` (and ideally a `params` fallback).
references/audio.md›
# Audio engine — voiceover, music, SFX, captions, transcription
For a full audio pass (TTS voiceover + background music + sound effects in one
shot), use the shared engine at `audio/scripts/audio.mjs`. It takes a neutral
`audio_request.json` and writes `audio_meta.json` plus assets under
`.media/audio/{voice,bgm,sfx}`:
```bash
node <SKILL_DIR>/audio/scripts/audio.mjs --request ./audio_request.json --out ./audio_meta.json
```
- **Request** `{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }`: `id` joins each line back to your model; `bgm.mode` = `retrieve | generate | none` (omit for auto). `--only tts,bgm,sfx` runs a subset and merges into an existing `--out`.
- **Output** `audio_meta.json` (id-keyed): `voices[].{path,duration_s,words[]}` (word timestamps for captions), `sfx[]`, `bgm`, `total_duration_s`.
- **HeyGen free-usage path**: HeyGen CLI auth unlocks TTS plus music/SFX retrieval. Local/provider-specific generators are explicit alternatives where installed; run `npx hyperframes media-use resolve --doctor` before assuming retrieval or TTS will work.
- If BGM took the generate path (`bgm_pending: true`), run `audio/scripts/wait-bgm.mjs` before final render.
Single-shot helpers: `audio/scripts/heygen-tts.mjs` (one voice file). Transcription / background removal / captions use the `hyperframes` CLI (`transcribe`, `remove-background`), see the per-topic guides in `audio/references/` (`tts.md`, `bgm.md`, `sfx.md`, `transcribe.md`, `remove-background.md`, `captions/`).
Transcription defaults to Parakeet (better than whisper.cpp: 6.05% vs 7.44% WER, 5-10x faster) via `scripts/transcribe.mjs`, with whisper.cpp auto-fallback (see `references/operations.md`).
references/grading.md›
# Color grading — grade blocks and LUTs
Use `grade` when you need a canonical HyperFrames grading/effects payload for
an `<img>` or `<video>`. Core presets and params-backed LUT entries resolve
locally; future CDN-backed LUT entries require network unless already
frozen. Persist a decided payload with the CLI rather than editing HTML by
hand:
For a vague but explicit polish request, do not jump directly from intent to a
preset name. Read `media-treatments.md`, choose a treatment whose subject and
avoid rules match the actual media, apply its conservative base with only
justified bounded tuning, then complete its visual verification steps. A named
owned treatment uses the exact preset/payload in its recipe; do not run the
generic grade/LUT resolver first.
Stop here and use that treatment workflow for requests such as retro, old home
video, camcorder, film, print, ASCII, glitch, privacy, or a media reveal. Do not
assemble those from a generic LUT plus handmade CSS vignette/grain/opacity.
**Never `cat`/read a `.cube` file into context.** A 3D LUT is ~size^3 lines of raw numbers (33^3 ≈ 36k lines at the default size). It bloats context and carries zero human/agent-legible signal. To understand or choose a LUT, use `hyperframes grade-compare` to see it rendered, or `cube-validate.mjs` for a one-line `{ok,size}` check. Read `.media/index.md` or `luts/index.json` for the description. Never read the LUT body itself.
```bash
npx hyperframes media-use resolve --type grade --intent "warm daylight" --project . --json
```
Preset-first output uses the core runtime vocabulary and does not freeze a file:
```json
{
"preset": "warm-daylight",
"intensity": 1
}
```
Apply that payload to one unambiguous real media element:
```bash
hyperframes media-treatment --project . --file index.html \
--selector '#hero' \
--grading '{"preset":"warm-daylight","intensity":1}' --apply --json
```
Use `--dry-run` before writing when scope is uncertain and `--clear` to remove
the treatment. The low-level persisted result is still normal HTML:
```html
<video
class="clip"
src="./media/scene.mp4"
data-color-grading='{"preset":"warm-daylight","intensity":1}'
></video>
```
Direct attribute authoring is a fallback for environments where the CLI is not
available, not the primary agent workflow.
To build a treatment that is not already represented by a recipe, inspect the
canonical toolbox first:
```bash
hyperframes media-treatment --capabilities --json
```
It reports a concise family map. Read `--capability grading` for the processing
order, then request only the focused family needed to get its legal controls
and ranges from Core. Compose one nested payload and pass it back through
`hyperframes media-treatment`; the command rejects unknown keys before
mutation. Do not generate or hand-edit a LUT merely to combine controls already
owned by the realtime shader.
For seek-safe effect motion, animate only the runtime-supported CSS properties
on that same real media element with its registered paused GSAP timeline:
| CSS property | Range |
| ---------------------------------- | ------- |
| `--hf-color-grading-intensity` | 0 to 1 |
| `--hf-color-grading-lut-intensity` | 0 to 1 |
| `--hf-color-grading-exposure` | -2 to 2 |
| `--hf-color-grading-blur` | 0 to 1 |
| `--hf-color-grading-bloom` | 0 to 3 |
| `--hf-color-grading-kuwahara` | 0 to 1 |
| `--hf-color-grading-pixelate` | 0 to 1 |
| `--hf-color-grading-ascii` | 0 to 1 |
| `--hf-color-grading-dither` | 0 to 1 |
Author the initial value directly in the media element's inline `style`, then
use finite `tl.to()` keyframes. Do not use a frame-zero `tl.set()`, CSS
animation clocks, timers, random values, or `onUpdate` callbacks. The static
`data-color-grading` payload remains the fallback and source of the other
controls.
For a reusable color transform beyond the preset vocabulary, freeze a validated
`.cube` under `.media/luts/` and return a block that references it:
```bash
npx hyperframes media-use resolve --type grade --intent "teal orange blockbuster" --project . --json
```
```json
{
"intensity": 1,
"lut": { "src": ".media/luts/grade_001.cube", "intensity": 0.85 }
}
```
Use `lut` when you only need the reusable `.cube` file:
```bash
npx hyperframes media-use resolve --type lut --intent "teal orange blockbuster" --project .
```
For a describable technical look, author an explicit parametric LUT with `--params`:
```bash
npx hyperframes media-use resolve --type lut --params '{"contrast":0.2,"temperature":-0.3}' --project .
npx hyperframes media-use resolve --type grade --params '{"exposure":0.2}' --project . --json
```
For a LUT generated by your own script, ingest it with `--from`; media-use validates it before registration and rejects invalid or oversized cubes:
```bash
npx hyperframes media-use resolve --type lut --from custom.cube --project .
```
Parametric math (`buildCube`) cannot reproduce real film stocks or emulsion
transforms. Use a CDN-backed scanned `.cube` entry or ingest a real scanned
`.cube` for those.
For visual selection, list reusable LUT candidates with
`resolve --type grade --candidates`, write the promising entries to a
`grades.json`, run
`hyperframes grade-compare --for <frame> --grades grades.json`, then commit the
winner with `resolve -t grade` as the final `data-color-grading` block.
For media already selected in a composition, use `media-treatment --analyze`
when you need side-effect-free `ffmpeg`/`ffprobe` signalstats evidence. It
returns source metadata, HDR/unknown-LOG warnings, and a bounded `adjust`
suggestion without modifying the composition. The suggestion is a starting
point for visual review, not an automatic neutralization of intentional color.
```bash
hyperframes media-treatment --project . --file index.html \
--selector '#hero' --analyze --json
```
For an unbound source file, `resolve --type grade --for ... --analyze` remains
available. Without `--analyze`, that resolver records a grade candidate in
`.media`; use that form only when you intend to keep the candidate.
Library LUT entries live in `luts/index.json`. Each entry keeps `id`,
`description`, `tags`, and `intensity`, then supplies either compact `params`
for on-demand `buildCube(params)` generation or a direct CDN `url` for future
scanned `.cube` files. Do not commit generated `.cube` bodies; resolve
validates generated or downloaded cubes as it freezes them under
`.media/luts/`.
```bash
npx hyperframes media-use resolve --type lut --intent "teal orange blockbuster" --project . --json
node skills/media-use/scripts/lib/cube-validate.mjs .media/luts/lut_001.cube
```
references/media-treatment-recipes.md›
# Media treatment recipes
These are optional tested seeds, not the complete capability surface. Read the
shared policy and choose one relevant section through `media-treatments.md`.
Agents may modify or combine a seed with compatible canonical controls after
inspecting the media, or assemble a bespoke payload from
`hyperframes media-treatment --capabilities --json` when no seed fits.
## Natural Portrait
Use for a talking head, interview, presenter, or people-focused photo whose
intended result is natural, polished, and restrained.
Do not use when the face is incidental or tiny, the source is intentionally
neon/monochrome/strongly stylized, or the requested result is beauty retouching.
This treatment changes the whole frame; it is not a face mask or skin-smoothing
effect.
Inspect face exposure, highlight retention, shadow detail, white balance, and
whether the existing look is intentional. Signalstats do not detect faces or
creative intent.
### Base payload
Start here, then tune only when the sampled frames justify it:
```json
{ "preset": "skin-soft", "intensity": 0.6 }
```
`skin-soft` is a global tonal/color preset whose vibrance math is reduced for
skin-like colors. It does not blur, retouch, segment, or track a face.
### Bounded tuning
Adjustment values are absolute values in the final payload, not deltas added to
the preset. Keep changes inside these conservative ranges unless the user asks
for a stylized result:
| Property | Natural Portrait range |
| ----------- | ---------------------- |
| intensity | 0.45 to 0.75 |
| exposure | -0.06 to 0.14 |
| contrast | -0.05 to 0.08 |
| highlights | -0.18 to -0.04 |
| shadows | 0.04 to 0.18 |
| whites | -0.10 to 0.04 |
| blacks | -0.06 to 0.06 |
| temperature | -0.05 to 0.10 |
| tint | -0.03 to 0.05 |
| vibrance | 0 to 0.06 |
| saturation | -0.04 to 0.06 |
Leave grain, blur, and pixelate at zero. A vignette is optional at `0` to
`0.05` only when it improves subject focus without looking like an effect.
Manual controls must stay inside their schema section; they are never
top-level keys. A tuned Natural Portrait payload looks like this:
```json
{
"preset": "skin-soft",
"intensity": 0.58,
"adjust": {
"highlights": -0.08,
"shadows": 0.08,
"temperature": 0.02,
"vibrance": 0.02
},
"details": { "vignette": 0.03 }
}
```
Use the same nested shape in `grade-compare` candidate files. `adjust` owns
tonal/color controls, `details` owns vignette/grain, and `effects` owns blur,
pixelate, chroma bleed, and the advanced treatment primitives below.
During the common comparison, reject any result that makes skin implausible,
loses highlight detail, flattens or desaturates dark skin, or casts clothing and
background colors accidentally.
## Product Polish
Use for photographed or filmed physical products when the goal is clean,
accurate, dimensional presentation. Protect product color, material texture,
label readability, specular highlights, and intentional lighting.
Do not use this treatment for literal app/site screenshots or screen captures;
follow UI Fidelity below. Do not neutralize a lifestyle scene's deliberate
ambient color, and do not infer exact brand-color correction without a neutral
reference or known product color.
Inspect the product separately from its background. Check white balance, label
legibility, surface texture, highlight clipping, shadow detail, white point,
and black point. Statistics cannot identify a white package, metallic
highlight, amber glass, or intentional warm light.
### Base payload
Compare this restrained correction against the untouched source:
```json
{
"intensity": 0.7,
"adjust": {
"exposure": 0.01,
"contrast": 0.06,
"highlights": -0.1,
"shadows": 0.04,
"whites": 0.02,
"blacks": -0.03,
"vibrance": 0.03,
"saturation": 0.02
}
}
```
This is a comparison starting point, not an instruction to change an already
finished source. If the original has accurate color, clean endpoints, and good
texture, leave the pixels unchanged and polish through framing or motion.
### Bounded tuning
| Property | Product Polish range |
| ----------- | -------------------- |
| intensity | 0.45 to 0.8 |
| exposure | -0.08 to 0.1 |
| contrast | 0 to 0.1 |
| highlights | -0.16 to 0 |
| shadows | 0 to 0.12 |
| whites | -0.08 to 0.05 |
| blacks | -0.06 to 0.04 |
| temperature | -0.05 to 0.05 |
| tint | -0.03 to 0.03 |
| vibrance | 0 to 0.06 |
| saturation | -0.04 to 0.05 |
Temperature and tint stay at zero unless the frames show a plausible cast.
Leave grain, vignette, blur, and pixelate at zero for catalog/e-commerce media.
For a lifestyle product shot, a vignette up to `0.04` is acceptable only when
it improves focus without changing the product itself.
During the common comparison, reject any result that clips white packaging,
muddies black products, shifts a known brand color, hides texture, or makes
labels harder to read. Report when preserving the original was the deliberate
decision.
## UI Fidelity
Use for literal app, website, dashboard, terminal, slide, or screen-recording
pixels whose colors and readability are part of the product being shown.
The default payload is **none**: do not add `data-color-grading`. Global color
changes affect brand colors, status colors, charts, screenshots, and tiny text
together, so even a tasteful photographic look can make the demonstration less
truthful.
Polish UI footage with crop, scale, pacing, cursor emphasis, surrounding DOM
overlays, or seek-safe motion outside the captured pixels. If the user
explicitly asks for a stylized UI look, preview it against the original and
state that exact UI color is no longer preserved. If a camera filmed a screen,
correct only a demonstrated capture cast or exposure issue and still verify
text and brand colors across representative frames.
## Film Memory
Use when the story explicitly calls for a warm memory, restrained flashback,
personal archive, or film-like recollection. This is not the default meaning of
"cinematic", and it is not scanned-film-stock emulation.
Do not use for literal UI, product catalog media, technical demonstrations, or
footage whose accurate current-day color is important. Use a separate camcorder
treatment for VHS/REC language. Do not add dust, scratches, light leaks, film
burns, or halation unless an owned component is available and the requested
story actually benefits from it.
Check that the source has enough highlight and shadow detail to tolerate a
faded treatment, and confirm nostalgia or temporal separation belongs in the
story. Compare the full moving treatment, not only a still preset card.
### Static pixel base
Start with this owned shader recipe:
```json
{
"preset": "vintage-wash",
"intensity": 0.6,
"details": {
"vignette": 0.12,
"grain": 0.12,
"grainSize": 0.2,
"grainRoughness": 0.6
}
}
```
Keep the static values inside these ranges:
| Property | Film Memory range |
| -------------- | ----------------- |
| intensity | 0.5 to 0.75 |
| vignette | 0.08 to 0.16 |
| grain | 0.08 to 0.16 |
| grainSize | 0.16 to 0.24 |
| grainRoughness | 0.5 to 0.7 |
The stronger end can flatten dark skin, black clothing, or already-faded
footage. Compare against the source and lower strength when it does.
### Temporal character
Use the existing registered paused GSAP timeline on the same media element:
- author `--hf-color-grading-exposure: 0` in the media element's inline
`style`;
- move it through a finite irregular sequence within `-0.03` to `0.03`, using
gentle `sine.inOut` segments around `0.45` to `0.8` seconds;
- for gate weave, keep `x`/`y` within `0.15%` of the shorter composition edge,
rotation within `0.03` degrees, and scale between `1.005` and `1.01` to
protect the frame edges;
- return close to the starting exposure and transform at the treatment end.
Do not use randomness, infinite CSS keyframes, timers, or `onUpdate`. Flicker
is a gentle exposure pulse, not a flash. Weave is slight mechanical drift, not
handheld shake.
Also run focused keyframe diagnostics and seek directly to the final-minus-frame
position. Reject brightness pumping, distracting drift, clipped edges, or
skin/detail loss. Report the motion ranges and describe this as an HF
film-memory treatment, not camera-stock emulation. If motion reads as an effect
before it reads as a memory, reduce or remove it.
## Creator Camcorder
Use when the story explicitly calls for a creator-camera recording, consumer
camcorder memory, or restrained digital-video character. This treatment is a
modern camcorder language, not VHS restoration, CRT simulation, surveillance,
or a promise to reproduce a specific camera model.
Do not apply it to literal UI, product catalog media, tiny media tiles, or
already compressed footage that has distracting color bleed. Do not add a REC
HUD merely because the source contains a person talking; the camera-device
language must support the story or the user's requested style.
Check skin, saturated edges, fine text, source compression, and whether the
source already has a deliberate camera look. Reject softened chroma that
damages labels, graphics, or identifying product color. Judge chroma softness
and grain in motion, not one still.
### Static pixel base
Start with the proven shader payload below, then tune only inside the bounded
ranges when representative frames justify it:
```json
{
"intensity": 0.72,
"adjust": {
"contrast": 0.08,
"highlights": -0.05,
"shadows": 0.02,
"whites": 0.03,
"blacks": -0.04,
"temperature": -0.03,
"tint": -0.015,
"vibrance": -0.03,
"saturation": -0.06
},
"details": {
"vignette": 0.06,
"grain": 0.08,
"grainSize": 0.18,
"grainRoughness": 0.58
},
"effects": { "chromaBleed": 0.55 }
}
```
| Property | Creator Camcorder range |
| -------------- | ----------------------- |
| intensity | 0.55 to 0.8 |
| contrast | 0.03 to 0.1 |
| highlights | -0.1 to 0 |
| shadows | 0 to 0.06 |
| whites | 0 to 0.05 |
| blacks | -0.08 to -0.01 |
| temperature | -0.06 to 0.04 |
| tint | -0.03 to 0.02 |
| vibrance | -0.06 to 0.02 |
| saturation | -0.12 to -0.02 |
| vignette | 0.03 to 0.1 |
| grain | 0.04 to 0.12 |
| grainSize | 0.14 to 0.24 |
| grainRoughness | 0.45 to 0.7 |
| chromaBleed | 0.35 to 0.7 |
Leave blur and pixelate at zero. Square pixels, scanlines, RGB splitting, and
tracking noise are different visual languages and are not defaults for this
treatment.
### Optional camera HUD
When the narrative benefits from explicit recording-device language, install
the Registry overlay block:
```bash
npx hyperframes add camcorder-hud --no-clipboard
```
Insert the printed `data-composition-src` host over the intended media range.
Edit the displayed date/time/mode/counter in
`compositions/camcorder-hud.html`. The block's paused GSAP timeline derives
its counter and REC blink from composition time, so play, scrub, and render
agree. Keep the HUD finite and scoped to the shot.
The HUD is an optional authored overlay. The pixel payload remains useful
without it, and the HUD alone is not evidence that the footage was treated.
### Optional source-to-camera reveal
Global grading intensity fades only primary correction and LUT output; it does
not fade the independent camcorder effects. For a visible source-to-camera
mode change, use two synchronized media layers and a finite opacity crossfade
from untreated to treated footage. Fade the HUD in on that same paused GSAP
timeline. Do not animate shader state with callbacks or an independent clock.
Also verify HUD placement and framing in each aspect ratio the project supports.
Report whether the HUD was used and describe this as an HF camcorder treatment,
not camera/VHS emulation. If an effect artifact is more noticeable than the
subject, reduce chroma bleed/grain or keep the source unchanged.
## VHS Playback
Use when the story explicitly calls for analog home-video tape, a dated archive,
or a visibly degraded VHS playback. This treatment is not Creator Camcorder,
generic pixelation, CRT display simulation, or a default retro look.
Do not use for literal UI, product catalog media, small text, clean modern
creator footage, or any source whose identifying color/detail must remain exact.
Inspect high-contrast vertical edges, faces, saturated objects, and the bottom
of the frame in motion. Analog damage must support the story without making the
subject hard to read.
### Pixel payload
Start with the complete proven combination, not `tapeDamage` alone:
```json
{
"intensity": 1,
"adjust": { "contrast": -0.04, "saturation": -0.08 },
"details": {
"grain": 0.16,
"grainSize": 0.12,
"grainRoughness": 0.72
},
"effects": {
"tapeDamage": 0.82,
"tapeTracking": 0.85,
"tapeNoise": 0.3,
"tapeSpeed": 0.5,
"chromaBleed": 0.5,
"chromaticAberration": 0.18,
"chromaticAngle": 0,
"scanlines": 0.35,
"scanlineCount": 0.17,
"scanlineSoftness": 1,
"digitalGlitch": 0.32,
"digitalGlitchColorSplit": 0,
"digitalGlitchLineTear": 0.08,
"digitalGlitchPixelate": 0,
"digitalGlitchBlockAmount": 0,
"digitalGlitchBlockDisplacement": 0,
"digitalGlitchBlockOpacity": 0,
"digitalGlitchSpeed": 0.5
}
}
```
| Property | VHS Playback range |
| --------------------- | ------------------ |
| intensity | 0.75 to 1 |
| contrast | -0.1 to 0 |
| saturation | -0.16 to 0 |
| grain | 0.08 to 0.18 |
| grainSize | 0.08 to 0.18 |
| grainRoughness | 0.55 to 0.8 |
| tapeDamage | 0.65 to 0.9 |
| tapeTracking | 0.5 to 0.9 |
| tapeNoise | 0.15 to 0.45 |
| tapeSpeed | 0.35 to 0.65 |
| chromaBleed | 0.35 to 0.65 |
| chromaticAberration | 0.08 to 0.22 |
| scanlines | 0.2 to 0.4 |
| scanlineCount | 0.14 to 0.2 |
| digitalGlitch | 0.2 to 0.4 |
| digitalGlitchLineTear | 0.04 to 0.1 |
`tapeDamage` owns deterministic horizontal line jitter, slow time-base wobble,
bottom-edge head switching, luma bandwidth loss, restrained ghosting, noise,
and sparse dropouts. Its subordinate tracking/noise/speed controls add bounded
moving tape tears and control their signal character without introducing a new
clock. `chromaBleed` separately reduces horizontal chroma detail. The restrained
scanline and chromatic settings supply the remaining tape-playback character.
The digital stage is used only for rare horizontal row tears: keep its color
split, pixelation, block displacement, block opacity, and corruption values at
zero. Leave blur, CRT curvature, generic pixelation, and a camera HUD off.
These values are an original HyperFrames recipe calibrated on the same public
Orange Cat source used for the external visual reference. They are not copied
shader code or a claim of pixel-identical output from the external reference. The scanline count is
mapped to the reference's approximately 127-cycle primary line pattern; the HF
tracking math stays bounded in media pixels and uses the composition clock.
The shader damage evolves from the existing deterministic media time, so it
needs no CSS loop or private timeline. Global grading intensity does not fade
tape damage or other independent effects. If the story requires a finite
source-to-tape reveal, crossfade synchronized untreated and treated media layers
on the host's paused GSAP timeline. During the common workflow, inspect dense
consecutive frames and reject hard edge tearing, face
smearing, frozen noise, square blocks, blank borders, or a bottom disturbance
that competes with the subject.
## 8mm Home Movie
Use for personal archive, family-memory, childhood, travel-memory, or explicit
small-gauge home-movie language. This is stronger and more materially film-like
than Film Memory, but it is still an owned HyperFrames treatment rather than a
claim to reproduce a named film stock, camera, or laboratory process.
Do not use for literal UI, technical demonstrations, catalog products, clean
interviews, or footage where dust/scratches would imply false provenance. Check
skin, highlights, dark clothing, and frame edges before applying it.
### Pixel payload
```json
{
"preset": "vintage-wash",
"intensity": 0.72,
"details": {
"vignette": 0.28,
"vignetteMidpoint": 0.54,
"vignetteFeather": 0.72,
"grain": 0.34,
"grainSize": 0.18,
"grainRoughness": 0.72
},
"effects": { "filmArtifacts": 0.62 }
}
```
| Property | 8mm Home Movie range |
| -------------- | -------------------- |
| intensity | 0.6 to 0.8 |
| vignette | 0.18 to 0.34 |
| grain | 0.22 to 0.42 |
| grainSize | 0.12 to 0.24 |
| grainRoughness | 0.6 to 0.8 |
| filmArtifacts | 0.35 to 0.7 |
`filmArtifacts` owns only deterministic sparse dust and short scratches. The
existing preset/details own color, vignette, and grain; the host's paused GSAP
timeline owns optional gate weave. Keep weave within `0.15%` of the shorter
composition edge, rotation within `0.03` degrees, and scale between `1.005` and
`1.015`. Use finite `sine.inOut` segments around `0.6` to `1` second, return
near the starting transform, and never use randomness, timers, `onUpdate`, or
an infinite CSS animation.
Reject a result when dust is constantly visible, scratches persist unnaturally,
the frame pumps, weave exposes an edge, highlights turn muddy, or the material
artifacts are more noticeable than the memory. For a subtler nostalgic result,
use Film Memory instead.
## Editorial Halftone
Use for print/editorial transitions, poster frames, comic/newsprint language,
stylized product or portrait beats, and graphic sequences where visible ink
screening is the point. This is a real four-angle CMYK raster treatment, not a
dotted DOM overlay.
Do not use on literal UI, dense text, tiny labels, footage that must remain
photorealistic, or a long talking-head segment unless the user explicitly asks
for strong print stylization. Preserve text/captions as ungraded DOM above the
media whenever they must stay readable.
### Pixel payload
```json
{
"intensity": 1,
"adjust": { "contrast": 0.04, "saturation": 0.04 },
"effects": { "halftone": 0.94, "halftoneSize": 0.36 }
}
```
| Property | Editorial Halftone range |
| ------------ | ------------------------ |
| intensity | 0.8 to 1 |
| contrast | -0.02 to 0.08 |
| saturation | -0.04 to 0.08 |
| halftone | 0.75 to 1 |
| halftoneSize | 0.15 to 0.55 |
The shader uses fixed C/M/Y/K screen angles of 15/75/0/45 degrees, separate ink
coverage, a warm paper base, and resolution-aware dot-cell sizing. Keep those
screen semantics fixed; tune only amount and size unless a future visual proof
justifies a broader schema. Judge the result at final output resolution because
browser zoom can misrepresent the screen. Reject unstable moire, unreadable
subjects, clipped ink detail, excessive dot size, or any treatment that looks
like a transparent dot texture laid over unchanged footage.
## Two-Ink Editorial Print
Use for poster frames, editorial portraits, music/social cutaways, zine
graphics, and bold print-led transitions where two visible spot inks are more
appropriate than photographic color. This is a fixed original HyperFrames
vermilion/teal treatment, not a claim to emulate a named printer, ink set, or
commercial print process.
Do not use for literal UI, brand-color-critical products, small labels, natural
talking heads, or media that must remain photorealistic. Keep captions and
graphics as normal DOM above the treated media.
### Pixel payload
```json
{
"intensity": 1,
"adjust": { "contrast": 0.08, "highlights": -0.06, "shadows": 0.04 },
"effects": { "twoInkPrint": 1, "twoInkPrintSize": 0.42 }
}
```
| Property | Two-Ink range |
| --------------- | ------------- |
| intensity | 0.8 to 1 |
| contrast | 0.02 to 0.1 |
| highlights | -0.1 to 0 |
| shadows | 0 to 0.08 |
| twoInkPrint | 0.8 to 1 |
| twoInkPrintSize | 0.18 to 0.55 |
The shader maps warm midtones to vermilion, deep/cool shadows to teal, and
shared dark coverage to a dark overprint on warm paper. It uses separate
15/75-degree screens, a subtle fixed registration offset, deterministic paper
texture, and resolution-aware dot sizing. Do not combine it with `halftone` or
a duotone LUT: that re-separates the result and defeats the two-ink contract.
Judge it at output resolution and across multiple frames. Reject missing second
ink, crushed faces, unstable moire, illegible silhouettes, or a result that
reads as a red tint with dots rather than two screened inks.
## Monochrome Screen Print
Use for graphic portrait beats, posterized social inserts, newspaper-like
screens, or a finite transition into visible monochrome cells. Keep captions
and typography as normal DOM above the treated media.
```json
{
"intensity": 1,
"effects": {
"monoScreen": 1,
"monoScreenSize": 0.35,
"monoScreenAngle": 0.25,
"monoScreenSpread": 0.3,
"monoScreenShape": 0,
"monoScreenInvert": 0
},
"palette": ["#111319", "#f2ecdc"]
}
```
Use `monoScreenShape` `0..4` for circle, square, diamond, triangle, or line.
Keep cell size within `0.15..0.55` and spread within `0.15..0.55`. Reject faces
that lose their silhouette, unstable moire, or cells too small to survive the
final encoded resolution.
## Engraved Illustration
Use for editorial portraits, historical/technical illustration, title-card
cutaways, or a source-to-line-art reveal. It is not routine correction and
should not be applied to literal UI or brand-color-critical product footage.
```json
{
"intensity": 1,
"effects": {
"engraving": 1,
"engravingSpacing": 0.4118,
"engravingMinThickness": 0.2,
"engravingMaxThickness": 0.4571,
"engravingAngle": 0.25,
"engravingContrast": 0.4667,
"engravingSharpness": 0.59,
"engravingWave": 0.2,
"engravingWaveFrequency": 0.2222
},
"palette": ["#101216", "#f3eddf"]
}
```
Preserve the calibrated base first. Tune spacing within `0.25..0.6`, contrast
within `0.3..0.65`, and wave within `0..0.35`. Reject squeezed framing, broken
contours, noisy flat backgrounds, or lines that flicker across moving frames.
## Crosshatched Sketch
Use for hand-rendered editorial beats, comic/documentary cutaways, and short
illustrative transformations where multiple line directions should preserve
the subject contour.
```json
{
"intensity": 1,
"effects": {
"crosshatch": 1,
"crosshatchSpacing": 0.28,
"crosshatchThickness": 0.25,
"crosshatchAngle": 0.25,
"crosshatchContrast": 0.3333,
"crosshatchEdges": 0.5,
"crosshatchLineWeight": 0,
"crosshatchWave": 0.33,
"crosshatchWaveFrequency": 0.2222
},
"palette": ["#101216", "#f3eddf"]
}
```
Tune spacing within `0.18..0.5`, edge detail within `0.3..0.7`, and wave within
`0.1..0.45`. Reject distorted aspect ratio, dense black fill that hides the
subject, or temporal shimmer stronger than the intended sketch language.
## CRT Display
Use when the media is intentionally shown as an older monitor, terminal, game
screen, or broadcast display. Curvature alone is geometry, not a complete CRT
treatment, so pair it with restrained scanlines and only slight channel
separation.
```json
{
"intensity": 1,
"effects": {
"crtCurvature": 0.2,
"scanlines": 0.35,
"scanlineCount": 0.17,
"scanlineSoftness": 1,
"chromaticAberration": 0.08,
"chromaticAngle": 0
}
}
```
Keep curvature within `0.08..0.28`, scanlines within `0.18..0.45`, and channel
separation within `0..0.12`. Reject excessive black corners, unreadable UI,
large color fringes, or applying the display language to ordinary footage when
the user only asked for correction.
## Procedural ASCII
Use for a deliberate terminal, code, data, surveillance, editorial, or
source-to-character reveal. This is a real shader-generated 5x7 glyph field,
not monospace text placed over unchanged footage.
Do not use as routine talking-head polish, on literal UI or dense text, or when
recognizing a face/product precisely matters. Keep captions and graphics as
normal DOM above the treated media.
Choose one of these proven starting points:
```json
{
"effects": { "ascii": 1, "asciiSize": 0.08, "asciiInvert": 1 },
"palette": ["#020605", "#38ff78"]
}
```
The first is **Terminal ASCII**: dark field, bright green glyphs, appropriate
for code/data/device language. For a warmer print-like **Editorial ASCII**, use:
```json
{
"effects": { "ascii": 1, "asciiSize": 0.066, "asciiInvert": 0 },
"palette": ["#0b0d0d", "#eee9db"]
}
```
Keep `ascii` between `0.75` and `1` for a fully readable treatment and
`asciiSize` between `0.04` and `0.15`. A finite reveal may author
`--hf-color-grading-ascii: 0` inline and tween it to `1` with the registered
paused GSAP timeline. Reject unstable cells, lost silhouette/face structure,
unreadable composition, or a palette that conflicts with the project.
## Ordered Palette Dither
Use for posterized social beats, music/editorial cutaways, pixel-art language,
or a finite source-to-palette reveal. The shader uses a stable 4x4 Bayer
threshold matrix and an explicit dark-to-light palette. Do not describe it as
Floyd-Steinberg, Atkinson, or another sequential error-diffusion process.
Do not use on literal UI, brand-color-critical products, tiny labels, or long
photorealistic sections. Start with one of these original palettes:
```json
{
"effects": { "dither": 1, "ditherSize": 0.25 },
"palette": ["#17121a", "#824c50", "#e09873", "#f7ddb1"]
}
```
The four-color option is **Warm Print**. For a louder social/music beat, use
the six-color **Electric Ink** palette:
```json
{
"effects": { "dither": 1, "ditherSize": 0.4 },
"palette": ["#080717", "#3c185f", "#7e2278", "#d9339f", "#ff6b66", "#aafae0"]
}
```
HyperFrames also owns these named ramps. The name is an authoring shortcut;
persist the listed colors through the existing `palette` array:
| Group | Palette ID | Ordered colors |
| ----------- | ---------------- | ---------------------------------------------------------------- |
| Classic | `noir` | `#000000`, `#ffffff` |
| Classic | `ink-paper` | `#1a1a2e`, `#f5f5dc` |
| Classic | `terminal` | `#001100`, `#00ff00` |
| Classic | `amber-glow` | `#1a0f00`, `#ffcc00` |
| Classic | `handheld-green` | `#0f380f`, `#306230`, `#8bac0f`, `#9bbc0f` |
| Mood | `golden-hour` | `#1a1205`, `#4a3510`, `#8b6914`, `#d4a017`, `#fff8dc` |
| Mood | `deep-sea` | `#0a1628`, `#1a3a5c`, `#2d6187`, `#5ba4c9`, `#a8dce8` |
| Mood | `arctic-night` | `#0a0a14`, `#1a2a4a`, `#3a5a8a`, `#6a9aca`, `#cae8ff` |
| Mood | `synthwave` | `#120458`, `#7b2cbf`, `#e040fb`, `#ff6ec7`, `#fff59d` |
| Mood | `vaporwave` | `#1a0a2e`, `#3d1a5c`, `#ff71ce`, `#01cdfe`, `#fffb96` |
| Mood | `forest` | `#1a2e1a`, `#2d4a2d`, `#4a7c4a`, `#7ab37a`, `#c8e6c8` |
| Mono | `sepia` | `#1a1610`, `#3d3020`, `#6b5a40`, `#a89070`, `#e8dcc8` |
| Mono | `blueprint` | `#001830`, `#003060`, `#0050a0`, `#0080e0`, `#e0f0ff` |
| HyperFrames | `warm-print` | `#17121a`, `#824c50`, `#e09873`, `#f7ddb1` |
| HyperFrames | `electric-ink` | `#080717`, `#3c185f`, `#7e2278`, `#d9339f`, `#ff6b66`, `#aafae0` |
Choose by inspected source and project language, not by palette name alone.
For example, `terminal` fits device/code language, `warm-print` fits editorial
print, and `synthwave` is an intentional stylization rather than generic polish.
`palette` must contain two to six exact `#RRGGBB` colors in authored order. Use
dark-to-light order for this treatment; the runtime validates colors but does
not reorder them, so reversing the array intentionally inverts the mapping.
Keep `dither` between `0.7` and `1` and `ditherSize` between `0.1` and `0.5`.
A finite reveal may author `--hf-color-grading-dither: 0` inline and tween it
to the chosen amount with GSAP. Judge the moving result at output resolution;
reject shimmer, lost subject structure, accidental muddy intermediate colors,
or a palette chosen without regard to the project's design language.
## Cached Error Diffusion
Use exact error diffusion for a deliberate 1-bit Macintosh, newspaper/print,
limited-palette game, or crunchy editorial treatment. It bakes a new image or
MP4 because every processed block depends on error from earlier blocks; it is
not a realtime shader setting.
Choose the algorithm by visible intent:
- `floyd-steinberg`: balanced default with organic fine texture.
- `atkinson`: higher-contrast, more open and distinctly early-Macintosh.
- `jarvis-judice-ninke`: smoother gradients with a wider 12-neighbor field.
- `stucki`: smooth, slightly sharper alternative to JJN.
- `burkes`: compact two-row texture.
- `sierra`, `sierra-lite`, `two-row-sierra`: progressively different
speed/texture tradeoffs; use only after comparing frames.
Run the exact processor and register its output through the existing media
ledger/cache:
```bash
node <SKILL_DIR>/scripts/dither.mjs \
--input .media/videos/video_001.mp4 \
--out .media/generated/video_001.atkinson.mp4 \
--algorithm atkinson \
--palette '#17121a,#824c50,#e09873,#f7ddb1' \
--point-size 3
npx hyperframes media-use resolve \
--from .media/generated/video_001.atkinson.mp4 --type video --project .
```
Use the registered output path on a real `<img>` or `<video>`. Keep text,
captions, logos, and interface graphics outside the processed media. For a
finite reveal, overlap the original and processed media with identical framing
and crossfade or wipe them using the registered paused GSAP timeline. Do not
label the realtime Bayer shader as Floyd-Steinberg/Atkinson, and do not process
PQ/HLG footage without an explicit SDR tone-map decision.
## Organic Light Leak
Use for one motivated memory beat, time shift, warm scene handoff, or tactile
transition. It is a finite deterministic CSS/GSAP overlay, not a looping
texture, generic flash, or film-stock emulation.
Install the Registry overlay block:
```bash
npx hyperframes add organic-light-leak-overlay --no-clipboard
```
Insert the printed `data-composition-src` host at the intended beat and keep
its duration finite. Its paused timeline owns one rise, peak, and complete
recovery and scales those phases to the placed duration. Inspect the source
before, at the brightest frame, and after recovery. Reject clipped faces, an
unmotivated warm wash, visible black from incorrect blend mode, or a leak that
conceals the subject longer than the transition needs.
## Freeze-Frame Cutout
Use for a social introduction, speaker emphasis, chapter punctuation, sports
or creator beat, or a scrapbook/editorial hold. This requires a real alpha
matte; decoration may not conceal a poor subject edge.
Extract the exact deterministic source frame first, then remove its background:
```bash
ffmpeg -ss <seconds> -i <source-video> -frames:v 1 -y .media/generated/freeze-source.png
npx hyperframes remove-background .media/generated/freeze-source.png \
-o .media/generated/freeze-cutout.png --json
npx hyperframes add freeze-frame-dressing --no-clipboard
```
Add the transparent result as a direct-root timed media layer and insert the
printed overlay block above the same time range. The block owns the paper,
tape, and flash; the host timeline only animates the real cutout:
```html
<img
id="hf-freeze-cutout"
class="clip"
src="./.media/generated/freeze-cutout.png"
alt=""
data-start="6"
data-duration="3"
data-track-index="20"
/>
```
```js
tl.fromTo(
"#hf-freeze-cutout",
{ y: 42, scale: 0.86, rotation: -2 },
{ y: 0, scale: 1, rotation: 0.4, duration: 0.5, ease: "back.out(1.35)" },
freezeAt,
);
```
Inspect the matte over both light and dark temporary plates before styling it.
Reject missing hair/fingers, background halos, a cutout that changes identity,
overly thick outline, exposed frame edges, or a flash that obscures the reveal.
If the matte is not acceptable, choose another frame or keep the original media.
## Social Flash / Editorial Reveal
Use this treatment for one meaningful high-energy cut, creator reveal, product
beat, or before/after handoff. It is not a default transition for every scene.
Avoid it for calm long-form footage, accessibility-sensitive contexts, already
clipped highlights, literal UI that must remain readable through the cut, or
any request for repeated strobing.
Inspect representative frames on both sides of the cut first. Grade each media
layer for its own subject using the appropriate contract above; the flash is
not a substitute for correction. For people, a restrained `skin-soft` payload
is a safe starting point. For literal UI, preserve the pixels and use only the
authored light/motion layers when they do not obscure required information.
Install the Registry overlay block:
```bash
npx hyperframes add editorial-flash-overlay --no-clipboard
```
Insert the printed `data-composition-src` host so the block's midpoint lands
on the cut. Its own paused timeline drives the finite flash. The host timeline
may coordinate outgoing and incoming media motion without reaching into the
block:
```js
tl.to(
"#outgoing-media",
{
scale: 1.035,
"--hf-color-grading-exposure": 0.82,
duration: 0.12,
ease: "power3.in",
},
cutAt - 0.16,
);
tl.fromTo(
"#incoming-media",
{ scale: 1.1 },
{ scale: 1, duration: 0.42, ease: "power3.out" },
cutAt,
);
tl.to(
"#incoming-media",
{
"--hf-color-grading-exposure": 0,
"--hf-color-grading-intensity": 0.58,
duration: 0.24,
ease: "power2.out",
},
cutAt,
);
```
When the shader steps are used, author
`--hf-color-grading-exposure: 0.72` and
`--hf-color-grading-intensity: 0` inline on the incoming media so a fresh seek
has the correct start state. Set the final intensity to the source-approved
value instead of copying `0.58` blindly. Skip the shader intensity step when
the incoming source should remain ungraded.
Keep the rise between roughly `0.035` and `0.055` seconds and the recovery
between `0.24` and `0.38` seconds. Default to one neutral/warm flash event,
never saturated red, never a looping strobe, and never more than one authored
flash inside a one-second treatment window. Verify frames immediately before,
at, and after the cut, then inspect moving playback and a rendered draft. The
peak must hide the cut; the recovery must reveal a correctly framed source with
no retained prior canvas, clipped face, or unexpected highlight damage.
references/media-treatments.md›
# Media treatments
A media treatment is a source-aware plan that composes existing HyperFrames
color, effect, timeline, and Registry primitives. It is not a second runtime
schema. Use this file to choose a primary direction. A matching recipe is an
optional tested seed; bespoke requests may assemble a validated treatment from
the canonical capability catalog.
## Permission and scope
- An explicit request such as "polish this", "make it look better", or "make it
fit the topic" delegates a conservative treatment. Apply it, verify it, and
report what changed.
- During an unsolicited opportunity scan, show or suggest the treatment first.
- Target meaningful photographic media. Skip text, SVG, logos, icons, UI
chrome, and intentionally stylized footage unless the user asks.
- Realtime grading and effects apply to the entire selected real `<img>` or
`<video>`. They do not isolate or track a face, plate, address, or other
region. For region-only work, first create a separate cropped/masked media
layer or use an external segmentation/tracking tool; never imply that a
whole-media Blur or Pixelate performed region isolation.
- The realtime treatment path is Rec.709/SDR. Do not silently process HDR, HLG,
PQ, or camera LOG sources through it.
## Classify the request
Choose the smallest lane that satisfies the request before choosing a recipe
or assembling a custom treatment:
| User intent | Lane |
| --------------------------------------------------- | ------------------------------------------ |
| too dark, flat, too warm, too many shadows | correction |
| shape shadows/highlights or selected colors | wheels, curves, or HSL secondary |
| polished, premium, warm, cinematic, fit the topic | preset or custom treatment |
| retro, print, ASCII, glitch, camcorder | shader effect or effect-bearing preset |
| obscure the whole selected media | privacy Blur or Pixelate |
| hide one face, plate, address, or screen region | separate crop/mask/asset or external tool |
| draw attention to media without changing its pixels | framing, motion, or optional overlay |
| reveal, focus, depixelate, fade the treatment | finite seek-safe treatment keyframes |
| REC HUD, light leak, flash, freeze-frame cutout | Registry overlay plus any justified pixels |
The lane identifies the primary reason for the change; it is not a one-feature
limit. A final treatment may combine correction, a preset, finishing, multiple
compatible shader effects, finite keyframed values, and optional overlays when
the inspected media and user intent justify the complete combination. Keep one
primary intent as the creative anchor so the result remains coherent and
deterministic.
Do not add a stylized Effect when correction solves the complaint. Do not
change color when the request is only temporal, and do not install an overlay
when the selected media alone communicates the result.
Match strength to intent. When the user explicitly names a bold look such as
VHS, glitch, ASCII, halftone, camcorder, print, or engraving, apply its
signature effects strongly enough to read unmistakably. The guard against
unrequested additions does not mean under-delivering an effect the user asked
for. Correction and polish stay restrained; named stylization must be obvious
in the after-frame.
### Translate vague feedback conservatively
| User feedback | First action | Add only when the frames justify it | Never infer |
| ------------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------- |
| too many shadows and a bit boring | lift shadows/protect highlights, then compare one restrained source-appropriate preset | mild contrast or vibrance | retro texture, HUD, or palette effect |
| make the product footage feel premium | protect product color and labels, compare Product Polish | restrained vignette on lifestyle footage | a cinematic LUT or crushed blacks |
| make this reveal cooler | preserve color and animate one supported effect or treatment value | a short owned overlay block | unrelated whole-clip styling |
| make it feel like an old home video | compare 8mm and VHS language against the source | finite weave/flicker or justified HUD | that every old-video request means VHS |
| hide this face | explain that realtime effects are whole-media; isolate the region first or use an external tool | whole-media Blur/Pixelate only when the user accepts that scope | face tracking or masking that was not performed |
| keep the brand colors exact | leave UI/logo pixels unchanged; use framing and motion | demonstrated exposure-only correction | stylized preset, palette, or LUT |
When more than one lane could fit, generate at most two candidates and choose
from inspected before/after evidence. Ordinary correction or polish starts with
one candidate; a second candidate is an escalation, not the default. Do not
stack effects merely to make the answer look more sophisticated.
## Seed or assemble
Use the table when a tested recipe directly fits. Read only that recipe
section. Recipes are optional macros, not a closed list of allowed results.
| Intent or source | Recipe heading to read |
| -------------------------------------------------------- | ------------------------------------------------- |
| Talking head, interview, presenter, people-focused photo | `Natural Portrait` |
| Product footage, lifestyle footage, clean social polish | `Product Polish` |
| Screen capture, dashboard, website, app UI | `UI Fidelity` |
| Warm memory, restrained nostalgia | `Film Memory` |
| Creator/UGC handheld camera character | `Creator Camcorder` |
| Analog tape playback | `VHS Playback` |
| Small-gauge home-movie character | `8mm Home Movie` |
| Editorial dots or ink print | `Editorial Halftone` or `Two-Ink Editorial Print` |
| Monochrome dot/line screen print | `Monochrome Screen Print` |
| Engraved or hand-hatched illustration | `Engraved Illustration` or `Crosshatched Sketch` |
| Curved scanlined display | `CRT Display` |
| Glyph-based art | `Procedural ASCII` |
| Realtime palette quantization | `Ordered Palette Dither` |
| Exact historical error diffusion | `Cached Error Diffusion` |
| Finite warm flare layer | `Organic Light Leak` |
| Held-frame graphic interruption | `Freeze-Frame Cutout` |
| Short exposure-flash transition | `Social Flash / Editorial Reveal` |
Use `rg -n '^## <heading>$' <SKILL_DIR>/references/media-treatment-recipes.md`,
then read only from that heading to the next `##`. Do not load the entire
cookbook for one request.
When source intent is unclear, inspect the concise capability overview:
```bash
hyperframes media-treatment --capabilities --json
```
It lists the complete surface by family with one-line descriptions. Then load
only the family, effect, preset, or palette relevant to the inspected source:
```bash
hyperframes media-treatment --capability <id> --json
```
The focused result provides legal controls, recommended apply values, render
cost, palette support, and the exact animation contract when supported. Use
`--all` only for tooling/tests or a genuinely exhaustive audit. Compose one
nested payload from these existing parts. Recipes and catalog-built payloads
use the same renderer and persistence contract.
Treat `renderLane: "multipass"` as a cost signal. Blur, Bloom, and Kuwahara are
bounded but more expensive than single-pass effects; avoid stacking several of
them across many simultaneous media elements unless the composition needs it,
then verify playback and a draft render.
Cost follows treated pixel area as well as element count. More than two
simultaneously visible full-frame multipass media layers requires a continuous
playback check on the target machine; simplify or pre-render the stack if it
drops frames. Do not impose or claim a universal hard cap from one machine.
## Common workflow
1. Confirm the target is a real `<img>` or `<video>` and inspect source color
metadata.
2. For an image, read it once. For video, capture early/middle/late output as
one labeled sheet and read that one image:
```bash
hyperframes snapshot <project> --frames 3 --no-end --describe false \
--output snapshots/treatment-before
```
Read `snapshots/treatment-before/contact-sheet.jpg`; do not spend separate
model turns reading each frame unless the sheet exposes a specific problem.
Do not infer semantics from signal statistics alone.
3. Choose one primary lane. Use one matching recipe as a tested seed, or read
the overview and one focused capability detail when the request is bespoke.
A seed may be changed or combined with compatible catalog controls when the
contact sheet justifies it. Do not invent keys, exceed reported ranges, or
stack effects without a visual reason. Do not run the generic grade/LUT
resolver first; it adds irrelevant candidates and may download an unused
LUT. Use `media-treatment --selector "#hero" --analyze --json` only when
correction needs measured signal evidence.
4. Persist pixel settings with `hyperframes media-treatment`; it validates and
merges a patch into the existing nested `data-color-grading` contract. Use registered
GSAP only for supported animated values and Registry overlay blocks only
for authored dressing.
```bash
hyperframes media-treatment --selector "<unique selector>" \
--grading '<nested JSON patch>' --apply --json
```
For a temporal reveal, use the focused capability result's `animation`
contract. If it is `null`, the capability is static. Author the starting CSS
property inline on the real media element and return temporary treatment
values to neutral so the finished shot preserves its existing pixels.
Prefer that bounded media animation first; if an overlay is justified,
install the owned Registry block instead of recreating it with bespoke
overlay markup.
For correction and ordinary polish, keep those values as editable
preset/adjustment JSON; do not generate a LUT for controls the realtime shader
already owns.
Use the canonical `details`/`effects` fields for vignette, grain, blur,
pixelate, and related primitives. Do not duplicate them with CSS filters,
SVG turbulence, opacity, or decorative DOM overlays.
5. When the treatment calls for an overlay, install that named block with
`hyperframes add <name> --dir <project> --no-clipboard --json`, inspect its
returned `data-composition-src` host, and place it once using the block's
timing contract. Check for the installed file and host element
ID before insertion; never duplicate an existing overlay block. This is one
treatment workflow: do not make the user discover Catalog or separately ask
for the recipe's justified overlay.
6. For ordinary correction/polish, capture one after-sheet with the same three
timestamps under `snapshots/treatment-after`, compare it to the before-sheet,
and stop when the result is clearly better. Run the normal project check;
do not encode a draft solely to prove a static correction.
7. Escalate only when evidence requires it. Read individual frames to diagnose a
specific visual problem. Preview and render moving evidence when judging
treatment keyframes, glitch/tape motion, overlays, playback smoothness, LUT
timing, or any other temporal behavior. HDR/LOG, privacy, and brand-sensitive
work also require the existing explicit caveats and stronger verification.
If the treatment is not clearly better, keep the source unchanged.
8. Report the selected media, primary intent, recipe seed if used, final
composed controls, optional overlays, and the frames/render that were
actually checked. Do not report visual quality from command success alone.
`media-treatment --analyze` provides deterministic clipping and signal
evidence for local composition media, not subject recognition or automatic
taste. It reports HDR/metadata caveats and a bounded primary-correction patch;
it does not invent wheels, curves, or HSL selections from statistics.
references/memory.md›
# User memory — preferences and recipes
## Preferences — remembered defaults
The lightweight tier of user memory: confirmed brief answers (destination, aspect, language, flow, storyboard, voice, style preset) persisted on the same two-tier split as assets — project `.media/preferences.json` (committed, the team inherits it) and personal `~/.media/preferences.json`. A value earns the personal tier by being confirmed in **two different projects**, so a one-off choice never pollutes the global defaults.
```bash
node <SKILL_DIR>/scripts/prefs.mjs get --hyperframes . --json # merged view (project overrides user)
node <SKILL_DIR>/scripts/prefs.mjs record --hyperframes . --key destination --value x-feed
node <SKILL_DIR>/scripts/prefs.mjs record --hyperframes . --key style_preset --value pin-and-paper --workflow faceless-explainer
```
Only what the user actually confirmed gets recorded — never an inferred or defaulted value. How workflows consume these (a remembered value becomes the recommended default with a receipt, and never skips a question) is the brief contract's rule: `hyperframes/references/brief-contract.md` § 2, Remembered defaults.
## Recipes — frozen video bundles
The heavyweight tier of user memory: one approved run frozen as a named, versioned bundle — `frame.md`, the storyboard skeleton (structure kept, content blanked to per-frame fill-ins), the brief skeleton (from `BRIEF.md` when the project has one — reusable frontmatter kept, run-shape and prose blanked), and the confirmed brief values. Same two tiers: project `.media/recipes/<name>/` (committed) and `~/.media/recipes/<name>/` (a freeze is already a confirmed bundle, so it promotes immediately — no two-project rule). Re-freezing a name bumps `version` and archives the old folder as `<name>@v<N>`.
```bash
node <SKILL_DIR>/scripts/recipe.mjs freeze --hyperframes . --name weekly-promo # workflow read from BRIEF.md (--workflow only for briefless projects)
node <SKILL_DIR>/scripts/recipe.mjs list --hyperframes . --workflow product-launch-video
node <SKILL_DIR>/scripts/recipe.mjs use --hyperframes . --name weekly-promo # also: resolve.mjs --type recipe --entity weekly-promo
```
The freeze is offered once after the final approval (`hyperframes/references/review-loop.md` § 4), and the intent layer (`/hyperframes` → `references/intent-interview.md`, step 1) checks for a match before its first question. Adopting a recipe fills the brief, the design spec, and the storyboard skeleton — and unlike preferences it may skip the questions it answers: the bundle was approved as a whole, and adoption itself is the question.
## Files
- `.media/manifest.jsonl`: machine SSOT, one JSON record per line
- `.media/index.md`: agent-readable table (id, type, dur, dims, path, description)
- `.media/preferences.json`: the project's remembered defaults (committed)
- `~/.media/`: global cross-project reuse cache (content-addressed, SHA-256)
- `~/.media/preferences.json`: personal remembered defaults (promoted after two projects)
- `.media/recipes/<name>/`: frozen video bundles — recipe.json + frame.md + storyboard skeleton (committed)
- `~/.media/recipes/<name>/`: personal recipe tier (promoted on freeze)
- `~/.media/misses.jsonl`: local-only resolve misses, including intent text for `--stats`
references/meta.md›
# Ownership matrix, usage stats, telemetry, privacy
Maintainer-facing reference. Nothing here changes how you resolve or operate on media.
## What it owns (the gaps HyperFrames leaves)
HyperFrames owns media _playback_; media-use owns everything else. Each row is enforced by `scripts/lib/coverage.test.mjs` so the claim can't rot.
| HyperFrames gap | media-use owns it via |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Audio-only, no image/icon | `resolve --type image\|icon` (heygen asset search) |
| No third-party brand logos | `resolve --type logo` (svgl → simple-icons → GitHub org avatar → domain favicon) |
| No voice / audio generation | `resolve --type voice` (HeyGen TTS free-usage path; optional local Kokoro) + the audio engine (`audio/scripts/audio.mjs`) |
| Scattered/duplicated audio engine | one consolidated engine under `audio/` (hyperframes-media retired) |
| No agent media-ops (cut/reframe/transform) | `references/operations.md` + `resolve --from` to register outputs |
| No transcript-driven cutting | `scripts/transcript-cut.mjs` compiles word-timestamp edits into cut lists |
| No auto-duck / publish loudness | `scripts/audio-duck.mjs` + `references/operations.md` loudnorm/sidechain recipes |
| No cross-project memory | global content-addressed cache + auto-promote (`~/.media`) |
| Grade recipes and LUT freezing | `resolve --type grade` emits a paste-ready recipe and `resolve --type lut` freezes validated `.cube` files; direct element analysis/authoring lives in `hyperframes media-treatment` |
| No image generation | RAM-graded local mflux (FLUX) via `scripts/lib/mflux-provider.mjs`, codex `image_gen` upsell (`scripts/lib/codex-provider.mjs`) |
| No video generation | `resolve --type video` — HeyGen avatar video first (free-usage path, sign-in nudge on auth failure), local LTX fallback (`videogen` in `scripts/lib/local-models.mjs`); image-to-video, photo-avatar, dub/translate remain manual `heygen` CLI recipes (`references/operations.md`) |
| Weak local-model defaults | HeyGen free-usage path via the `heygen` CLI; local open-source tools only as opt-in alternatives (`scripts/lib/local-run.mjs`) |
## Usage stats
Use `resolve --stats` for a local, shareable report over the current project's `.media/` manifest, the global `~/.media/` cache, and local resolve misses. Human output is compact; add `--json` for a single machine-readable object, and `--days N` to window timestamped records.
```bash
npx hyperframes media-use resolve --stats --project . --days 7
# media-use stats
# total resolves: 12
# misses: 2
# hit rate: 86%
```
## Telemetry
`resolve` and the edit tools (transcribe / transcript-cut / audio-duck) send an
anonymous usage event to PostHog (`scripts/lib/telemetry.mjs`), so we can see
which capabilities are actually used. It records only the media TYPE, the
resolution SOURCE, and the winning PROVIDER: never the intent text, file names,
or paths, and `$ip:null` so no IP is stored. Best-effort and non-blocking (a
resolve never waits on or fails from telemetry).
Opt out with `DO_NOT_TRACK=1` or `HYPERFRAMES_NO_TELEMETRY=1` (also off in CI and
dev). Same public PostHog project key and opt-outs as the `hyperframes` CLI.
HeyGen request tagging: every generating `heygen` call (TTS, avatar video, catalog
search) carries the allowlisted `X-HeyGen-Client-Source: media-use` header, sourced
from one shared constant (`HEYGEN_CLIENT_SOURCE_ARGV` in `scripts/lib/heygen-cli.mjs`)
so a future call site can't silently ship untagged. Read-only discovery calls
(`voice list`, `avatar list`) are intentionally left untagged.
## Privacy
media-use uses the same shared install id as the `hyperframes` CLI/studio
(`~/.hyperframes/config.json`). When you are signed in to HeyGen, usage is
linked to your account email, or username when email is unavailable, matching
the CLI behavior. The events stay coarse: media type, source, provider, and
small counts only; intent text and paths stay local. Disable telemetry with
`HYPERFRAMES_NO_TELEMETRY=1` or `DO_NOT_TRACK=1`.
references/operations.md›
# Media operations: agent guidance
media-use resolves and remembers assets. For **operating** on them: cutting,
reframing, stitching, transforming, it does not wrap every action as a bespoke
command. Instead it points you at the right local tool (decision OP1). Run the
tool, then register the output with `resolve --from <output> --type <type>` so the
result lands in the ledger and the global cache like any other asset.
All tools below are local and free. ffmpeg is assumed present (it backs the
engine already).
## Cut / trim: keep a slice
```bash
ffmpeg -i in.mp4 -ss 00:00:12 -to 00:00:20 -c copy out.mp4 # 0:12–0:20, no re-encode
```
In-composition trimming usually needs **no new file**: a clip plays a sub-window
via `data-media-start` + `data-duration` (see hyperframes-core). Only cut a
physical file when exporting/assembling outside the composition.
## Reframe / crop: change aspect ratio
```bash
# 16:9 -> 9:16, crop centered
ffmpeg -i in.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" out.mp4
```
For a non-destructive crop, set a `clip-path` on the element in the composition
itself (render-time, source file untouched) instead of re-encoding with ffmpeg.
## Montage / stitch: join clips
```bash
printf "file '%s'\n" a.mp4 b.mp4 c.mp4 > list.txt
ffmpeg -f concat -safe 0 -i list.txt -c copy out.mp4
```
## Silence-cut / highlight: trim dead air, grab the best moment
```bash
auto-editor in.mp4 --edit audio:threshold=4% -o tight.mp4 # pip install auto-editor
scenedetect -i in.mp4 detect-adaptive list-scenes # pip install scenedetect
```
## Transforms with a quality choice (process)
These have a local option AND a higher-quality HeyGen-CLI option. Run the local
one for free/offline; use the HeyGen CLI when quality matters. Showing the user
a **side-by-side** (local vs HeyGen) is the honest way to let them choose.
| Op | Local (free) | HeyGen CLI (quality) |
| ------------------ | -------------------------------------------------- | --------------------------- |
| Background removal | `hyperframes remove-background in.png` (u2net) | `heygen background-removal` |
| Upscale | `realesrgan-ncnn-vulkan -i in.png -o out.png -s 4` | n/a |
| Lipsync (dub) | n/a | `heygen lipsync` |
| Translate | n/a | `heygen video-translate` |
After any op: `resolve --from out.ext --type <type>` to register the derived
asset (it records provenance and auto-promotes to the global cache).
> ponytail: media-use doesn't re-wrap ffmpeg/heygen here, that's deliberate
> (OP1). The value it adds is the ledger + global reuse on the _output_, via
> `--from`. Add a thin `process` verb only if agents repeatedly fumble these
> recipes.
## Exact error-diffusion dither
Use the local processor when the requested look specifically calls for
Floyd-Steinberg, Atkinson/Macintosh, Jarvis-Judice-Ninke, Stucki, Burkes, or a
Sierra variant. These are sequential error-diffusion algorithms, not the
realtime Bayer `effects.dither` shader.
```bash
node <SKILL_DIR>/scripts/dither.mjs \
--input source.mp4 \
--out source.atkinson.mp4 \
--algorithm atkinson \
--palette '#0f380f,#306230,#8bac0f,#9bbc0f' \
--point-size 3
npx hyperframes media-use resolve \
--from source.atkinson.mp4 --type video --project .
```
Available algorithms: `floyd-steinberg`, `atkinson`,
`jarvis-judice-ninke`, `stucki`, `burkes`, `sierra`, `sierra-lite`, and
`two-row-sierra`. The default is balanced Floyd-Steinberg with a black/white
palette. Palettes contain 2-6 `#rrggbb` colors in authored dark-to-light order;
reversing the order intentionally inverts the mapping. `--point-size` controls
1-20px blocks; `--brightness` and `--contrast` accept 0.5-2; `--detail` accepts
0.1-1.
The processor supports ordinary SDR images and MP4 video, preserves video
audio, and emits BT.709 MP4. It rejects tagged PQ/HLG input rather than silently
tone-mapping it. To animate the transformation, keep the original and processed
files as two real media layers and use the seek-safe GSAP timeline to reveal or
crossfade between them. Use the realtime Bayer shader instead when the dither
amount itself must animate continuously.
## Transcription (default: Parakeet, better than whisper.cpp)
`transcribe.mjs` is the default local transcription path. It runs **NVIDIA
Parakeet-TDT via parakeet-mlx**, which beats whisper.cpp on the Open ASR
Leaderboard (avg WER ~6.05% vs 7.44%; on NOISY audio 4.73% vs 5.96%, where
whisper-large-v3 hallucinated to 308% WER on meetings) and is 5-10x faster.
It emits `{ text, words:[{text,start,end}] }` with word timestamps (merged from
Parakeet's sub-word tokens), feeding transcript-cut, captions, and the audio
engine directly.
```bash
# install once: uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx
node <SKILL_DIR>/scripts/transcribe.mjs --input talk.mp4 --out talk.transcribe.json
# equivalently, the hyperframes CLI has Parakeet built in (auto-detects it, whisper fallback):
npx hyperframes transcribe talk.mp4 --engine parakeet # or --engine auto (default)
```
VERIFIED on 24GB: accurate, ~3s (cached) for 8s audio. Parakeet covers English +
25 European languages. For other languages, or when parakeet-mlx is not
installed, transcribe.mjs auto-falls-back to whisper.cpp (99 languages) via
`hyperframes transcribe`. `--engine parakeet|whisper` forces one. (Cohere
Transcribe tops the leaderboard on paper but its mlx-audio quants produced
garbage and ran 40-70x slower on a Mac in testing, so it is not wired in.)
## Text-based editing (transcript cut)
`transcript-cut.mjs` is a compiler, not a wrapper: it turns word timestamps and
agent cut decisions into exact kept segments. It is provided even though the rest
of this file is guidance-only.
```bash
node <SKILL_DIR>/scripts/transcript-cut.mjs \
--input talk.mp4 \
--transcript talk.transcribe.json \
--remove "12.41-15.02,88.3-91.7" \
--remove-fillers "um,uh,like" \
--cut-silence 0.8 \
--out talk.cut.mp4
resolve --from talk.cut.mp4 --type video
```
Use `--plan` first when you want to inspect the kept segment JSON before encoding.
## Ducking (declare in-composition / bake for export)
B1, declare ducking in the composition. `audio-duck.mjs` emits a volume lane
as a `data-automation` attribute. Add it to the background `<audio>` element;
the source file stays untouched. Lane times are clip-local, so pass
`--composition` to let the script subtract the element's `data-start`.
```bash
node <SKILL_DIR>/scripts/audio-duck.mjs \
--meta audio_meta.json \
--target "#bgm" \
--composition index.html
```
```html
<!-- auto-duck: #bgm under narration; add to its <audio> element -->
data-automation='{"version":1,"lanes":[{"target":"volume","points":[{"t":0,"v":0.6},{"t":3.42,"v":0.6},{"t":3.57,"v":0.15},{"t":9.87,"v":0.15},{"t":10.27,"v":0.6}]}]}'
```
B2, bake ducking only for exported or standalone files.
```bash
ffmpeg -i bgm.mp3 -i voice.wav \
-filter_complex "[0][1]sidechaincompress=threshold=0.03:ratio=8:attack=200:release=400[ducked]" \
-map "[ducked]" bgm.ducked.wav
```
Declare inside compositions. Bake only for assets leaving the hyperframes
pipeline.
## Publish loudness
Two-pass `loudnorm` measures first, then applies the measured values with the
target LUFS baked in.
Socials target, -14 LUFS:
```bash
ffmpeg -i mix.wav \
-af loudnorm=I=-14:TP=-1.5:LRA=11:print_format=json \
-f null -
ffmpeg -i mix.wav \
-af loudnorm=I=-14:TP=-1.5:LRA=11:measured_I=<input_i>:measured_TP=<input_tp>:measured_LRA=<input_lra>:measured_thresh=<input_thresh>:offset=<target_offset>:linear=true:print_format=summary \
mix.social.wav
```
Podcast target, -16 LUFS:
```bash
ffmpeg -i mix.wav \
-af loudnorm=I=-16:TP=-1.5:LRA=11:print_format=json \
-f null -
ffmpeg -i mix.wav \
-af loudnorm=I=-16:TP=-1.5:LRA=11:measured_I=<input_i>:measured_TP=<input_tp>:measured_LRA=<input_lra>:measured_thresh=<input_thresh>:offset=<target_offset>:linear=true:print_format=summary \
mix.podcast.wav
```
## Generate: images (local first, cloud upsell)
`resolve --type image` retrieves from the HeyGen catalog first; on a miss it
GENERATES. Two paths, best-for-the-machine picked automatically:
1. **Local (default, free, private): mflux** (FLUX-on-MLX). `resolve` spec-checks
AVAILABLE RAM and runs the best FLUX-class model that fits, via
`scripts/lib/local-models.mjs` (`imagegen` ladder) + `mflux-provider.mjs`.
The RAM ladder (agent sees it via `describeModelLadder("imagegen", specs)`):
| Tier | Model | Needs (available RAM) | Notes |
| ------ | -------------------- | --------------------- | ----------------------------------- |
| medium | FLUX.1 schnell int4 | ~8GB (`--low-ram`) | ~20s/512px on 24GB. VERIFIED. Fast. |
| large | FLUX.2 Klein 4B int4 | ~32GB | higher quality, full-resident |
| xlarge | Qwen-Image | ~64GB | top quality, 64GB+ Macs only |
Gotchas baked into the table: the official FLUX repos are HF-gated, so it
points at non-gated community 4-bit re-uploads; and `--low-ram` is MANDATORY
at the medium tier (without it a 768x512 run swap-thrashed to 90 minutes on
24GB; with it, 20 seconds).
2. **Cloud upsell (better quality): the `codex` CLI** `image_gen` tool, on the
user's ChatGPT subscription (codex owns auth, no key here, no per-call
charge). It is the automatic fallback when no local model fits AND the
explicit "make it better" choice on any machine. Users who just want codex
can ask for it directly. Verified: prompt -> raster -> frozen + ledgered.
`--local-only` keeps mflux (once cached) and skips codex (network).
## Generate: video (`resolve --type video`, HeyGen avatar first)
`resolve --type video "<intent>"` is the default path. It generates a
script-driven HeyGen avatar video first (the free-usage allowance — OAuth
sessions ride the web-plan free avatar-video quota where eligible, API keys
follow normal API billing), falling back to local generative LTX only when
HeyGen is unavailable, uncredentialed, or `--local-only` is passed. The two
are non-substitutable outputs (a real presenter vs. a generic generative
clip), so treat the fallback as "HeyGen wasn't reachable," not "upgrade the
quality":
- **HeyGen avatar video (default, free for new API users):**
`heygenVideoGenerate` (`scripts/lib/heygen-video-provider.mjs`) shells the
`heygen` CLI — never the raw API — auto-picking a public avatar and a
starfish voice (override with `--avatar-id`/`--voice-id`, threaded through
as `ctx.avatarId`/`ctx.voiceId`). If the CLI reports `not_authenticated`,
the provider prints an onboarding recommendation (avatar video is free for
new API users — sign in) to stderr and falls through to LTX instead of
hard-failing.
- **Local fallback: LTX 2.3 on MLX** via `dgrauet/ltx-2-mlx`, the `videogen`
ladder in `local-models.mjs` (`ltx-video-provider.mjs`). Generative clips
(t2v), spec-gated to RAM. Verified on 24GB: 512x320 x 33f with audio.
Every generating `heygen` call from media-use — TTS, avatar video, and
catalog search — sends the allowlisted `X-HeyGen-Client-Source: media-use`
header (persistent flag, works on every subcommand) via the shared
`HEYGEN_CLIENT_SOURCE_ARGV` constant (`scripts/lib/heygen-cli.mjs`), so usage
tags correctly in billing/resource meta and shows up in the API dashboards.
Read-only discovery (`avatar list`, `voice list`) doesn't need it.
For structured bodies `resolve --type video` doesn't expose yet (a specific
`avatar_id`/`voice_id` combination beyond the ctx overrides, or a
pre-recorded `audio_url` instead of a script), the raw `heygen video create`
recipe below remains the escape hatch:
```bash
# discover an avatar + a starfish voice, then create + wait
heygen avatar list --ownership public --limit 5
heygen voice list --engine starfish --limit 5
heygen video create --headers "X-HeyGen-Client-Source: media-use" --wait -d '{
"type": "avatar",
"avatar_id": "<avatar-id>",
"script": "Your narration here.",
"voice_id": "<voice-id>"
}'
```
Avatar videos are deterministic + script-driven (lip-sync from a script or a
pre-recorded `audio_url`), distinct from the generative LTX clips. After a
manual recipe renders, `resolve --from <downloaded.mp4> --type video` to
ledger it (not needed when generating via `resolve --type video` directly —
that already ledgers the result).
### Image-to-video (animate any still into a talking clip)
Not wired into `resolve --type video` (deferred — the `avatar` type covers
the default script-driven case). `heygen video create` takes the raw
`POST /v3/videos` body, so switching `type`
from `avatar` to `image` animates **any image of a person** into a lip-synced
talking video, with no avatar/photo-avatar creation step first. Point `image` at a
public URL or an uploaded `asset_id`, and drive speech with a `script`+`voice_id`
or a pre-recorded `audio_url`:
```bash
heygen video create --headers "X-HeyGen-Client-Source: media-use" --wait -d '{
"type": "image",
"image": { "type": "url", "url": "https://example.com/person.jpg" },
"script": "Your narration here.",
"voice_id": "<voice-id>"
}'
```
Common optional fields: `title`, `resolution` (`4k`/`1080p`/`720p`),
`aspect_ratio`, `remove_background`, `background`, `voice_settings`,
`motion_prompt` + `expressiveness` (photo-avatar animation), and
`callback_url`/`callback_id` for webhooks. Don't hardcode these from memory: the
CLI self-documents the full, current body with
`heygen video create --request-schema` (a discriminated union keyed on `type`),
so read the schema rather than trusting a stale field list. For a still you'll
reuse across many scripts, create a reusable **Photo Avatar** once instead
(`heygen avatar create`). Ledger the result with
`resolve --from <downloaded.mp4> --type video`. Docs:
<https://developers.heygen.com/image-to-video>.
## HEVC / H.265 sources
HEVC/H.265 sources need no conversion for **render** (FFmpeg pre-decodes all
input video) or for **preview** (auto-proxy transcodes and caches an H.264
copy on first use, disable with `--no-proxy` or `media.autoProxy: false` in
hyperframes.json). A manual H.264 proxy via `ffmpeg -i in.mp4 -c:v libx264
-crf 18 proxy.mp4`, registered with `resolve --from`, remains available for
edge cases (e.g. auto-proxy disabled, or ffmpeg unavailable at preview time).
references/resolve.md›
# Resolve — command, flags, reuse, adopt, inventory
```bash
npx hyperframes media-use resolve --type <type> --intent "<description>" --project <dir>
```
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`
## Types
| Type | What it finds | Provider / cascade |
| ------- | -------------------------------- | ------------------------------------------------------------ |
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
| `icon` | Icons, symbols | HeyGen asset search (type=icon) |
| `logo` | Official brand marks | svgl → simple-icons → GitHub org avatar → domain favicon |
| `voice` | TTS voiceover | HeyGen TTS free-usage path; optional local Kokoro |
| `grade` | HyperFrames color-grading blocks | Core preset → look index params/CDN LUT → deterministic cube |
| `lut` | Reusable `.cube` LUT files | Look index params/CDN LUT → deterministic cube |
## Examples
```bash
# Background music
npx hyperframes media-use resolve --type bgm --intent "upbeat tech launch" --project .
# → resolved bgm_001 → .media/audio/bgm/bgm_001.mp3 (bgm, 25s)
# Sound effect
npx hyperframes media-use resolve --type sfx --intent "whoosh" --project .
# → resolved sfx_001 → .media/audio/sfx/sfx_001.mp3 (sfx, 0.57s)
# Image
npx hyperframes media-use resolve --type image --intent "gradient tech background" --project .
# → resolved image_001 → .media/images/image_001.jpg (image)
# Icon
npx hyperframes media-use resolve --type icon --intent "rocket" --project .
# → resolved icon_001 → .media/images/icon_001.png (icon, transparent)
# Brand logo (official mark — never redrawn by hand)
npx hyperframes media-use resolve --type logo --entity linkedin --intent "LinkedIn logo" --project .
# → resolved logo_001 → .media/images/logo_001.svg (logo, official mark)
# Color grade block
npx hyperframes media-use resolve --type grade --intent "warm daylight" --project . --json
# → {"ok":true,"preset":"warm-daylight","grading":{"preset":"warm-daylight","intensity":1},...}
# LUT file
npx hyperframes media-use resolve --type lut --intent "teal orange blockbuster" --project .
# → resolved lut_001 → .media/luts/lut_001.cube (lut)
```
## Flags
| Flag | Description |
| --------------- | ------------------------------------------------------------------------------------ |
| `--type, -t` | Media type: bgm, sfx, image, icon, logo, voice, grade, lut |
| `--intent, -i` | What you need (natural language) |
| `--entity, -e` | Entity name for cache matching (optional) |
| `--project, -p` | Project directory (default: .) |
| `--candidates` | List reusable assets (project + global cache) for `--type`; no download, no mutation |
| `--reuse <sha>` | Import a specific global-cache asset (by content sha/prefix, from `--candidates`) |
| `--from` | Freeze a local file or direct public URL (ingest) |
| `--for` | Analyze a local image/video and add measured adjust suggestions (`grade` only) |
| `--local-only` | Offline: skip every network provider (cache + local only) |
| `--provider` | Force one generator (e.g. `codex`, `mflux`, `kokoro`, `heygen`) |
| `--adopt` | Bulk-import existing assets/ into manifest |
| `--doctor` | Check local CLI dependencies; no manifest changes |
| `--stats` | Print local usage stats from `.media/` and `~/.media`; no manifest changes |
| `--days N` | Limit `--stats` to timestamped records/misses from the last N days |
| `--json` | Output JSON instead of one-line result |
## Reuse before you resolve
Before resolving bgm/sfx/image/icon/logo/grade/lut, **check what already exists and reuse it when it fits.** media-use does not semantically match for you — you are the judge. It surfaces candidates; you decide.
```bash
npx hyperframes media-use resolve --type bgm --intent "upbeat tech launch" --candidates --project .
# [project] upbeat tech launch (25s, heygen.audio.sounds)
# .media/audio/bgm/bgm_001.wav
# [global] energetic tech intro (22s, heygen.audio.sounds)
# --reuse 06e052c075fd2b80
```
Read the list and judge semantic fit yourself — "upbeat tech launch" ≈ "energetic tech intro" is a call only you can make from the descriptions. Then:
- **A project candidate fits** → just reference its path in your composition. Nothing else to run.
- **A global candidate fits** → `resolve --type bgm --reuse <sha>` copies it into this project (self-contained render) and records it.
- **Nothing fits** → resolve fresh (`--type ... --intent ...`).
**Trust guardrail — when unsure, resolve fresh.** A redundant download is cheap; shipping the wrong asset is not. Judge fit from description + prompt + type + duration/dims. For **brand/entity** assets, reuse a _global_ candidate only when the entity matches exactly — the global cache aggregates every project you have worked on, so a `--candidates` list can surface another client's brand mark and its prompt text. Never reuse a cross-project brand asset on a loose match.
The deterministic floor still runs automatically: an identical (case/whitespace-insensitive) repeat auto-reuses with no `--candidates` step. `--candidates` is only for the semantic layer above that floor — and a fuzzy match is **never** auto-applied; reuse is always your explicit call. On a resolve that misses the floor and is about to fetch, media-use prints a one-line stderr hint when similar cached assets exist, pointing you back here.
## How it works
`resolve` runs an automatic floor, then falls through to fetching:
1. Check project `.media/manifest.jsonl` for a prompt match (case- and whitespace-insensitive) — auto-reuse
2. Scan existing `assets/` directory for unregistered files that share a word with the need
3. Check global cache `~/.media/` for a reusable asset matched on the same normalized prompt — auto-reuse
4. Search via provider (HeyGen audio catalog, HeyGen asset search), or resolve color locally
5. Freeze file to `.media/<type>/`, register in manifest, regenerate `index.md`, auto-promote to `~/.media/`
Steps 1 and 3 are the **deterministic floor**: they only auto-reuse an exact-normalized match, never a fuzzy one. Semantic reuse ("close enough") is the agent's explicit call via [Reuse before you resolve](#reuse-before-you-resolve) — it never happens automatically. The agent gets back **one line**; candidates, scores, provenance stay on disk.
## Stamping provenance when mounting a resolved video
`resolve` never writes composition HTML itself — mounting a `<video>` element
into a composition is always the agent's own edit. When the mounted video's
manifest record has `provenance.provider === "heygen.video"` (check
`.media/manifest.jsonl` or the one-line resolve output for the provider name),
add `data-media-source="heygen"` to that `<video>` tag. Leave the attribute off
entirely for every other provider (`ltx.local`, an adopted/local file, etc.) —
this is not a general provider taxonomy, just the one signal render telemetry
tracks today.
## Adopt existing projects
Most HyperFrames projects already have assets in `assets/`. media-use adopts them:
```bash
npx hyperframes media-use resolve --adopt --project .
# → adopted 9 assets from assets/
# bgm_001 → assets/bgm/mango-fizz.mp3 (bgm, 146.6s)
# image_001 → assets/images/avatar.jpg (image, 400×400)
```
`ffprobe` extracts real duration and dimensions. During resolve, unregistered files in `assets/` matching the intent are adopted on the fly.
## Reading the inventory
After resolve or adopt, read `.media/index.md` for the full inventory:
```
# .media · 4 assets
id type dur dims path description
bgm_001 bgm 25s - .media/audio/bgm/bgm_001.mp3 upbeat tech launch
sfx_001 sfx 0.6s - .media/audio/sfx/sfx_001.mp3 whoosh
image_001 image - 1920×1080 .media/images/image_001.jpg gradient tech background
icon_001 icon - 200×200 .media/images/icon_001.png rocket
```
## Cross-project reuse
Assets are cached automatically on resolve. Every resolved/ingested asset is auto-promoted to the global cache at `~/.media/`, so subsequent resolves for the same (or near-identical) prompt, in any project, hit the cache with no re-download and no provider call.
For a _semantically_ similar (not identical) need in another project, the exact-match floor won't fire — use [Reuse before you resolve](#reuse-before-you-resolve): `--candidates` lists the global assets, and `--reuse <sha>` imports the one you pick. This is how a track resolved in one project gets reused in the next when the wording differs.
references/setup-providers.md›
# Setup and providers — install, auth, RAM ladders, forcing a provider
## Setup — install heygen first (free-usage path)
Install the HeyGen CLI through its [verified release instructions](https://developers.heygen.com/cli), then run:
```bash
heygen update # free usage needs the OAuth-capable CLI (v0.3.0+)
heygen auth login --oauth # OAuth = free subscription credits; --api-key bills API credits
```
This unlocks the FREE path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar videos. Sign in with `--oauth` — the free allowance rides on the OAuth session (an API key bills API credits instead). **media-use requires heygen >= v0.3.0 uniformly** (the OAuth free-usage path needs it), so `--doctor` nudges older CLIs to update even for API-key-only use. Before resolving anything, verify setup with:
```bash
npx hyperframes media-use resolve --doctor
```
## Providers
media-use holds no keys; every external tool owns its auth. Generation is
centered on the HeyGen CLI free-usage path. Install and authenticate `heygen`
before resolving bgm/sfx/image/icon/voice/avatar-video. Local tools are opt-in
alternatives where they exist: mflux for image, Kokoro for voice, Parakeet for
transcription, and LTX for local video generation. `resolve` spec-checks
AVAILABLE RAM for those local ladders (`describeModelLadder`); the agent can
see the ladder and override.
| Type | Provider / path |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| bgm/sfx | heygen catalog free-usage path |
| image | heygen search free-usage path; optional local mflux; codex `image_gen` upsell |
| voice | heygen tts free-usage path; optional local **Kokoro** (free, on-device) |
| icon | heygen asset search free-usage path |
| logo | svgl, then simple-icons, then GitHub org avatar, then domain favicon (all free) |
| grade/lut | local core-preset map, params/CDN look index, deterministic `buildCube` fallback |
| video | heygen avatar video free-usage path (sign-in nudge on auth failure); optional local LTX (`videogen` ladder). Image-to-video / photo-avatar / dub stay manual `heygen` recipes |
Local Kokoro (voice), mflux (image), and LTX (video) run on-device (free,
private, offline once cached). The `codex` CLI remains the ChatGPT-sub image
upsell. Cost rule (X4): the agent confirms before an agent-initiated paid call;
a user-requested one just runs — `heygen.video` is flagged paid (metered free
allowance) so an agent-initiated `resolve --type video` confirms first.
To force a specific generator (e.g. a user says "make this image with codex"),
pass `--provider codex`: it pins resolution to that provider and skips the
free-usage default. See `references/operations.md` for the RAM ladders and
provider recipes.
`--local-only` skips every network provider, including the free HeyGen ones,
leaving the project + global cache and any installed local provider. For
HeyGen-only types, that means no fresh resolve.
## CLI tools used (what to run, and how to enable each)
`resolve` auto-cascades; each provider shells one CLI. HeyGen is the
free-usage path for bgm/sfx/image/icon catalog search, TTS (voice), and avatar
video, so those capabilities need `heygen` installed and authenticated. Local
tools are OPT-IN alternatives where they exist; install one to unlock its free,
private, on-device path instead of or ahead of HeyGen for that type. Only
`ffmpeg`/`ffprobe` are strictly required for the tool to run at all.
| Tool | Serves | Install |
| ------------------ | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `ffmpeg`/`ffprobe` | adopt probing, smart-grade signalstats, cut, duck bake, loudnorm | system package (`brew install ffmpeg`) |
| `heygen` | catalog (bgm/sfx/image/icon) + TTS (voice) + avatar video — the free-usage path | install through [verified HeyGen release instructions](https://developers.heygen.com/cli), then `heygen auth login --oauth` (needs >= v0.3.0) |
| `mflux-generate` | local image gen (FLUX), best-for-RAM | `uv venv ~/.venvs/mflux && VIRTUAL_ENV=~/.venvs/mflux uv pip install mflux==0.9.6` |
| `codex` | image gen upsell (ChatGPT sub) | Codex CLI, logged in via ChatGPT (owns its own auth) |
| `parakeet-mlx` | local transcription (default ASR, best) | `uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx` |
| `ltx-2-mlx` | local video gen | `git clone https://github.com/dgrauet/ltx-2-mlx && cd ltx-2-mlx && uv sync --all-extras` |
| `npx hyperframes` | Kokoro TTS (voice), whisper.cpp (transcribe fallback), remove-background | via the hyperframes CLI; whisper.cpp is built on first use (Homebrew on macOS, else git+cmake), models download from HuggingFace |
The RAM-graded local-model shortlist + exact per-tier install/invoke lives in
`scripts/lib/local-models.mjs` (the agent can read `describeModelLadder(cap, specs)`
to see which model fits this machine). Without a tool on PATH, its provider
prints a one-line diagnostic to stderr and resolve falls through where another
provider exists (e.g. no `mflux` -> codex image upsell; no `parakeet-mlx` -> whisper.cpp).
`heygen asset search` is a pre-launch command hidden from `heygen --help`, but it
runs; providers tag requests with the allowlisted `X-HeyGen-Client-Source` header
(v0.3.0+).
references/telemetry-dashboard.md›
# media-use usage dashboard
Reproducible definition of the media-use usage dashboard. The dashboard answers
"how much is media-use used, for what, is reuse working, and what can't it
satisfy" from the telemetry `scripts/lib/telemetry.mjs` already emits. Build it
in an authorized HyperFrames analytics project; this doc is the source of truth
so it can be recreated. Local complement: `resolve --stats` (same questions,
from `.media/` + `~/.media`, no dashboard access needed).
## Identity (see `scripts/lib/telemetry.mjs`)
Events attribute to the **same person as the hyperframes CLI and studio**
— the shared install id in `~/.hyperframes/config.json` (`anonymousId`), stitched
to the HeyGen account (`$identify`, `distinct_id` = email/username) on sign-in.
Not fully anonymous by design; pseudonymous before sign-in, account-linked after.
`$ip:null`. Opt-out: `HYPERFRAMES_NO_TELEMETRY=1` / `DO_NOT_TRACK=1` (also CI, dev).
## Event catalog (verified present in-project)
Every event carries `surface: "media-use"`. Event **properties are coarse** —
never intent text, file names, or paths.
| Event | Fires on | Key properties |
| ---------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
| `media_use_resolve` | a resolve that produced/returned an asset | `type`, `source`, `provider`, `via`, `local_only`, `provider_override` |
| `media_use_resolve_miss` | a resolve that found nothing | `type`, `local_only`, `provider_override` (no intent) |
| `media_use_candidates` | `--candidates` / `--dry-run` listing | `type`, counts |
| `media_use_doctor_run` | `--doctor` | `ok`, `checks_failed`, `failed[]` |
| `media_use_compare` | `grade-compare` / `compare` | `command`, `cells`, `truncated`, `total`, `render_ready_timed_out` |
| `media_use_transcribe` · `media_use_duck` · `media_use_transcript_cut` | audio-engine ops | op-specific |
## Dashboard tiles
1. **Invocation volume** — `query-trends`, count of `media_use_resolve` over time (daily). "How much."
2. **By media type** — `media_use_resolve` broken down by `type` (bgm/sfx/image/icon/logo/voice/grade/lut). "For what."
3. **Resolve hit-rate** — trends formula: `A / (A + B)` where A = `media_use_resolve`, B = `media_use_resolve_miss`. "Is the catalog covering needs."
4. **Provider mix** — `media_use_resolve` broken down by `provider`; a second tile by `via` (`url` / `params-fallback` / `params`) to catch CDN→params LUT downgrades.
5. **Top misses** — `media_use_resolve_miss` broken down by `type` (the tuning signal — pair with local `resolve --stats`, which also shows the missed _intents_ that telemetry deliberately omits).
6. **Doctor health** — `media_use_doctor_run` broken down by `failed[]` (which dependency check fails most) + `checks_failed` distribution.
7. **Compare cost** — `media_use_compare` by `command`, plus `truncated` / `render_ready_timed_out` rates (observe before lifting the 16-cell cap).
8. **Adoption (optional)** — if the `first_run` property ships (plan U5), segment `media_use_resolve` first-run vs repeat.
## Recreate in an analytics dashboard
For each tile, confirm the event/property schema, build its trend or breakdown,
then add it to a dashboard. Keep names prefixed `media-use:` so the dashboard is
greppable. Cross-surface note: because identity is shared with CLI/studio, you
can also break these down by the same person across `cli_command*` and `studio:*`
events.
scripts/audio-duck.mjs›
#!/usr/bin/env node
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import { parseArgs } from "node:util";
import { duckKeyframes, duckLane, speechSpans } from "./lib/duck.mjs";
import { track } from "./lib/telemetry.mjs";
const { values: args } = parseArgs({
options: {
meta: { type: "string" },
target: { type: "string" },
duck: { type: "string", default: "0.25" },
attack: { type: "string", default: "0.15" },
release: { type: "string", default: "0.4" },
"merge-gap": { type: "string", default: "0.6" },
sequential: { type: "boolean", default: false },
gap: { type: "string", default: "0" },
offsets: { type: "string" },
composition: { type: "string" },
json: { type: "boolean", default: false },
help: { type: "boolean", short: "h", default: false },
},
strict: true,
});
if (args.help) {
console.log(`media-use audio-duck — generate a volume ducking lane (data-automation)
Usage:
node audio-duck.mjs --meta audio_meta.json --target "#bgm"
Options:
--meta audio_meta.json or JSON word transcript
--target GSAP selector for the background audio element
--duck Duck multiplier (default: 0.25)
--attack Duck-in duration seconds (default: 0.15)
--release Restore duration seconds (default: 0.4)
--merge-gap Bridge speech gaps smaller than this many seconds (default: 0.6)
--sequential Place multi-line meta back to back at composition time
--gap Extra seconds between sequential lines (default: 0)
--offsets Explicit placement, "l1=0,l2=3.4" (voice id = start seconds)
--composition Read target data-volume and data-start from this HTML file
--json Output { spans, keyframes, lane }
--help, -h Show this help`);
process.exit(0);
}
try {
run();
await track("media_use_duck", { sequential: !!args.sequential });
} catch (err) {
if (args.json) console.log(JSON.stringify({ ok: false, error: err.message }));
else console.error(`error: ${err.message}`);
process.exit(1);
}
function run() {
if (!args.meta || !args.target) throw new Error("--meta and --target are required");
const meta = JSON.parse(readFileSync(resolve(args.meta), "utf8"));
const target = args.target;
const { baseVolume, clipStart } = readTargetAttrs(args.composition, target);
const offsets = args.offsets
? Object.fromEntries(
args.offsets.split(",").map((pair) => {
const [id, t] = pair.split("=");
return [id.trim(), Number(t)];
}),
)
: undefined;
const spans = speechSpans(meta, {
mergeGap: Number(args["merge-gap"]),
sequential: args.sequential,
gap: Number(args.gap),
offsets,
});
const keyframes = duckKeyframes(spans, {
duck: Number(args.duck),
attack: Number(args.attack),
release: Number(args.release),
baseVolume,
});
const lane = duckLane(keyframes, { clipStart, baseVolume });
if (args.json) {
console.log(JSON.stringify({ spans, keyframes, lane }));
return;
}
console.log(`<!-- auto-duck: ${target} under narration; add to its <audio> element -->`);
console.log(`data-automation='${JSON.stringify(lane)}'`);
}
function readTargetAttrs(composition, target) {
if (!composition || !target.startsWith("#")) return { baseVolume: 1, clipStart: 0 };
const id = target.slice(1);
const html = readFileSync(resolve(composition), "utf8");
// ponytail: regex is enough here because this only reads one attribute from
// one user-authored composition element, not arbitrary HTML.
const tag = html.match(new RegExp(`<[^>]*\\bid=["']${escapeRegExp(id)}["'][^>]*>`, "i"))?.[0];
const attr = (name) => Number(tag?.match(new RegExp(`\\b${name}=["']([^"']+)["']`, "i"))?.[1]);
const volume = attr("data-volume");
const start = attr("data-start");
return {
baseVolume: Number.isFinite(volume) ? volume : 1,
clipStart: Number.isFinite(start) ? start : 0,
};
}
function escapeRegExp(value) {
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
}
scripts/compatibility.test.mjs›
import { cpSync, mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { execFileSync } from "node:child_process";
import test from "node:test";
function run(scriptsDir, script, args) {
return execFileSync(process.execPath, [join(scriptsDir, script), ...args], {
encoding: "utf8",
stdio: ["ignore", "pipe", "pipe"],
});
}
test("published memory scripts run without reaching outside the skill", () => {
const scratchDir = mkdtempSync(join(tmpdir(), "media-use-compat-"));
const skillDir = join(scratchDir, "media-use");
cpSync(new URL("..", import.meta.url), skillDir, { recursive: true });
const scriptsDir = join(skillDir, "scripts");
const projectDir = join(scratchDir, "project");
try {
run(scriptsDir, "prefs.mjs", ["get", "--hyperframes", projectDir, "--json"]);
run(scriptsDir, "recipe.mjs", ["list", "--hyperframes", projectDir, "--json"]);
run(scriptsDir, "transcribe.mjs", ["--help"]);
} finally {
rmSync(scratchDir, { recursive: true, force: true });
}
});
scripts/dither.mjs›
#!/usr/bin/env node
import { execFileSync, spawn } from "node:child_process";
import { once } from "node:events";
import { existsSync, mkdirSync, renameSync, rmSync } from "node:fs";
import { dirname, extname, resolve } from "node:path";
import { text } from "node:stream/consumers";
import { parseArgs } from "node:util";
import {
ERROR_DIFFUSION_ALGORITHMS,
applyErrorDiffusionRgba,
errorDiffusionBufferLength,
} from "./lib/error-diffusion.mjs";
const IMAGE_EXTENSIONS = new Set([".png", ".jpg", ".jpeg", ".webp", ".bmp", ".tif", ".tiff"]);
const OUTPUT_IMAGE_EXTENSIONS = new Set([".png", ".jpg", ".jpeg", ".webp"]);
const { values: args } = parseArgs({
options: {
input: { type: "string", short: "i" },
out: { type: "string", short: "o" },
algorithm: { type: "string", short: "a", default: "floyd-steinberg" },
palette: { type: "string", default: "#000000,#ffffff" },
"point-size": { type: "string", default: "3" },
brightness: { type: "string", default: "1" },
contrast: { type: "string", default: "1.2" },
detail: { type: "string", default: "1" },
json: { type: "boolean", default: false },
help: { type: "boolean", short: "h", default: false },
},
strict: true,
});
if (args.help) {
console.log(`media-use dither — exact cached error-diffusion for image or MP4 media
Usage:
node dither.mjs --input in.mp4 --out out.mp4 [options]
Options:
--algorithm, -a ${Object.keys(ERROR_DIFFUSION_ALGORITHMS).join(" | ")}
--palette 2-6 authored-order #rrggbb colors, comma-separated
--point-size Block size in pixels, 1-20 (default: 3)
--brightness 0.5-2 (default: 1)
--contrast 0.5-2 (default: 1.2)
--detail Diffused-error strength, 0.1-1 (default: 1)
--json Output JSON status
--help, -h Show this help
Video output uses the source average frame rate as CFR; VFR cadence is normalized.
After processing, register the output with:
node resolve.mjs --from <output> --type image|video`);
process.exit(0);
}
try {
const result = await run();
if (args.json) console.log(JSON.stringify({ ok: true, ...result }));
else {
console.log(`dithered ${result.input} -> ${result.out} (${result.algorithm})`);
console.log(`next: resolve --from ${result.out} --type ${result.type}`);
}
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
if (args.json) console.log(JSON.stringify({ ok: false, error: message }));
else console.error(`error: ${message}`);
process.exit(1);
}
async function run() {
if (!args.input || !args.out) throw new Error("--input and --out are required");
const inputPath = resolve(args.input);
const outPath = resolve(args.out);
if (!existsSync(inputPath)) throw new Error(`input file not found: ${inputPath}`);
if (inputPath === outPath) throw new Error("--out must differ from --input");
const metadata = probe(inputPath);
if (metadata.colorTransfer === "smpte2084" || metadata.colorTransfer === "arib-std-b67") {
throw new Error(
`HDR ${metadata.colorTransfer} input is not supported by the 8-bit SDR dither processor; tone-map to Rec.709 first`,
);
}
const options = {
algorithm: args.algorithm,
palette: args.palette.split(",").map((color) => color.trim()),
pointSize: Number(args["point-size"]),
brightness: Number(args.brightness),
contrast: Number(args.contrast),
detail: Number(args.detail),
};
// Validate before starting FFmpeg or creating an output file.
applyErrorDiffusionRgba(new Uint8ClampedArray(4), 1, 1, options, new Float32Array(3));
mkdirSync(dirname(outPath), { recursive: true });
const inputIsImage = IMAGE_EXTENSIONS.has(extname(inputPath).toLowerCase());
if (inputIsImage) {
if (!OUTPUT_IMAGE_EXTENSIONS.has(extname(outPath).toLowerCase())) {
throw new Error("image output must use .png, .jpg, .jpeg, or .webp");
}
processImage(inputPath, outPath, metadata, options);
} else {
if (extname(outPath).toLowerCase() !== ".mp4") throw new Error("video output must use .mp4");
await processVideo(inputPath, outPath, metadata, options);
}
return {
input: inputPath,
out: outPath,
type: inputIsImage ? "image" : "video",
algorithm: options.algorithm,
palette: options.palette,
point_size: options.pointSize,
brightness: options.brightness,
contrast: options.contrast,
detail: options.detail,
};
}
function probe(filePath) {
const raw = execFileSync(
"ffprobe",
["-v", "error", "-print_format", "json", "-show_streams", "-show_format", "--", filePath],
{ encoding: "utf8", timeout: 10_000 },
);
const parsed = JSON.parse(raw);
const video = parsed.streams?.find((stream) => stream.codec_type === "video");
if (!video?.width || !video?.height)
throw new Error(`no readable video/image stream: ${filePath}`);
const fps = usableFrameRate(video.avg_frame_rate) ?? usableFrameRate(video.r_frame_rate) ?? "30";
return {
width: video.width,
height: video.height,
fps,
colorTransfer: video.color_transfer || "",
};
}
function processImage(inputPath, outPath, metadata, options) {
const frameBytes = metadata.width * metadata.height * 4;
const rgba = execFileSync(
"ffmpeg",
[
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-i",
inputPath,
"-frames:v",
"1",
"-f",
"rawvideo",
"-pix_fmt",
"rgba",
"-",
],
{ maxBuffer: frameBytes + 1024 },
);
if (rgba.length !== frameBytes)
throw new Error(`decoded ${rgba.length} bytes; expected ${frameBytes}`);
applyErrorDiffusionRgba(rgba, metadata.width, metadata.height, options);
const temporary = temporaryOutput(outPath);
try {
execFileSync(
"ffmpeg",
[
"-y",
"-hide_banner",
"-loglevel",
"error",
"-f",
"rawvideo",
"-pix_fmt",
"rgba",
"-s:v",
`${metadata.width}x${metadata.height}`,
"-i",
"-",
"-frames:v",
"1",
temporary,
],
{ input: rgba, maxBuffer: frameBytes + 1024 },
);
renameSync(temporary, outPath);
} finally {
rmSync(temporary, { force: true });
}
}
async function processVideo(inputPath, outPath, metadata, options) {
const temporary = temporaryOutput(outPath);
const frameBytes = metadata.width * metadata.height * 4;
const keyframeInterval = String(Math.max(1, Math.round(frameRateNumber(metadata.fps))));
const errors = new Float32Array(
errorDiffusionBufferLength(metadata.width, metadata.height, options.pointSize),
);
const decoder = spawn("ffmpeg", [
"-hide_banner",
"-loglevel",
"error",
"-nostdin",
"-i",
inputPath,
"-map",
"0:v:0",
"-f",
"rawvideo",
"-pix_fmt",
"rgba",
"-",
]);
const encoder = spawn("ffmpeg", [
"-y",
"-hide_banner",
"-loglevel",
"error",
"-f",
"rawvideo",
"-pix_fmt",
"rgba",
"-s:v",
`${metadata.width}x${metadata.height}`,
"-r",
metadata.fps,
"-i",
"-",
"-i",
inputPath,
"-map",
"0:v:0",
"-map",
"1:a?",
"-map_metadata",
"1",
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"18",
"-g",
keyframeInterval,
"-keyint_min",
keyframeInterval,
"-sc_threshold",
"0",
"-pix_fmt",
"yuv420p",
"-x264-params",
"colorprim=bt709:transfer=bt709:colormatrix=bt709",
"-color_primaries:v",
"bt709",
"-color_trc:v",
"bt709",
"-colorspace:v",
"bt709",
"-color_range",
"tv",
"-c:a",
"aac",
"-b:a",
"192k",
"-shortest",
"-movflags",
"+faststart",
temporary,
]);
const decoderError = text(decoder.stderr);
const encoderError = text(encoder.stderr);
const decoderDone = once(decoder, "close").then(([code]) => code ?? 1);
const encoderDone = once(encoder, "close").then(([code]) => code ?? 1);
try {
const frame = Buffer.allocUnsafe(frameBytes);
let frameOffset = 0;
for await (const chunk of decoder.stdout) {
let chunkOffset = 0;
while (chunkOffset < chunk.length) {
const length = Math.min(frameBytes - frameOffset, chunk.length - chunkOffset);
chunk.copy(frame, frameOffset, chunkOffset, chunkOffset + length);
chunkOffset += length;
frameOffset += length;
if (frameOffset !== frameBytes) continue;
applyErrorDiffusionRgba(frame, metadata.width, metadata.height, options, errors);
await writeFrame(encoder.stdin, frame);
frameOffset = 0;
}
}
if (frameOffset)
throw new Error(`decoder returned a partial RGBA frame (${frameOffset} bytes)`);
encoder.stdin.end();
const [decoderCode, encoderCode] = await Promise.all([decoderDone, encoderDone]);
if (decoderCode !== 0) throw new Error(`FFmpeg decode failed: ${(await decoderError).trim()}`);
if (encoderCode !== 0) throw new Error(`FFmpeg encode failed: ${(await encoderError).trim()}`);
renameSync(temporary, outPath);
} catch (error) {
decoder.kill("SIGKILL");
encoder.kill("SIGKILL");
throw error;
} finally {
rmSync(temporary, { force: true });
}
}
function writeFrame(stream, frame) {
return new Promise((resolveWrite, reject) => {
stream.write(frame, (error) => (error ? reject(error) : resolveWrite()));
});
}
function usableFrameRate(value) {
if (!value || value === "0/0") return null;
const number = frameRateNumber(value);
return Number.isFinite(number) && number > 0 ? value : null;
}
function frameRateNumber(value) {
const [numerator, denominator = "1"] = value.split("/");
return Number(numerator) / Number(denominator);
}
function temporaryOutput(outPath) {
const extension = extname(outPath);
return `${outPath.slice(0, -extension.length)}.part-${process.pid}${extension}`;
}
scripts/dither.test.mjs›
import assert from "node:assert/strict";
import { execFileSync, spawnSync } from "node:child_process";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import test from "node:test";
import { fileURLToPath } from "node:url";
const SCRIPT = fileURLToPath(new URL("./dither.mjs", import.meta.url));
const HAS_FFMPEG =
spawnSync("ffmpeg", ["-version"], { stdio: "ignore" }).status === 0 &&
spawnSync("ffprobe", ["-version"], { stdio: "ignore" }).status === 0;
function fixture() {
const dir = mkdtempSync(join(tmpdir(), "media-use-dither-"));
return { dir, cleanup: () => rmSync(dir, { recursive: true, force: true }) };
}
function ffmpeg(args) {
execFileSync("ffmpeg", ["-y", "-hide_banner", "-loglevel", "error", ...args]);
}
function run(args) {
return spawnSync(process.execPath, [SCRIPT, ...args, "--json"], { encoding: "utf8" });
}
test("processes an image into the requested ordered palette", { skip: !HAS_FFMPEG }, (t) => {
const { dir, cleanup } = fixture();
t.after(cleanup);
const input = join(dir, "source.png");
const output = join(dir, "dithered.png");
ffmpeg(["-f", "lavfi", "-i", "testsrc2=size=16x12:rate=1", "-frames:v", "1", input]);
const result = run([
"--input",
input,
"--out",
output,
"--algorithm",
"floyd-steinberg",
"--palette",
"#000000,#ffffff",
"--point-size",
"3",
]);
assert.equal(result.status, 0, result.stderr || result.stdout);
assert.equal(JSON.parse(result.stdout).type, "image");
const rgb = execFileSync("ffmpeg", [
"-hide_banner",
"-loglevel",
"error",
"-i",
output,
"-frames:v",
"1",
"-f",
"rawvideo",
"-pix_fmt",
"rgb24",
"-",
]);
for (let index = 0; index < rgb.length; index += 3) {
const value = rgb[index];
assert.ok(value === 0 || value === 255);
assert.equal(rgb[index + 1], value);
assert.equal(rgb[index + 2], value);
}
});
test("processes moving MP4 frames, audio, and BT.709 metadata", { skip: !HAS_FFMPEG }, (t) => {
const { dir, cleanup } = fixture();
t.after(cleanup);
const input = join(dir, "source.mp4");
const output = join(dir, "dithered.mp4");
ffmpeg([
"-f",
"lavfi",
"-i",
"testsrc2=size=16x12:rate=3:duration=2",
"-f",
"lavfi",
"-i",
"sine=frequency=440:duration=3",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p",
"-c:a",
"aac",
input,
]);
const result = run([
"--input",
input,
"--out",
output,
"--algorithm",
"atkinson",
"--palette",
"#0f380f,#306230,#8bac0f,#9bbc0f",
]);
assert.equal(result.status, 0, result.stderr || result.stdout);
const probe = JSON.parse(
execFileSync(
"ffprobe",
["-v", "error", "-print_format", "json", "-show_streams", "-show_format", "--", output],
{
encoding: "utf8",
},
),
);
const video = probe.streams.find((stream) => stream.codec_type === "video");
const audio = probe.streams.find((stream) => stream.codec_type === "audio");
assert.equal(video.width, 16);
assert.equal(video.height, 12);
assert.equal(video.nb_frames, "6");
assert.equal(video.color_space, "bt709");
assert.equal(video.color_transfer, "bt709");
assert.equal(video.color_primaries, "bt709");
assert.equal(video.color_range, "tv");
assert.equal(audio.codec_name, "aac");
assert.ok(Number(probe.format.duration) < 2.5, "audio must not outlive processed video");
const keyframes = execFileSync(
"ffprobe",
[
"-v",
"error",
"-select_streams",
"v:0",
"-skip_frame",
"nokey",
"-show_entries",
"frame=best_effort_timestamp_time",
"-of",
"csv=p=0",
"--",
output,
],
{ encoding: "utf8" },
)
.trim()
.split(/\s+/)
.map((value) => Number.parseFloat(value));
assert.deepEqual(keyframes, [0, 1]);
});
test("rejects tagged PQ or HLG instead of silently producing SDR", { skip: !HAS_FFMPEG }, (t) => {
const { dir, cleanup } = fixture();
t.after(cleanup);
const input = join(dir, "hlg.mp4");
ffmpeg([
"-f",
"lavfi",
"-i",
"color=c=white:size=16x12:rate=1:duration=1",
"-c:v",
"libx264",
"-pix_fmt",
"yuv420p10le",
"-x264-params",
"colorprim=bt2020:transfer=arib-std-b67:colormatrix=bt2020nc",
"-color_primaries:v",
"bt2020",
"-color_trc:v",
"arib-std-b67",
"-colorspace:v",
"bt2020nc",
input,
]);
const result = run(["--input", input, "--out", join(dir, "wrong.mp4")]);
assert.equal(result.status, 1);
assert.match(JSON.parse(result.stdout).error, /HDR arib-std-b67 input is not supported/);
});
scripts/eval.mjs›
#!/usr/bin/env node
/**
* media-use eval — compare baseline (no media-use) vs. with media-use
* on real registry blocks. Produces an HTML report.
*/
import {
mkdtempSync,
cpSync,
rmSync,
readFileSync,
readdirSync,
existsSync,
writeFileSync,
} from "node:fs";
import { join, basename, resolve, dirname } from "node:path";
import { execFileSync } from "node:child_process";
import { tmpdir } from "node:os";
import { fileURLToPath } from "node:url";
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
const REPO_ROOT = resolve(SCRIPT_DIR, "..", "..", "..");
const RESOLVE_SCRIPT = join(SCRIPT_DIR, "resolve.mjs");
const TEST_BLOCKS = [
"registry/blocks/nyc-paris-flight",
"registry/blocks/macos-tahoe-liquid-glass",
"registry/blocks/blue-sweater-intro-video",
"registry/blocks/vpn-youtube-spot",
"registry/blocks/apple-money-count",
"registry/blocks/liquid-glass-notification",
"registry/blocks/instagram-follow",
];
// Run resolve.mjs with args as a literal argv array (no shell), so values
// interpolated from manifest metadata (--intent prompt, --type) can't inject
// shell. Mirrors the execFileSync fix in probe.mjs / heygen-search.mjs.
function run(args, opts = {}) {
try {
return {
ok: true,
output: execFileSync(process.execPath, [RESOLVE_SCRIPT, ...args], {
encoding: "utf8",
timeout: 15000,
stdio: "pipe",
...opts,
}).trim(),
};
} catch (err) {
return { ok: false, output: (err.stdout || "") + (err.stderr || ""), code: err.status };
}
}
function countAssetFiles(dir) {
const assetsDir = join(dir, "assets");
if (!existsSync(assetsDir)) return { count: 0, files: [] };
const files = [];
function walk(d, base = "") {
for (const e of readdirSync(d, { withFileTypes: true })) {
const rel = base ? `${base}/${e.name}` : e.name;
if (e.isDirectory()) walk(join(d, e.name), rel);
else files.push(rel);
}
}
walk(assetsDir);
return { count: files.length, files };
}
function evalBlock(blockPath) {
const fullPath = join(REPO_ROOT, blockPath);
if (!existsSync(fullPath)) return null;
const name = basename(blockPath);
const tmp = mkdtempSync(join(tmpdir(), `mu-eval-${name}-`));
try {
cpSync(fullPath, tmp, { recursive: true });
// baseline: what the agent sees WITHOUT media-use
const baseline = countAssetFiles(tmp);
const htmlFiles = readdirSync(tmp).filter((f) => f.endsWith(".html"));
// parse compositions for asset references
const assetRefs = [];
for (const hf of htmlFiles) {
const html = readFileSync(join(tmp, hf), "utf8");
const srcMatches = html.matchAll(/src=["']([^"']+?)["']/g);
for (const m of srcMatches) {
const ref = m[1];
if (ref.startsWith("data:") || ref.startsWith("http")) continue;
assetRefs.push({ composition: hf, ref });
}
const urlMatches = html.matchAll(/url\(["']?([^"')]+?)["']?\)/g);
for (const m of urlMatches) {
const ref = m[1];
if (ref.startsWith("data:") || ref.startsWith("http") || ref.startsWith("#")) continue;
assetRefs.push({ composition: hf, ref });
}
}
// with media-use: run --adopt
const adoptResult = run(["--adopt", "--project", tmp, "--json"]);
let adopted = { ok: false, adopted: 0, assets: [] };
if (adoptResult.ok) {
try {
adopted = JSON.parse(adoptResult.output);
} catch {
/* */
}
}
// read the generated index
const indexPath = join(tmp, ".media", "index.md");
const indexContent = existsSync(indexPath)
? readFileSync(indexPath, "utf8")
: "(no index generated)";
// read manifest for detail
const manifestPath = join(tmp, ".media", "manifest.jsonl");
const manifest = existsSync(manifestPath)
? readFileSync(manifestPath, "utf8")
.trim()
.split("\n")
.map((l) => {
try {
return JSON.parse(l);
} catch {
return null;
}
})
.filter(Boolean)
: [];
// test resolve cache hit: try resolving something that was adopted
let resolveTest = null;
if (manifest.length > 0) {
const first = manifest[0];
const prompt = first.provenance?.prompt || first.description;
const r = run(["--type", first.type, "--intent", prompt, "--project", tmp, "--json"]);
if (r.ok) {
try {
resolveTest = JSON.parse(r.output);
} catch {
/* */
}
}
}
// test resolve miss: try resolving something that doesn't exist
const missResult = run([
"--type",
"bgm",
"--intent",
"nonexistent query xyz",
"--project",
tmp,
"--json",
]);
let resolveMiss = null;
if (!missResult.ok) {
try {
resolveMiss = JSON.parse(missResult.output);
} catch {
/* */
}
}
// coverage: which composition refs are covered by the manifest
const manifestPaths = new Set(manifest.map((m) => m.path));
const coverage = assetRefs.map((r) => ({
...r,
covered: manifestPaths.has(r.ref),
}));
return {
name,
baseline: { fileCount: baseline.count, files: baseline.files, htmlCount: htmlFiles.length },
compositions: htmlFiles,
assetRefs: coverage,
adopted: { count: adopted.adopted, assets: adopted.assets || [] },
index: indexContent,
manifest,
resolveTest,
resolveMiss,
};
} finally {
rmSync(tmp, { recursive: true, force: true });
}
}
function generateReport(results) {
const all = results.filter(Boolean);
const passed = all.filter((r) => r.adopted.count > 0);
const rows = results
.filter(Boolean)
.map((r) => {
const hasMetadata = r.manifest.some((m) => m.duration || m.width);
const cacheHit = r.resolveTest?._source === "cached";
const missHandled = r.resolveMiss?.ok === false;
return `<tr>
<td><strong>${r.name}</strong></td>
<td>${r.baseline.fileCount} files, ${r.baseline.htmlCount} comp${r.baseline.htmlCount === 1 ? "" : "s"}</td>
<td>${r.adopted.count} adopted</td>
<td>${hasMetadata ? "<span class='pass'>with metadata</span>" : "<span class='warn'>no metadata</span>"}</td>
<td>${cacheHit ? "<span class='pass'>cache hit</span>" : "<span class='warn'>no hit</span>"}</td>
<td>${missHandled ? "<span class='pass'>handled</span>" : "<span class='fail'>unexpected</span>"}</td>
</tr>`;
})
.join("\n");
const details = results
.filter(Boolean)
.filter((r) => r.adopted.count > 0)
.map((r) => {
const assetRows = r.manifest
.map((m) => {
const dur = m.duration != null ? `${m.duration}s` : "—";
const dims = m.width && m.height ? `${m.width}×${m.height}` : "—";
return `<tr><td>${m.id}</td><td>${m.type}</td><td>${dur}</td><td>${dims}</td><td class="path">${m.path}</td><td>${m.description || ""}</td></tr>`;
})
.join("\n");
const coveredCount = r.assetRefs.filter((c) => c.covered).length;
const totalRefs = r.assetRefs.length;
const coveragePct = totalRefs > 0 ? Math.round((coveredCount / totalRefs) * 100) : 100;
const refRows = r.assetRefs
.map(
(c) =>
`<tr><td class="path">${c.composition}</td><td class="path">${c.ref}</td><td>${c.covered ? "<span class='pass'>covered</span>" : "<span class='warn'>not in manifest</span>"}</td></tr>`,
)
.join("\n");
return `<div class="block-detail">
<h3>${r.name}</h3>
<p style="font-size:13px;color:var(--muted)">${r.compositions.length} composition${r.compositions.length === 1 ? "" : "s"}: ${r.compositions.join(", ")}</p>
<div class="comparison">
<div class="col">
<h4>Baseline (no media-use)</h4>
<p>Agent sees: ${r.baseline.fileCount} raw files in assets/<br>No metadata, no type info, no relationship to compositions.</p>
<pre class="file-list">${r.baseline.files.join("\n") || "(no assets)"}</pre>
</div>
<div class="col">
<h4>With media-use (after --adopt)</h4>
<p>Agent reads index.md — structured, typed, with metadata:</p>
<pre class="index">${escapeHtml(r.index)}</pre>
</div>
</div>
${
totalRefs > 0
? `<h4>Composition → asset coverage <span class="${coveragePct === 100 ? "pass" : "warn"}">${coveragePct}%</span> (${coveredCount}/${totalRefs} refs)</h4>
<table class="manifest">
<thead><tr><th>composition</th><th>asset reference</th><th>in manifest?</th></tr></thead>
<tbody>${refRows}</tbody>
</table>`
: ""
}
<h4>Manifest records</h4>
<table class="manifest">
<thead><tr><th>id</th><th>type</th><th>dur</th><th>dims</th><th>path</th><th>description</th></tr></thead>
<tbody>${assetRows}</tbody>
</table>
</div>`;
})
.join("\n");
return `<title>media-use eval report</title>
<style>
:root { --bg: #fafaf7; --text: #1b1b18; --muted: #7a756a; --accent: #0d7377; --good: #1a7a3a; --warn: #b45309; --fail: #dc2626; --border: #e8e5df; --surface: #fff; --mono: ui-monospace, 'SF Mono', Menlo, Consolas, monospace; --sans: system-ui, -apple-system, sans-serif; --serif: Georgia, serif }
* { box-sizing: border-box; margin: 0 } body { background: var(--bg); color: var(--text); font-family: var(--serif); line-height: 1.6; font-size: 15px; padding: 40px 24px }
.wrap { max-width: 1100px; margin: 0 auto }
h1 { font-family: var(--sans); font-size: 28px; font-weight: 700; margin-bottom: 8px; letter-spacing: -.02em }
h2 { font-family: var(--sans); font-size: 20px; font-weight: 650; margin: 32px 0 12px; letter-spacing: -.01em }
h3 { font-family: var(--sans); font-size: 17px; font-weight: 650; margin: 24px 0 8px }
h4 { font-family: var(--sans); font-size: 14px; font-weight: 600; margin: 16px 0 6px; color: var(--muted) }
p { margin-bottom: 10px }
.meta { font-family: var(--mono); font-size: 12px; color: var(--muted); margin-bottom: 24px }
.summary { display: flex; gap: 16px; margin: 16px 0; flex-wrap: wrap }
.stat { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 18px; flex: 1; min-width: 140px }
.stat .num { font-family: var(--sans); font-size: 28px; font-weight: 700; color: var(--accent) }
.stat .label { font-family: var(--mono); font-size: 11px; color: var(--muted); text-transform: uppercase; letter-spacing: .1em }
table { width: 100%; border-collapse: collapse; font-size: 13px; font-family: var(--sans); margin: 8px 0 }
th { text-align: left; font-family: var(--mono); font-size: 10px; letter-spacing: .08em; text-transform: uppercase; color: var(--muted); border-bottom: 2px solid var(--border); padding: 6px 8px; font-weight: 700 }
td { border-bottom: 1px solid var(--border); padding: 7px 8px; vertical-align: top }
td.path { font-family: var(--mono); font-size: 12px; color: var(--muted); max-width: 300px; overflow: hidden; text-overflow: ellipsis }
.pass { color: var(--good); font-weight: 600 } .warn { color: var(--warn); font-weight: 600 } .fail { color: var(--fail); font-weight: 600 }
.comparison { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin: 12px 0 }
@media(max-width:700px) { .comparison { grid-template-columns: 1fr } }
.col { background: var(--surface); border: 1px solid var(--border); border-radius: 8px; padding: 14px 16px }
.col h4 { margin-top: 0 }
pre { font-family: var(--mono); font-size: 12px; background: #1b1b18; color: #d4d0c8; border-radius: 6px; padding: 12px 14px; overflow-x: auto; margin: 6px 0; line-height: 1.5 }
pre.file-list { background: var(--bg); color: var(--muted); border: 1px solid var(--border) }
pre.index { white-space: pre; }
.block-detail { border-top: 1px solid var(--border); padding-top: 20px; margin-top: 20px }
.verdict { margin-top: 24px; padding: 16px 20px; border-radius: 8px; font-family: var(--sans); font-size: 15px }
.verdict.ship { background: #edfbf0; border: 1px solid #1a7a3a; color: #1a7a3a }
.verdict.wait { background: #fff3ec; border: 1px solid #d94f04; color: #d94f04 }
</style>
<div class="wrap">
<h1>media-use eval report</h1>
<p class="meta">${new Date().toISOString().slice(0, 10)} · ${all.length} blocks evaluated · baseline vs. media-use --adopt</p>
<div class="summary">
<div class="stat"><div class="num">${all.length}</div><div class="label">blocks tested</div></div>
<div class="stat"><div class="num">${passed.length}</div><div class="label">with assets</div></div>
<div class="stat"><div class="num">${all.reduce((s, r) => s + r.adopted.count, 0)}</div><div class="label">assets adopted</div></div>
<div class="stat"><div class="num">${all.filter((r) => r.manifest.some((m) => m.duration || m.width)).length}</div><div class="label">with ffprobe metadata</div></div>
<div class="stat"><div class="num">${(() => {
const refs = all.flatMap((r) => r.assetRefs);
const covered = refs.filter((c) => c.covered).length;
return refs.length > 0 ? Math.round((covered / refs.length) * 100) + "%" : "—";
})()}</div><div class="label">composition coverage</div></div>
</div>
<h2>Results matrix</h2>
<table>
<thead><tr><th>Block</th><th>Baseline</th><th>Adopted</th><th>Metadata</th><th>Cache hit</th><th>Miss handling</th></tr></thead>
<tbody>${rows}</tbody>
</table>
<h2>Before / after comparisons</h2>
${details}
<div class="verdict ${passed.length >= 3 ? "ship" : "wait"}">
${
passed.length >= 3
? `<strong>Ship it.</strong> ${passed.length}/${all.length} blocks adopted successfully with metadata. Resolve cache hits work. Miss handling is clean.`
: `<strong>Needs work.</strong> Only ${passed.length} blocks adopted. Check the failures above.`
}
</div>
</div>`;
}
function escapeHtml(str) {
return str.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">");
}
console.log("media-use eval · running against registry blocks...\n");
const results = [];
for (const block of TEST_BLOCKS) {
const fullPath = join(REPO_ROOT, block);
if (!existsSync(fullPath)) {
console.log(` skip ${basename(block)} (not found)`);
results.push(null);
continue;
}
process.stdout.write(` ${basename(block)}...`);
const result = evalBlock(block);
if (result) {
console.log(
` ${result.adopted.count} adopted, ${result.manifest.filter((m) => m.duration || m.width).length} with metadata`,
);
} else {
console.log(" failed");
}
results.push(result);
}
const report = generateReport(results);
const outPath = join(SCRIPT_DIR, "..", "eval-report.html");
writeFileSync(outPath, report);
console.log(`\nReport: ${outPath}`);
scripts/lib/cutlist.mjs›
import { normalizeWords } from "./words.mjs";
const MIN_SEGMENT_SECONDS = 0.2;
const SILENCE_PAD_SECONDS = 0.15;
export function compileCutList(transcript, opts = {}) {
const words = normalizeWords(transcript);
if (opts.keep != null && hasRemovalSource(opts)) {
throw new Error("--keep is mutually exclusive with removal options");
}
if (opts.keep != null) {
const duration = durationFrom(words, opts);
const ranges = parseTimeRanges(opts.keep);
return finalizeKept(duration != null ? clampRanges(ranges, duration) : ranges);
}
const duration = durationFrom(words, opts);
if (!duration) return [];
const removals = [
...parseTimeRanges(opts.remove),
...wordIndexRanges(words, opts.removeWords),
...fillerRanges(words, opts.removeFillers),
...silenceRanges(words, opts.cutSilence),
];
const mergedRemovals = mergeRanges(clampRanges(removals, duration));
return finalizeKept(invertRanges(mergedRemovals, duration));
}
function hasRemovalSource(opts) {
return (
opts.remove != null ||
opts.removeWords != null ||
opts.removeFillers != null ||
opts.cutSilence != null
);
}
function durationFrom(words, opts) {
const explicit = Number(opts.duration ?? opts.totalDuration);
if (Number.isFinite(explicit) && explicit > 0) return explicit;
const last = words.at(-1);
return last && Number.isFinite(last.end) && last.end > 0 ? last.end : null;
}
function parseTimeRanges(value) {
if (value == null || value === false || value === "") return [];
if (typeof value === "string") {
return value
.split(",")
.map((part) => part.trim())
.filter(Boolean)
.map(parseRangeString);
}
if (!Array.isArray(value)) throw new Error("range list must be a string or array");
return value.map((range) => {
if (Array.isArray(range)) return cleanRange(Number(range[0]), Number(range[1]));
return cleanRange(Number(range?.start), Number(range?.end));
});
}
function parseRangeString(value) {
const match = value.match(/^([0-9]*\.?[0-9]+)\s*-\s*([0-9]*\.?[0-9]+)$/);
if (!match) throw new Error(`invalid range: ${value}`);
return cleanRange(Number(match[1]), Number(match[2]));
}
function cleanRange(start, end) {
if (!Number.isFinite(start) || !Number.isFinite(end)) {
throw new Error("range start/end must be finite numbers");
}
if (end < start) throw new Error(`range end ${end} is before start ${start}`);
return { start, end };
}
function wordIndexRanges(words, value) {
if (value == null || value === false || value === "") return [];
const ranges = typeof value === "string" ? value.split(",") : value;
if (!Array.isArray(ranges)) throw new Error("--remove-words must be a string or array");
return ranges
.map((range) => (typeof range === "string" ? range.trim() : range))
.filter(Boolean)
.map((range) => {
const [first, last = first] =
typeof range === "string" ? range.split("-").map((n) => n.trim()) : range;
const startIndex = Number(first);
const endIndex = Number(last);
if (!Number.isInteger(startIndex) || !Number.isInteger(endIndex)) {
throw new Error(`invalid word range: ${range}`);
}
if (startIndex < 0 || endIndex < startIndex || endIndex >= words.length) {
throw new Error(`word range out of bounds: ${range}`);
}
return { start: words[startIndex].start, end: words[endIndex].end };
});
}
function fillerRanges(words, value) {
if (value == null || value === false || value === "") return [];
const fillers = Array.isArray(value)
? value
: String(value)
.split(",")
.map((s) => s.trim());
const set = new Set(fillers.filter(Boolean).map(bareToken));
if (set.size === 0) return [];
// Whisper emits words with attached punctuation and arbitrary case
// ("UM," / "Um."), so compare bare tokens.
return words
.filter((word) => set.has(bareToken(word.text)))
.map((word) => ({ start: word.start, end: word.end }));
}
function bareToken(text) {
return String(text)
.toLowerCase()
.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, "");
}
function silenceRanges(words, value) {
if (value == null || value === false || value === "") return [];
const threshold = Number(value);
if (!Number.isFinite(threshold) || threshold <= 0) {
throw new Error("--cut-silence must be a positive number");
}
const ranges = [];
for (let i = 0; i < words.length - 1; i++) {
const current = words[i];
const next = words[i + 1];
const gap = next.start - current.end;
if (gap <= threshold) continue;
const start = current.end + SILENCE_PAD_SECONDS;
const end = next.start - SILENCE_PAD_SECONDS;
if (end > start) ranges.push({ start, end });
}
return ranges;
}
function clampRanges(ranges, duration) {
return ranges
.map((range) => ({
start: Math.max(0, Math.min(duration, range.start)),
end: Math.max(0, Math.min(duration, range.end)),
}))
.filter((range) => range.end > range.start);
}
function mergeRanges(ranges) {
const sorted = ranges
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
.sort((a, b) => a.start - b.start || a.end - b.end);
const merged = [];
for (const range of sorted) {
const prev = merged.at(-1);
if (prev && range.start <= prev.end) {
prev.end = Math.max(prev.end, range.end);
} else {
merged.push({ ...range });
}
}
return merged;
}
function invertRanges(removals, duration) {
const kept = [];
let cursor = 0;
for (const range of removals) {
if (range.start > cursor) kept.push({ start: cursor, end: range.start });
cursor = Math.max(cursor, range.end);
}
if (cursor < duration) kept.push({ start: cursor, end: duration });
return kept;
}
function finalizeKept(ranges) {
return mergeRanges(ranges)
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
.filter((range) => round3(range.end - range.start) >= MIN_SEGMENT_SECONDS);
}
function round3(n) {
return Math.round(Number(n) * 1000) / 1000;
}
scripts/lib/duck.mjs›
import { wordListsFromMediaMeta } from "./words.mjs";
// audio_meta.json word times are relative to each line's own file.
export function speechSpans(meta, { mergeGap = 0.6, offsets, sequential = false, gap = 0 } = {}) {
const merge = Number(mergeGap);
const lists = wordListsFromMediaMeta(meta);
const voices = Array.isArray(meta?.voices) ? meta.voices : [];
if (lists.length > 1 && !offsets && !sequential) {
throw new Error(
"audio_meta has multiple voice lines with file-relative times; pass --sequential or --offsets so spans land at composition time",
);
}
const intervals = [];
let cursor = 0;
for (let i = 0; i < lists.length; i++) {
const voice = voices[i];
let offset = 0;
if (offsets) {
const id = voice?.id ?? String(i);
if (!(id in offsets)) throw new Error(`--offsets is missing voice "${id}"`);
offset = Number(offsets[id]) || 0;
} else if (sequential) {
offset = cursor;
const lineDuration = Number(voice?.duration_s) || Math.max(...lists[i].map((w) => w.end), 0);
cursor += lineDuration + (Number(gap) || 0);
}
for (const word of lists[i]) {
if (word.end > word.start)
intervals.push({ start: word.start + offset, end: word.end + offset });
}
}
return mergeIntervals(intervals, Number.isFinite(merge) && merge >= 0 ? merge : 0.6);
}
export function duckKeyframes(
spans,
{ duck = 0.25, attack = 0.15, release = 0.4, baseVolume = 1 } = {},
) {
const base = finiteOr(baseVolume, 1);
const ducked = round3(base * finiteOr(duck, 0.25));
const keyframes = [];
for (const span of spans) {
keyframes.push({
time: round3(Math.max(0, finiteOr(span.start, 0))),
volume: ducked,
duration: round3(finiteOr(attack, 0.15)),
});
keyframes.push({
time: round3(Math.max(0, finiteOr(span.end, 0))),
volume: round3(base),
duration: round3(finiteOr(release, 0.4)),
});
}
return keyframes.sort((a, b) => a.time - b.time);
}
/** Volume lane for `data-automation`: composition-time keyframes as clip-local ramps. */
export function duckLane(keyframes, { clipStart = 0, baseVolume = 1 } = {}) {
const start = finiteOr(clipStart, 0);
const points = [{ t: 0, v: round3(finiteOr(baseVolume, 1)) }];
const push = (t, v) => {
if (t > points.at(-1).t) points.push({ t: round3(t), v });
};
for (const kf of keyframes) {
const t = Math.max(0, kf.time - start);
push(t, points.at(-1).v);
push(t + kf.duration, kf.volume);
}
return { version: 1, lanes: [{ target: "volume", points }] };
}
function mergeIntervals(intervals, mergeGap) {
const sorted = intervals
.map((range) => ({ start: round3(range.start), end: round3(range.end) }))
.sort((a, b) => a.start - b.start || a.end - b.end);
const merged = [];
for (const range of sorted) {
const prev = merged.at(-1);
if (prev && (range.start <= prev.end || range.start - prev.end < mergeGap)) {
prev.end = Math.max(prev.end, range.end);
} else {
merged.push({ ...range });
}
}
return merged;
}
function finiteOr(value, fallback) {
const n = Number(value);
return Number.isFinite(n) ? n : fallback;
}
function round3(n) {
return Math.round(Number(n) * 1000) / 1000;
}
scripts/lib/error-diffusion.mjs›
export const ERROR_DIFFUSION_ALGORITHMS = {
"floyd-steinberg": {
kernel: [
[1, 0, 7],
[-1, 1, 3],
[0, 1, 5],
[1, 1, 1],
],
divisor: 16,
},
atkinson: {
kernel: [
[1, 0, 1],
[2, 0, 1],
[-1, 1, 1],
[0, 1, 1],
[1, 1, 1],
[0, 2, 1],
],
divisor: 8,
},
"jarvis-judice-ninke": {
kernel: [
[1, 0, 7],
[2, 0, 5],
[-2, 1, 3],
[-1, 1, 5],
[0, 1, 7],
[1, 1, 5],
[2, 1, 3],
[-2, 2, 1],
[-1, 2, 3],
[0, 2, 5],
[1, 2, 3],
[2, 2, 1],
],
divisor: 48,
},
stucki: {
kernel: [
[1, 0, 8],
[2, 0, 4],
[-2, 1, 2],
[-1, 1, 4],
[0, 1, 8],
[1, 1, 4],
[2, 1, 2],
[-2, 2, 1],
[-1, 2, 2],
[0, 2, 4],
[1, 2, 2],
[2, 2, 1],
],
divisor: 42,
},
burkes: {
kernel: [
[1, 0, 8],
[2, 0, 4],
[-2, 1, 2],
[-1, 1, 4],
[0, 1, 8],
[1, 1, 4],
[2, 1, 2],
],
divisor: 32,
},
sierra: {
kernel: [
[1, 0, 5],
[2, 0, 3],
[-2, 1, 2],
[-1, 1, 4],
[0, 1, 5],
[1, 1, 4],
[2, 1, 2],
[-1, 2, 2],
[0, 2, 3],
[1, 2, 2],
],
divisor: 32,
},
"sierra-lite": {
kernel: [
[1, 0, 2],
[-1, 1, 1],
[0, 1, 1],
],
divisor: 4,
},
"two-row-sierra": {
kernel: [
[1, 0, 4],
[2, 0, 3],
[-2, 1, 1],
[-1, 1, 2],
[0, 1, 3],
[1, 1, 2],
[2, 1, 1],
],
divisor: 16,
},
};
const DEFAULTS = {
algorithm: "floyd-steinberg",
brightness: 1,
contrast: 1.2,
detail: 1,
palette: ["#000000", "#ffffff"],
pointSize: 3,
};
export function errorDiffusionBufferLength(width, height, pointSize) {
return Math.ceil(width / pointSize) * Math.ceil(height / pointSize) * 3;
}
export function applyErrorDiffusionRgba(data, width, height, options = {}, errorBuffer) {
if (!Number.isInteger(width) || width < 1 || !Number.isInteger(height) || height < 1) {
throw new Error("width and height must be positive integers");
}
if (!data || data.length !== width * height * 4) {
throw new Error(`RGBA data must contain ${width * height * 4} bytes`);
}
const algorithm = options.algorithm ?? DEFAULTS.algorithm;
const diffusion = ERROR_DIFFUSION_ALGORITHMS[algorithm];
if (!diffusion) throw new Error(`unknown error-diffusion algorithm: ${algorithm}`);
const pointSize = integerInRange(options.pointSize ?? DEFAULTS.pointSize, 1, 20, "pointSize");
const brightness = numberInRange(options.brightness ?? DEFAULTS.brightness, 0.5, 2, "brightness");
const contrast = numberInRange(options.contrast ?? DEFAULTS.contrast, 0.5, 2, "contrast");
const detail = numberInRange(options.detail ?? DEFAULTS.detail, 0.1, 1, "detail");
const palette = parsePalette(options.palette ?? DEFAULTS.palette);
const blockColumns = Math.ceil(width / pointSize);
const blockRows = Math.ceil(height / pointSize);
const errorLength = errorDiffusionBufferLength(width, height, pointSize);
const errors = errorBuffer ?? new Float32Array(errorLength);
if (!(errors instanceof Float32Array) || errors.length !== errorLength) {
throw new Error(`errorBuffer must be a Float32Array of length ${errorLength}`);
}
errors.fill(0);
const centerOffset = Math.floor(pointSize / 2);
for (let blockRow = 0; blockRow < blockRows; blockRow++) {
const blockY = blockRow * pointSize;
for (let blockColumn = 0; blockColumn < blockColumns; blockColumn++) {
const blockX = blockColumn * pointSize;
const centerX = Math.min(blockX + centerOffset, width - 1);
const centerY = Math.min(blockY + centerOffset, height - 1);
const rgbaIndex = (centerY * width + centerX) * 4;
const errorIndex = (blockRow * blockColumns + blockColumn) * 3;
const red = correctedChannel(data[rgbaIndex], errors[errorIndex], brightness, contrast);
const green = correctedChannel(
data[rgbaIndex + 1],
errors[errorIndex + 1],
brightness,
contrast,
);
const blue = correctedChannel(
data[rgbaIndex + 2],
errors[errorIndex + 2],
brightness,
contrast,
);
const luminance = 0.299 * red + 0.587 * green + 0.114 * blue;
const output = palette[Math.min(palette.length - 1, Math.floor(luminance * palette.length))];
for (let y = blockY; y < Math.min(blockY + pointSize, height); y++) {
for (let x = blockX; x < Math.min(blockX + pointSize, width); x++) {
const outputIndex = (y * width + x) * 4;
data[outputIndex] = Math.round(output[0] * 255);
data[outputIndex + 1] = Math.round(output[1] * 255);
data[outputIndex + 2] = Math.round(output[2] * 255);
}
}
for (const [dx, dy, weight] of diffusion.kernel) {
const targetColumn = blockColumn + dx;
const targetRow = blockRow + dy;
if (
targetColumn < 0 ||
targetColumn >= blockColumns ||
targetRow < 0 ||
targetRow >= blockRows
) {
continue;
}
const target = (targetRow * blockColumns + targetColumn) * 3;
const scale = (weight / diffusion.divisor) * detail;
errors[target] += (red - output[0]) * scale;
errors[target + 1] += (green - output[1]) * scale;
errors[target + 2] += (blue - output[2]) * scale;
}
}
}
return data;
}
function correctedChannel(byte, error, brightness, contrast) {
return Math.min(1, Math.max(0, ((byte / 255 - 0.5) * contrast + 0.5) * brightness + error));
}
function parsePalette(colors) {
if (!Array.isArray(colors) || colors.length < 2 || colors.length > 6) {
throw new Error("palette must contain 2 to 6 colors");
}
return colors.map((color) => {
const match = /^#([0-9a-f]{6})$/i.exec(color);
if (!match) throw new Error(`palette color must use #rrggbb: ${color}`);
const value = Number.parseInt(match[1], 16);
return [(value >> 16) / 255, ((value >> 8) & 255) / 255, (value & 255) / 255];
});
}
function numberInRange(value, min, max, name) {
const number = Number(value);
if (!Number.isFinite(number) || number < min || number > max) {
throw new Error(`${name} must be between ${min} and ${max}`);
}
return number;
}
function integerInRange(value, min, max, name) {
const number = Number(value);
if (!Number.isInteger(number) || number < min || number > max) {
throw new Error(`${name} must be an integer between ${min} and ${max}`);
}
return number;
}
scripts/lib/index-gen.mjs›
import { writeFileSync, mkdirSync } from "node:fs";
import { dirname } from "node:path";
import { readManifest, indexPath } from "./manifest.mjs";
function pad(str, len) {
return String(str ?? "").padEnd(len);
}
function formatDur(record) {
if (record.duration == null) return "—";
return `${record.duration}s`;
}
function formatDims(record) {
if (record.width && record.height) return `${record.width}×${record.height}`;
if (record.type === "icon" && record.transparent) return "svg";
return "—";
}
export function generateIndexContent(records) {
const count = records.length;
const header = `# .media · ${count} asset${count === 1 ? "" : "s"}\n`;
if (count === 0) return header;
const cols = { id: 4, type: 5, dur: 4, dims: 5, path: 5, desc: 11 };
for (const r of records) {
cols.id = Math.max(cols.id, (r.id ?? "").length);
cols.type = Math.max(cols.type, (r.type ?? "").length);
cols.dur = Math.max(cols.dur, formatDur(r).length);
cols.dims = Math.max(cols.dims, formatDims(r).length);
cols.path = Math.max(cols.path, (r.path ?? "").length);
}
const heading =
pad("id", cols.id + 2) +
pad("type", cols.type + 2) +
pad("dur", cols.dur + 2) +
pad("dims", cols.dims + 2) +
pad("path", cols.path + 2) +
"description";
const lines = [header, heading];
for (const r of records) {
lines.push(
pad(r.id, cols.id + 2) +
pad(r.type, cols.type + 2) +
pad(formatDur(r), cols.dur + 2) +
pad(formatDims(r), cols.dims + 2) +
pad(r.path, cols.path + 2) +
(r.description ?? ""),
);
}
return lines.join("\n") + "\n";
}
export function regenerateIndex(projectDir) {
const records = readManifest(projectDir);
const content = generateIndexContent(records);
const p = indexPath(projectDir);
mkdirSync(dirname(p), { recursive: true });
writeFileSync(p, content);
return content;
}
scripts/lib/manifest.mjs›
import {
readFileSync,
appendFileSync,
mkdirSync,
existsSync,
readdirSync,
openSync,
closeSync,
writeFileSync,
rmSync,
statSync,
} from "node:fs";
import { join } from "node:path";
const MANIFEST_FILE = "manifest.jsonl";
const INDEX_FILE = "index.md";
const TYPE_DIRS = {
bgm: "audio/bgm",
sfx: "audio/sfx",
voice: "audio/voice",
image: "images",
icon: "images",
logo: "images",
brand: "images",
video: "video",
grade: "luts",
lut: "luts",
recipe: "recipes",
};
export function mediaDir(projectDir) {
return join(projectDir, ".media");
}
export function manifestPath(projectDir) {
return join(mediaDir(projectDir), MANIFEST_FILE);
}
export function indexPath(projectDir) {
return join(mediaDir(projectDir), INDEX_FILE);
}
export function typeSubdir(type) {
const sub = TYPE_DIRS[type];
if (!sub) throw new Error(`unknown media type: ${type}`);
return sub;
}
export function typeDirPath(projectDir, type) {
return join(mediaDir(projectDir), typeSubdir(type));
}
export function readManifest(projectDir) {
const p = manifestPath(projectDir);
if (!existsSync(p)) return [];
const raw = readFileSync(p, "utf8");
const records = [];
for (const line of raw.split(/\r?\n/)) {
const trimmed = line.trim();
if (!trimmed) continue;
try {
records.push(JSON.parse(trimmed));
} catch {
// ponytail: skip malformed lines, don't crash
}
}
return records;
}
export function appendRecord(projectDir, record) {
const dir = mediaDir(projectDir);
mkdirSync(dir, { recursive: true });
const typeDir = typeDirPath(projectDir, record.type);
mkdirSync(typeDir, { recursive: true });
const p = manifestPath(projectDir);
const line = JSON.stringify(record) + "\n";
appendFileSync(p, line);
}
// Match prompts forgivingly. Agents rarely re-emit a byte-identical intent, so
// keying cache lookups on exact equality meant "Calm piano" and "calm piano"
// re-searched and re-downloaded. Normalize (trim, lowercase, collapse internal
// whitespace) on both sides; the raw prompt is still stored for audit.
export function normalizePrompt(prompt) {
return String(prompt ?? "")
.trim()
.toLowerCase()
.replace(/\s+/g, " ");
}
export function findByPrompt(projectDir, prompt, type) {
const key = normalizePrompt(prompt);
if (!key) return null;
const records = readManifest(projectDir);
return (
records.find(
(r) => normalizePrompt(r.provenance?.prompt) === key && (type == null || r.type === type),
) || null
);
}
export function findByEntity(projectDir, entity) {
const lower = entity.toLowerCase();
const records = readManifest(projectDir);
return records.find((r) => r.entity && r.entity.toLowerCase() === lower) || null;
}
export function nextId(projectDir, type) {
const records = readManifest(projectDir);
const prefix = type;
let max = 0;
for (const r of records) {
if (r.type !== type) continue;
const m = r.id?.match(new RegExp(`^${prefix}_(\\d+)$`));
if (m) max = Math.max(max, parseInt(m[1], 10));
}
return `${prefix}_${String(max + 1).padStart(3, "0")}`;
}
// Sync sleep (no busy-spin) for the allocation lock retry.
function sleepMs(ms) {
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
}
// Coarse per-project lock so concurrent resolves don't race on id allocation.
// ponytail: one lock file with a 15s stale-steal (a crashed holder can't wedge
// the project); fine for agent-scale concurrency — revisit if throughput needs
// finer locking. Date.now() is available here (a normal Node CLI, not a
// workflow DSL), so mtime-based staleness is safe.
const LOCK_STALE_MS = 15000;
const LOCK_TIMEOUT_MS = 20000;
function withLock(dir, fn) {
const lock = join(dir, ".lock");
const start = Date.now();
for (;;) {
try {
closeSync(openSync(lock, "wx")); // O_EXCL: atomic acquire
break;
} catch (err) {
if (err.code !== "EEXIST") throw err;
try {
if (Date.now() - statSync(lock).mtimeMs > LOCK_STALE_MS) {
rmSync(lock, { force: true }); // steal a stale lock from a dead holder
continue;
}
} catch {
continue; // lock vanished between check and stat — retry the acquire
}
if (Date.now() - start > LOCK_TIMEOUT_MS) {
throw new Error("media-use: timed out acquiring .media/.lock");
}
sleepMs(25);
}
}
try {
return fn();
} finally {
rmSync(lock, { force: true });
}
}
// Atomically allocate the next free id for `type` AND reserve its file, so a
// slow download/copy between allocation and appendRecord can't let a concurrent
// caller grab the same id (the MU-23 clobber). Under the lock we take the max id
// across BOTH the manifest and any already-reserved files in the type dir, then
// O_EXCL-create an empty placeholder at the target path; freeze/copy overwrites
// it. Returns { id, localPath }.
export function allocateId(projectDir, type, ext) {
mkdirSync(mediaDir(projectDir), { recursive: true });
const typeDir = typeDirPath(projectDir, type);
mkdirSync(typeDir, { recursive: true });
return withLock(mediaDir(projectDir), () => {
const re = new RegExp(`^${type}_(\\d+)`);
let max = 0;
for (const r of readManifest(projectDir)) {
if (r.type !== type) continue;
const m = r.id?.match(re);
if (m) max = Math.max(max, parseInt(m[1], 10));
}
for (const f of readdirSync(typeDir)) {
const m = f.match(re);
if (m) max = Math.max(max, parseInt(m[1], 10)); // skip ids reserved but not yet appended
}
const id = `${type}_${String(max + 1).padStart(3, "0")}`;
const localPath = `.media/${typeSubdir(type)}/${id}${ext}`;
writeFileSync(join(projectDir, localPath), "", { flag: "wx" }); // durable reservation
return { id, localPath };
});
}
function reservedFile(projectDir, type, ext) {
const allocation = allocateId(projectDir, type, ext);
return { ...allocation, fullPath: join(projectDir, allocation.localPath) };
}
function rollbackReservation(reservation) {
rmSync(reservation.fullPath, { force: true });
}
// A reservation is committed only when populate returns a non-null value.
// Throwing/rejecting or returning null means no usable asset was produced, so
// the placeholder must be released. Keeping this transaction beside allocateId
// prevents individual provider/cache/LUT paths from forgetting the rollback.
export function withReservedFileSync(projectDir, type, ext, populate) {
const reservation = reservedFile(projectDir, type, ext);
try {
const result = populate(reservation);
if (result == null) rollbackReservation(reservation);
return result;
} catch (error) {
rollbackReservation(reservation);
throw error;
}
}
export async function withReservedFile(projectDir, type, ext, populate) {
const reservation = reservedFile(projectDir, type, ext);
try {
const result = await populate(reservation);
if (result == null) rollbackReservation(reservation);
return result;
} catch (error) {
rollbackReservation(reservation);
throw error;
}
}
scripts/lib/media-fetch.mjs›
// Compatibility copy for the audio helpers that still import the historical
// skill-relative path. The CLI-owned engine is the source of truth.
export {
fetchMedia,
isPublicMediaUrl,
} from "../../../../packages/cli/src/media-use/lib/media-fetch.mjs";
scripts/lib/npx-sync.mjs›
import { existsSync } from "node:fs";
import { dirname, join } from "node:path";
function resolveNpxCliPath(env, pathExists) {
const npmExecPath = env.npm_execpath;
const nodeExecPath = env.npm_node_execpath || process.execPath;
if (npmExecPath) {
const fileName = npmExecPath.replace(/\\/g, "/").split("/").pop()?.toLowerCase();
const candidate =
fileName === "npx-cli.js" ? npmExecPath : join(dirname(npmExecPath), "npx-cli.js");
if (pathExists(candidate)) return candidate;
}
const besideNode = join(dirname(nodeExecPath), "node_modules", "npm", "bin", "npx-cli.js");
return pathExists(besideNode) ? besideNode : null;
}
// Sync-spawn analog of the audio engine's spawnP, for execFileSync call sites
// that must hard-fail (rather than fall through to another provider) when npx
// cannot be resolved. On Windows a bare "npx" is npx.cmd, which
// execFileSync/spawnSync cannot exec (spawnSync npx ENOENT) —
// resolveSpawnCommand reroutes it through node + npx-cli.js, no shell:true.
//
// `platform`/`env`/`pathExists` params (defaulting to the real values) exist
// so tests can exercise the win32 branch without mocking node:child_process
// (its ESM exports are non-configurable) — same idiom as spawnP and
// localTtsGenerate.
export function resolveNpxInvocation(
argv,
opts,
platform = process.platform,
env = process.env,
pathExists = existsSync,
) {
const resolved =
platform !== "win32"
? { cmd: "npx", args: argv, opts: { stdio: "ignore", ...opts } }
: (() => {
const nodeExecPath = env.npm_node_execpath || process.execPath;
const npxCliPath = resolveNpxCliPath(env, pathExists);
return npxCliPath
? {
cmd: nodeExecPath,
args: [npxCliPath, ...argv.map((arg) => String(arg))],
opts: { stdio: "ignore", windowsHide: true, ...opts },
}
: null;
})();
if (!resolved) {
// npx-on-win32 with no resolvable npx-cli.js — same terminal condition
// spawnP warns about, surfaced as a throw for callers with no fallback.
throw new Error(
"cannot run npx on Windows: npm's npx-cli.js was not found " +
"(install npm with Node, or run via npx/npm run so npm_execpath is set)",
);
}
return resolved;
}
scripts/lib/parakeet-words.mjs›
// Merge Parakeet-MLX token timestamps into word timestamps.
//
// parakeet-mlx JSON emits SUB-WORD tokens (" H", "ello", ...) with per-token
// start/end. Captions + transcript-cut need WORD timestamps, so join tokens
// into words on the space boundary: a token whose text starts with a space
// (or the very first token) begins a new word; the rest append. Output matches
// the { words: [{ text, start, end }] } shape the rest of media-use consumes
// (see words.mjs / cutlist.mjs).
export function mergeTokensToWords(parakeet) {
const sentences = Array.isArray(parakeet?.sentences) ? parakeet.sentences : [];
const words = [];
for (const s of sentences) {
for (const t of s.tokens ?? []) {
const raw = typeof t.text === "string" ? t.text : "";
const startsWord = raw.startsWith(" ") || words.length === 0;
if (startsWord) {
words.push({ text: raw.trim(), start: t.start, end: t.end });
} else {
const w = words[words.length - 1];
w.text += raw;
w.end = t.end;
}
}
}
return { text: (parakeet?.text ?? "").trim(), words: words.filter((w) => w.text.length > 0) };
}
scripts/lib/prefs-store.mjs›
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { basename, dirname, join, resolve } from "node:path";
/**
* Remembered defaults — the lightweight tier of HyperFrames user memory.
*
* Two files, same shape as the rest of media-use's storage split:
* - project `.media/preferences.json` — committed with the repo, so the whole
* team inherits it; written every time a brief answer is confirmed.
* - user `~/.media/preferences.json` — personal, cross-repo. A key is promoted
* here only once the same value has been confirmed in two different projects
* (`PROMOTE_AT`), so a one-off choice never pollutes the global defaults.
* Pre-promotion evidence accumulates in the user file's `sightings` ledger —
* project files can't see each other, so the cross-project count has to live
* user-side.
*
* Consumption contract (brief-contract § 2, Remembered defaults): a remembered
* value becomes the recommended option with a receipt naming its source — it
* never skips a question, and explicit request content always wins.
*/
const PREFS_FILE = "preferences.json";
/** Keys the brief contract records; `style_preset` is stored per workflow. */
export const PREFERENCE_KEYS = [
"destination",
"aspect",
"language",
"flow",
"storyboard",
"voice",
"style_preset",
];
/** A value must be confirmed in this many distinct projects to go user-tier. */
export const PROMOTE_AT = 2;
export function projectPrefsPath(projectDir) {
return join(resolve(projectDir), ".media", PREFS_FILE);
}
export function userPrefsPath() {
return join(homedir(), ".media", PREFS_FILE);
}
function emptyFile() {
return { version: 1, preferences: {}, sightings: {} };
}
function isRecord(value) {
return typeof value === "object" && value !== null && !Array.isArray(value);
}
/** Tolerant read — a missing or malformed file counts as empty. */
function readPrefsFile(path) {
try {
if (!existsSync(path)) return emptyFile();
const parsed = JSON.parse(readFileSync(path, "utf8"));
if (!isRecord(parsed)) return emptyFile();
return {
version: 1,
preferences: isRecord(parsed.preferences) ? parsed.preferences : {},
sightings: isRecord(parsed.sightings) ? parsed.sightings : {},
};
} catch {
return emptyFile();
}
}
/** Atomic write (tmp + rename) so a crash never leaves a torn file. */
function writePrefsFile(path, file) {
mkdirSync(dirname(path), { recursive: true });
const tmp = `${path}.tmp`;
writeFileSync(tmp, `${JSON.stringify(file, null, 2)}\n`);
renameSync(tmp, path);
}
/** `style_preset` entries are stored per workflow as `style_preset.<workflow>`. */
export function preferenceKeyFor(key, workflow) {
return key === "style_preset" && workflow ? `style_preset.${workflow}` : key;
}
function validEntry(entry) {
return isRecord(entry) && typeof entry.value === "string" && entry.value.length > 0;
}
/**
* The merged view the brief reads: user-tier promoted entries first, project
* entries on top (project wins). Each entry carries `source` plus the receipt
* material (`confirmed_in`, `updated_at`).
*/
export function mergedPreferences(projectDir) {
const user = readPrefsFile(userPrefsPath());
const project = readPrefsFile(projectPrefsPath(projectDir));
const merged = {};
for (const [key, entry] of Object.entries(user.preferences)) {
if (validEntry(entry)) merged[key] = { ...entry, source: "user" };
}
for (const [key, entry] of Object.entries(project.preferences)) {
if (validEntry(entry)) merged[key] = { ...entry, source: "project" };
}
return merged;
}
function dedupe(list) {
return [...new Set(list)];
}
/**
* Project tier: same value accumulates confirmations; a changed value starts
* provenance over (the old confirmations vouched for the old value).
*/
function recordProjectTier(projectDir, fullKey, value, projectName, now) {
const path = projectPrefsPath(projectDir);
const file = readPrefsFile(path);
const previous = file.preferences[fullKey];
const keepProvenance = validEntry(previous) && previous.value === value;
const confirmedIn = keepProvenance
? dedupe([...(Array.isArray(previous.confirmed_in) ? previous.confirmed_in : []), projectName])
: [projectName];
file.preferences[fullKey] = { value, confirmed_in: confirmedIn, updated_at: now };
writePrefsFile(path, file);
return confirmedIn;
}
/**
* User tier: accumulate this sighting in the ledger, and promote the key once
* the same value has been confirmed in PROMOTE_AT distinct projects.
*/
function recordUserSighting(fullKey, value, projectName, now) {
const path = userPrefsPath();
const file = readPrefsFile(path);
const keySightings = isRecord(file.sightings[fullKey]) ? file.sightings[fullKey] : {};
const seenIn = dedupe([
...(Array.isArray(keySightings[value]) ? keySightings[value] : []),
projectName,
]);
keySightings[value] = seenIn;
file.sightings[fullKey] = keySightings;
const promoted = seenIn.length >= PROMOTE_AT;
if (promoted) {
file.preferences[fullKey] = { value, confirmed_in: seenIn, updated_at: now };
}
writePrefsFile(path, file);
return promoted;
}
/**
* Record one confirmed brief answer. Always writes the project tier; feeds the
* user tier's sightings ledger and promotes once the same value has been
* confirmed in PROMOTE_AT distinct projects. Idempotent per project.
*/
export function recordPreference({ projectDir, key, value, workflow }) {
if (!PREFERENCE_KEYS.includes(key)) {
throw new Error(`unknown preference key: "${key}" (known: ${PREFERENCE_KEYS.join(", ")})`);
}
if (typeof value !== "string" || !value.trim()) {
throw new Error("a preference needs a non-empty string value");
}
if (key === "style_preset" && (!workflow || !String(workflow).trim())) {
throw new Error("style_preset is stored per workflow — pass --workflow <w>");
}
const fullKey = preferenceKeyFor(key, workflow);
const projectName = basename(resolve(projectDir));
const trimmed = value.trim();
const now = new Date().toISOString();
const confirmedIn = recordProjectTier(projectDir, fullKey, trimmed, projectName, now);
// Best-effort — a read-only home directory must never fail a brief.
let promoted = false;
try {
promoted = recordUserSighting(fullKey, trimmed, projectName, now);
} catch {
// The project record already landed; promotion just waits for next time.
}
return { key: fullKey, value: trimmed, confirmed_in: confirmedIn, promoted };
}
scripts/lib/recipe-store.mjs›
import {
cpSync,
existsSync,
mkdirSync,
readdirSync,
readFileSync,
renameSync,
rmSync,
writeFileSync,
} from "node:fs";
import { homedir } from "node:os";
import { basename, join, resolve } from "node:path";
import { appendRecord, mediaDir, nextId } from "./manifest.mjs";
import { regenerateIndex } from "./index-gen.mjs";
import { mergedPreferences } from "./prefs-store.mjs";
/**
* Recipes — the heavyweight tier of HyperFrames user memory.
*
* A recipe is the full confirmed bundle for one video type: the frozen design
* spec (`frame.md`), the storyboard skeleton (structure with the content
* blanked), and the confirmed brief values — frozen after the run's final
* approval, reused to start the next video of the same type from everything
* already approved.
*
* Storage is **named folders**, not content-addressed cache entries: a recipe
* is an evolving bundle with a `version`, so re-freezing the same name bumps
* the version and archives the old folder as `<name>@v<N>`. Two tiers, same
* split as everything else in media-use: project `.media/recipes/<name>/`
* (committed) and user `~/.media/recipes/<name>/` (a freeze is already a
* confirmed bundle, so it promotes immediately — no two-project rule here).
*/
/** Frontmatter keys that describe THIS video, not the reusable type. */
const FRONTMATTER_CONTENT_KEYS = new Set(["message", "audience", "mode"]);
/** BRIEF.md frontmatter keys that describe this run, not the reusable type —
* a recipe never locks the run's shape, so the intent layer always re-asks. */
const BRIEF_CONTENT_KEYS = new Set(["flow", "storyboard", "message", "audience"]);
/** Per-frame metadata that is content, not structure. */
const FRAME_CONTENT_KEYS = new Set([
"voiceover",
"vo",
"voice_over",
"narration",
"scene",
"description",
"summary",
"caption",
"asset_candidates",
]);
const FRAME_HEADING_RE = /^(#{2,3})\s+(?:frame|beat|scene)\s+\d+/i;
export function projectRecipesDir(projectDir) {
return join(mediaDir(projectDir), "recipes");
}
export function userRecipesDir() {
return join(homedir(), ".media", "recipes");
}
export function slugifyRecipeName(name) {
const slug = String(name ?? "")
.trim()
.toLowerCase()
.replace(/[\s_]+/g, "-")
.replace(/[^a-z0-9-]/g, "")
.replace(/-+/g, "-")
.replace(/^-|-$/g, "");
if (!slug) throw new Error(`recipe name "${name}" has no usable characters`);
return slug;
}
function frameTitle(headingLine) {
const dash = headingLine.split(/\s+—\s+/)[1];
if (dash && dash.trim()) return dash.trim();
return headingLine.replace(/^#+\s*/, "").trim();
}
/** Frontmatter: drop the content keys, keep structure/style keys verbatim. */
function skeletonFrontmatter(lines, out, contentKeys = FRONTMATTER_CONTENT_KEYS) {
if (lines[0]?.trim() !== "---") return 0;
out.push(lines[0]);
let i = 1;
while (i < lines.length && lines[i].trim() !== "---") {
const key = lines[i].match(/^(\w+)\s*:/)?.[1]?.toLowerCase();
if (!key || !contentKeys.has(key)) out.push(lines[i]);
i++;
}
if (i < lines.length) {
out.push(lines[i]); // closing ---
i++;
}
return i;
}
/** One line inside a frame section — returns the replacement lines (may be none). */
function skeletonFrameLine(line, state, out) {
const bulletKey = line.match(/^-\s+(\w+)\s*:/)?.[1]?.toLowerCase();
if (bulletKey) {
if (bulletKey === "status") out.push("- status: outline");
else if (!FRAME_CONTENT_KEYS.has(bulletKey)) out.push(line);
return;
}
if (!line.trim()) {
out.push(line);
return;
}
// Frame prose: one placeholder per frame in place of the narrative.
if (!state.proseReplaced) {
out.push(
`<fill in: this video's content for the "${state.title}" beat — keep the layout role, replace the words.>`,
);
state.proseReplaced = true;
}
}
/**
* Skeletonize a STORYBOARD.md: keep the reusable structure (frame count,
* durations, transitions, src paths, the Video direction block, style-ish
* frontmatter), reset every status to `outline`, and blank the content
* (message/audience, narration guides, per-frame prose) down to a fill-in
* placeholder that names the frame's role.
*/
/**
* Skeletonize a BRIEF.md: keep the frontmatter's reusable keys (workflow,
* destination, aspect, language, length, angle…), drop the run-shape and
* content keys (flow, storyboard, message, audience), and blank each body
* section down to a fill-in placeholder under its kept heading.
*/
export function skeletonizeBrief(source) {
const lines = String(source ?? "").split(/\r?\n/);
const out = [];
let i = skeletonFrontmatter(lines, out, BRIEF_CONTENT_KEYS);
for (; i < lines.length; i++) {
const heading = lines[i].match(/^##\s+(.+)$/);
if (heading) {
out.push(lines[i], "");
out.push(
`<fill in: this video's ${heading[1].trim().toLowerCase()} — the recipe keeps the shape, this run supplies the specifics.>`,
);
out.push("");
}
}
return out.join("\n").replace(/\n{3,}/g, "\n\n");
}
export function skeletonizeStoryboard(source) {
const lines = String(source ?? "").split(/\r?\n/);
const out = [];
const state = { inFrame: false, proseReplaced: false, title: "" };
for (let i = skeletonFrontmatter(lines, out); i < lines.length; i++) {
const line = lines[i];
if (/^#{2,3}\s/.test(line)) {
state.inFrame = FRAME_HEADING_RE.test(line);
state.proseReplaced = false;
state.title = state.inFrame ? frameTitle(line) : "";
out.push(line);
} else if (!state.inFrame) {
out.push(line);
} else {
skeletonFrameLine(line, state, out);
}
}
return out.join("\n").replace(/\n{3,}/g, "\n\n");
}
/** The run's workflow as BRIEF.md records it — the source of truth a freeze
* must not contradict. Undefined when no BRIEF.md (or no `workflow:`) exists. */
function briefWorkflow(root) {
const brief = join(root, "BRIEF.md");
if (!existsSync(brief)) return undefined;
const lines = readFileSync(brief, "utf8").split(/\r?\n/);
if (lines[0]?.trim() !== "---") return undefined;
for (let i = 1; i < lines.length && lines[i].trim() !== "---"; i++) {
const match = lines[i].match(/^workflow\s*:\s*(.+?)\s*$/);
if (match) return match[1].replace(/^["']|["']$/g, "") || undefined;
}
return undefined;
}
function readRecipeJson(dir) {
try {
const parsed = JSON.parse(readFileSync(join(dir, "recipe.json"), "utf8"));
if (typeof parsed !== "object" || parsed === null || typeof parsed.name !== "string") {
return null;
}
return parsed;
} catch {
return null;
}
}
function prefValue(prefs, key) {
return prefs[key]?.value;
}
/**
* Freeze the current project's approved run as a named recipe. Writes the
* project-tier folder + a manifest record, then copies to the user tier (a
* freeze is already confirmed — it promotes immediately).
*/
export function freezeRecipe({ projectDir, name, workflow, blocks }) {
const slug = slugifyRecipeName(name);
const root = resolve(projectDir);
const fromBrief = briefWorkflow(root);
const fromFlag = workflow && String(workflow).trim() ? String(workflow).trim() : undefined;
// BRIEF.md decides; the flag only covers projects briefed before it existed.
const resolvedWorkflow = fromBrief ?? fromFlag;
if (!resolvedWorkflow) {
throw new Error("no workflow found — BRIEF.md names none and no --workflow was given");
}
const frameSpec = join(root, "frame.md");
const storyboard = join(root, "STORYBOARD.md");
if (!existsSync(frameSpec)) throw new Error("no frame.md to freeze — run the design step first");
if (!existsSync(storyboard)) throw new Error("no STORYBOARD.md to freeze");
const dir = join(projectRecipesDir(root), slug);
let version = 1;
const previous = existsSync(dir) ? readRecipeJson(dir) : null;
if (previous) {
version = (Number.isInteger(previous.version) ? previous.version : 1) + 1;
const archive = `${dir}@v${previous.version ?? 1}`;
rmSync(archive, { recursive: true, force: true });
renameSync(dir, archive);
}
mkdirSync(dir, { recursive: true });
const prefs = mergedPreferences(root);
const recipe = {
version,
name: slug,
workflow: resolvedWorkflow,
approved_at: new Date().toISOString(),
source_project: basename(root),
destination: prefValue(prefs, "destination"),
aspect: prefValue(prefs, "aspect"),
language: prefValue(prefs, "language"),
voice: prefValue(prefs, "voice"),
// The bare-key fallback tolerates records made before the store required
// style_preset to be workflow-scoped.
style_preset:
prefValue(prefs, `style_preset.${resolvedWorkflow}`) ?? prefValue(prefs, "style_preset"),
blocks: Array.isArray(blocks) && blocks.length > 0 ? blocks : undefined,
};
writeFileSync(join(dir, "recipe.json"), `${JSON.stringify(recipe, null, 2)}\n`);
cpSync(frameSpec, join(dir, "frame.md"));
writeFileSync(
join(dir, "storyboard-skeleton.md"),
`${skeletonizeStoryboard(readFileSync(storyboard, "utf8")).trimEnd()}\n`,
);
// Best-effort fourth artifact — projects briefed before BRIEF.md existed
// (or by workflows that don't write one) freeze fine without it.
const brief = join(root, "BRIEF.md");
const briefSkeleton = existsSync(brief);
if (briefSkeleton) {
writeFileSync(
join(dir, "brief-skeleton.md"),
`${skeletonizeBrief(readFileSync(brief, "utf8")).trimEnd()}\n`,
);
}
const id = nextId(root, "recipe");
appendRecord(root, {
id,
type: "recipe",
path: `.media/recipes/${slug}/recipe.json`,
entity: slug,
description: `recipe: ${slug} (${recipe.workflow}, v${version})`,
provenance: { provider: "recipe.freeze", version, source_project: recipe.source_project },
});
regenerateIndex(root);
// User tier — best-effort, like every other promotion.
try {
const userDir = join(userRecipesDir(), slug);
mkdirSync(userDir, { recursive: true });
cpSync(dir, userDir, { recursive: true, force: true });
} catch {
// The project-tier freeze already landed.
}
return {
id,
slug,
version,
dir,
briefSkeleton,
workflow: resolvedWorkflow,
workflowOverridden: Boolean(fromBrief && fromFlag && fromBrief !== fromFlag),
};
}
function scanRecipesDir(dir, source) {
if (!existsSync(dir)) return [];
const found = [];
for (const entry of readdirSync(dir, { withFileTypes: true })) {
if (!entry.isDirectory() || entry.name.includes("@v")) continue;
const recipe = readRecipeJson(join(dir, entry.name));
if (recipe) found.push({ ...recipe, source, dir: join(dir, entry.name) });
}
return found;
}
/** Two-tier merged listing (project wins), newest approval first. */
export function listRecipes({ projectDir, workflow }) {
const merged = new Map();
for (const recipe of scanRecipesDir(userRecipesDir(), "user")) merged.set(recipe.name, recipe);
for (const recipe of scanRecipesDir(projectRecipesDir(resolve(projectDir)), "project")) {
merged.set(recipe.name, recipe);
}
let list = [...merged.values()];
if (workflow) list = list.filter((r) => r.workflow === workflow);
return list.sort((a, b) =>
String(b.approved_at ?? "").localeCompare(String(a.approved_at ?? "")),
);
}
/**
* Adopt a recipe into the current project: import the folder from the user
* tier when the project doesn't have it, copy its frame.md over the project's,
* and hand back the values + the skeleton path for the storyboard draft.
*/
export function useRecipe({ projectDir, name }) {
const slug = slugifyRecipeName(name);
const root = resolve(projectDir);
let dir = join(projectRecipesDir(root), slug);
if (!readRecipeJson(dir)) {
const userDir = join(userRecipesDir(), slug);
if (!readRecipeJson(userDir)) {
const known = listRecipes({ projectDir: root }).map((r) => r.name);
throw new Error(
`no recipe named "${slug}"${known.length ? ` (known: ${known.join(", ")})` : ""}`,
);
}
mkdirSync(dir, { recursive: true });
cpSync(userDir, dir, { recursive: true, force: true });
const imported = readRecipeJson(dir);
appendRecord(root, {
id: nextId(root, "recipe"),
type: "recipe",
path: `.media/recipes/${slug}/recipe.json`,
entity: slug,
description: `recipe: ${slug} (${imported.workflow}, v${imported.version})`,
provenance: { provider: "recipe.local", imported_from: "user-tier" },
});
regenerateIndex(root);
}
const recipe = readRecipeJson(dir);
cpSync(join(dir, "frame.md"), join(root, "frame.md"));
return {
recipe,
dir,
frameSpecPath: "frame.md",
skeletonPath: `.media/recipes/${slug}/storyboard-skeleton.md`,
// Recipes frozen before BRIEF.md existed have no brief skeleton — degrade.
briefSkeletonPath: existsSync(join(dir, "brief-skeleton.md"))
? `.media/recipes/${slug}/brief-skeleton.md`
: undefined,
};
}
scripts/lib/telemetry.mjs›
// Usage tracking shares the CLI and Studio identity. Properties stay coarse and
// never carry intent text, file names, or paths.
import { randomUUID } from "node:crypto";
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { join } from "node:path";
const POSTHOG_API_KEY = "phc_zjjbX0PnWxERXrMHhkEJWj9A9BhGVLRReICgsfTMmpx";
const POSTHOG_HOST = "https://us.i.posthog.com";
const TIMEOUT_MS = 1500;
let identifiedAccount = false;
let warnedNonDefaultHost = false;
function isTestOrCiContext() {
return (
process.env.CI === "true" ||
process.env.CI === "1" ||
process.env.NODE_ENV === "test" ||
process.env.NODE_ENV === "development"
);
}
function posthogHost() {
const override = process.env.MEDIA_USE_TELEMETRY_HOST;
if (override && !warnedNonDefaultHost && !isTestOrCiContext()) {
warnedNonDefaultHost = true;
console.error(
`media-use: telemetry is redirected to a non-default host via MEDIA_USE_TELEMETRY_HOST (${override}) — unset it unless this is intentional.`,
);
}
return override || POSTHOG_HOST;
}
/** True when telemetry must NOT be sent (opt-out envs, CI, dev). */
export function optedOut() {
return (
process.env.HYPERFRAMES_NO_TELEMETRY === "1" ||
process.env.DO_NOT_TRACK === "1" ||
process.env.CI === "true" ||
process.env.CI === "1" ||
process.env.NODE_ENV === "development"
);
}
// Read and write the shared config so media-use keeps one identity per install.
function sharedConfigPath() {
return join(homedir(), ".hyperframes", "config.json");
}
function readSharedConfig() {
try {
const file = sharedConfigPath();
if (existsSync(file)) {
const parsed = JSON.parse(readFileSync(file, "utf8"));
if (parsed && typeof parsed === "object" && !Array.isArray(parsed)) return parsed;
}
} catch {
// unreadable config → treat as empty; never throw
}
return {};
}
function writeSharedConfig(config) {
const dir = join(homedir(), ".hyperframes");
if (!existsSync(dir)) mkdirSync(dir, { recursive: true });
writeFileSync(join(dir, "config.json"), JSON.stringify(config, null, 2) + "\n");
}
// Adopt a pre-existing media-use-only id (~/.media/anon-id from before this
// change) so upgraders keep their PostHog persona instead of resetting to a new
// one — otherwise cross-surface continuity would start over on upgrade.
function legacyMediaAnonId() {
try {
const file = join(homedir(), ".media", "anon-id");
if (existsSync(file)) {
const id = readFileSync(file, "utf8").trim();
if (id) return id;
}
} catch {
// ignore
}
return null;
}
// Stable per-machine id from the shared config; seeds it (adopting a legacy
// media-use id when present) if absent.
function anonymousId() {
try {
const config = readSharedConfig();
if (typeof config.anonymousId === "string" && config.anonymousId.trim()) {
return config.anonymousId.trim();
}
const id = legacyMediaAnonId() || randomUUID();
writeSharedConfig({ ...config, anonymousId: id });
return id;
} catch {
return "anon"; // best-effort; a shared bucket is fine if the fs is read-only
}
}
function heygenAccountDistinctId() {
const file = join(process.env.HEYGEN_CONFIG_DIR || join(homedir(), ".heygen"), "credentials");
try {
if (!existsSync(file)) return null;
const raw = readFileSync(file, "utf8").trim();
if (!raw.startsWith("{")) return null;
const parsed = JSON.parse(raw);
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return null;
const user = parsed.user;
if (!user || typeof user !== "object" || Array.isArray(user)) return null;
const id = typeof user.email === "string" && user.email.trim() ? user.email : user.username;
// Lowercased so this joins with the CLI's own identify call regardless of
// the account's stored email casing — two different-case distinct ids
// would otherwise split one person across two PostHog profiles.
return typeof id === "string" && id.trim() ? id.trim().toLowerCase() : null;
} catch {
return null;
}
}
function showTelemetryNotice() {
if (optedOut()) return;
try {
const config = readSharedConfig();
// Shared with the CLI (config.telemetryNoticeShown): shown once per person
// across surfaces, not once per tool.
if (config.telemetryNoticeShown === true) return;
console.error(
[
"media-use sends usage telemetry: media type, resolution source, and provider; never intent text, file names, or paths.",
"If you sign in to HeyGen, usage links to your account email or username. Opt out with HYPERFRAMES_NO_TELEMETRY=1 or DO_NOT_TRACK=1.",
].join("\n"),
);
writeSharedConfig({ ...config, telemetryNoticeShown: true });
} catch {
// notice is best-effort; never surface into the command
}
}
async function postBatch(batch) {
try {
await fetch(`${posthogHost()}/batch/`, {
method: "POST",
headers: { "Content-Type": "application/json", Connection: "close" },
body: JSON.stringify({ api_key: POSTHOG_API_KEY, batch }),
signal: AbortSignal.timeout(TIMEOUT_MS),
});
} catch {
// telemetry is best-effort; never surface into the command
}
}
async function postEvent(event, properties, distinctId) {
await postBatch([
{
event,
properties: { ...properties, surface: "media-use", $ip: null },
distinct_id: distinctId,
timestamp: new Date().toISOString(),
},
]);
}
async function identifyAccount(anonId) {
if (optedOut() || identifiedAccount) return;
const distinctId = heygenAccountDistinctId();
if (!distinctId) return;
identifiedAccount = true;
await postEvent("$identify", { $anon_distinct_id: anonId }, distinctId);
}
/**
* Fire-and-forget a single event to PostHog. Best-effort: awaited with a short
* timeout so a short-lived script flushes before exit, but any failure (offline,
* opted out) is swallowed. `properties` must be non-PII (no intent/paths).
*/
export async function track(event, properties = {}) {
if (optedOut()) return;
showTelemetryNotice();
const anonId = anonymousId();
await identifyAccount(anonId);
await postEvent(event, properties, anonId);
}
export function __anonymousIdForTest() {
return anonymousId();
}
export function __resetTelemetryForTest() {
identifiedAccount = false;
warnedNonDefaultHost = false;
}
scripts/lib/words.mjs›
export function normalizeWords(input) {
const raw = Array.isArray(input) ? input : Array.isArray(input?.words) ? input.words : [];
return raw
.map((w, index) => {
const text = String(w?.text ?? w?.word ?? "").trim();
const start = Number(w?.start);
const end = Number(w?.end);
if (!text || !Number.isFinite(start) || !Number.isFinite(end)) return null;
return { id: w?.id ?? `w${index}`, text, start, end };
})
.filter(Boolean);
}
export function wordListsFromMediaMeta(input) {
if (Array.isArray(input) || Array.isArray(input?.words)) return [normalizeWords(input)];
if (!Array.isArray(input?.voices)) return [];
return input.voices.map((voice) => normalizeWords(voice)).filter((words) => words.length > 0);
}
scripts/prefs.mjs›
#!/usr/bin/env node
/**
* Remembered defaults CLI — the lightweight tier of HyperFrames user memory.
*
* node prefs.mjs get --hyperframes . [--json]
* Print the merged view (project `.media/preferences.json` over user
* `~/.media/preferences.json`), each key with its source and the receipt
* material (confirmed_in, updated_at).
*
* node prefs.mjs record --hyperframes . --key destination --value x-feed [--workflow <w>]
* Record one confirmed brief answer into the project tier; the same value
* confirmed in two different projects promotes the key to the user tier.
*
* Consumption rules live in hyperframes/references/brief-contract.md § 2
* (Remembered defaults): a remembered value becomes the recommended option
* with a receipt — it never skips a question.
*/
import { parseArgs } from "node:util";
import { mergedPreferences, recordPreference } from "./lib/prefs-store.mjs";
const { values: args, positionals } = parseArgs({
options: {
hyperframes: { type: "string", default: "." },
key: { type: "string" },
value: { type: "string" },
workflow: { type: "string" },
json: { type: "boolean", default: false },
},
allowPositionals: true,
});
const verb = positionals[0];
function fail(message) {
console.error(message);
process.exit(1);
}
if (verb === "get") {
const merged = mergedPreferences(args.hyperframes);
if (args.json) {
console.log(JSON.stringify(merged, null, 2));
} else if (Object.keys(merged).length === 0) {
console.log("no remembered preferences yet");
} else {
for (const [key, entry] of Object.entries(merged)) {
console.log(
`${key} = ${entry.value} (${entry.source}; confirmed in ${entry.confirmed_in.join(", ")})`,
);
}
}
} else if (verb === "record") {
if (!args.key || !args.value) fail("record needs --key and --value");
try {
const result = recordPreference({
projectDir: args.hyperframes,
key: args.key,
value: args.value,
workflow: args.workflow,
});
const promotion = result.promoted ? "; promoted to user tier" : "";
console.log(`recorded ${result.key} = ${result.value} (project${promotion})`);
} catch (err) {
fail(err instanceof Error ? err.message : String(err));
}
} else {
fail(
"usage: prefs.mjs <get|record> --hyperframes . [--key <k> --value <v> --workflow <w>] [--json]",
);
}
scripts/recipe.mjs›
#!/usr/bin/env node
/**
* Recipes CLI — the heavyweight tier of HyperFrames user memory.
*
* node recipe.mjs freeze --hyperframes . --name <n> [--workflow <w>] [--blocks a,b,c]
* Freeze the current approved run as a named recipe: frame.md + the
* storyboard skeleton (structure kept, content blanked) + the brief
* skeleton (when BRIEF.md exists) + the confirmed brief values.
* The workflow is read from BRIEF.md; --workflow only covers projects
* briefed before BRIEF.md existed. Re-freezing the same name bumps the
* version and archives the old folder as <name>@v<N>. Promotes to
* ~/.media/recipes/ immediately.
*
* node recipe.mjs list --hyperframes . [--workflow <w>] [--json]
* Two-tier merged listing (project wins), newest approval first.
*
* node recipe.mjs use --hyperframes . --name <n> [--json]
* Adopt a recipe: import it from the user tier if needed, copy its
* frame.md over the project's, and print the brief values + skeleton path.
*
* When recipes are offered/consumed is the review loop's and the intent
* layer's business — see hyperframes/references/review-loop.md § 4 and
* the intent layer's recipe check (hyperframes SKILL.md § 4).
*/
import { parseArgs } from "node:util";
import { freezeRecipe, listRecipes, useRecipe } from "./lib/recipe-store.mjs";
const { values: args, positionals } = parseArgs({
options: {
hyperframes: { type: "string", default: "." },
name: { type: "string" },
workflow: { type: "string" },
blocks: { type: "string" },
json: { type: "boolean", default: false },
},
allowPositionals: true,
});
const verb = positionals[0];
function fail(message) {
console.error(message);
process.exit(1);
}
try {
if (verb === "freeze") {
if (!args.name) fail("freeze needs --name");
const frozen = freezeRecipe({
projectDir: args.hyperframes,
name: args.name,
workflow: args.workflow,
blocks: args.blocks
? args.blocks
.split(",")
.map((b) => b.trim())
.filter(Boolean)
: undefined,
});
if (args.json) console.log(JSON.stringify({ ok: true, ...frozen }));
else {
console.log(
`froze recipe ${frozen.slug} (v${frozen.version}, ${frozen.workflow}) → ${frozen.dir}`,
);
if (frozen.workflowOverridden)
console.log(` (BRIEF.md says "${frozen.workflow}" — the --workflow flag was ignored)`);
if (!frozen.briefSkeleton)
console.log(" (no BRIEF.md in the project — brief skeleton skipped)");
}
} else if (verb === "list") {
const list = listRecipes({ projectDir: args.hyperframes, workflow: args.workflow });
if (args.json) console.log(JSON.stringify(list));
else if (list.length === 0) console.log("no recipes yet");
else {
for (const r of list) {
console.log(
`${r.name} v${r.version} ${r.workflow} (${r.source}; approved ${String(r.approved_at ?? "?").slice(0, 10)})`,
);
}
}
} else if (verb === "use") {
if (!args.name) fail("use needs --name");
const used = useRecipe({ projectDir: args.hyperframes, name: args.name });
if (args.json) console.log(JSON.stringify({ ok: true, ...used }));
else {
console.log(
`using recipe ${used.recipe.name} (v${used.recipe.version}, ${used.recipe.workflow})`,
);
console.log(` frame spec → ${used.frameSpecPath} (copied over)`);
console.log(` storyboard skeleton → ${used.skeletonPath}`);
if (used.briefSkeletonPath) console.log(` brief skeleton → ${used.briefSkeletonPath}`);
for (const key of ["destination", "aspect", "language", "voice", "style_preset"]) {
if (used.recipe[key]) console.log(` ${key}: ${used.recipe[key]}`);
}
}
} else {
fail(
"usage: recipe.mjs <freeze|list|use> --hyperframes . [--name <n>] [--workflow <w>] [--blocks a,b] [--json]",
);
}
} catch (err) {
fail(err instanceof Error ? err.message : String(err));
}
scripts/resolve.mjs›
#!/usr/bin/env node
import { spawnSync } from "node:child_process";
import { existsSync } from "node:fs";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
const root = join(fileURLToPath(new URL("../../../", import.meta.url)));
const dist = join(root, "packages/cli/dist/cli.js");
const result = existsSync(dist)
? spawnSync(process.execPath, [dist, "media-use", "resolve", ...process.argv.slice(2)], {
stdio: "inherit",
})
: spawnSync(
"bun",
[join(root, "packages/cli/src/cli.ts"), "media-use", "resolve", ...process.argv.slice(2)],
{ stdio: "inherit" },
);
process.exit(result.status ?? 1);
scripts/transcribe.mjs›
#!/usr/bin/env node
import { execFileSync } from "node:child_process";
import {
existsSync,
readFileSync,
writeFileSync,
copyFileSync,
renameSync,
mkdtempSync,
rmSync,
} from "node:fs";
import { homedir, tmpdir } from "node:os";
import { basename, extname, join, resolve } from "node:path";
import { parseArgs } from "node:util";
import { mergeTokensToWords } from "./lib/parakeet-words.mjs";
import { track } from "./lib/telemetry.mjs";
import { resolveNpxInvocation } from "./lib/npx-sync.mjs";
// The DEFAULT local transcription path. Prefers NVIDIA Parakeet-TDT via
// parakeet-mlx, which beats whisper.cpp on the Open ASR Leaderboard (~6.05% vs
// 7.44% avg WER, and 4.73% vs 5.96% on noisy test-other) and is 5-10x faster
// with native punctuation. Emits { text, words:[{text,start,end}] } (word
// timestamps merged from Parakeet's sub-word tokens) for transcript-cut /
// captions / the audio engine.
//
// Parakeet v3 covers English + 25 European languages. For other languages, or
// when parakeet-mlx is not installed, it falls back to whisper.cpp via
// `hyperframes transcribe` (99 languages; the CLI resolves/builds whisper.cpp
// on first use — it is not bundled). `--engine` forces one.
const { values: args } = parseArgs({
options: {
input: { type: "string", short: "i" },
out: { type: "string", short: "o" },
engine: { type: "string", default: "auto" }, // auto | parakeet | whisper
model: { type: "string", default: "mlx-community/parakeet-tdt-0.6b-v3" },
json: { type: "boolean", default: false },
help: { type: "boolean", short: "h", default: false },
},
strict: true,
});
if (args.help) {
console.log(`media-use transcribe: better-than-whisper local ASR (Parakeet), whisper.cpp fallback
Usage:
node transcribe.mjs --input audio.wav [--out audio.transcribe.json] [--engine auto|parakeet|whisper]
Parakeet (default) beats whisper.cpp on accuracy + speed for English/European
languages; whisper.cpp (99 languages) is the fallback. Install Parakeet once:
uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx`);
process.exit(0);
}
if (!args.input) {
console.error("error: --input is required");
process.exit(2);
}
const inputPath = resolve(args.input);
if (!existsSync(inputPath)) {
console.error(`error: input not found: ${inputPath}`);
process.exit(2);
}
const outPath = resolve(
args.out || `${inputPath.slice(0, -extname(inputPath).length)}.transcribe.json`,
);
// Locate the parakeet-mlx runner the same way the CLI does: env override, then
// the documented ~/.venvs/parakeet install, then PATH. Checking the venv (not
// just PATH) is what keeps a user who followed the install docs verbatim from
// silently falling through to whisper. Returns the runner path, or null.
function resolveParakeet() {
for (const p of [
process.env.HYPERFRAMES_PARAKEET,
join(homedir(), ".venvs", "parakeet", "bin", "parakeet-mlx"),
]) {
if (p && existsSync(p)) return p;
}
try {
execFileSync("parakeet-mlx", ["--help"], {
stdio: ["ignore", "ignore", "ignore"],
timeout: 20000,
});
return "parakeet-mlx";
} catch {
return null;
}
}
// Write via a sibling temp + atomic rename so a SIGKILL mid-write can't leave a
// truncated transcript at outPath (downstream reads it as valid JSON).
function atomicWrite(target, data) {
const tmp = `${target}.tmp-${process.pid}`;
writeFileSync(tmp, data);
renameSync(tmp, target);
}
function report(engine, wordCount) {
if (args.json) console.log(JSON.stringify({ ok: true, out: outPath, engine, words: wordCount }));
else
console.log(
`transcribed ${basename(inputPath)} -> ${outPath}${wordCount != null ? ` (${wordCount} words,` : " ("}${engine})`,
);
}
function runParakeet(runner) {
const workDir = mkdtempSync(join(tmpdir(), "media-use-asr-"));
try {
execFileSync(
runner,
[inputPath, "--model", args.model, "--output-format", "json", "--output-dir", workDir],
{ stdio: ["ignore", "pipe", "pipe"], timeout: 1_800_000 },
);
const jsonPath = join(workDir, `${basename(inputPath, extname(inputPath))}.json`);
if (!existsSync(jsonPath)) throw new Error("parakeet produced no JSON");
const merged = mergeTokensToWords(JSON.parse(readFileSync(jsonPath, "utf8")));
atomicWrite(outPath, JSON.stringify(merged, null, 2));
report("parakeet", merged.words.length);
} finally {
rmSync(workDir, { recursive: true, force: true });
}
}
// whisper.cpp via the hyperframes CLI (fetched/built on first use — see
// SKILL.md): writes transcript.json into --dir; relocate to --out.
function runWhisper() {
const workDir = mkdtempSync(join(tmpdir(), "media-use-whisper-"));
try {
// On Windows a bare "npx" is npx.cmd, which execFileSync cannot exec
// (spawnSync npx ENOENT) — resolveNpxInvocation reroutes it through
// node + npx-cli.js (and throws actionably when it can't), same
// mechanism as the audio engine's TTS spawns.
const resolved = resolveNpxInvocation(
["hyperframes", "transcribe", inputPath, "--dir", workDir],
{ stdio: ["ignore", "pipe", "pipe"], timeout: 1_800_000 },
);
execFileSync(resolved.cmd, resolved.args, resolved.opts);
const produced = join(workDir, "transcript.json");
if (!existsSync(produced)) throw new Error("whisper produced no transcript.json");
const tmp = `${outPath}.tmp-${process.pid}`;
copyFileSync(produced, tmp);
renameSync(tmp, outPath); // atomic publish
let words;
try {
const t = JSON.parse(readFileSync(outPath, "utf8"));
words = Array.isArray(t?.words) ? t.words.length : undefined;
} catch {
/* leave undefined */
}
report("whisper", words);
} finally {
rmSync(workDir, { recursive: true, force: true });
}
}
try {
const parakeetBin = resolveParakeet();
const engine =
args.engine === "parakeet" || args.engine === "whisper"
? args.engine
: parakeetBin
? "parakeet"
: "whisper";
if (engine === "parakeet") {
if (!parakeetBin) {
throw new Error(
"parakeet-mlx not found (checked $HYPERFRAMES_PARAKEET, ~/.venvs/parakeet, and PATH). Install: uv venv ~/.venvs/parakeet && VIRTUAL_ENV=~/.venvs/parakeet uv pip install parakeet-mlx (or use --engine whisper)",
);
}
runParakeet(parakeetBin);
} else {
runWhisper();
}
await track("media_use_transcribe", { engine });
} catch (err) {
if (args.json) console.log(JSON.stringify({ ok: false, error: err.message }));
else console.error(`error: transcription failed: ${err.message}`);
process.exit(1);
}
scripts/transcript-cut.mjs›
#!/usr/bin/env node
import { execFileSync } from "node:child_process";
import { mkdtempSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { dirname, extname, join, resolve } from "node:path";
import { parseArgs } from "node:util";
import { compileCutList } from "./lib/cutlist.mjs";
import { track } from "./lib/telemetry.mjs";
const { values: args } = parseArgs({
options: {
input: { type: "string" },
transcript: { type: "string" },
remove: { type: "string" },
"remove-words": { type: "string" },
"remove-fillers": { type: "string" },
"cut-silence": { type: "string" },
keep: { type: "string" },
copy: { type: "boolean", default: false },
plan: { type: "boolean", default: false },
out: { type: "string" },
json: { type: "boolean", default: false },
help: { type: "boolean", short: "h", default: false },
},
strict: true,
});
if (args.help) {
console.log(`media-use transcript-cut — compile transcript edits into video cuts
Usage:
node transcript-cut.mjs --input in.mp4 --transcript transcript.json --remove "12-15" --out out.mp4
Options:
--input Source video/audio file
--transcript JSON word transcript, array or { words: [...] }
--remove Time ranges to remove, seconds: a-b,c-d
--remove-words Word-index ranges to remove: 12-18,40-41
--remove-fillers Comma list of filler words to remove
--cut-silence Remove inter-word gaps longer than this many seconds
--keep Inverse mode: direct kept ranges, mutually exclusive with removal
--copy Use stream copy for faster, keyframe-snapped cuts
--plan Print kept segment JSON and exit without ffmpeg
--out Output file
--json Output JSON status
--help, -h Show this help`);
process.exit(0);
}
try {
run();
await track("media_use_transcript_cut", {
mode: args.plan ? "plan" : "encode",
remove_fillers: !!args["remove-fillers"],
cut_silence: !!args["cut-silence"],
ranges: !!args.remove,
keep: !!args.keep,
});
} catch (err) {
if (args.json) console.log(JSON.stringify({ ok: false, error: err.message }));
else console.error(`error: ${err.message}`);
process.exit(1);
}
function run() {
if (!args.transcript) throw new Error("--transcript is required");
const transcript = JSON.parse(readFileSync(resolve(args.transcript), "utf8"));
const segments = compileCutList(transcript, {
remove: args.remove,
removeWords: args["remove-words"],
removeFillers: args["remove-fillers"],
cutSilence: args["cut-silence"],
keep: args.keep,
});
if (args.plan) {
console.log(JSON.stringify(segments));
return;
}
if (!args.input || !args.out)
throw new Error("--input and --out are required unless --plan is set");
if (segments.length === 0) throw new Error("cut list has no kept segments");
const inputPath = resolve(args.input);
const outPath = resolve(args.out);
mkdirSync(dirname(outPath), { recursive: true });
const tmpDir = mkdtempSync(join(tmpdir(), "media-use-cut-"));
const keptSeconds = sumDurations(segments);
const totalSeconds = probeDuration(inputPath);
try {
const parts = segments.map((segment, index) => {
const out = join(
tmpDir,
`segment-${String(index).padStart(4, "0")}${extname(outPath) || ".mp4"}`,
);
cutSegment(inputPath, segment, out, Boolean(args.copy));
return out;
});
const listPath = join(tmpDir, "list.txt");
writeFileSync(
listPath,
parts.map((part) => `file '${escapeConcatPath(part)}'`).join("\n") + "\n",
);
// Encode to a sibling temp (same extension so ffmpeg picks the right muxer),
// then atomic-rename so a SIGKILL mid-encode can't leave a truncated outPath.
const tmpOut = `${outPath}.part${extname(outPath) || ".mp4"}`;
execFileSync(
"ffmpeg",
["-y", "-f", "concat", "-safe", "0", "-i", listPath, "-c", "copy", tmpOut],
{
stdio: "ignore",
},
);
renameSync(tmpOut, outPath);
} finally {
rmSync(tmpDir, { recursive: true, force: true });
}
// Stream copy can only cut on keyframes; on sparse-keyframe footage the snap
// can silently swallow the whole cut. Compare, then surface the drift in BOTH
// the stderr warning (human) and the --json result (pipelines).
let copyDrift = null;
if (args.copy) {
const outSeconds = probeDuration(outPath);
if (Math.abs(outSeconds - keptSeconds) > 1) {
copyDrift = { produced_s: round3(outSeconds), expected_s: round3(keptSeconds) };
if (!args.json) {
console.error(
`warning: --copy keyframe snapping produced ${round3(outSeconds)}s instead of ${round3(keptSeconds)}s kept; drop --copy for frame-accurate cuts`,
);
}
}
}
if (args.json) {
console.log(
JSON.stringify({
ok: true,
input: inputPath,
out: outPath,
segments,
kept_s: round3(keptSeconds),
total_s: round3(totalSeconds),
...(copyDrift && { copy_drift: copyDrift }),
}),
);
return;
}
console.log(
`cut ${inputPath} -> ${outPath} (${segments.length} segments, ${fmt(keptSeconds)}s kept of ${fmt(
totalSeconds,
)}s)`,
);
console.log(`next: resolve --from ${outPath} --type <type>`);
}
function cutSegment(inputPath, segment, outPath, copy) {
const argv = [
"-y",
"-nostdin",
"-ss",
fmt(segment.start),
"-i",
inputPath,
"-to",
fmt(segment.end - segment.start),
];
if (copy) {
argv.push("-c", "copy", "-avoid_negative_ts", "make_zero");
} else {
argv.push(...encodeArgsFor(extname(outPath).toLowerCase()));
}
argv.push(outPath);
execFileSync("ffmpeg", argv, { stdio: "ignore" });
}
// Codec set per output container. Audio-only outputs must not get the
// video-centric aac/x264 set (aac inside .wav breaks timing entirely).
function encodeArgsFor(ext) {
if (ext === ".wav") return ["-c:a", "pcm_s16le"];
if (ext === ".mp3") return ["-c:a", "libmp3lame", "-q:a", "2"];
if (ext === ".m4a" || ext === ".aac") return ["-c:a", "aac"];
if (ext === ".flac") return ["-c:a", "flac"];
return [
"-c:v",
"libx264",
"-preset",
"veryfast",
"-crf",
"18",
"-c:a",
"aac",
"-movflags",
"+faststart",
];
}
function probeDuration(filePath) {
const raw = execFileSync(
"ffprobe",
[
"-v",
"error",
"-show_entries",
"format=duration",
"-of",
"default=noprint_wrappers=1:nokey=1",
"--",
filePath,
],
{ encoding: "utf8" },
);
const duration = Number(raw.trim());
if (!Number.isFinite(duration) || duration <= 0)
throw new Error(`could not probe duration: ${filePath}`);
return duration;
}
function escapeConcatPath(filePath) {
return filePath.replace(/'/g, "'\\''");
}
function sumDurations(segments) {
return segments.reduce((sum, segment) => sum + (segment.end - segment.start), 0);
}
function fmt(n) {
return round3(n)
.toFixed(3)
.replace(/\.?0+$/, "");
}
function round3(n) {
return Math.round(Number(n) * 1000) / 1000;
}
SKILL.md›
---
name: media-use
description: Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.
---
# media-use
The media OS for HyperFrames: resolve · generate · operate · remember — every media type, one skill, zero context noise.
First run: install and sign in to the `heygen` CLI (the free-usage path), then verify with `npx hyperframes media-use resolve --doctor`. Setup and providers: `references/setup-providers.md`.
## Resolve — the one verb
```bash
npx hyperframes media-use resolve --type <type> --intent "<description>" --project <dir>
```
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`. All search noise stays on disk.
| Type | One-line intent |
| ------- | ----------------------------------------------------------------------------------- |
| `bgm` | background music (HeyGen catalog, 10k+ tracks) |
| `sfx` | sound effects (bundled 19-file library + catalog) |
| `image` | photos, backgrounds (HeyGen asset search, 75k+ vectors) |
| `icon` | icons, symbols (transparent) |
| `logo` | official brand marks (svgl → simple-icons → GitHub avatar → favicon; never redrawn) |
| `voice` | TTS voiceover (HeyGen free-usage path; optional local Kokoro) |
| `grade` | measured correction candidate; broad polish/stylization follows Media Treatments |
| `lut` | user-provided or explicitly chosen reusable validated `.cube` file |
Before resolving fresh, list reusable candidates with `--candidates` and judge fit yourself — reuse rules, all flags, ingest (`--from`), and adopt are in `references/resolve.md`.
## Treat broad visual feedback as media intent
When a user explicitly asks to fix, polish, stylize, obscure, emphasize, or
reveal photographic media, read `references/media-treatments.md` even if they
do not name color grading or an effect. Inspect the real `<img>`/`<video>`,
choose one primary intent, then use deterministic persistence and verification.
Use a matching recipe as an optional tested seed, or inspect
`hyperframes media-treatment --capabilities --json`, then request one relevant
family/effect with `--capability <id>` and assemble a custom treatment from
canonical controls. Never load `--all` for ordinary authoring. A treatment may
compose correction, a preset, finishing, compatible shader effects, supported
keyframes, and optional Registry overlays. Add only source-justified bounded
tuning and compatible parts, never effects merely to make the result look more
sophisticated. Persist the final combined payload with
`hyperframes media-treatment`.
Use one progressively escalating workflow. For video, inspect one labeled
early/middle/late contact sheet rather than reading frames separately. Apply one
candidate and inspect one after-sheet for ordinary correction or polish.
Escalate to individual frames or moving draft evidence only when the result is
ambiguous, temporal, stylized, LUT-based, HDR/LOG-sensitive, private, or
brand-critical.
For ordinary correction or polish, persist the final treatment's
preset/adjustment JSON.
Do not generate a `.cube` LUT merely to encode exposure, shadows, contrast, or
warmth. Use a LUT only when the user supplies one or the selected treatment
explicitly owns one. `resolve --type grade --for ... --analyze` is measurement
evidence, not permission to replace the chosen treatment with a generated LUT.
Do not recreate supported vignette, grain, blur, pixelate, color, or treatment
effects with CSS/SVG overlays; that bypasses Studio controls and the canonical
preview/render shader path.
## Be proactive — run a media opportunity pass
The human usually can't tell which media would lift the piece. You can. When you build or review a composition, do **one** grounded scan and then **ask once** — don't silently add, and don't nag per asset.
Surface an opportunity only when a concrete signal is present:
| Signal detected | Offer |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| On-screen text / a script with no voiceover | TTS voiceover (audio engine) |
| Emoji or a `<div>` styled as an icon | resolve real `icon`s |
| Image that is a placeholder, tiny, or upscaled-looking | a better `image` (and/or upscale — see `references/operations.md`) |
| Hard scene cuts / transitions with no sound | transition `sfx` |
| A piece over ~10s with no music bed | `bgm` |
| Footage that reads under/over-exposed or color-cast | a corrective grade (inspect it with `hyperframes media-treatment --selector '#hero' --analyze --json`) |
| Photographic media that feels visually flat or off-topic | one specific source-appropriate preset or custom treatment, with the intended target named |
| A meaningful media entrance/reveal that feels static | one supported seek-safe treatment animation; preserve color unless the request also justifies a preset |
Rules that keep this a help, not nagware: **grounded, not generic** (no signal → no suggestion); **opinionated + concrete** (propose the specific fix with defaults chosen — the human approves **all / some / none**); **once per project** (one consolidated ask; respect "leave it"); **surface, never silently mutate** (color grades especially: propose and preview — a gray-world "correction" ruins an intentional sunset or neon look).
## Where to look — read only the file your task needs
| Task | Read |
| ------------------------------------------------------------------------- | -------------------------------- |
| resolve / reuse / adopt / ingest, flags, cascade, inventory | `references/resolve.md` |
| color grading, LUTs, smart grade (`--for`), grade-compare | `references/grading.md` |
| voiceover / TTS, music, SFX, captions, transcription (audio engine) | `references/audio.md` |
| cut / reframe / transform existing media, exact error diffusion, HEVC | `references/operations.md` |
| source-aware creative treatments, realtime effects, overlays, reveals | `references/media-treatments.md` |
| install + auth, provider table, RAM ladders, `--local-only`, `--provider` | `references/setup-providers.md` |
| remembered preferences + frozen recipes (user memory) | `references/memory.md` |
| ownership matrix, usage stats, telemetry, privacy (maintainer-facing) | `references/meta.md` |