argent-device-interact

# Chromium (CDP) Targets

Everything in the related SKILL.md still applies; this page covers what differs.

## Multi-tab / windows

A Chromium device may have several tabs / BrowserWindows. Use `chromium-tabs` to `list` them (stable ids `t1`, `t2`, …, optional labels), open a `new` one, `select` which is active, or `close` one.

Every other tool (`describe`, `gesture-tap`, `screenshot`, `debugger-evaluate`, `open-url`, …) acts on the **active** tab, so `chromium-tabs action=select` before driving a different tab.

Note: a cross-process navigation (some redirects) can swap a tab's underlying CDP target; re-run `chromium-tabs action=list` to pick it up under a fresh id.

## Cookies & storage

`chromium-cookies` reads/writes cookies via the Network domain (so HttpOnly cookies are visible):

- `action=get` (optionally scoped by `url`)
- `set` (`name`, `value`, + `url`/`domain`, optional `secure`/`httpOnly`/`sameSite`/`expires`)
- `delete` (`name`)
- `clear` (all)

`chromium-storage` reads/writes Web Storage for the active page: `store=local|session`, `action=get` (one `key` or all entries), `set`, `remove`, `clear`.

Both are per-origin / active-tab. Handy for seeding auth before an e2e test or asserting app state after one.

# Gesture Examples

Detailed sequences and parameters for the gesture tool family. All coordinates, distances, and radii are **normalized 0.0–1.0** (fractions of screen width/height, not pixels); same coordinate space across all gesture tools.

## gesture-pinch: Two-finger pinch

Pinch out (zoom in):

```json
{ "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.2, "endDistance": 0.6 }
```

Pinch in (zoom out):

```json
{ "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.6, "endDistance": 0.2 }
```

Vertical pinch (angle = 90):

```json
{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "startDistance": 0.2,
  "endDistance": 0.6,
  "angle": 90
}
```

## gesture-rotate: Two-finger rotation

Clockwise 90 degrees:

```json
{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "radius": 0.15,
  "startAngle": 0,
  "endAngle": 90
}
```

Counter-clockwise 45 degrees:

```json
{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "radius": 0.15,
  "startAngle": 0,
  "endAngle": -45
}
```

## gesture-custom: Custom touch sequences

### Long Press (800ms hold)

```json
{
  "udid": "<UDID>",
  "events": [
    { "type": "Down", "x": 0.5, "y": 0.5 },
    { "type": "Up", "x": 0.5, "y": 0.5, "delayMs": 800 }
  ]
}
```

### Pinch Out (zoom in): manual events

```json
{
  "udid": "<UDID>",
  "events": [
    { "type": "Down", "x": 0.4, "y": 0.5, "x2": 0.6, "y2": 0.5 },
    { "type": "Move", "x": 0.2, "y": 0.5, "x2": 0.8, "y2": 0.5 },
    { "type": "Up", "x": 0.2, "y": 0.5, "x2": 0.8, "y2": 0.5 }
  ]
}
```

### Pinch Out with interpolation: smooth version

```json
{
  "udid": "<UDID>",
  "events": [
    { "type": "Down", "x": 0.4, "y": 0.5, "x2": 0.6, "y2": 0.5 },
    { "type": "Up", "x": 0.2, "y": 0.5, "x2": 0.8, "y2": 0.5 }
  ],
  "interpolate": 15
}
```

### Drag and Drop

```json
{
  "udid": "<UDID>",
  "events": [
    { "type": "Down", "x": 0.3, "y": 0.4 },
    { "type": "Move", "x": 0.3, "y": 0.4, "delayMs": 500 },
    { "type": "Move", "x": 0.7, "y": 0.6 },
    { "type": "Up", "x": 0.7, "y": 0.6 }
  ]
}
```

Add a `delayMs` on the first `Move` to simulate a hold before dragging; this is required for some drag-and-drop implementations that only activate after a sustained press.

# HarmonyOS Targets

HarmonyOS targets exist only after `argent enable harmonyos`; with the flag off `list-devices` lists none and every `harmony-*` id is refused with `HarmonyOS support is off; turn it on with argent enable harmonyos`.

Everything in the related SKILL.md still applies; this page covers what differs.

## Ids and discovery

`list-devices` tags HarmonyOS entries `platform: "harmony"` and gives each a `kind`:

| `kind`     | id                        | state                   | What it is                                                                                                |
| ---------- | ------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------- |
| `device`   | `harmony-<connectKey>`    | `Connected` / `Offline` | A target connected over `hdc` (a phone on USB, or a booted emulator). The interaction tools drive this id |
| `emulator` | `harmony-emulator-<name>` | `running` / `stopped`   | A DevEco Studio instance. Only `boot-device` accepts it; never use it as an interaction target            |

- A running emulator appears under both kinds, as a running Android AVD appears in both `avds` and the device list.
- A plugged-in phone (`state: "Connected"`) is listed as ready alongside booted simulators. Pick a `harmony` entry only when the user asked for HarmonyOS, and pick the `kind: "device"` one.
- Every tool auto-dispatches on the id: `harmony-<connectKey>` is a HarmonyOS device, `harmony-emulator-<name>` is a boot target.

## Booting an emulator

A phone on `hdc` is ready as-is. Otherwise call `boot-device` with `harmonyInstance`: the instance `name` from the `kind: "emulator"` entry (e.g. `Phone_1`), or its `harmony-emulator-<name>` id. It hands the instance to DevEco Studio's Emulator manager, waits for it to register with `hdc`, and returns the `harmony-<connectKey>` id to drive. An instance that is already running is left alone unless you pass `force: true`. Read the `note` when one comes back: it says what the boot left unproven (no key and why, or a key not answering yet). Huawei serves emulator images only within mainland China, so a host outside it usually has no instance to start.

## What runs and what is refused

The device drives the tap / swipe / pinch / type / read loop with the same tool names as iOS and Android: `describe`, `screenshot`, `await-ui-element`, `await-screen-idle`, `gesture-tap`, `gesture-swipe`, `gesture-pinch`, `keyboard`, `button`, `launch-app`, `restart-app`, `reinstall-app`, `open-url`, `run-sequence`, `screenshot-diff`.

Refused as unsupported on harmony:

- `gesture-rotate` and `gesture-custom`: the device moves each contact along one straight line per call, so there is no arc and no long-press. There is no long-press tool on HarmonyOS.
- `gesture-drag` and `gesture-scroll` (Chromium-only), and `rotate`: the device has no orientation command.
- The app-scoped inspectors `native-describe-screen`, `native-view-at-point`, `native-user-interactable-view-at-point` and `debugger-component-tree`, which declare no harmony support and answer `Tool '<id>' is not supported on harmony device` before running. The rest of the debugger (`debugger-evaluate`, `debugger-log-registry`), `view-network-logs` and the profilers are refused too.
- `paste`, so a `{{secret:...}}` placeholder goes through `keyboard` only.
- Screen recording: HarmonyOS exposes no readable frame stream. Use `screenshot` for a still frame.

`gesture-pinch` sends both fingers' endpoints in one `uinput` call and the device interpolates the move; `durationMs` is wall-clock time, clamped to 1-15000ms. `gesture-swipe` is likewise one on-device call, and its `durationMs` sets speed, clamped to `uitest`'s 200-40000 px/s: a short slow swipe ends early, a long fast one runs late. `gesture-tap` with `clickCount: 2` is one double-tap gesture; 3 or more land as separate taps.

## Discovery

`describe` runs the device's own `uitest dumpLayout` (`source: "harmony-uitest"`) and returns it in the same DescribeNode shape as the other platforms. Each top-level child is a window carrying its bundle as `identifier`, so an app plus the status bar reads as two windows. `describe` and `screenshot` are the whole discovery surface, and when `describe` fails `screenshot` is the only fallback.

An empty tree comes with a `hint` that the layout dump contains no windows and the foreground app may still be starting. That is the expected read during app startup: call `describe` again rather than switching tools.

`screenshot` takes no `rotation`: `uitest screenCap` captures the display in its current orientation, so a `rotation` is refused with 400 and no image comes back. The same holds for a live capture in `screenshot-diff`. Rotate the device itself, or drop the parameter.

## Suspended panel

A suspended display still dumps its last-composited windows, which look like a live screen. `describe` then carries a `hint` that the panel is off, and the waits pass on that frame:

- `await-ui-element` with `success: true` and a `note`: the tree was read off a suspended panel. It is not clearance to act.
- `await-screen-idle` settles at once on the last frame; a `note` on `settled: true` says the screen is not live.

Taps land nowhere and input tools refuse against the panel until `button` with `power` wakes it.

## Keyboard keys

Only `enter` (alias `return`), `backspace` (alias `delete`), `space`, `arrow-left` and `arrow-right` exist; any other key is refused with 400 `KEYBOARD_KEY_UNSUPPORTED`. `delayMs` is ignored: `uitest uiInput text` types the text in one shot. Text cannot contain a newline or another control character; type the text, then send `key: "enter"`.

## Buttons

`button` accepts `home`, `back` and `power` only; any other button is rejected.

## App launch and open-url

- `launch-app` and `restart-app` take the bundle name as listed by `bm dump -a` (e.g. `com.huawei.hmos.calculator`). `activity` is ignored: the ability comes from the bundle's own `abilityInfos`. A launch fails if the app is not installed.
- `reinstall-app` takes one `.hap` file as `appPath` and the bundle name from its `module.json` as `bundleId`; a mismatch is refused before anything is uninstalled. A multi-module `.app` bundle is not accepted.
- `open-url` resolves any scheme through `aa start -U`, and every result carries a `note`: `aa` reports success whenever the system accepts the URI. A web URL can leave the screen unchanged, and `tel:` / `mailto:` can leave a "No options to open with" dialog over it. Confirm with `describe` or `screenshot` before treating the link as followed. It fails whenever `aa` refuses the URI, as it does a scheme nothing claims.

## E2E tests

The e2e runner (`e2e-execute`, skill `argent-create-e2e`) runs on HarmonyOS only when you name the device: `device: "harmony-<connectKey>"` (`--device` on the CLI). A HarmonyOS device is never auto-resolved, and `platform` / `--platform` does not accept `harmony`. HarmonyOS is not an authorable platform either: `when: { platform: harmony }` and a `harmony:` launch-map key are rejected when the YAML is parsed, so launch with a bare app id or `native:`.

The runner has no UI tree on HarmonyOS (`describe` reads the `uitest` layout dump, but no runner tree source exists), so no selector resolves:

- Every selector step errors with a reason containing `ui-tree matching is not supported on platform "harmony"`: a selector `tap:` / `swipe:` / `pinch:`, `type:` (selector-only), `scroll-to:`, `await:` / `assert:` and `when:` UI conditions, and `snapshot:` with `cropOn:`.
- `await: { idle: true }` errors immediately, with no polling, since no relaunch can restore a tree that does not exist. It names `wait:` as the replacement and skips the rest of the run.
- `long-press:` and `rotate:` are refused up front, with or without a selector: `uitest uiInput` sends one whole gesture per call with no hold duration, and `uinput -T -m`, the two-contact injection `pinch:` rides on, moves each contact along one straight line, never an arc.

What runs: a coordinate `tap:`, a selector-less `swipe:` and `pinch:`, `launch:`, `snapshot:` without `cropOn:`, `wait:`, `echo:`, `script:`, and raw `tool:` steps for the tools that support HarmonyOS. For anything that needs a selector, run the scenario as a live interact-screenshot-verify loop instead.

# Typing Secrets

Where a `{{secret:<NAME>}}` placeholder gets its value, and the rules for using one. See the `keyboard` section of SKILL.md for the placeholder syntax itself.

## Where the value comes from

The placeholder is resolved on the machine running the tool-server, from the first of these that defines the name:

| #   | Source                                        | Which keys it exposes                                                |
| --- | --------------------------------------------- | -------------------------------------------------------------------- |
| 1   | `ARGENT_SECRET_<NAME>` environment variable   | prefixed vars only; the CI-native path                               |
| 2   | `<project>/.argent/secrets.env`               | every key (`APP_PASSWORD=…`); gitignore this file                    |
| 3   | `<project>/.env.local`, then `<project>/.env` | only `ARGENT_SECRET_`-prefixed keys, so app config stays unreachable |
| 4   | `~/.argent/secrets.env`                       | every key, per-user, works in any project                            |

## Rules

- The result echoes the placeholder, never the value. An unknown name fails with the list of available secret _names_ and every source it looked in, with paths; read that list before asking the user anything.
- The auto-screenshot after the call is skipped so the typed value cannot re-enter your context as pixels. Do **not** `describe` or `screenshot` a non-secure field you just filled with a secret; submit or navigate away first, then verify the resulting screen. To submit, put the text step and the Enter step in **one `run-sequence`**. The skip covers a whole batch that contains the placeholder, but a second bare `keyboard` call gets its own screenshot of the filled field.
- Nothing outside those sources is reachable; never ask the user to paste a secret value into the conversation. Ask them to put it in a secrets file instead: a file edit applies to the next call, while an exported env var only reaches a tool-server started afterwards.
- The project sources are found by walking up from the tool-server's working directory. If a project file is not being picked up, the failure's source list shows the paths actually consulted; `~/.argent/secrets.env` needs no project and always applies.

---
name: argent-device-interact
description: Interact with an iOS simulator, Android emulator, or Chromium (CDP) app using argent MCP tools. Use when tapping UI elements, performing gestures, scrolling/swiping, typing text, pressing hardware buttons, launching apps, opening URLs, taking screenshots, waiting for an element to appear or disappear, or checking visible app state after interactions. Not for TV targets.
---

## Unified tool surface

All interaction tools below accept a `udid` parameter and auto-dispatch iOS vs Android based on its shape (UUID → iOS simulator, `chromium-cdp-<port>` → Chromium (CDP) app, anything else → Android adb serial). You use the same tool names on every platform.

**Chromium (CDP) app** = an Electron app or a Chromium-family browser (Chrome/Brave/Edge) exposing a Chrome DevTools Protocol endpoint. The same describe/tap/keyboard/screenshot surface drives it, but scrolling, tabs, cookies and storage differ: **read `references/chromium.md` before driving a `chromium` target.**

## 1. Before You Start

If you delegate simulator tasks to sub-agents, make sure they have MCP permissions.

Use `list-devices` to get a target id. Results are tagged with `platform` (`ios`, `android`, or `chromium`); booted/ready devices come first. For a `harmony` target, read `references/harmonyos.md` before driving it. Pick the first entry that matches the platform you need; if none are ready, call `boot-device` with `udid` (iOS), `avdName` (Android), or `electronAppPath` (boots an Electron app as a `chromium` device). A Chromium browser already running with a CDP port shows up directly; no `boot-device` needed. See `argent-ios-simulator-setup` / `argent-android-emulator-setup` for full setup flow.

**Load tool schemas before first use.** Gesture tools (`gesture-tap`, `gesture-swipe`, `gesture-pinch`, `gesture-rotate`, `gesture-custom`) may be deferred: their parameter schemas are not loaded until fetched. Always use ToolSearch to load the schemas of all gesture tools you plan to use **before** calling any of them. If you skip this step, parameters may be coerced to strings instead of numbers, causing validation errors.

## 2. Best Practices

1. **Always refer to tapping_rule** from your argent.md rule before tapping.
2. Before performing interactions, consider whether they can be **dispatched sequentially** - more on that in `run-sequence`.
3. **Use `gesture-swipe` for lists/scrolling**, not `gesture-custom`, unless you need non-linear movement. On Chromium use `gesture-scroll` instead; `gesture-swipe` is touch-only. Consider whether you need multiple swipes, if yes - use `run-sequence`. Pass `momentum: false` when the swipe should decelerate before ending for a precise movement.
4. **Tap a text field before typing**, then use `keyboard` to enter text.
5. **Coordinates are normalized**: always 0.0–1.0, not pixels.
6. **For app navigation, use the element tree returned after each action** (`--- Elements after action (describe) ---`); call `describe` only when no fresh tree is available for the current screen. It works on any screen without app restart. Do not navigate from screenshot pixels on regular in-app screens unless the tree failed to expose a reliable target. Use `native-describe-screen` only when you need app-scoped UIKit properties.

## 3. Opening Apps

**Never navigate to an app by tapping home-screen icons.** Use `launch-app` or `open-url`: they are instant and reliable.

### launch-app: by bundle ID

```json
{ "udid": "<UDID>", "bundleId": "com.apple.MobileSMS" }
```

Common IDs: `com.apple.MobileSMS` (Messages), `com.apple.mobilesafari` (Safari), `com.apple.Preferences` (Settings), `com.apple.Maps`, `com.apple.Photos`, `com.apple.mobilemail`, `com.apple.mobilenotes`, `com.apple.MobileAddressBook` (Contacts)

### open-url: by URL scheme

```json
{ "udid": "<UDID>", "url": "messages://" }
```

Common schemes: `messages://`, `settings://`, `maps://?q=<query>`, `tel://<number>`, `mailto:<address>`, `https://...` (Safari)

## 4. Choosing the Right Tool

| Action            | Tool                | Notes                                                             |
| ----------------- | ------------------- | ----------------------------------------------------------------- |
| Multiple actions  | `run-sequence`      | Batch steps in one call (no intermediate screenshots)             |
| Open an app       | `launch-app`        | **Always: never tap home-screen icons**                           |
| Restart an app    | `restart-app`       | Terminate and relaunch by bundle ID                               |
| Open URL/scheme   | `open-url`          | Web pages, deep links, URL schemes                                |
| Single tap        | `gesture-tap`       | Buttons, links, checkboxes                                        |
| Scroll/swipe      | `gesture-swipe`     | Straight-line scroll or swipe                                     |
| Scroll (Chromium) | `gesture-scroll`    | Wheel-based; deltas are window fractions, positive deltaY = down  |
| Drag (Chromium)   | `gesture-drag`      | Sliders, drag-and-drop, text selection                            |
| Long press        | `gesture-custom`    | Context menus, drag start                                         |
| Drag & drop       | `gesture-custom`    | Complex drag interactions                                         |
| Pinch/zoom        | `gesture-pinch`     | Two-finger pinch with auto-interpolation                          |
| Rotation          | `gesture-rotate`    | Two-finger rotation with auto-interpolation                       |
| Custom gesture    | `gesture-custom`    | Arbitrary touch sequences, optional interpolation                 |
| Hardware key      | `button`            | Home, back, power, volume, appSwitch, actionButton                |
| Type text         | `keyboard`          | Every platform. Text or one named key per call, never both        |
| Paste text        | `paste`             | Only where a user would paste (OTP code, long link). Sim/emu only |
| Rotate device     | `rotate`            | Orientation changes                                               |
| Fold device       | `fold`              | Foldable simulator/emulator: closed / half-open / open, or angle  |
| Shake device      | `shake`             | Shake handlers (sim/emu only), Undo-typing prompt, RN dev menu    |
| Wait for UI       | `await-ui-element`  | Block until an element is visible/hidden/exists/contains text     |
| Wait for idle     | `await-screen-idle` | Block until a non-empty screen tree stops changing                |

## 5. Finding Tap Targets

IMPORTANT. When moved to a different screen after an action or do not know the coordinates of component, **always** perform proper discovery first.

| App type                          | Discovery tool            | What it returns                                                                                                                                                                                                                                    |
| --------------------------------- | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Target app discovery              | `describe`                | Accessibility element tree for the current device screen (iOS AX-service, Android uiautomator, or Chromium DOM walker) with normalized frame coordinates. Works on any app, system dialogs, and Home screen; no app restart or `bundleId` required |
| React Native                      | `debugger-component-tree` | React component tree with names, text, testID, and (tap: x,y)                                                                                                                                                                                      |
| App-scoped native                 | `native-describe-screen`  | Low-level app-scoped accessibility elements with normalized and raw coordinates; requires `bundleId`                                                                                                                                               |
| Permission / system modal overlay | `describe`                | `describe` detects system dialogs automatically and returns dialog buttons with tap coordinates. Fall back to `screenshot` only if `describe` does not expose the controls                                                                         |
| Final visual fallback             | `screenshot`              | Use only when discovery tools cannot inspect the current UI reliably. Do not derive routine in-app navigation targets from screenshots                                                                                                             |

Point follow-up native diagnostics after you already have a candidate point:

- `native-user-interactable-view-at-point`: deepest native view that would receive touch at a known raw iOS point; requires `bundleId`
- `native-view-at-point`: deepest visible native view at a known raw iOS point; requires `bundleId`

### If `describe` Tool Fails

Read the exact error and choose the action that matches it:

- Error mentions `ax-service` not available or daemon startup failure:
  the ax-service daemon could not start. Check that the simulator is booted. Use `screenshot` as a temporary fallback, or use `native-describe-screen` with an explicit `bundleId` if the app has native devtools injected.
- `describe` returns an empty element list:
  the screen may be blank, loading, or showing content without accessibility labels. Use `screenshot` to see what is visible, then retry after the content has loaded.
- `describe` succeeds but is not detailed enough for a React Native app:
  use `debugger-component-tree` next.
- You need app-scoped inspection with full UIKit properties (`accessibilityIdentifier`, `viewClassName`):
  use `native-describe-screen` with an explicit `bundleId`. This requires native devtools (dylib) injection.
- You already have a candidate point and want to confirm what would actually receive touch:
  use `native-user-interactable-view-at-point`. Use `native-view-at-point` when you want the visually deepest view instead of the hit-test target.

## 6. Tool Usage

### gesture-tap: Single tap at a point

```json
{ "udid": "<UDID>", "x": 0.5, "y": 0.5 }
```

Coordinates: `0.0` = left/top, `1.0` = right/bottom.

Before tapping near the bottom of the screen in React Native apps, check that "Open Debugger to View Warnings" banners are not visible: tapping them breaks the debugger connection. Close them with the X icon if present.

### gesture-swipe: Straight-line gesture

```json
{ "udid": "<UDID>", "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 }
```

Swipe **up** (`fromY > toY`) = scroll content **down**. Default duration: 300ms. Optional: `"durationMs": 500` for slower swipe.

`"momentum"` defaults to `true` (a natural flinging swipe). Pass `"momentum": false` for a momentum-free swipe: the finger decelerates into the end point, resulting in little to no fling. It needs `durationMs` of at least 150 and is rejected below it.

### gesture-pinch: Two-finger pinch

```json
{ "udid": "<UDID>", "centerX": 0.5, "centerY": 0.5, "startDistance": 0.2, "endDistance": 0.6 }
```

All values are normalized 0.0–1.0 (fractions of screen, not pixels), same as all other gesture tools. `startDistance: 0.2` means fingers start 20% of the screen apart; `endDistance: 0.6` means they end 60% apart. `startDistance < endDistance` = pinch out (zoom in). `startDistance > endDistance` = pinch in (zoom out). Defaults: `angle: 0` (horizontal), `durationMs: 300`. Optional: `"angle": 90` for vertical axis, `"durationMs": 500` for slower pinch, `"endCenterX"`/`"endCenterY"` to let the centroid drift to a new center over the gesture (omitted = fixed center).

### gesture-rotate: Two-finger rotation

```json
{
  "udid": "<UDID>",
  "centerX": 0.5,
  "centerY": 0.5,
  "radius": 0.15,
  "startAngle": 0,
  "endAngle": 90
}
```

All positions and radii are normalized 0.0–1.0 (fractions of screen, not pixels). `radius: 0.15` means each finger is 15% of the screen away from center. `endAngle > startAngle` = clockwise. Default duration: 300ms. Optional: `"durationMs": 500` for slower rotation, and `"radiusX"`/`"radiusY"` (fractions of screen width/height; give both: they override `radius`) with `radiusX·width = radiusY·height` for a physically circular orbit; a single `radius` traces a physical ellipse on a non-square screen, coupling a slight pinch into the turn.

### gesture-custom: Custom touch sequence

For long-press, drag-and-drop, and other complex sequences, see `references/gesture-examples.md`. Set `"interpolate": 10` to auto-generate smooth intermediate Move events between keyframes.

### button: Hardware button press

```json
{ "udid": "<UDID>", "button": "home" }
```

Values: `home`, `back`, `power`, `volumeUp`, `volumeDown`, `appSwitch`, `actionButton`

### keyboard: Type text or press special keys

```json
{ "udid": "<UDID>", "text": "search query" }
```

One call does one action. `text` and `key` are mutually exclusive, and a call that carries both is rejected with nothing typed. To type and then submit, send two `keyboard` steps in one `run-sequence` (§ 8): `{ "text": "search query" }`, then `{ "key": "enter" }`. Two separate calls do the same work, but cost an extra round-trip.

Special keys: `enter`, `escape`, `backspace`, `tab`, `space`, `arrow-up`, `arrow-down`, `arrow-left`, `arrow-right`, `f1`–`f12`. Optional: `"delayMs": 100` between keystrokes (default 50ms); applies to the iOS simulator and Chromium; it is ignored on Android phones/tablets (typed via `adb input text`, no per-key cadence), on Vega, and on TV targets.

**Typing secrets.** To enter a credential without its plaintext ever entering your context, transcript, or logs, use a secret placeholder in `text` (works in `keyboard`, `paste`, `run-sequence` keyboard steps, and e2e `type` steps):

```json
{ "udid": "<UDID>", "text": "{{secret:APP_PASSWORD}}" }
```

Where the value is read from, and the rules for using a placeholder (including not screenshotting the field afterwards) are in `references/secrets.md`. Read it before typing any credential.

### paste: Paste text into the focused field

```json
{ "udid": "<UDID>", "text": "482913" }
```

Puts `text` on the **device** clipboard and triggers the platform's paste shortcut. iOS simulator and Android emulator only; a TV target, a physical device, Chromium and Vega are rejected.

`paste` is **not** a faster `keyboard`. `keyboard` types the way a user types and stays the default for every text entry: a search query, a login, a form field. Reach for `paste` only where a real user would paste: a 2FA / OTP code copied from another app, a long link or token, or to test how the app handles pasted input. It also carries what `keyboard` can't type on a given platform (multi-line text, non-ASCII on Android), but that alone is not a reason to paste; ask whether the user would.

Tap the field first so it has focus; pasting with no focused field is a silent no-op, as with `keyboard`. `text` accepts the same `{{secret:<NAME>}}` placeholders as `keyboard`, with the same auto-screenshot skip.

### rotate: Change orientation

```json
{ "udid": "<UDID>", "orientation": "LandscapeLeft" }
```

Values: `Portrait`, `LandscapeLeft`, `LandscapeRight`, `PortraitUpsideDown`

On an unfolded foldable simulator, the value sets the orientation of the device, not of the UI. `Portrait` gives a landscape UI. `LandscapeLeft` gives a portrait UI.

### fold: Fold or unfold a foldable device

```json
{ "udid": "<UDID>", "posture": "open" }
```

Give `posture` (`closed`, `half-open` or `open`) or `angle` (0–180). Do not give both.

Use `fold` only on a foldable iOS simulator (the iPhone Duo) or Android emulator (a Pixel Fold or generic foldable AVD). `list-devices` marks them with `foldable: true`. Other devices reject the call.

Closed, the cover panel shows the UI. Half-open and open, the inner panel shows the UI. All tools use the active panel (the panel that shows the UI), also after a fold made outside argent. `fold` returns when the device accepts input again, so the next tap lands. This is also true for the next step in `run-sequence`.

Rules:

- The coordinates change with the panel. Before you tap, read the element tree in the `fold` result, or run `describe` again. Do not use frames from before the fold.
- The screenshot size changes with the panel. Keep one screenshot-diff baseline for each posture.
- iOS: unfolded, the UI is landscape. The screenshot shows the UI turned 90 degrees, and the `describe` frames use the same axes, as on a rotated iPhone.
- iOS: a fold between two angles that are not 0 or 180 can keep the current panel. The result names the active panel. To change panels, fold to `closed` or `open`.
- Fold between gestures, not during a gesture. A gesture stays on the panel where it started.
- Android: the result gives `deviceState` (`CLOSED`, `HALF_OPENED`, `OPENED`). A lock screen that the fold to closed shows is dismissed; a `warning` says when it cannot be, or when a lock screen is up after the fold. Unlock the device before the next action.
- Android: only a Pixel Fold AVD has a cover panel. When a generic foldable AVD is closed, the emulator shows only the left part of the display. `describe` frames do not match tap coordinates. Argent can reliably tap only an element that covers the visible part, for example a full-width row. Use a Pixel Fold AVD to test the cover panel.
- iOS: if argent cannot find the active panel, it uses the cover panel. The tool result then has a `warning`, and a screenshot-diff summary has a `panel:` line. Take a screenshot to see what the device shows, then do the check that the warning gives.

### await-ui-element: Block until a UI element reaches a state

**Never poll `screenshot`/`describe` in a loop to wait for something.** Use `await-ui-element`: it blocks server-side on the same tree `describe` reads. It has no bare-timer mode by design; for a plain pause, use your own harness sleep.

```json
{ "udid": "<UDID>", "condition": "visible", "selector": { "text": "Continue" } }
```

The tool's own description carries the conditions, selector matching, defaults and return shape. What it does not tell you:

- A `hidden` check that succeeds **immediately** may be a false pass: its `note` then says the selector never matched anything at all. Treat that as a failed check and fix the selector; do not read it as "the element went away".
- The synthetic `ROOT` container `describe` prints is never matched, so a `role` like `AXGroup`/`html` won't trivially "match the screen".
- To disambiguate a loose selector, pin the `role` to a text role like `StaticText`; that skips a same-named button.
- On a `text` timeout the `note` quotes the text of the element the check actually read, so you can see which match it landed on.

### await-screen-idle: Block until the screen stops changing

Use after launch/navigation and before a raw tap, when an early-painted element may still be moving:

```json
{ "udid": "<UDID>", "timeoutMs": 3000, "minStableMs": 250 }
```

On local iOS, Android, and Chromium, the tool waits for a non-empty `describe` tree to stop changing. Continue only when `settled: true`. Pair it with a destination-specific `await-ui-element`; stillness does not identify a screen.

Use it only for live diagnosis. Do not record it or put it in `run-sequence`. E2E tests use `await: { idle: true }`, which also compares pixels. This live tool can return during a presentation-layer animation.

---

## 7. Screenshots

Use the explicit `screenshot` tool only when:

- You need the initial screen state before any action.
- You are about to edit visible UI and need a baseline capture before making changes.
- The auto-attached screenshot shows a transitional or loading frame.
- You require extra context.
- You want to check state after a delay (e.g. waiting for a network response).
- A permission dialog, system alert, or native modal overlay is visible and `describe` did not expose reliable targets.

When using `screenshot` for permission or native modal navigation:

- Do not switch to screenshot-driven navigation just because a modal is visible. On regular app screens and in-app modals, keep using `describe`.
- Prefer obvious, centered alert buttons such as `Allow`, `OK`, `Don't Allow`, `Not Now`, or `Continue`.
- Tap one control at a time and inspect the returned auto-screenshot before doing anything else.
- After the modal is dismissed, return to normal discovery with `describe`, `native-describe-screen`, or `debugger-component-tree`.

> **Prefer the dialog over the Settings tool.** When the app triggers its own permission prompt, answering it here is the real user path; do that. Reach for the `settings-permissions` tool only when you can't get to the change through the app: pre-authorize/deny a permission _before_ the app asks, re-enable one the user already denied (iOS won't re-prompt), or reset it so the prompt reappears. See the `argent-settings-permissions` skill.

Optional rotation parameter: `{ "udid": "<UDID>", "rotation": "LandscapeLeft" }`: rotates the capture without changing simulator orientation.

Screenshots are downscaled by default (30% of original resolution) to reduce context size. Use the normal downscaled screenshot for UI context and state checks. `scale` accepts values from 0.01 to 1.0, but do not use `scale: 1.0` as a general readability or tapping aid.

Use full-resolution screenshots only when saving baseline/current PNG files for comparison. In that case, suppress the image block so the full-size PNG is not loaded into agent context:

```json
{ "udid": "<UDID>", "scale": 1.0, "includeImageInContext": false }
```

For visual regression checks, before/after screenshot comparisons, and detailed `screenshot-diff` parameter guidance, use the `argent-screenshot-diff` skill. Keep this skill focused on device interaction mechanics and screenshot capture.

### Troubleshooting

| Problem                 | Solution                                                      |
| ----------------------- | ------------------------------------------------------------- |
| Screenshot times out    | Restart the simulator-server via `stop-simulator-server` tool |
| No booted iOS simulator | Call `boot-device` with the iOS `udid`                        |
| No ready Android device | Call `boot-device` with `avdName`                             |

---

## 8. Action Sequencing with `run-sequence`

Use `run-sequence` to batch multiple interaction steps into **a single tool call**. Only one screenshot is returned, after all steps complete.

Do **not** use `run-sequence` when any step depends on observing the result of a previous step.

### Use cases

- "scroll to bottom", "scroll to top", "scroll until X" -> sequence 3-5 scrolls
- form interactions, "clear and retype field" -> triple-tap to select all, then type the new value
- "submit form" → fill all fields in sequence, tap submit
- "go back to X" → defined tap sequence for the navigation

### Allowed tools inside `run-sequence`

`gesture-tap`, `gesture-swipe`, `gesture-scroll`, `gesture-drag`, `gesture-custom`, `gesture-pinch`, `gesture-rotate`, `button`, `keyboard`, `paste`, `rotate`, `shake`, `fold`, `tv-remote`, `await-ui-element`

The `udid` is shared; do **not** include it in each step's `args`. Optional `delayMs` per step (default 100ms).

Add an `await-ui-element` step to gate a later tap on a screen transition (e.g. tap → wait for the next screen's button → tap it). If its condition is **not** met before the timeout, the sequence stops at that step and the following steps do **not** run, so a mistimed tap can't fire against a screen that never settled.

### Examples

Scroll down three times:

```json
{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } },
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } },
    { "tool": "gesture-swipe", "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 } }
  ]
}
```

Type into a focused field and submit. This is the only way to mix text and a key, because one `keyboard` call cannot carry both:

```json
{
  "udid": "<UDID>",
  "steps": [
    { "tool": "keyboard", "args": { "text": "hello world" } },
    { "tool": "keyboard", "args": { "key": "enter" } }
  ]
}
```

Tap a known button, then scroll down:

```json
{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.15 } },
    {
      "tool": "gesture-swipe",
      "args": { "fromX": 0.5, "fromY": 0.7, "toX": 0.5, "toY": 0.3 },
      "delayMs": 300
    }
  ]
}
```

Tap, wait for the next screen, then act on it: the `await-ui-element` step **gates** the tap after it:

```json
{
  "udid": "<UDID>",
  "steps": [
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.9 } },
    {
      "tool": "await-ui-element",
      "args": { "condition": "visible", "selector": { "text": "Continue" } }
    },
    { "tool": "gesture-tap", "args": { "x": 0.5, "y": 0.5 } }
  ]
}
```

Prefer this over a fixed `delayMs` when a step depends on a screen transition: it adapts to real load time, and if the condition is not met before the timeout the sequence **stops there** so the next tap can't fire against a screen that never settled.

Stops on the first error (or unmet `await-ui-element` condition) and returns partial results.

---

## 9. Platform-specific notes

### Android

- **Metro reachability**: run `adb reverse tcp:8081 tcp:8081` on the device before the RN app starts, or Metro won't be reachable from the device. See `argent-metro-debugger` for the full workflow. Re-run if the device restarts.
- **First-launch permission prompts**: `reinstall-app` on Android always installs with `-g` so runtime permissions are pre-granted on first launch; no flag to pass.
- **Locked screen / secure surfaces**: `describe` throws a clear error if it can't capture (keyguard, DRM, Play Integrity). Unlock the device or fall back to `screenshot`.
- **APK vs .app in `reinstall-app`**: pass `.apk` absolute path on Android; `.app` directory on iOS.

### Chromium

See `references/chromium.md` (tabs, cookies/storage).

### iOS

_(no iOS-only gotchas collected here yet; add them as they come up)_