firecrawl-monitor

# Writing monitor goals and queries

Reference for authoring `--goal` (all monitors) and `--queries` (web monitors). Read from [SKILL.md](SKILL.md) when creating or tuning a monitor.

## Writing a good `--goal`

The goal is what the AI change judge uses to decide whether a page is `changed` vs `same`. Convert the user's intent into a concise 2-3 sentence goal:

- Start with `Alert when ...` and state the trigger using the user's wording.
- Restate any scope they mentioned: top N, price, role type, region, company, topic, status, or a specific entity.
- Add an `Ignore ...` sentence **only** for intent-specific exclusions (e.g. points/comments for rankings, marketing copy for pricing, general company-page updates for job listings). The judge already handles generic noise — whitespace, casing, punctuation, encoding, formatting-only changes, request/session IDs, cache busters, tracking params, generic metadata, and unrelated page chrome — so leave those out.
- Include only page-specific sections, entities, thresholds, exclusions, or business rules the user actually mentioned.
- If the user is vague or asks for "any change", keep the goal broad with no exclusions. If the user mentions noise they do not care about, include that explicitly.

| User says                   | Good goal                                                                                                                                                             |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `top 10 hackernews stories` | `Alert when stories enter, leave, or change rank within the Hacker News top 10. Ignore points, comments, and timestamps. Do not alert on changes outside the top 10.` |
| `pricing changes`           | `Alert when pricing information changes, including prices, plan names, billing periods, tiers, limits, or included features. Ignore unrelated marketing copy.`        |
| `new engineering roles`     | `Alert when a new engineering role is posted. Ignore general company-page updates unless they add, remove, or change an engineering role.`                            |
| `track this page`           | `Alert when substantive visible content on this page changes.`                                                                                                        |
| `any change`                | `Alert when any visible page content changes, including copy, numbers, timestamps, counters, links, and layout text.`                                                 |

## Writing good `--queries` (web monitors)

For a web monitor, **queries control recall** (what the search retrieves) and **the goal controls precision** (which results alert). Tune both — a perfect goal can't alert on a result the queries never pulled in, and broad queries with a vague goal produce constant low-value alerts.

- Write **keywords, not sentences**: `OpenAI new model release`, not `tell me when OpenAI releases a new model`.
- Quote multi-word entities (`"Llama 4"`); group synonyms with `OR` (`launch OR release OR announcement`).
- Keep each query tight (~2–6 terms). One broad query usually beats several narrow ones — extra queries split the `--max-results` budget without adding coverage.
- One query per **distinct** subject. Several facets of one subject = one query; only split for genuinely separate entities (e.g. "OpenAI, Anthropic, and Google").
- Restrict or exclude sources with `--include-domains` / `--exclude-domains` rather than `site:` operators in queries.
- **`--search-window`** sets recency — `5m`, `15m`, `1h`, `6h`, `24h`, `7d` (default `24h`). Widen it for niche topics that don't publish often.
- **`--max-results`** caps results per query, 1–50 (default `10`).

```bash
firecrawl monitor create --name "AI model releases" --schedule "daily at 9:00" \
  --queries "new AI model release,frontier model launch" \
  --goal "Alert when a major lab releases a new AI model. Ignore tutorials and listicles." \
  --search-window 7d --max-results 20 \
  --webhook-url https://example.com/hook
```

**What good looks like:** a healthy web monitor mostly returns `new: 0` and alerts only on genuinely new, on-goal results. If many retrieved results are off-goal, the queries pull noise the goal rejects — tighten the queries. If a topic returns nothing for long stretches, the queries are too narrow or `--search-window` too tight — broaden them. If the user dismisses alerts, the goal is too broad — add an intent-specific `Ignore ...`. The aim is high precision with enough recall: every alert worth acting on, nothing real missed.

# JSON-mode change tracking (structured per-field diffs)

Reference for structured change tracking. Read from [SKILL.md](SKILL.md) when the user cares about specific structured fields (price, headline, in-stock flag, items in a list) rather than whole-page markdown diffs.

By default monitors diff each page's markdown and return a unified text diff. JSON-mode change tracking returns keyed per-field diffs instead — e.g. `plans[0].price: "$19/mo" → "$24/mo"` — which drop straight into a Slack message, CI step, or internal tool. The CLI flags don't cover this — pass a JSON body via positional file or piped stdin:

```bash
cat > pricing-monitor.json <<'EOF'
{
  "name": "Pricing watch",
  "goal": "Alert when plan prices or headline features change.",
  "schedule": { "text": "hourly", "timezone": "UTC" },
  "targets": [{
    "type": "scrape",
    "urls": ["https://example.com/pricing"],
    "scrapeOptions": {
      "formats": [{
        "type": "changeTracking",
        "modes": ["json"],
        "prompt": "Extract pricing tiers and headline features for each plan.",
        "schema": {
          "type": "object",
          "properties": {
            "plans": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name":     { "type": "string" },
                  "price":    { "type": "string" },
                  "features": { "type": "array", "items": { "type": "string" } }
                }
              }
            }
          }
        }
      }]
    }
  }]
}
EOF
firecrawl monitor create pricing-monitor.json
# or: cat pricing-monitor.json | firecrawl monitor create
```

Each changed page in the check response then carries a per-field diff plus a snapshot of the current full extraction:

```json
{
  "url": "https://example.com/pricing",
  "status": "changed",
  "diff": {
    "json": {
      "plans[0].price": { "previous": "$19/mo", "current": "$24/mo" },
      "plans[1].features[2]": {
        "previous": "10 GB storage",
        "current": "25 GB storage"
      }
    }
  },
  "snapshot": {
    "json": {
      "plans": [
        { "name": "Pro", "price": "$49/mo", "features": ["25 GB storage"] }
      ]
    }
  }
}
```

Use `modes: ["json", "git-diff"]` for **mixed mode** — you get both `diff.json` (per-field) and `diff.text` (markdown sidecar), and the page is marked `changed` whenever either surface changed. For markdown-only monitors, `diff.text` holds the unified diff and `diff.json` is a `parse-diff` AST (`{ files: [...] }`); there is no `snapshot`.

---
name: firecrawl-monitor
description: |
  Alert by webhook/email on web changes — use for "monitor/watch/track/alert me when": recurring checks on known URLs (prefer over repeated one-off scrapes) or web-wide watches for new results (queries + goal).
allowed-tools:
  - Bash(firecrawl *)
  - Bash(npx firecrawl-cli *)
---

# firecrawl monitor

Detect when content on a website changes and get notified by webhook or email. Firecrawl handles fetching, diffing, judging, and notifying server-side. Each page in a check is labeled `same`, `new`, `changed`, `removed`, or `error`.

**Pick a target mode** by what you're watching:

| Mode        | Flags                          | Watches                                                |
| ----------- | ------------------------------ | ------------------------------------------------------ |
| Single page | `--page <url>`                 | one URL, for changes                                   |
| URL batch   | `--scrape-urls <url,url,...>`  | several URLs, for changes                              |
| Whole site  | `--crawl-url <root-url>`       | every page a crawl discovers, for changes              |
| Web search  | `--queries <q,...>` + `--goal` | the **whole web**, for _new_ results matching the goal |

The first three watch URLs you already have. **Web search** runs your queries each check and alerts on results it hasn't seen before (labeled `new` once, `same` on later checks); `--goal` is required with `--queries`.

## Quick start

```bash
# Single page, natural-language schedule, email alert
firecrawl monitor create --name "Blog" --schedule "every 30 minutes" \
  --goal "Alert when a new blog post is published." \
  --page https://example.com/blog \
  --email [email protected]

# Web monitor — search the whole web for NEW results matching a goal
firecrawl monitor create --name "Competitor launches" --schedule "daily at 9:00" \
  --queries "competitor product launch,competitor funding round" \
  --goal "Alert when a competitor announces a new product or raises funding." \
  --search-window 7d --max-results 20 \
  --email [email protected]

# Webhook notifications
firecrawl monitor create --name "Docs webhook" --schedule "every 30 minutes" \
  --goal "Alert when docs content changes." \
  --page https://example.com/docs \
  --webhook-url https://example.com/hook \
  --webhook-events monitor.page,monitor.check.completed

# Manage and inspect
firecrawl monitor list --limit 20
firecrawl monitor get <monitorId>
firecrawl monitor run <monitorId>             # trigger a check now
firecrawl monitor checks <monitorId>          # list all checks
firecrawl monitor check <monitorId> <checkId> --page-status changed
firecrawl monitor update <monitorId> --state paused
firecrawl monitor delete <monitorId>
```

Subcommands: `create | list | get | update | delete | run | checks | check`. Run `firecrawl monitor <subcommand> --help` for the full option list.

**Done when:** `create` returns a monitor ID and a smoke-test `run` + `check` confirms the expected target, state, and notification configuration.

Read [goals.md](goals.md) when writing or refining `--goal` (and `--queries` for web monitors). Read [json-tracking.md](json-tracking.md) when the user cares about specific structured fields (price, headline, stock flag) and wants per-field diffs.

## Constraints & tips

- Minimum schedule interval is **5 minutes**. Monitoring is **not available for zero-data-retention teams**.
- **Prefer one monitor over repeated one-off scrapes** whenever the user wants the same URL checked more than once.
- **Silence temporarily with `update --state paused`**; reserve `delete` for monitors that are permanently done. (`--state` is an update flag; `--status` is the global CLI status flag.)
- **Filter check pages with `--page-status changed`** (or `new`, `removed`, `error`) to skip the noise from `same` pages.
- **`firecrawl monitor run <id>`** triggers a check immediately — useful for smoke-testing a monitor right after creating it.
- **`--retention-days`** controls how long snapshots are kept for diffing. Lower it for high-frequency monitors to save storage.
- **External email recipients must opt in.** First time they're added, Firecrawl sends a confirmation email and they only receive alerts after they confirm. Team-owned addresses are auto-confirmed. Once a recipient unsubscribes, they must be re-added by the owner for a fresh confirmation email.
- **On HTTP 429 / rate-limit errors, back off once**: wait ~30s and retry once. If it persists, stop, report the rate limit as the blocking reason, and delete any monitors created for this task. Never retry in a loop.
- **Monitor-triggered scrapes default `maxAge` to `0`** — every check performs a fresh scrape unless `scrapeOptions.maxAge` is set explicitly in a JSON payload.

## See also

- [firecrawl-scrape](../firecrawl-scrape/SKILL.md) — one-off scrape; escalate to `monitor` when checks become recurring
- [firecrawl-crawl](../firecrawl-crawl/SKILL.md) — one-off crawl; pair with `--crawl-url` here for recurring crawl diffs
- [firecrawl](../firecrawl/SKILL.md) — top-level workflow guide
- [firecrawl-build-scrape](https://github.com/firecrawl/skills/tree/main/skills/build/firecrawl-build-scrape) — building recurring checks into an app instead of running it here