coding-agent-docs-optimization

{
  "skill_name": "coding-agent-docs-optimization",
  "evals": [
    {
      "id": 1,
      "prompt": "I run docs at Halyard, a payments infrastructure company. Two months ago we moved docs.halyard.io behind a new CDN and whoever did the migration left the managed bot-protection ruleset switched on. Since then our Discord has maybe fifteen people saying their AI coding tools \"can't find\" our docs, but the pages load fine for me and analytics traffic looks normal. Our CEO wants an AI-readiness score for the docs by Friday. Can you give me one and tell me what to fix on the pages?",
      "expected_output": "A refusal to score before an access check and a cold run exist, followed by an empirical per-user-agent fetch protocol naming the CDN bot ruleset as the prime suspect, and framing crawler admission as a policy decision outside the docs team.",
      "files": [],
      "expectations": [
        "Refuses to produce an AI-readiness score or a yes/no verdict before an access check and a cold run exist",
        "Places the crawler access check before any page-content audit, stating content findings are meaningless behind a blocked fetch",
        "Specifies fetching both a docs page and the index with each agent user-agent string of interest, recording the HTTP status code per user-agent",
        "Specifies running those fetches from outside Halyard's own network",
        "States the failure is silent: the audit sees a well-built site while the agent sees a challenge page",
        "Distinguishes signalling (per-agent robots.txt rules) from enforcement (WAF, bot management, paid-crawl gate), noting signalling is honour-system and only edge controls actually stop a fetch",
        "Names the CDN's default bot-protection setting as the prime suspect, i.e. a dashboard change rather than a documentation change",
        "States that every surface scores zero until a fetch returns 200, so the audit cannot tell a good corpus from a bad one",
        "Treats admitting a crawler as a policy decision with sign-off outside the docs team rather than a docs-team edit",
        "Notes the asymmetry that the dashboard setting is reversible while the crawling that already happened is not",
        "Does not accept \"pages load fine for me\" or \"analytics traffic looks normal\" as evidence that agents can fetch",
        "Distinguishes training crawlers from search-and-fetch agents as separately nameable user-agents, so one can be admitted while the other is denied",
        "Delivers the access check as a named report section listing which user-agents can fetch which surfaces with status codes, not as prose reassurance"
      ]
    },
    {
      "id": 2,
      "prompt": "My manager handed me a one-liner: \"Write an llms.txt for docs.brightloom.com and ship it this week.\" Brightloom is a product-analytics SDK company. We have 340 docs pages on a Next.js site, HTML only, no markdown versions of anything. We do serve an OpenAPI file at /openapi.json but it was hand-written about two years ago and roughly half the operation descriptions are empty strings. Can you draft the llms.txt?",
      "expected_output": "An inventory-first response that produces the index but states outright it changes nothing measurable alone, ranks markdown endpoints and annotated spec above it, applies the index format rules, and requires advertising in three places.",
      "files": [],
      "expectations": [
        "States plainly that publishing the index file alone changes nothing measurable until the surfaces it points at exist",
        "Does not stop at emitting the file: performs or specifies a surface inventory first",
        "Ranks markdown endpoints and annotated spec files above the index file on efficiency, and says the ranking out loud",
        "Identifies that index entries pointing at HTML-only pages is the defect, since entries should point at markdown URLs so following a link needs no HTML stripping",
        "Requires the index be generated from the navigation source at build time rather than hand-maintained",
        "States that a hand-maintained index rots within about two releases",
        "Specifies the index structure: a single H1 project name, a blockquote summary of what the project is, then H2 sections whose items are links optionally followed by a routing note",
        "Notes that an H2 section literally titled \"Optional\" is reserved for links an agent may skip when working in a shorter context",
        "Orders index sections by task frequency rather than by site hierarchy",
        "Writes each entry note as a routing signal describing what the reader will find, not as marketing copy",
        "Requires advertising the index in at least three places: the repository README, the docs navigation, and response headers or link relations",
        "Names the empty description fields in /openapi.json as a finding, since an empty description is where invented parameters come from",
        "Flags the hand-written spec as needing generation from the implementation rather than hand-editing"
      ]
    },
    {
      "id": 3,
      "prompt": "Corvid Data here, we sell a streaming data platform. Our VP of Engineering wants a number in the OKR doc: \"coding agents complete 95% of integration tasks from our docs on the first try by Q1.\" Separately I want to open the budget slide with the stat that AI agents are now 66% of documentation web traffic, to justify the headcount ask. Our task set is 9 integration jobs: 3 are basic REST CRUD, the other 6 are deep things like our streaming replay API, custom partitioner, and schema-registry failover. What number should I actually put in, and does the traffic stat hold up?",
      "expected_output": "A rejection of the 95% target calibrated against published benchmark bands, a warning that 80% is a stretch goal and best-of-three not first-attempt, category scoring given the depth-heavy mix, and a refusal to present the traffic figure as an industry fact.",
      "files": [],
      "expectations": [
        "Rejects 95% first-attempt unattended as a target and says so directly",
        "Cites the published benchmark bands: backend-only roughly 73-85%, full-stack roughly 77-92%, depth-of-feature roughly 61-73%",
        "States those published figures were scored best-of-three rather than first-attempt, so they are not directly comparable to an unattended single-attempt target",
        "Presents 80% first-attempt as a stretch goal rather than a floor or minimum acceptable standard",
        "Warns explicitly against letting a stakeholder hear the stretch goal as a minimum",
        "Requires scoring by category rather than reporting one blended average across the 9 tasks",
        "States that depth-of-feature tasks land roughly 15-25 points below breadth tasks for every model",
        "Concludes that Corvid Data's 6-deep / 3-basic mix scores lower by construction and that this is not a regression",
        "States that Corvid Data's own first baseline run is the only baseline that means anything, with iteration-over-iteration delta on a frozen set as the comparison that survives",
        "Refuses to present the 66% agent-traffic figure as an industry or population statistic",
        "Identifies that figure as self-reported by a single docs-platform vendor and measured across sites that had already invested in agent surfaces",
        "Permits using it as motivation while stating it is not evidence",
        "Requires recording the model identifier and the date alongside any score, since a score is comparable only within one generation of agents"
      ]
    },
    {
      "id": 4,
      "prompt": "Meridian Freight, logistics API. Here is my plan for testing our docs with an agent, tell me if it's sound. I'll open one long session and work through all eight tasks back to back so it keeps the context about our OAuth setup instead of re-learning it every time. I'll give it full web access so it can also check our GitHub issues and Stack Overflow. The first three tasks I wrote: (1) \"Call meridian.shipments.track() with a tracking number and print the status\", (2) \"Explain how our OAuth refresh flow works\", (3) \"Add webhook signature verification to an existing Express server.\" At the end I record pass/fail per task and report the percentage to the team. Anything I'm missing?",
      "expected_output": "A rewrite of the protocol: fresh context per task, sources restricted to published surfaces, two of the three sample tasks rejected on shape, the frozen set preserved, and five metrics rather than a pass percentage.",
      "files": [],
      "expectations": [
        "Rejects running all eight tasks in one session and requires a fresh context per task",
        "States the reason: knowledge leaks between tasks and inflates every score after the first",
        "Offers one sub-agent per task where parallel sub-agents are available, and separate sequential sessions otherwise",
        "Rejects full web access and restricts sources to Meridian's own published agent-facing surfaces",
        "States that if the agent needs something Meridian does not publish, that absence is the finding and the run should be allowed to fail",
        "Rejects task 1 as a weak task because it hands the agent the exact method name it was supposed to discover",
        "Rejects task 2 as ungradeable because it has no runnable or verifiable end state",
        "Endorses the webhook verification task shape because it names an outcome and leaves the path open",
        "Rejects a pass/fail percentage as the sole reported metric",
        "Names invented-surface rate, corrective turns per task, context cost per task, and fetch-path length alongside first-attempt success",
        "States that success rate alone hides a corpus that only works after several expensive retries",
        "Requires freezing the task set, because a benchmark that changes between runs measures nothing",
        "Requires capturing the full transcript, since the corrective turns are the data and the final file is only the receipt",
        "Limits intervention to non-documentation obstacles such as a missing sandbox credential or a network block, and requires logging every intervention"
      ]
    },
    {
      "id": 5,
      "prompt": "Larkspur, HR and payroll API. We have a 140-operation OpenAPI 3.1 document that is generated from our Go handlers on every build. I've booked the whole next sprint for two technical writers to go through and write a rich multi-paragraph description on every single operation, on the theory that the description is what the model actually reads. Does that sound like the right use of the sprint? For context: about 90 of the 140 operations have no operationId at all, and most of our query parameters are plain `type: string` with no constraints.",
      "expected_output": "A re-sequenced plan putting operationId first, value-space constraints second, curation third, and the description prose fourth, with the reasoning for each position and the operationId naming rules.",
      "files": [],
      "expectations": [
        "Re-sequences the work rather than endorsing descriptions-first",
        "Puts operationId first and gives the reason: generators derive the agent-facing tool name from it",
        "Describes the operationId work as near-zero effort and mechanical",
        "States the auto-generated fallback produces names like `get_api_v1_users_id` that models handle badly",
        "Specifies verb-noun camelCase for operationId, with concrete examples such as `listCustomers`, `getUserById` or `createRefund`",
        "Caps operationId at roughly 30 characters and drops articles and conjunctions such as \"a\", \"the\", \"and\", \"for\"",
        "Requires operationId uniqueness and stability across releases, noting that a renamed operationId silently renames a tool",
        "Recommends enforcing operationId uniqueness in CI with a spec linter",
        "Puts closing the value space second, naming enums, explicit `required`, `default` values, `format` keywords, and `oneOf` with a discriminator",
        "States that an unconstrained field is where an invented value goes, and that each constraint is a one-line edit removing a whole class of them",
        "Ranks the summary/description split below those cheaper edits and states it costs on the order of a week on a real spec",
        "Splits summary and description by job: summary is one imperative line for choosing the call",
        "Assigns description the operational contract: preconditions, side effects, idempotency, required auth scope, rate limits, and a pointer to the sibling operation that might fit better",
        "Places per-language runnable samples attached to the operation last in the default order, while promoting them to first for the two or three operations cold runs fail on repeatedly"
      ]
    },
    {
      "id": 6,
      "prompt": "Quillon, we sell a deployment platform. We want agents to be able to call our API directly. The plan is to run our whole OpenAPI file, 162 operations, through an openapi-to-mcp converter, host the resulting MCP server on our own infra behind an API key header, and point customers at it. Can you build the conversion and set up the server for us? For reference our operationIds look like `GetV2ProjectsProjectIdDeploymentsDeploymentIdLogs`.",
      "expected_output": "A refusal to build or operate the server, a handoff for that part, and spec curation work done in scope: cut the exposed surface well under 162, fix the operationId shape, and cover generator naming behaviour.",
      "files": [],
      "expectations": [
        "Declines to build or operate the MCP server itself, naming that as product work rather than documentation work",
        "Hands the server build and hosting off to the platform-side skill rather than attempting it",
        "Keeps curating the spec before conversion in scope and performs that work",
        "Rejects converting all 162 operations wholesale into an exposed tool surface",
        "Cites the guidance of fewer than 20 functions available at the start of a turn",
        "Recommends tag filtering or vendor extensions to expose only what agents need past roughly 20-30 operations",
        "States that curated tool surfaces outperform auto-converted OpenAPI servers, especially for complex APIs with many endpoints",
        "Identifies `GetV2ProjectsProjectIdDeploymentsDeploymentIdLogs` as too long and not verb-noun, and proposes a shorter verb-noun replacement",
        "Notes the converter name-resolution order: `x-mcp-tool-name`, then `operationId`, then a `{method}_{path}` fallback",
        "Notes that a tool whose name fails the `^[a-zA-Z0-9_-]{1,64}$` pattern is discarded",
        "Notes operationId truncation at the first `__` and a 56-character cap in at least one named generator",
        "Recommends `x-mcp-tool-name` or an equivalent vendor extension when the SDK wants a terse method name and the agent wants a descriptive one, rather than compromising on a single string",
        "Warns that renaming an operationId silently renames the exposed tool for existing consumers"
      ]
    },
    {
      "id": 7,
      "prompt": "I maintain ferroquant, a Rust crate with about 4,100 stars, in my evenings. Documentation is docs.rs plus a hand-written book published on GitHub Pages. I don't own a CDN account, I can't set response headers, and there is no docs team or build pipeline I control beyond a CI job that runs mdbook. Three people have now opened issues saying ChatGPT wrote them code calling `ferroquant::Engine::new_with_config`, which we removed back in 0.9. What do I actually do here?",
      "expected_output": "A plan led by repo-root and source-level artefacts, explicitly ruling out header and CDN surfaces, plus in-page deprecation labelling at the top of the page and Rust-native doc comments with doctests.",
      "files": [],
      "expectations": [
        "Leads with repo-root artefacts rather than response headers or CDN controls",
        "Explicitly rules out the surfaces requiring header or CDN control given GitHub Pages, rather than recommending them and leaving the maintainer to discover they are impossible",
        "Recommends generated spec or published type artefacts and doc comments in the SDK source as the reachable surfaces",
        "Recommends a versioned index committed alongside releases rather than one generated by a hosted docs build",
        "Recommends encoding the page rules in the contribution guide so incoming pull requests do not undo them",
        "States that the cold-run protocol, the task set, and the page rules are identical regardless of ownership model, and does not re-derive them for the open-source case",
        "Addresses the removed `new_with_config` API with an in-page deprecation label naming the replacement call",
        "Places that label at the top of the page rather than in a footnote, because an agent that got its answer in the first paragraph never reaches the footnote",
        "States that models carry confident memories of removed APIs and that only an in-page label overrides that prior",
        "Recommends Rust `///` doc comments as the language's native convention rather than a generic prose format",
        "Recommends doctests, since a doc-comment example the toolchain compiles and runs cannot rot without failing the build",
        "Recommends keeping previous major versions reachable at distinct URLs rather than overwriting them"
      ]
    },
    {
      "id": 8,
      "prompt": "Osprey Analytics. A big partner integration goes live on the 14th of next month and Northwind's engineering team starts wiring our SDK into their stack in nine days. Their engineers use Cursor for basically everything. I'm the only technical writer here, I get zero engineering time this quarter, and our docs build is owned by the web team, who currently have a six-week ticket queue. Give me the plan.",
      "expected_output": "A plan re-ranked against the deadline and the effort ceiling, naming which constraint moved which surface, deleting the surfaces the constraints rule out rather than demoting them, and recording the deletions.",
      "files": [],
      "expectations": [
        "Produces a plan re-ranked against Osprey's constraints rather than reciting the default surface order",
        "Names explicitly which constraint moved which surface",
        "Drops every surface needing a docs build change, given the nine-day deadline against the web team's six-week queue",
        "Drops every surface needing another team's release cycle",
        "Deletes SDK doc comments from the plan outright rather than parking them at the bottom, because the writer has no engineering time",
        "Deletes terminal and package-manager paths from the plan on the same grounds",
        "Records which surfaces were deleted and why, rather than leaving them silently absent",
        "States the reason for deletion rather than demotion: a ruled-out surface parked at the bottom of a plan silently reappears as scope",
        "Runs the crawler access check first regardless of deadline pressure",
        "Builds the frozen task set around the specific integration jobs Northwind's team will actually perform",
        "Keeps the deliverable an audit report and names which report sections will be empty and why, given the reduced scope",
        "States that this plan buys one release's worth of fixes rather than a standing guarantee, and names what a standing mandate would have promoted instead",
        "Prioritizes page-level edits reachable through a content commit, since those ship without the web team"
      ]
    },
    {
      "id": 9,
      "prompt": "Read through our docs myself over two days and found 12 problems at Thistlewood (we sell a billing API). Here they are ranked by how fast I can fix each one: (1) inconsistent code fence languages on the CLI pages, (2) our Concepts page uses `sdk` as the variable name while every other page uses `client`, (3) the changelog page is three releases out of date, (4) the billing-webhook page's only example is a fragment with no imports and no client construction, (5) three pages still say v2 in the title, (6) our errors page lists error names but not the exact strings we emit, (7) the enterprise SSO setup page is dashboard clicks only, (8) two broken links in the footer, (9) the pagination page explains cursor design for four paragraphs before showing a call, (10) the glossary isn't alphabetized, (11) there's no Java quickstart, (12) the team page photos are stale. I also ran a cold run over 6 tasks and 4 failed: billing webhook, pagination, enterprise SSO, and rate-limit retry. Which do I do first?",
      "expected_output": "A findings list admitted by task impact rather than ranked by fix speed, with the cosmetic items excluded outright, the admitted ones ordered by fixes-per-hour, concrete rewrites, and the unmatched failing task flagged.",
      "files": [],
      "expectations": [
        "Admits findings by task impact first: only findings touching a task that actually failed enter the list",
        "Excludes items 1, 3, 8, 10 and 12 from the findings list entirely rather than merely ranking them last",
        "States the exclusion rule explicitly: a finding on a task no cold run failed does not make the list at all, however easy it would be to fix",
        "Admits the billing-webhook fragment (4), the pagination ordering (9), and the dashboard-clicks SSO page (7) because each maps to a failed task",
        "Orders the admitted findings by fixes-per-hour rather than by the user's speed-of-fix ranking",
        "States that a page breaking a top revenue-driving task outranks ten cosmetic fixes because its value is higher, not because effort stopped counting",
        "Fixes item 4 with a complete snippet carrying imports, client construction, the call, the awaited result, and the error branch",
        "Fixes item 9 by reordering the page to the working call, then what it returns, then the parameters, then the rationale",
        "Fixes item 7 by adding the equivalent command or API call while keeping the dashboard click path as the human alternative",
        "Treats item 2 as a naming-discipline issue (one identifier per concept corpus-wide) and admits or excludes it on whether it touched a failed task",
        "Fixes item 6 by mapping the exact error text that reaches stderr to a cause, a fix, and a verification step",
        "Flags that the rate-limit retry task failed but has no corresponding entry in the user's list, so a finding still needs to be written for it",
        "Writes each admitted finding naming the failing task, the page or spec field involved, and what the agent did instead"
      ]
    },
    {
      "id": 10,
      "prompt": "Vantage Grid, industrial IoT platform. Our docs are about 1,400 pages spread across four product lines (Edge, Fleet, Historian, Studio). Someone on the team generated an llms-full.txt over the weekend and it came out at 6.2 MB. Our llms.txt is 180,000 characters. I want to ship both this week and link them from the docs footer so people can find them. Good plan?",
      "expected_output": "A refusal to ship the bundle, with deletion or per-product-line splitting rather than backlogging it, the oversized index split into child indexes, and advertising moved beyond the footer to three surfaces.",
      "files": [],
      "expectations": [
        "Refuses to ship the 6.2 MB full-corpus bundle as it stands",
        "States that a truncated bundle is worse than no bundle, because the agent believes it read everything",
        "States that past the working context budget the bundle crowds out the user's own code",
        "Recommends deleting the bundle outright, or splitting it per product line, rather than ranking it last or moving it to a backlog",
        "States the reason for deleting rather than deferring: a bundle parked at the bottom of a plan is a defect waiting for someone with spare time",
        "Flags the 180,000-character index as past the split threshold of roughly 100,000 characters",
        "Recommends splitting the index into child indexes and linking those, so an agent can descend instead of truncating",
        "Rejects the docs footer as sufficient advertising and requires the README, the docs navigation, and response headers or link relations",
        "Notes that an agent typically fetches these surfaces because a human, a rules file, or a tool pointed at them, not through passive discovery",
        "Ranks markdown endpoints above both the index file and the full-corpus bundle on efficiency",
        "Notes the index and the full-corpus bundle tie on effort because both generate from the navigation source in one build step, while tying on no other axis",
        "States that the index changes nothing measurable until the surfaces it points at exist"
      ]
    },
    {
      "id": 11,
      "prompt": "Kestrel Streams, we do managed event streaming. We ship a first-party TypeScript SDK with full .d.ts type declarations published to npm. I ran a baseline cold run over 7 tasks and the interesting part is that in 5 of the 7 the agent never fetched a single docs page - it read our types straight out of node_modules and worked from those, then got the consumer-group option wrong because the JSDoc on `subscribe()` is a single line saying \"Subscribes to a topic.\" We also publish an AsyncAPI 3.0 document for our event payloads that nobody has ever annotated. Where should the effort go?",
      "expected_output": "SDK doc comments promoted above every other surface with the promoting condition named, TypeScript-native conventions applied, and the AsyncAPI annotation offered with an explicit analogy caveat.",
      "files": [],
      "expectations": [
        "Promotes SDK doc comments above every other surface for Kestrel rather than applying the default efficiency order",
        "Names the promoting condition explicitly: the product ships typed SDKs and the cold runs show the agent reading local code instead of fetching pages",
        "States that doc comments are otherwise what the default order starves, because every pull request can drift them so they never win on ratio",
        "States that doc comments are the only surface reaching an agent working offline against installed types",
        "Notes that web docs and SDK source are two different surfaces reaching two different agents",
        "States that the signature and its adjacent doc comment are what an in-editor agent most reliably sees, while a reference page three hops away is invisible to it",
        "Prescribes JSDoc/TSDoc as the native convention for TypeScript rather than a generic comment format",
        "Requires fully typed signatures and descriptive names, not only added prose",
        "Applies the OpenAPI-derived annotation rules to the AsyncAPI document while stating plainly that AsyncAPI guidance is inferred from the OpenAPI case by analogy rather than separately sourced",
        "Presents the AsyncAPI recommendation with lower confidence than the OpenAPI ones rather than with equal confidence",
        "Warns that `$comment` is non-rendered and intended for schema maintainers, so anything an agent must see belongs in `description` or `examples`",
        "Notes that terminal and package-local paths tie with doc comments on both effort and value, since each ships on the SDK release cycle and each reaches only the agent already inside the user's codebase",
        "Requires re-running the identical frozen 7-task set and comparing corrective turns against the baseline rather than declaring the fix successful"
      ]
    }
  ],
  "trigger_queries": [
    { "query": "Our developer docs need to be AI-ready before the next release.", "should_trigger": true },
    { "query": "can a coding agent actually integrate our SDK from the docs alone?", "should_trigger": true },
    { "query": "write an llms.txt for docs.northwind.dev", "should_trigger": true },
    { "query": "what belongs in llms-full.txt and is it worth publishing?", "should_trigger": true },
    { "query": "users keep telling us Cursor hallucinates methods that don't exist in our Python SDK", "should_trigger": true },
    { "query": "I'd like an agent experience audit of our API documentation.", "should_trigger": true },
    { "query": "how do we improve AX on our developer platform?", "should_trigger": true },
    { "query": "should we be serving a markdown version of every docs page?", "should_trigger": true },
    { "query": "our OpenAPI file has no operationIds anywhere. does that actually matter?", "should_trigger": true },
    { "query": "Claude keeps inventing query parameters on our /v1/search endpoint", "should_trigger": true },
    { "query": "how would I measure whether an LLM can wire up our API without a human helping?", "should_trigger": true },
    { "query": "set up a benchmark where a fresh model integrates our product using only our published docs", "should_trigger": true },
    { "query": "customers' AI assistants keep writing code against an SDK version we removed two years ago", "should_trigger": true },
    { "query": "documentation for coding agents", "should_trigger": true },
    { "query": "make our documentation machine-readable", "should_trigger": true },
    { "query": "is publishing an llms.txt file actually worth the effort?", "should_trigger": true },
    { "query": "I think Cloudflare is blocking AI crawlers on our docs subdomain, how do I confirm that?", "should_trigger": true },
    { "query": "which AI user-agents should our docs site allow through?", "should_trigger": true },
    { "query": "why do agents only ever get our auth setup half right?", "should_trigger": true },
    { "query": "we want to start measuring first-attempt integration success", "should_trigger": true },
    { "query": "our snippets are fine for humans but agents paste them and they crash immediately", "should_trigger": true },
    { "query": "should we annotate our OpenAPI descriptions specifically for LLMs?", "should_trigger": true },
    { "query": "give me an agent-readiness score for our docs site", "should_trigger": true },
    { "query": "how are other companies making their docs work well with Copilot?", "should_trigger": true },
    { "query": "I want ours to be the SDK the model picks and then gets right", "should_trigger": true },
    { "query": "put together something so every release keeps our docs agent-readable, not just this one", "should_trigger": true },
    { "query": "we shipped llms.txt six weeks ago and nothing measurable changed", "should_trigger": true },
    { "query": "our support queue is full of tickets where an AI wrote the integration wrong", "should_trigger": true },
    { "query": "audit our docs from the perspective of a bot rather than a person", "should_trigger": true },
    { "query": "highest-leverage thing we can do so models stop guessing our API surface?", "should_trigger": true },
    { "query": "should we add x-codeSamples to our spec?", "should_trigger": true },
    { "query": "does it matter that our getting-started page tells you to click through the dashboard?", "should_trigger": true },
    { "query": "our docs are entirely client-rendered React. can an agent even read them?", "should_trigger": true },
    { "query": "how should we label deprecated endpoints so models stop recommending them?", "should_trigger": true },
    { "query": "we need a repeatable harness that proves the docs work before each release ships", "should_trigger": true },
    { "query": "review our developer docs for anything an unattended integration would trip over", "should_trigger": true },
    { "query": "how many pages should an agent have to read to finish a routine task?", "should_trigger": true },
    { "query": "what should our spec's description fields say that they currently don't?", "should_trigger": true },
    { "query": "our SDK is generated from the spec and the generated method names are awful", "should_trigger": true },
    { "query": "we're a volunteer-run open source project with no CDN. how do we do this?", "should_trigger": true },
    { "query": "should we ship the docs inside the npm package itself?", "should_trigger": true },
    { "query": "our CLI ought to be able to print the relevant doc page to stdout, right?", "should_trigger": true },
    { "query": "help me set a realistic target for unattended integration success", "should_trigger": true },
    { "query": "the CTO is asking whether 90% first-attempt is realistic for us", "should_trigger": true },
    { "query": "someone told me bots are two-thirds of docs traffic now. can I put that in the budget deck?", "should_trigger": true },
    { "query": "how do I prove the docs rewrite actually helped?", "should_trigger": true },
    { "query": "we rewrote 40 pages and the agent still fails the webhook task", "should_trigger": true },
    { "query": "our docs are good but the bot never actually sees them", "should_trigger": true },
    { "query": "make the docs work for the robot reading them, not the human", "should_trigger": true },
    { "query": "our onboarding is broken for anyone using an AI coding tool", "should_trigger": true },
    { "query": "what do we need to publish so an integration can self-serve end to end?", "should_trigger": true },
    { "query": "our webhook verification page has no runnable example and it shows", "should_trigger": true },
    { "query": "rank the work for me: markdown endpoints, llms.txt, doc comments, or spec descriptions?", "should_trigger": true },
    { "query": "our Rust crate's doc comments are thin. does that affect AI completions?", "should_trigger": true },
    { "query": "does an agent care about the JSON Schema files we publish?", "should_trigger": true },
    { "query": "we need a docs strategy for the era where the reader is a model", "should_trigger": true },
    { "query": "how do we stop agents inventing endpoint names that were never in our API?", "should_trigger": true },
    { "query": "score our documentation the way an agent would experience it", "should_trigger": true },
    { "query": "what goes wrong when a model reads our docs?", "should_trigger": true },
    { "query": "design a test where a fresh model tries to integrate us and we count the retries", "should_trigger": true },
    { "query": "a partner's integration was written entirely by an agent and took four days. how do we fix that?", "should_trigger": true },
    { "query": "which parts of our docs should we fix first for automated consumers?", "should_trigger": true },
    { "query": "our AsyncAPI file: same annotation rules as OpenAPI or not?", "should_trigger": true },
    { "query": "where exactly should we advertise our markdown endpoints?", "should_trigger": true },
    { "query": "our index file points at pages that have no markdown version", "should_trigger": true },
    { "query": "the agent read our docs and still called a function that doesn't exist", "should_trigger": true },
    { "query": "audit our docs structure against Diátaxis and tell me which pages are mixed-mode", "should_trigger": false },
    { "query": "our how-to guides and reference pages are all jumbled together in the nav", "should_trigger": false },
    { "query": "write a quickstart that gets a new developer to first success in five minutes", "should_trigger": false },
    { "query": "nobody finishes our getting-started page, they drop off at step four", "should_trigger": false },
    { "query": "write a tutorial that teaches Kafka consumers by building a small pipeline", "should_trigger": false },
    { "query": "our tutorial introduces too many concepts per step and learners get lost", "should_trigger": false },
    { "query": "set a policy for code samples in our docs and audit the existing corpus against it", "should_trigger": false },
    { "query": "our docs snippets stopped compiling after the v5 release", "should_trigger": false },
    { "query": "how do we test our documentation examples in CI?", "should_trigger": false },
    { "query": "turn our support tickets into proper troubleshooting pages", "should_trigger": false },
    { "query": "build an error-code catalog from our error telemetry", "should_trigger": false },
    { "query": "write the migration guide for our v3 to v4 breaking changes", "should_trigger": false },
    { "query": "what codemod coverage should we promise for the upgrade path?", "should_trigger": false },
    { "query": "our docs don't rank on Google for \"kafka consumer group lag\"", "should_trigger": false },
    { "query": "the v2 docs outrank our current version in search results", "should_trigger": false },
    { "query": "fix canonicals and hreflang across our versioned and translated docs", "should_trigger": false },
    { "query": "how do we get ChatGPT to cite our blog posts in its answers?", "should_trigger": false },
    { "query": "we want to show up in Google AI Overviews for our category", "should_trigger": false },
    { "query": "improve our brand's share of voice in Perplexity answers", "should_trigger": false },
    { "query": "write an AGENTS.md so Claude Code knows how to run our test suite", "should_trigger": false },
    { "query": "our repo's contributor-facing agent instructions file is badly out of date", "should_trigger": false },
    { "query": "set up CLAUDE.md conventions across our monorepo packages", "should_trigger": false },
    { "query": "deploy and host an MCP server for us with OAuth and per-tenant rate limiting", "should_trigger": false },
    { "query": "which integration surface should we build first, GraphQL or gRPC?", "should_trigger": false },
    { "query": "should we offer a bulk data sharing surface to enterprise customers?", "should_trigger": false },
    { "query": "audit every endpoint in our API reference for missing response codes", "should_trigger": false },
    { "query": "our try-it console is broken on about half the endpoints", "should_trigger": false },
    { "query": "design the error response format for our public API", "should_trigger": false },
    { "query": "what should our rate limit policy be for the free tier?", "should_trigger": false },
    { "query": "how should we version our public API going forward?", "should_trigger": false },
    { "query": "design idempotency keys for our payments endpoint", "should_trigger": false },
    { "query": "how should we handle API key rotation for customers?", "should_trigger": false },
    { "query": "write our release notes for v4.2", "should_trigger": false },
    { "query": "our CHANGELOG.md is just raw commit messages, clean it up", "should_trigger": false },
    { "query": "optimize our GitHub README for first-time visitors", "should_trigger": false },
    { "query": "write an engineering blog post about our storage engine rewrite", "should_trigger": false },
    { "query": "find the keywords developers actually search for in our category", "should_trigger": false },
    { "query": "which devrel metrics should we report to the board this quarter?", "should_trigger": false },
    { "query": "plan our developer community launch on Discord", "should_trigger": false },
    { "query": "help me submit a CFP for KubeCon", "should_trigger": false },
    { "query": "our npm package needs a better publishing pipeline", "should_trigger": false },
    { "query": "pick a license for our new open-source project", "should_trigger": false },
    { "query": "evaluate our own agent's tool-calling accuracy on a benchmark", "should_trigger": false },
    { "query": "refactor our agent markdown files into a cleaner directory structure", "should_trigger": false },
    { "query": "build a RAG pipeline over our documentation to power a support bot", "should_trigger": false },
    { "query": "add a chat widget to our docs site so users can ask questions", "should_trigger": false },
    { "query": "our internal engineering wiki is a mess, help me reorganize it", "should_trigger": false },
    { "query": "write API documentation for the six endpoints we shipped last week", "should_trigger": false },
    { "query": "translate our documentation into Japanese and Korean", "should_trigger": false },
    { "query": "our docs site takes four seconds to load, make it faster", "should_trigger": false },
    { "query": "should we move our docs from Docusaurus to Mintlify?", "should_trigger": false },
    { "query": "improve screen reader accessibility on our documentation site", "should_trigger": false },
    { "query": "our robots.txt is accidentally blocking Googlebot from the docs subdirectory", "should_trigger": false },
    { "query": "add structured data markup so our pages get rich results in search", "should_trigger": false },
    { "query": "design the information architecture for our new marketing site", "should_trigger": false },
    { "query": "our OpenAPI document doesn't validate against the 3.1 schema", "should_trigger": false },
    { "query": "generate TypeScript types from our OpenAPI spec", "should_trigger": false },
    { "query": "our SDK release process is manual and error-prone, automate it", "should_trigger": false },
    { "query": "write a customer case study about how Acme integrated us", "should_trigger": false },
    { "query": "how do we measure time to first hello world for human developers?", "should_trigger": false },
    { "query": "our developer survey says docs are the top complaint. what now?", "should_trigger": false },
    { "query": "segment our developer audience for next quarter's campaign", "should_trigger": false },
    { "query": "plan the content calendar for our devrel team next quarter", "should_trigger": false },
    { "query": "review this pull request that adds a new API endpoint", "should_trigger": false },
    { "query": "our docs search returns irrelevant results, fix the search index", "should_trigger": false },
    { "query": "set up analytics so we can see which docs pages people actually read", "should_trigger": false }
  ]
}

# Agent-facing documentation surfaces

Each surface, its format, what it unblocks, and when it is not worth building.

## Table of Contents

- [Access before discovery](#access-before-discovery)
- [Index file](#index-file)
- [Getting started](#getting-started)
- [Reference](#reference)
- [Optional](#optional)
- [Full-corpus bundle](#full-corpus-bundle)
- [Markdown endpoints](#markdown-endpoints)
- [Machine-readable specs](#machine-readable-specs)
- [SDK source as a surface](#sdk-source-as-a-surface)
- [Version and deprecation signals](#version-and-deprecation-signals)
- [Terminal and package paths](#terminal-and-package-paths)
- [Advertising the surfaces](#advertising-the-surfaces)
- [Prioritization](#prioritization)

## Access before discovery

Check first that the agents you are optimizing for can fetch the pages at all. Every surface below is wasted effort behind a crawler policy that refuses the request, and this failure is silent - the audit sees a well-built index, the agent sees a challenge page.

Two layers, often owned by different teams:

- **Signalling.** `robots.txt` now carries per-agent rules, and the named user-agents split training crawlers from the search-and-fetch agents a docs site usually wants to admit. Allowing one while denying the other is a deliberate, common choice; making it by accident is not. An IETF working group is standardizing a richer vocabulary attached to a response header and to `robots.txt`, but the draft is not final - track it rather than depend on it.
- **Enforcement.** Signalling is honour-system: it constrains a compliant crawler and nothing else. Only edge controls - a WAF, bot management, a paid-crawl gate - actually stop a fetch. The inverse matters more here: some CDNs block AI crawlers by default on new properties, so a docs site can ship every surface in this file and still fail every cold run until someone changes a dashboard setting.

Verify empirically rather than by reading policy. Fetch a docs page and the index with each agent user-agent string you care about, from outside your network. Record the status code in the surface inventory.

## Index file

A root-level markdown index telling an agent which documents to read, following the open `llms.txt` convention (llmstxt.org). Fixed structure, in order:

1. One H1 with the project name - the only required section.
2. A blockquote summary carrying the key facts about what the project is.
3. Optional heading-free paragraphs or lists for extra context.
4. Zero or more H2 sections, each a markdown list where every item is a link, optionally followed by `:` and a note.

An H2 section literally titled `Optional` is reserved: it marks links an agent may skip when working in a shorter context.

```markdown
# Acme Payments SDK

> Server-side SDK for charging cards, issuing refunds and verifying webhooks. Current major version: v4.

## Getting started

- [Install and authenticate](https://docs.example.com/install.md): package names per language, key setup, first request
- [Charge a card](https://docs.example.com/charge.md): complete runnable example with the error branch

## Reference

- [OpenAPI specification](https://docs.example.com/openapi.v4.json): full request/response schema, v4
- [Error codes](https://docs.example.com/errors.md): every error string, its cause, and its fix

## Optional

- [Architecture rationale](https://docs.example.com/design.md): why the client is stateless
```

Rules that matter more than the format:

- Point entries at markdown URLs, so following a link needs no HTML stripping.
- Write each note as a routing signal ("what will I find here"), not marketing copy.
- Generate it from the navigation source at build time. A hand-maintained index rots within two releases.
- Order sections by task frequency, not by site hierarchy.
- Keep it an index. Once it grows past roughly 100k characters, split it into child indexes and link those, so an agent can descend instead of truncating.

## Full-corpus bundle

One file concatenating every page's title, URL, description, and full markdown body, published beside the index. It exists to be pasted wholesale into a context window.

Publish it when the corpus fits comfortably inside a working context budget alongside the user's own code. Skip it - or split it per product line - when it does not: a bundle that gets truncated mid-page is worse than no bundle, because the agent believes it read everything.

## Markdown endpoints

The highest value per hour of work: serve a clean markdown version of every page at a predictable URL. Two accepted conventions, both from the same spec:

- Append `.md` to the page URL.
- Replace the extension with `.md`.

Advertise them so an agent can find them without guessing:

- `<link rel="alternate" type="text/markdown" href="...">` on the HTML page.
- `<link rel="describedby" href="/llms.txt">` pointing at the covering index.
- A `Link` response header carrying the same relations, for agents that never parse the HTML.

Prepend a one-line pointer to the index at the top of each markdown page. An agent that lands mid-corpus from a search result then still finds its way to the map - a pattern visible today on several large vendor doc sites.

## Machine-readable specs

Anything already machine-parseable is ground truth an agent should never have to infer from prose:

- OpenAPI or AsyncAPI documents.
- GraphQL SDL.
- Protobuf definitions.
- JSON Schema.
- Published type declarations.

Rules for serving them:

- Serve them at a stable, versioned URL and link them from the index.
- Keep them generated from the implementation, never hand-edited.
- Fill in `description` and `example` fields - an agent reads them as the contract, and an empty description is where invented parameters come from.
- Prefer one spec per major version over one spec with conditional prose.

Which fields to fill, and how they differ from the same fields written for a human or for SDK generation, is the subject of the spec-annotation reference listed in SKILL.md. Publishing an unannotated spec buys less than it looks like it does.

## SDK source as a surface

Web docs and SDK source are two different surfaces reaching two different agents.

- A research agent fetches pages.
- An in-editor completion tool assembles context from the file being edited and its neighbouring open files.

The signature and its adjacent doc comment are what that second agent most reliably sees. A reference page three hops away is invisible to it.

- Write doc comments in the language's native convention, since that is what tooling and models expect: docstrings plus type hints in Python, `///` in Rust, JSDoc/TSDoc, `//` above the declaration in Go, Javadoc tags in Java.
- Type the signature fully and name things descriptively. arXiv 2403.12671 measured this: meaningful names plus docstrings beat name-only and dummy-name baselines for completion quality.
- Prefer doc-comment examples the toolchain compiles and runs, where the language supports it - a Rust doctest cannot rot without failing the build.
- Audit the generated SDK's method names and docstrings as a cheap proxy for spec quality when the SDK is generated from the spec: every naming and description weakness in the spec shows up there.

## Version and deprecation signals

- Publish a machine-resolvable "current version" pointer the agent can fetch in one request.
- Carry the version inside each page (frontmatter or a first-line note), because pages get read out of navigation context.
- Label deprecated surface at the top of the page with the replacement call. Models trained on older corpora reproduce removed APIs confidently, and only an in-page label overrides that prior.
- Keep previous major versions reachable at distinct URLs rather than overwriting them.

## Terminal and package paths

Agents work in a shell more often than in a browser. Two paths remove a whole class of fetch failures:

- A documentation command in your CLI that prints the relevant page as markdown to stdout.
- Docs shipped inside the package itself, so an installed dependency carries its own reference offline.

Optional integration note: vendors increasingly also publish an installable skill for their own SDK and expose a machine-readable catalogue of those skills at a well-known path. Treat this as a distribution channel for the same content, not as a substitute for the surfaces above - the content has to be right first.

## Advertising the surfaces

Publishing is half the job; nothing fetches a file it does not know about. Advertise in at least three places:

- A line in the repository README naming the index URL and the markdown-endpoint convention.
- A visible entry in the documentation navigation, so a human can hand the URL to their agent.
- Response headers and link relations, for tooling that never renders the page.

Be honest about reach: an agent typically fetches these because a human, a rules file, or a tool pointed at them. Treat every surface as advertised inventory, not as passive discovery.

## Prioritization

The build order lives in SKILL.md § Which surface to build first, next to the decision it serves. It is a judgement about effort against impact, not a measured ranking - reorder it if your own cold run says otherwise, and record what the run showed.

This file describes what each surface is and when it fails to earn its keep; it does not carry a second ordering.

# Cold-run protocol

How to prove - before and after any rewrite - that an agent can integrate the product from the published documentation alone. This is a friction log where the newcomer is an agent.

Contents: [Building the task set](#building-the-task-set) · [Run rules](#run-rules) · [Metrics](#metrics) · [Calibrating the target](#calibrating-the-target) · [Recording a finding](#recording-a-finding) · [Report skeleton](#report-skeleton) · [Keeping it honest](#keeping-it-honest)

## Table of Contents

- [Building the task set](#building-the-task-set)
- [Run rules](#run-rules)
- [Metrics](#metrics)
- [Calibrating the target](#calibrating-the-target)
- [Recording a finding](#recording-a-finding)
- [Report skeleton](#report-skeleton)
- [Task set and baseline](#task-set-and-baseline)
- [Surface inventory](#surface-inventory)
- [Findings](#findings)
- [Fix plan](#fix-plan)
- [Re-run](#re-run)
- [Keeping it honest](#keeping-it-honest)

## Building the task set

Five to ten jobs, each phrased the way a developer would ask, each with a verifiable end state.

Compose the set from three sources:

- The two or three integrations that drive the product's revenue or adoption.
- The top-traffic documentation pages, since that is where failures cost the most.
- The known-painful areas: webhook verification, pagination, auth refresh, retries, anything support answers weekly.

Good tasks name an outcome and leave the path open:

- "Add webhook signature verification to an existing Express server."
- "Paginate through every customer created in the last 30 days and write them to CSV."
- "Handle a rate-limit response with backoff on the search endpoint."

Weak tasks smuggle in the answer or cannot be graded:

- "Call `client.webhooks.constructEvent`." - the task hands over the surface the agent was supposed to discover.
- "Explain how authentication works." - no runnable end state, so nothing can fail.

Freeze the set. It is a benchmark, and a benchmark that changes between runs measures nothing.

## Run rules

1. Start each task in a fresh context with no memory of previous runs and no cached copy of the repository.
2. Restrict sources to the published agent-facing surfaces. If the agent needs something you do not publish, that absence is the finding - record it and let the run fail.
3. Run unattended to the first verifiable outcome: it compiles, it executes, it returns the documented result.
4. Intervene only to unblock a non-documentation obstacle (a missing sandbox credential, a network block). Log the intervention.
5. Capture the full transcript. The corrective turns are the data; the final file is just the receipt.
6. Record the model identifier and the date with every score. A score is comparable only within one generation of agents.

If your environment supports parallel sub-agents, run one task per agent to keep runs independent. If it does not, run them sequentially in separate sessions - never several tasks in one session, because knowledge leaks between them and inflates every score after the first.

## Metrics

| Metric                    | Definition                                                      | What a bad value points at                                                       |
| ------------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| First-attempt success     | Tasks reaching the verifiable end state with no corrective turn | The headline number; see Calibrating the target below before picking a pass mark |
| Invented-surface rate     | Runs calling a method, parameter, or field that does not exist  | A missing or ambiguous reference page, or an unpublished spec file               |
| Corrective turns per task | Re-reads and retries before success                             | Scattered answers, incomplete snippets, missing error mapping                    |
| Context cost per task     | Tokens fetched before first success                             | An over-linking index, or cross-reference hops that force extra fetches          |
| Fetch-path length         | Distinct documents opened                                       | More than three for a routine task means the answer is spread too thin           |

Track all five. Success rate alone hides a corpus that works only after six expensive retries.

## Calibrating the target

One vendor publishes an open benchmark of this exact shape: twelve agentic evals of real integration tasks, graders hidden from the agent, documentation access restricted to a single search endpoint. Its 2026-02-26 results across four frontier models, scored best-of-three rather than first-attempt, with best-run turn counts spanning 17 to 163:

- Backend-only tasks: 73-85%.
- Full-stack tasks: 77-92%.
- Depth-of-feature task sets: 61-73%.

Read three things off that:

- **80% first-attempt, unattended, is a stretch goal, not a floor.** A vendor with unusually mature agent-facing surfaces reaches 85% on its easiest category with three attempts allowed. Set the bar there if you want somewhere to climb to; do not let a stakeholder hear it as a minimum acceptable standard.
- **Score by category, not as one average.** Depth tasks land 15-25 points below breadth tasks for every model. A set weighted toward niche cases scores lower by construction, and reporting one blended number invites the wrong conclusion.
- **Your own first run is the only baseline that means anything.** Absolute numbers move with the model generation, the task mix, and how strictly you restrict sources. Iteration-over-iteration delta on a frozen set is the comparison that survives.

If your first baseline lands well under these figures, that is the expected result for a corpus that has never been audited - it is the starting point the rest of the protocol exists to move.

## Recording a finding

One entry per failure, written so someone who did not watch the run can act on it:

```markdown
### Webhook verification fails on the first attempt

- Task: "Add webhook signature verification to an existing Express server."
- Outcome: failed - agent called `verifySignature(payload, header)`; the real method takes a third argument (the endpoint secret).
- Where it looked: /docs/webhooks (HTML only, no markdown version), then guessed.
- Root cause: the signature helper appears only in a prose paragraph; no snippet, and the OpenAPI file does not cover SDK helpers.
- Fix: add a complete runnable snippet with all three arguments and the failure branch; publish the type declarations.
- Impact: top-3 task, invented surface - fix before any cosmetic work.
```

## Report skeleton

```markdown
# Agent-readiness audit - <product> - <date>

## Task set and baseline

Model, date, tasks, per-task result, five metrics.

## Surface inventory

Surface | Status (present/partial/missing) | Effort | Unblocks

## Findings

Task impact decides which findings make the list; within it, order by fixes-per-hour as in SKILL.md § Workflow step 4. One block per finding, in the format above.

## Fix plan

Ships now (content edits) | Needs a build or infrastructure change

## Re-run

Same task set, same metrics, delta against baseline, what still fails and why.
```

## Keeping it honest

- Re-run after every structural change; the score decays silently as the corpus drifts. The cost of that drift is measured, not hypothetical: on a benchmark that deliberately mutates tool interfaces, frontier models lose over 13 points against the unmutated originals.
- Never grade a task the same session has already solved.
- Vary the agent between iterations where you can, so you are not tuning the corpus to one model's habits.
- Fix the pages, not the prompt. A prompt that carries the missing information proves only that you knew it.
- Publish the score alongside the docs when you can. A visible number is what keeps the protocol from being run once and forgotten.

# Page rewrite patterns

Before/after pairs for the failures that cost the most first-attempt successes. Each pattern names the agent behaviour it corrects.

Contents: [Self-containment](#self-containment) · [Snippet completeness](#snippet-completeness) · [Naming discipline](#naming-discipline) · [UI paths](#ui-paths) · [Error mapping](#error-mapping) · [Version labelling](#version-labelling) · [Answer-first ordering](#answer-first-ordering) · [Page frontmatter](#page-frontmatter)

## Table of Contents

- [Self-containment](#self-containment)
- [Snippet completeness](#snippet-completeness)
- [Naming discipline](#naming-discipline)
- [UI paths](#ui-paths)
- [Error mapping](#error-mapping)
- [Version labelling](#version-labelling)
- [Answer-first ordering](#answer-first-ordering)
- [Page frontmatter](#page-frontmatter)

## Self-containment

An agent usually arrives mid-corpus, from a search result or a pasted URL. Backward references point at context it never loaded.

Avoid:

> Using the client you configured in the previous section, call `charge()` with the amount in cents.

Prefer:

````md
Requires an initialized client (see Install and authenticate) and a secret key with `charges:write`.

```python
from acme import Client            # acme>=4.2,<5

client = Client(api_key=os.environ["ACME_API_KEY"])
charge = client.charges.create(amount_cents=1999, currency="eur", source="tok_visa")
print(charge.id, charge.status)    # ch_1A2B3C succeeded
```
````

The prerequisite is restated, the forward link stays for the human, and the snippet no longer depends on a page the agent skipped.

## Snippet completeness

An incomplete snippet is not treated as an excerpt - it is pasted as-is, and the missing parts get invented.

Avoid:

```js
const result = await client.customers.list({ limit: 100 });
```

Prefer:

```js
// acme-node@^4.2  - npm i acme-node@^4.2
import { Client } from "acme-node";

const client = new Client({ apiKey: process.env.ACME_API_KEY });

try {
  // Returns { data: Customer[], has_more: boolean, next_cursor: string | null }
  const page = await client.customers.list({ limit: 100 });
  console.log(page.data.length, page.has_more);
} catch (err) {
  if (err.code === "rate_limited") {
    // Retry after err.retry_after seconds; see Rate limits.
  }
  throw err;
}
```

The block the agent copies now carries:

- Imports.
- Version constraint.
- Auth source.
- The shape of the result.
- The error branch.

When a fragment genuinely cannot run on its own, say so on the line above it - an unlabelled fragment is indistinguishable from a complete example:

```md
Illustrative fragment - not runnable on its own; see the full example above.
```

## Naming discipline

`client`, `sdk`, `acme`, and `handle` read as four different objects to an agent building a mental model from examples.

- Pick one identifier per concept and use it in every language sample and every page.
- Use one placeholder convention corpus-wide (`<YOUR_API_KEY>`), so a single substitution rule covers everything.
- Disambiguate overloaded nouns on first use per page: "a `Session` here is a checkout session, not an HTTP session."

## UI paths

An agent cannot click, and cannot see a screenshot. A setup step that exists only as an interface walkthrough is a dead end.

Avoid:

> Open **Settings → API keys**, click **Create key**, and copy the value.

Prefer:

````md
Create a key from the dashboard (Settings → API keys → Create key), or from the CLI:

```bash
acme keys create --name "local dev" --scope charges:write
# prints: sk_test_… (shown once)
```
````

Both readers are served, and the agent has an executable path.

## Error mapping

Agents arrive by pasting the error string they just hit. Match that string exactly.

```md
### `invalid_signature: timestamp outside tolerance`

- Conditions: webhook endpoint behind a proxy that buffers requests, or a host clock skewed by more than 5 minutes.
- Cause: the signed timestamp is compared against system time with a 300-second tolerance.
- Fix: sync the host clock (NTP), or raise the tolerance with `constructEvent(payload, header, secret, { toleranceSeconds: 600 })`.
- Verify: replay the event with `acme webhooks resend <event_id>` and expect HTTP 200.
```

Cause, fix, and a verification step - an agent that cannot verify keeps retrying variations of the same call.

## Version labelling

Models carry confident memories of removed APIs. Only an in-page label overrides that prior.

```md
> **Deprecated in v4.** `client.charges.refund()` was removed. Use `client.refunds.create({ charge_id })`.
> Removal shipped 2026-03-01; v3 documentation stays at /v3/.
```

Put the label at the top of the page, not in a footnote - an agent that got its answer in the first paragraph never reaches the footnote.

## Answer-first ordering

Reference material buried under rationale forces extra fetches and invites guessing.

Order every task page:

1. The working call.
2. What it returns.
3. The parameters.
4. The rationale.

Keep parameters, limits, and enum values in tables. They survive chunking and truncation better than prose, and an agent reading half a table still reads whole rows.

## Page frontmatter

Machine-readable page metadata is what index generators and retrieval tools route on. Keep it uniform:

```yaml
---
title: Verify webhook signatures
description: Validate the signature header before trusting a webhook payload.
version: v4
languages: [node, python, go]
deprecated: false
---
```

The description doubles as the index entry's routing note, so write it as a statement of what the reader will find, not as a teaser.

# Published findings

Every figure, threshold and named claim this skill relies on, with its source and how much weight it carries. Read this before quoting a number to a stakeholder, and before treating any target here as an industry standard.

Contents: [Calibration data](#calibration-data) · [Evidence that doc quality moves agent success](#evidence-that-doc-quality-moves-agent-success) · [Traffic and adoption figures](#traffic-and-adoption-figures) · [Named concepts and who named them](#named-concepts-and-who-named-them) · [Self-set baselines](#self-set-baselines) · [Claims to avoid](#claims-to-avoid)

## Calibration data

**Stripe integration benchmarks** (`github.com/stripe/ai`, `benchmarks/`, results dated 2026-02-26) - the only vendor-published, open-sourced suite measuring coding agents completing real integration tasks against a vendor's own documentation. Twelve evals in three categories. Scoring is best-of-3 per task, infrastructure failures discarded, best transcript human-reviewed.

| Category            | gpt-5 | sonnet-4.5 | gpt-5.2 | opus-4.5 |
| ------------------- | ----- | ---------- | ------- | -------- |
| Backend-only        | 73%   | 75%        | 80%     | 85%      |
| Full-stack          | 86%   | 80%        | 77%     | 92%      |
| Gym (feature depth) | 61%   | 61%        | 73%     | 65%      |

Best-run turn counts spanned 17-163. Documentation access was restricted to a single channel - Stripe's docs-search endpoint - which is what makes the run measure the docs rather than the model's priors.

Two things follow:

- A single-attempt unattended target above 80% is ambitious even for a vendor with mature agent-facing docs. Present 80% as a stretch goal rather than a floor.
- Depth-of-feature tasks score 15-25 points below breadth tasks for every model, so a task set weighted toward niche cases scores lower by construction. That is not a regression.

## Evidence that doc quality moves agent success

The published benchmarks measure description quality together with retriever quality and tool count, so the link between prose quality and agent success is directional rather than directly measured. Present it that way. The supporting evidence is indirect but consistent:

- **Gorilla / APIBench** - swapping a weak retriever for an oracle retriever moves API-call accuracy to 67.20% (TorchHub) and 91.26% (HuggingFace). Retrieval quality alone accounts for a large share of the outcome.
- **Anthropic**, "Writing effective tools for agents" - "Claude Sonnet 3.5 achieved state-of-the-art performance on the SWE-bench Verified evaluation after we made precise refinements to tool descriptions, dramatically reducing error rates."
- **BFCL v4** carries a "format sensitivity" category that penalizes schema serialization and phrasing brittleness, and a "missing function" category that penalizes calling a tool when none actually fits.
- **MCP-Bench** tests retrieving the right tool "from fuzzy instructions without explicit tool names" - the closest published proxy for whether a description does its job.
- **MCPEvol-Bench** - frontier models drop 13.7% and 14.4% on evolved MCP servers versus their original versions. Interface and doc drift has a measurable cost, which is the argument for re-running the cold run after every structural change rather than once a year.
- **arXiv 2403.12671** - meaningful function names plus docstrings outperform name-only and dummy-name baselines for code completion. Completion quality differs before versus after paraphrasing in about 70% of cases, so phrasing is not a rounding error.

## Traffic and adoption figures

- **Mintlify** reports agents at 66% of measured web traffic across docs it powers, with over 213 million agent requests against 105 million human page loads in one month, up from 15.2% at the start of 2026. Self-reported, single-vendor, and biased toward docs sites that already invested in agent surfaces. Use it to motivate the work, never as a population statistic.
- **AGENTS.md** is claimed at 60,000+ open-source projects, stewarded by the Agentic AI Foundation under the Linux Foundation since December 2025. Relevant as context for why contributor-facing agent instructions are a separate, already-solved problem - not as a surface this skill produces.

## Named concepts and who named them

- **Agent Experience (AX)** - coined by Mathias Biilmann (Netlify) in January 2025: "the holistic experience AI agents will have as the user of a product or platform." Three pillars:
  - Simple access and permissions.
  - Clean APIs.
  - Machine-optimized documentation.

  This skill executes the third pillar only. The first two are product and API design.

- Biilmann again, on why this is a distinct discipline: "LLMs understand your product differently than human developers. And writing documentation for LLMs is not the same as for humans."
- **Agent Experience Engineer** is a real, hired title, which matters when arguing for staffing.
- Reducto's posting (Glassdoor, $200K-$300K, reporting to the CTO) scopes it as "own Reducto's agent-facing surfaces… CLI, MCP server, SDKs, and other integration points."
- AgentMail's (LinkedIn) frames the reader plainly: "Our biggest user is not a person. It's an agent that reads our docs, asks a model what to use, and signs up on its own."
- **Agent-Friendly Documentation Spec (AFDocs)** - Dachary Carey's specification, first published February 2026 and developed with community contributors, with Fern Labs partnering on the reference benchmark. Its own scope statement: "This spec defines 23 checks across 7 categories that evaluate how well a documentation site serves agent consumers." The seven categories: content discoverability, markdown availability, page size and truncation risk, content structure, URL stability and redirects, observability and content health, and authentication and access. Fern's "Agent Score" tool scores any docs URL against it from 0 to 100. This is a narrower, docs-only counterpart to Biilmann's third pillar rather than a competing account of AX as a whole.
- **Agent Readability: A Specification for AI-Optimized Websites** - published by Vercel in March 2026. Three areas: discovery, structure, and context - a different three-part shape than Biilmann's pillars, and scoped to a whole site rather than documentation alone (llms.txt and markdown mirrors, structured data and heading hierarchy, and an AGENTS.md file for coding agents specifically).

## Self-set baselines

These are self-set rather than published standards: defensible starting points that a team with its own data should replace.

- Fetch-path ceiling of three documents for a routine task.
- Zero invented surface on the three highest-traffic tasks. Defensible as a policy rather than a measurement: on a top task an invented call reaches production.
- "Corrective turns must fall between iterations" as the improvement signal, with Stripe's 17-163 range as the only published turn-count reference point.
- The surface build order in SKILL.md § Which surface to build first, the spec-rule order, and the page-rule order are all judgements about effort against impact, not measured rankings.
- A task set of 5-10 integration jobs (SKILL.md interview question 4, [./cold-run-protocol.md](./cold-run-protocol.md)). A smaller set misses failure modes; a larger one stops being cheap to rerun each iteration.
- The 30-character `operationId` cap in [./spec-annotation-rules.md](./spec-annotation-rules.md). APIMatic's guidance covers dropping articles and conjunctions, not a character count; FastMCP's own 56-character truncation is the only sourced ceiling.
- The 100k-character index-split threshold and "a hand-maintained index rots within two releases" in [./agent-surfaces.md](./agent-surfaces.md) § Index file.

## Claims to avoid

Treat these as judgement calls and never answer them with a number:

- The effect of publishing an index file on agent behaviour in the wild. Agents mostly fetch one because a human, a rules file, or a tool pointed at it.
- A population-level baseline for first-attempt integration success across vendors.
- Whether doctests and type hints get surfaced in completion suggestions - plausible from how completion context is assembled, but not demonstrated.
- Whether AsyncAPI or bare JSON Schema respond to the same annotation rules as OpenAPI - inferred by analogy only.

# Annotating a machine-readable spec for agent consumption

A spec written for SDK generation and a spec written for agent consumption differ in a handful of fields. This file covers only those fields. Endpoint-level reference completeness, resource modelling and versioning strategy are API-design questions, not documentation ones.

Contents: [Why the spec outranks the prose](#why-the-spec-outranks-the-prose) · [Operation identifiers](#operation-identifiers) · [Summary versus description](#summary-versus-description) · [Constraining the value space](#constraining-the-value-space) · [Embedded samples](#embedded-samples) · [Curate before exposing](#curate-before-exposing) · [A negative example](#a-negative-example) · [AsyncAPI and bare JSON Schema](#asyncapi-and-bare-json-schema)

## Table of Contents

- [Why the spec outranks the prose](#why-the-spec-outranks-the-prose)
- [Operation identifiers](#operation-identifiers)
- [Summary versus description](#summary-versus-description)
- [Constraining the value space](#constraining-the-value-space)
- [Embedded samples](#embedded-samples)
- [Curate before exposing](#curate-before-exposing)
- [A negative example](#a-negative-example)
- [AsyncAPI and bare JSON Schema](#asyncapi-and-bare-json-schema)

## Why the spec outranks the prose

An OpenAPI, AsyncAPI, GraphQL SDL, protobuf or JSON Schema document is already parseable, already generated from the implementation, and already the contract. Every field an agent can read there is a field it never has to infer from a paragraph.

Published measurements back the priority. On Gorilla/APIBench, swapping a weak retriever for an oracle one moves API-call accuracy to 67.20% (TorchHub) and 91.26% (HuggingFace). Anthropic reports that "Claude Sonnet 3.5 achieved state-of-the-art performance on the SWE-bench Verified evaluation after we made precise refinements to tool descriptions, dramatically reducing error rates."

## Operation identifiers

`operationId` is the single highest-leverage field, because generators turn it into the agent-facing tool name. `openapi-to-mcp` resolves the name in this order:

1. `x-mcp-tool-name`.
2. `operationId`.
3. A `{method}_{path}` fallback.

It discards any tool whose name fails `^[a-zA-Z0-9_-]{1,64}$`.

FastMCP truncates `operationId` at the first `__`, slugifies it, and caps it at 56 characters. When `operationId` is absent, the fallback produces names like `get_api_v1_users_id` that tooling vendors describe as confusing models.

Rules:

- Give every operation an `operationId`. Verb-noun camelCase: `listCharacters`, `getUserById`, `createRefund`.
- Keep it to 30 characters or fewer and drop articles and conjunctions - APIMatic's guidance: "avoid using 'a', 'an', 'the', 'and', 'for' and similar words as they unnecessarily make names longer".
- Keep it unique and stable across releases; a renamed `operationId` silently renames a tool. Enforce uniqueness in CI (`redocly lint --rule operation-operationId-unique`).
- Use `x-mcp-tool-name` / `x-speakeasy-mcp` when the SDK wants a terse method name and the agent wants a descriptive one, rather than compromising on a single string.

## Summary versus description

The two fields do different jobs once the reader is choosing which call to make.

- `summary` - one imperative line. It drives disambiguation between similarly named operations.
- `description` - the operational contract:
  - Preconditions.
  - Side effects.
  - Idempotency.
  - Required auth scope.
  - Rate limits.
  - Response shape.
  - An explicit pointer to the sibling operation that might be a better fit.

Write both from the model's perspective, not a human browsing a reference page. A worked cross-reference line, from Speakeasy's tool-design guidance: "Use this endpoint when you need complete information about a task. If you only need a list of task IDs and titles, use listTasks instead."

Put a disambiguating example inside the description for any argument whose meaning is guessable-but-wrong: "`user_id` is the numeric user ID (e.g. 12345), not the username."

## Constraining the value space

An unconstrained field is where an invented value goes. OpenAI's function-calling guide states the principle directly: "Use enums and object structures to prevent invalid states" and "don't make the model fill arguments you already know."

- Prefer a closed `enum` over a free-form string wherever the value space is finite.
- Mark `required` explicitly instead of relying on convention.
- Supply `default` so the agent never has to guess a safe value.
- Use `format` keywords (`uuid`, `date-time`) rather than describing the shape in prose.
- Use `oneOf` with a `discriminator` so a polymorphic body resolves to one concrete variant.
- Add `x-enumDescriptions` where an enum value's meaning is not obvious from its spelling.

## Embedded samples

`x-codeSamples` attaches per-language snippets directly to an operation - each entry carries `lang`, `source` (inline or a `$ref` to a file), and `label`. This colocates the runnable example with the contract, so an agent reading the spec never needs a second fetch to see the call in context. Native `example` / `examples` cover request and response payloads.

## Curate before exposing

Publishing a spec is documentation. Converting it wholesale into a callable tool surface is not, and it fails at scale.

OpenAI's guidance: "Aim for fewer than 20 functions available at the start of a turn at any one time… Use tool search to defer large or infrequently used parts of your tool surface instead of exposing everything up front." FastMCP is blunter: models "achieve significantly better performance with well-designed and curated MCP servers than with auto-converted OpenAPI servers… especially true for complex APIs with many endpoints."

Working rule - this skill's own widening of OpenAI's figure, not separately sourced: past roughly 20-30 operations, tag-filter or use `x-mcp` extensions to expose only what agents actually need. Building and operating that tool server is product work - hand it to the platform-side skill named in `SKILL.md` § References.

## A negative example

Tempting, and wrong, because every field is technically filled:

```yaml
/v1/customers/{id}:
  get:
    summary: Get
    description: Returns the customer.
    parameters:
      - name: id
        in: path
        required: true
        schema: { type: string }
      - name: expand
        in: query
        schema: { type: string }
```

- No `operationId`, so the tool name becomes `get_v1_customers_id`.
- `summary` disambiguates nothing against the other eleven `Get` operations.
- `description` states no auth scope, no rate limit, and no alternative.
- `expand` is a free-form string whose accepted values exist only in prose elsewhere, so the agent invents one.

The same operation, agent-readable:

```yaml
/v1/customers/{id}:
  get:
    operationId: getCustomer
    summary: Retrieve one customer by ID
    description: >
      Returns the full customer object. Read-only, idempotent, no side effects.
      Requires a key with the `customers:read` scope. Rate limit: 100 req/s per key.
      Responds 404 with error code `resource_missing` when the ID does not exist.
      To fetch many customers, use `listCustomers` instead  - this endpoint accepts one ID.
    parameters:
      - name: id
        in: path
        required: true
        description: Customer ID returned by createCustomer, e.g. cus_9s6XKzkNRiz8i3  - not the email address.
        schema: { type: string, pattern: "^cus_[A-Za-z0-9]+$" }
      - name: expand
        in: query
        description: Sub-resources to inline in the response.
        schema:
          type: array
          items:
            type: string
            enum: [subscriptions, default_source, tax_ids]
```

## AsyncAPI and bare JSON Schema

AsyncAPI reuses JSON Schema for payloads and supports `title`, `description` and vendor extensions, so the rules above transfer to both by analogy. Say so when you apply them; do not present them with the confidence of the OpenAPI findings.

One concrete trap does hold across both: `$comment` is explicitly non-rendered and intended for schema maintainers. Anything an agent must see belongs in `description` or `examples`.

---
name: coding-agent-docs-optimization
description: Makes SDK, API or protocol documentation something a coding agent can integrate from unattended - crawler access, machine-readable entry points (llms.txt, markdown endpoints, OpenAPI and JSON Schema files), self-contained copy-paste-safe pages, and a measured first-attempt agent success rate. Use whenever the user mentions agent-readable or AI-ready docs, agent experience or AX, llms.txt or llms-full.txt, docs for coding agents, agents inventing API calls that do not exist, or asks whether an agent could wire up their SDK from the docs alone - even if they never say "agent". Not for AI-search citation (samber/developer-relations-skills@developer-docs-ai-visibility) or a repo's contributor agent instructions. For human docs architecture use samber/developer-relations-skills@developer-docs-structure-audit.
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.0.1"
---

# Coding Agent Docs Optimization

You are a documentation engineer whose reader is a coding agent integrating someone else's SDK, API, or protocol. Make that integration succeed on the first attempt, unattended, and prove it with a repeatable run.

This is the documentation half of **agent experience (AX)** - the term Mathias Biilmann (Netlify) coined in January 2025 for "the holistic experience AI agents will have as the user of a product or platform". His three pillars:

- Simple access and permissions.
- Clean APIs.
- Machine-optimized documentation - this skill owns that one.

The first two are product and API design. Pointing at them as the real cause of a failure is a legitimate finding, not a deflection.

Biilmann again, on why this is a distinct discipline: "LLMs understand your product differently than human developers. And writing documentation for LLMs is not the same as for humans."

Treat the agent as a real user persona with fixed traits:

- No memory of the previous page.
- No eyes for screenshots.
- A hard context budget.
- A strong habit of inventing plausible API surface when the real one is unstated.

## Scope

Cover the vendor side: the surfaces a developer's agent reads while wiring your technology into their codebase.

- Docs pages.
- Machine-readable specs.
- SDK source-level documentation.

Leave out:

- Getting cited by AI search engines. That is a visibility problem with different metrics: samber/developer-relations-skills@developer-docs-ai-visibility.
- The project's own contributor-facing agent instructions - build, test, lint conventions for people hacking on the repo. That file has its own open specification and its own well-served tooling.
- Building a callable tool surface for agents (tool server, function-calling API). Operating the server is product work. Curating a spec down before conversion stays in scope.
- General documentation architecture and content-type placement (see References).

## Interview

Follow these interview rules:

- Ask one question at a time.
- Prefer multiple choice.
- Stop once you can name the task set.

Skip anything the conversation already answered.

**Memory: Store a few info about the current task or project context. When memory lives in a file, use `devrel-context.md`; if a different memory system is in use, rely on that instead. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**
1. What is being documented - client SDKs, a hosted API, a protocol or spec, a CLI, or several of these?
2. Is there a machine-readable spec (OpenAPI, AsyncAPI, GraphQL SDL, protobuf, published types), is it generated from the implementation, and is it served at a public URL?
3. Where do the docs live: a generated site, markdown in the repo, or both? Can you change the build output, the server headers, and the CDN's bot policy?
4. Which 5-10 integration jobs matter most? Name them the way a developer would phrase the request.
5. What already goes wrong: invented method names, stale versions, snippets that do not run, missing auth setup, silent breaking changes?
6. Who maintains this - a volunteer maintainer team merging contributor PRs, or a vendor with a docs team and CDN control?
7. Which languages must reach parity, and how is the product versioned?
8. Is there a date this has to be working by - a launch, a partner integration, a customer escalation - or is it ongoing quality work?
9. Do you want one release's worth of fixes, or the surfaces and the cold-run harness standing so every release stays agent-readable?
10. What is your ceiling - your own hours, a docs build change, engineering time inside the SDK release cycle, a bot-policy decision someone else signs off?

Re-rank the surface order below against 8-10, and name which answer moved which surface.

- A hard date drops every surface that needs a build change or another team's release cycle.
- A standing mandate promotes SDK doc comments and the cold-run harness, which pay back per release instead of once.
- A ceiling of your own hours deletes doc comments and terminal paths from the plan rather than parking them at the bottom.

## Workflow

1. Verify an agent can fetch the docs at all before auditing what they say. If you can make outbound HTTP requests, fetch a page and the index with each agent user-agent string you care about and record the status codes; otherwise have the docs owner run those requests and report them. A CDN that blocks AI crawlers by default fails every later step silently. See [./references/agent-surfaces.md](./references/agent-surfaces.md).
2. Run a baseline cold run. A fresh agent, no prior knowledge, sources restricted to the published surfaces, one run per task in the frozen set. Log every corrective turn. See [./references/cold-run-protocol.md](./references/cold-run-protocol.md).
3. Inventory what an agent can already reach: index file, markdown versions of pages, machine-readable specs, versioned entry point, SDK doc comments, terminal or package-manager paths. Mark each present, partial, or missing, then order the missing ones with Which surface to build first.
4. Let task impact decide what enters the findings list, then order that list by what each fix returns per hour spent. A finding on a task no cold run failed does not make the list at all, however easy it would be to fix; among the findings that do qualify, a page breaking the top revenue-driving task outranks ten cosmetic fixes because its value is higher, not because effort stopped counting.
5. Rewrite the pages the failed runs actually touched, applying the page rules below.
6. Annotate the spec where an invented parameter or a wrong call traces back to it. See [./references/spec-annotation-rules.md](./references/spec-annotation-rules.md).
7. Publish the missing entry points and advertise each one in the README, the docs navigation, and the response headers. An agent fetches an index because a human, a rules file, or a tool pointed at it - publishing without advertising changes nothing.
8. Re-run the identical task set and compare. Iterate until the exit criteria below hold.

If your environment has persistent memory, store the task set, the surface inventory, and the last scores, so the next run measures against the same baseline instead of a new one.

## Which surface to build first

Crawler access is not on this list. It is the gate: until a fetch returns 200 for the agents you care about, every surface below scores zero, and the audit cannot tell the difference.

Treat opening that gate as its own decision with its own review. Admitting or refusing a given crawler is a policy call about who may read and train on the corpus. It is usually signed off outside the docs team, and reversible from a dashboard in a way the crawling that already happened is not.

Everything after the gate is a real menu, and the axes disagree:

- effort, least first: `index file == full-corpus bundle > version labels > markdown endpoints > spec files annotated > SDK doc comments == terminal and package paths`
- value, most first: `spec files annotated > markdown endpoints > version labels > SDK doc comments > terminal and package paths > index file > full-corpus bundle`
- efficiency, do first: `markdown endpoints > spec files annotated > version labels > index file > SDK doc comments == terminal and package paths > full-corpus bundle`

The index file and the full-corpus bundle tie on effort. Both are generated from the navigation source in one build step, and neither requires a decision about content. They do not tie anywhere else.

The index's position is the whole point of ranking this menu:

- It is the cheapest surface.
- It is the one users ask for by name.
- It changes nothing measurable until the surfaces it points at exist.

Leading with it produces a tidy file above a corpus an agent still cannot use.

SDK doc comments and terminal or package-local access tie on both effort and value. Each ships on the SDK release cycle rather than the docs build. Each reaches only the agent already working inside the user's codebase, never the one fetching from the web.

Default to markdown endpoints and annotated spec files. Move down to version labels once a cold run stops failing on invented calls and starts failing on version mismatch, which is the usual second failure.

SDK doc comments are what this order starves. They are a standing job: every pull request can drift them, so they never win on ratio. They are also the only surface that reaches an agent working offline against installed types.

Promote them above everything when the product ships typed SDKs and the cold runs show the agent reading local code instead of fetching pages.

Delete the full-corpus bundle rather than ranking it last when the corpus does not fit a working context budget alongside the user's own code. A truncated bundle is worse than none, because the agent believes it read everything. A bundle parked at the bottom of a plan is a defect waiting for someone with spare time.

This order is a default, not a law: it shifts with who executes it, since ownership shapes exactly which rows are reachable at all.

Re-rank it against what you already know:

- A vendor whose docs build already emits markdown gets that rung free and should start at the spec.
- A project whose spec is generated from the implementation gets annotation far cheaper than one hand-maintaining it.

Delete every surface answer 10's ceiling rules out, and record which one you removed.

## Exit criteria

Hold the run open until all four hold. The first is calibrated against published data. The other three are this skill's own baselines: replace them the moment the team has data of its own.

- First-attempt task success reaches the agreed target, unattended. Calibrated: set the target from [./references/cold-run-protocol.md](./references/cold-run-protocol.md) § Calibrating the target rather than picking a round number - 80% first-attempt is a stretch goal there, not a floor.
- Zero invented API surface on the three highest-traffic tasks. Self-set, and a policy rather than a measurement: on a top task, an invented call reaches production.
- No more than three documents fetched for a routine task. Self-set: raise it if your corpus is legitimately layered, but raise it deliberately.
- Corrective turns per task lower than the previous iteration. Self-set: direction matters more than the absolute number.

Fix the documentation, never the prompt. A task that only passes with a hand-tuned prompt has not passed - a real developer will not write that prompt.

## Page rules

Apply these to every page a failed run touched.

All ten cost the same thing: one editing pass over a page you are already opening. Effort is flat across the list, so frequency alone decides the efficiency order. That is the one place in this skill where a single axis is honest rather than lazy - there is no cheap-versus-valuable trade to split out.

The order reflects how often each failure shows up in cold runs. It is a judgement call, not a measured ranking. Start at the top when time is short, and reorder against what your own runs keep failing on.

Full before/after pairs live in [./references/page-rewrite-patterns.md](./references/page-rewrite-patterns.md).

- Make each page standalone: restate the prerequisite instead of pointing back to an earlier section.
- Pin the package name and version range inside the snippet, not on a distant setup page.
- Replace UI click paths with the equivalent command or API call; keep the click path as the human alternative.
- Ship complete snippets: imports, client construction, the call, the awaited result, the error branch.
- State what the snippet returns or prints, so the agent can verify rather than assume.
- Label illustrative fragments as non-runnable - an unmarked fragment gets pasted verbatim.
- Use one name per concept across the whole corpus, and one placeholder convention.
- Map each error string the product emits to a cause and a fix, using the exact text that reaches stderr.
- Put the version the page describes in the page, machine-readably, and label deprecated surface at the top with its replacement.
- Put the working call first, the explanation after.

## Spec rules

A handful of spec fields decide whether an agent picks the right call and fills it correctly, and a spec written for SDK generation leaves most of them empty. Work them in the order below - ranked by wrong calls prevented per hour of annotation - and read [./references/spec-annotation-rules.md](./references/spec-annotation-rules.md) before editing:

1. Give every operation a stable, verb-noun `operationId`. Near-zero effort, mechanical, and generators derive the agent-facing tool name from it - the auto-generated fallback produces names models handle badly.
2. Close the value space - prefer these over free-form strings:
   - Enums.
   - Explicit `required`.
   - `default` values.
   - `format` keywords.
   - `oneOf` with a discriminator.

   An unconstrained field is where an invented value goes, and each constraint is a one-line edit that removes a whole class of them.

3. Curate the spec before converting it into tools, when a conversion is planned. Cheap, and skipped entirely otherwise. OpenAI's function-calling guidance: fewer than 20 functions available at the start of a turn - past that, filter by tag or vendor extension instead of exposing everything.
4. Split `summary` and `description` by job:
   - `summary`: one imperative line for choosing the call.
   - `description`: preconditions, side effects, idempotency, auth scope, rate limits, and a pointer to the sibling operation that might fit better.

   This is per-operation prose, so it costs a week on a real spec and lands below three cheaper edits despite being what an agent reads to choose at all.

5. Attach per-language runnable samples to the operation itself, so the contract and the call live in one fetch. This is what the order starves: a sample per language per operation, kept running, is a standing job that never wins on ratio. Promote it to first for the two or three operations your cold runs fail on repeatedly - there, one worked sample outperforms every description you could write.

One caveat before applying these beyond OpenAPI: guidance for AsyncAPI and bare JSON Schema is inferred from the OpenAPI case, not separately sourced. Say so when you apply it.

## Ownership shapes the plan

The failures are the same; the achievable fixes are not.

- Maintainer-run open-source project: you control repo files, not headers or a docs CDN. Lead with repo-root artefacts, generated spec files, doc comments in the SDK source, and a versioned index committed alongside releases. Encode the page rules in the contribution guide so PRs do not undo them.
- Commercial vendor with a docs platform: you control build output, headers, redirects, and the bot policy. Lead with the access check, markdown endpoints, and an index generated by the docs build, so the surfaces cannot drift from the pages.
- Both: the cold-run protocol, the task set, and the page rules are identical. Do not re-derive them per ownership model.

## Invocation examples

Typical requests, and what each one should produce:

- "Our users keep telling us their coding agent hallucinates methods that don't exist in our Python SDK." → Run the interview, build a task set around the surfaces being invented, cold-run it, and return the audit report. Expect the root cause in the spec's `description` fields or in prose-only helper documentation, not in the pages that look worst.
- "Write an llms.txt for docs.example.com." → Do not just emit the file. Inventory first, generate the index from the navigation source, advertise it in three places, and say plainly that the file alone changes nothing measurable.
- "Are our docs AI-ready?" → An agent-readiness audit: baseline scores, surface inventory table, ranked findings, fix plan. Refuse to answer with a yes or a score before a cold run exists.
- "Make our OpenAPI spec work better with MCP servers." → Apply the spec rules and the curation step. Building or hosting the server itself is out of scope; hand that off (see References).

The deliverable is always the audit report below, plus the page and spec edits it justifies. When only part of the workflow is wanted, say which sections of the report will be empty and why.

## Output shape

Produce one audit report, not a pile of page edits:

1. Task set and baseline scores, with the model and date recorded.
2. Access check: which agent user-agents can fetch which surfaces, with status codes.
3. Surface inventory table: surface, status, effort, what it unblocks.
4. Findings, admitted by task impact and ordered by fixes-per-hour, each naming the failing task, the page or spec field, and what the agent did instead.
5. Fix plan split into what ships this week and what needs a build change, naming which of the owner's answers to questions 8-10 moved which item and which surfaces were deleted from the plan outright.
6. Re-run scores against the baseline, plus what remains failing and why.

## Failure modes

- Auditing content on a site that blocks the agent at the edge. Every score is zero for a reason no page rewrite can fix. Check access first.
- Publishing an index file and stopping. Advertise it in the README, the docs navigation, and the server response, or it never gets fetched.
- Publishing a spec with empty `description` fields and calling the spec surface done. The agent still guesses - now inside a schema-valid call.
- Dumping the entire corpus into one giant context file. Past the agent's budget it crowds out the user's own code and gets truncated mid-page.
- Optimizing pages that no task in the set touches. Traffic and task impact, not tidiness, decide the queue.
- Restating the public specification in your own docs instead of linking it. It goes stale and doubles the maintenance surface.
- Grading a re-run on a task the same session already solved. Cached knowledge inflates the score; start clean every time.
- Treating a single model's behaviour as the population's. Vary the agent between runs where you can, and record which one produced each score.
- Quoting a vendor's own traffic statistic as an industry fact to justify the budget. The widely-shared figure that agents are now the majority of docs traffic is self-reported by one docs platform, measured across sites that already invested in agent surfaces. It is a motivation, not evidence.

## References

- `samber/developer-relations-skills@docs-code-sample-standards` - runnable-sample policy, language parity, and sample testing
- `samber/developer-relations-skills@developer-docs-structure-audit` - content-type placement and site information architecture
- `samber/developer-relations-skills@developer-troubleshooting-docs` - building the error-to-fix pages this skill assumes exist
- `samber/developer-platform-skills@agent-skill-offering` - see also: shipping an Agent Skill inside the SDK or CLI repo, versioned with each release
- `samber/developer-relations-skills@version-migration-guide` - the upgrade guides an agent needs when a major version lands
- `samber/developer-platform-skills@api-reference-quality` - endpoint-level reference completeness audited against the spec surface
- `samber/developer-platform-skills` - API design questions and agent-callable tool surface operations

See [./references/agent-surfaces.md](./references/agent-surfaces.md) for the access layer, each machine-readable surface, its format, and when it earns its keep.
See [./references/cold-run-protocol.md](./references/cold-run-protocol.md) for task set design, run rules, metric definitions, target calibration, and a worked finding entry.
See [./references/page-rewrite-patterns.md](./references/page-rewrite-patterns.md) for before/after page rewrites, including what not to do.
See [./references/spec-annotation-rules.md](./references/spec-annotation-rules.md) for the spec fields that decide tool naming, selection, and parameter validity, with a negative example.
See [./references/published-findings.md](./references/published-findings.md) for every figure and threshold this skill uses, its source, and which targets are self-set baselines.