samber/developer-platform-skills已通过检查
SKILL DETAIL
api-reference-quality
samber/developer-platform-skills/api-reference-quality
Audit a published API reference at the endpoint level against its spec surface - every operation, parameter, response code, and error documented, request/response examples present, per-language code snippets in parity, try-it affordances working - and install the source-of-truth gates that stop reference drift (spec-driven generation, OpenAPI lint, contract tests, snippet parity in CI). Use whenever the user mentions API reference docs, OpenAPI docs completeness, undocumented endpoints or error codes, stale or drifting docs, or docs CI gates - even if they never say "reference quality". Endpoint-level reference only. Do NOT use for portal IA and onboarding - use samber/developer-platform-skills@developer-portal-design instead.
安装量 · 331查看来源
Installation
npx skills add https://github.com/samber/developer-platform-skills --skill api-reference-quality
技能文件
SKILL.md
最近同步 · 2026年10月23日
evals/evals.json›
{
"skill_name": "api-reference-quality",
"evals": [
{
"id": 1,
"prompt": "I run developer experience at Ledgerline, a B2B payouts API. Engineering keeps an OpenAPI 3.1 spec current from code annotations - 52 operations in it today. Our public reference at docs.ledgerline.com is written by two tech writers in Contentful; they update pages when product managers remember to tell them. I pulled the page list this morning: the reference has 49 endpoint pages. We ship Python and Node SDKs, and we have a full sandbox environment integrators can hit. I'd like a proper audit of the reference. Honestly the pages read really well and get solid traffic, so I mostly expect polish findings.",
"expected_output": "An endpoint-level audit that censuses the 52-operation spec against the 49 published pages in both directions, diagnoses hand-maintained drift as structural in the report's opening, scores coverage as percentages with severity-tiered findings, benchmarks against Stripe/Twilio mechanisms, and promotes the full generated-docs migration.",
"files": [],
"expectations": [
"Diffs the spec's 52 operations against the 49 published pages in both directions, checking for documented-but-removed pages as well as live-but-undocumented operations",
"Classifies any endpoint documented in the reference but absent from the spec or live API as a blocker (documented-but-removed destroys trust), not a minor cleanup item",
"Uses the spec as the census source and audits the published reference against it, never auditing the rendered pages against themselves",
"Recommends probing for undocumented-but-valid surface beyond the spec (gateway logs, SDK internals, or support tickets), rather than treating the spec as the whole truth",
"Delivers the census as a table with columns for operation, in spec, in published reference, in SDKs, and discrepancy",
"States in the report's first paragraph or verdict that the reference is hand-maintained, so every finding will recur until pages render from the spec",
"Promotes the full generated-docs migration tier because the reference is hand-maintained, instead of stopping at the lint plus contract-test default",
"Reports coverage as percentages against the census (operations documented, parameters fully documented, responses including at least one 4xx, error codes with cause and fix, snippet parity), not as a flat issue list",
"Assigns each finding a severity tier using blocker, major, and minor definitions",
"Benchmarks against Stripe and Twilio by checkable mechanisms (spec-generated reference, embedded personalized test keys, CI-tested snippet repository, per-endpoint error tables), not by reputation",
"Counts snippet parity only for samples that exercise the Python and Node SDKs, excluding relabeled curl or generic HTTP blocks",
"Pushes back on the expectation of only polish findings by citing that incompleteness is the measured industry norm (for example the 67.11 percent mean documentation quality or parameter examples absent from 86.95 percent of specs)",
"Recommends installing a standing CI gate behind the audit rather than shipping the findings as a one-off fix list"
]
},
{
"id": 2,
"prompt": "We're Parcelry, a freight-booking API - 38 endpoints, OpenAPI spec exists and drives the docs site through Redocly, so pages regenerate on every merge. Docs quality still regressed twice this quarter: someone merged endpoints with no description and no error documentation at all. There is no staging or sandbox environment - every API call books real freight and costs real money, and the platform team says a sandbox is at least two quarters away. Board meeting is in 3 weeks and my VP wants 'docs can't regress again' shipped before it. Propose the fix and be specific about tooling. Also, I've read that mature API teams run contract tests, so put those in the plan for later this year.",
"expected_output": "A lint-gate-only remediation plan with a named linter and minimum ruleset, contract tests removed from the plan entirely (not scheduled for later) with an explicit re-promotion trigger, and honest framing of what a lint gate does and does not verify.",
"files": [],
"expectations": [
"Chooses the lint-gate-only remediation tier for now",
"Removes contract testing from the plan entirely rather than scheduling it for later this year, because no safely-callable environment exists to run it",
"States the explicit re-promotion trigger for contract tests: a sandbox or staging surface a test suite may call",
"Justifies the removal: an option nothing can execute is not a slower option, and a parked option quietly reappears in every plan without shipping",
"Notes the 3-week deadline independently favors the lint-only rung as a first increment",
"Recommends a specific spec linter by name (Spectral, Redocly CLI, Vacuum, or RateMyOpenAPI) as an instance of the linter category",
"Specifies a minimum ruleset covering per-operation description and operationId, documented responses including a 4xx, and examples on parameters and schemas",
"Advises starting the linter at warn severity and flipping to fail-on-warn once the spec is clean",
"If a numeric 80 out of 100 lint bar is cited, it is labeled as RateMyOpenAPI's or Zuplo's own tool default, never as an industry average",
"Acknowledges the lint gate only verifies the spec describes itself, not that the spec tells the truth about the running API",
"Does not treat the Redocly docs site rebuilding green as a quality gate",
"Frames the lint gate as running in CI on every PR touching the spec, not as a one-off cleanup pass"
]
},
{
"id": 3,
"prompt": "I'm putting together three slides for our CPO at Corven to fund a documentation-quality push for our logistics API. I need hard numbers: how bad API documentation is across the industry, what it costs teams, and the ROI of adding an interactive try-it console - I saw a vendor claim developers get 10x faster with one. Give me the stats with sources so I don't get shredded in the review.",
"expected_output": "A sourced set of figures where every number carries its evidence-quality label: peer-reviewed corpus measurements, survey self-reports, and vendor case studies clearly distinguished, with vendor try-it multipliers never presented as controlled results and a recommendation to measure the company's own time-to-first-call.",
"files": [],
"expectations": [
"Cites a mean documentation quality of 67.11 percent measured across 2,529 public OpenAPI specs (OASQuali)",
"Cites parameter examples absent from 86.95 percent of specs",
"Labels the spec-corpus figures as peer-reviewed corpus measurement",
"Cites the Postman 2024 State of the API findings: 39 percent call inconsistent docs their biggest roadblock and/or 44 percent dig through source code to understand APIs",
"Labels the Postman survey figures as large-sample survey self-report, distinct from spec measurement",
"Presents the 1.7x to 56x faster time-to-first-call range as Postman's own vendor experiment, never as a controlled or independent result",
"Presents the roughly 10x faster and 225 percent registration figures as the Postman Moneris customer story, a named vendor case study",
"States that the try-it figures conflate the console with sandbox testing and better onboarding generally, so neither isolates the console's effect",
"Recommends measuring the company's own before-and-after time-to-first-call instead of reusing any vendor multiplier",
"Does not present the roughly 10 percent undocumented-exception crash figure as a REST or HTTP API statistic; if cited at all it is attributed as an Android platform-API study",
"If a time-to-first-call target appears, the under-10-minutes bound is labeled practitioner consensus, not a measured industry statistic",
"Every figure presented carries a named source rather than appearing as an unattributed statistic",
"Positions Stripe and Twilio as practitioner-consensus benchmarks compared by checkable mechanisms, not as controlled comparison evidence"
]
},
{
"id": 4,
"prompt": "Hexavolt here - we sell an energy-metering API. The docs are a Docusaurus site the team edits by hand. No OpenAPI linting, no contract tests, though we do have a swagger file that's mostly right, and a staging cluster anyone on the team can safely hit. The docs pages already have language tabs for Python, TypeScript and Java - we ship official SDKs in all three - and the tabs are generated by a widget that converts our curl examples into each language's HTTP-client syntax. Leadership just approved one full quarter of docs investment and wants us to spend it shipping an embedded try-it console and code samples that feel like Stripe's. Lay out the plan.",
"expected_output": "A staged plan that refuses to jump to the try-it and samples stage first, scores the converted language tabs as zero real parity, sequences source-of-truth generation, lint gate, and contract tests before snippet parity and try-it, and notes the approved investment is the condition that promotes the full migration.",
"files": [],
"expectations": [
"Refuses to spend the quarter going straight to the try-it console and samples while earlier stages are missing, and sequences the rollout stages strictly in order",
"Identifies the language tabs as converted or relabeled HTTP snippets that do not count as per-language parity because they never exercise the shipped SDKs",
"Scores current snippet parity at or near zero rather than crediting the three existing tabs",
"Puts adopting the source of truth first: the reference renders from the spec and pages stop being hand-edited",
"Notes the already-approved quarter of docs investment is exactly the condition that promotes the full generated-docs migration, since the marginal cost of the platform move is small",
"Places a spec lint gate in CI before contract testing in the sequence",
"Includes contract testing because a safely-callable staging environment exists, naming a property-based or spec-driven tool such as Schemathesis, Portman, or Prism",
"Sets the expectation that a first contract-test run on a production schema typically surfaces 5 to 15 issues, framed as normal rather than a tool failure",
"Achieves sample parity through x-codeSamples produced by an SDK generator (Speakeasy, Fern, Stainless, APIMatic, or OpenAPI Generator) or through a Twilio-style CI-tested snippet repository",
"If the Twilio snippet-repository model is described, it includes the fake API server validating every snippet in CI before merge, not just a shared snippets repo",
"Specifies the try-it console ships with pre-filled test credentials, the Stripe model",
"Warns the mostly-right swagger file must pass lint and contract verification before it becomes the source of truth, because an unverified spec passes every lie into the reference",
"States the parity pass condition as every shipped language having a working sample on every operation"
]
},
{
"id": 5,
"prompt": "Follow-up audit for our team at Brightmast, a transit-data API. Six months ago we did what an audit recommended: the reference now renders from the spec, and Vacuum lint runs on every PR at fail-on-warn with zero errors today. We still have no environment that's safe for generated traffic - our API writes into municipal systems - so we stopped at the lint rung. Current numbers: all 41 spec operations are documented, every parameter has type, required-ness, description and example, every operation documents its responses including at least one 4xx, every error code has a cause and a fix, and there are no ghost pages for removed endpoints. Snippet parity measures 58 percent across the Ruby and Go SDKs we ship. Median time-to-first-call from our analytics is 14 minutes. Grade us against the gates - my director wants a pass/fail scorecard.",
"expected_output": "A scorecard scoped to the lint-only tier: census and lint gates pass, contract gate reported as out of scope rather than failed, snippet parity recorded as a measured percentage and finding rather than a failed gate, and time-to-first-call treated as a trend, never a gate.",
"files": [],
"expectations": [
"Reports the census coverage gate as a pass at 100 percent",
"Reports the lint gate as a pass with zero errors",
"Reports the contract gate as out of scope at the lint-only tier, never as a failure",
"Explains why the contract gate is out of scope (no safely-callable environment) and states it returns to scope the moment one exists",
"Records the 58 percent snippet parity as a measured percentage and a finding, not a failed gate, because the team sits below the full generated-docs migration tier",
"Does not hold the team to the 100 percent snippet-parity gate at their current tier",
"Treats time-to-first-call as a trend to watch, never as a pass/fail gate on the scorecard",
"If the 10-minute bound is referenced against the 14-minute median, it is labeled a practitioner-consensus target and the gap flagged as a trend concern, not a gate failure",
"The scorecard explicitly scopes which gates apply to the remediation tier the team chose",
"Classifies the missing shipped-language snippets at major severity per the rubric, not blocker and not minor",
"Recommends a standing re-audit cadence, such as quarterly review against support tickets and docs-search analytics"
]
},
{
"id": 6,
"prompt": "Here's the reference page for POST /v1/transfers on our treasury API at Fenwick & Vale. The writer says it's our most polished page and I agree it reads beautifully - tell me if it's done.\n\n---\nPOST /v1/transfers - Creates a transfer between two ledger accounts.\nParameters: source_account_id (string, required) - the funding account, e.g. acct_8f2. destination_account_id (string, required). amount_cents (integer, required) - amount in cents, minimum 1. currency (string, optional, default USD). memo (string, optional, max 140 characters).\nReturns: 200 with the transfer object (a full example JSON response is shown).\nExample: a complete curl request creating a $25.00 transfer.\nAbout our ledger: Fenwick & Vale uses double-entry accounting internally; every transfer produces two ledger entries. [The page continues with four paragraphs on the ledger architecture and why we chose double-entry - it's my favorite part of the page.]\n---\n\nFor context on the endpoint itself: it honors an Idempotency-Key header, and support tells me most failed calls come back as 402 insufficient_funds or 422 currency_mismatch, usually because people attempt transfers before the source account has cleared KYC verification. Give me the verdict, row by row if you have a rubric.",
"expected_output": "A row-by-row rubric verdict that the page is not done: blocker on undocumented response and error codes, majors on the missing idempotency note, ordering constraint, version statement, auth requirements and parameter depth, the architecture prose flagged as quadrant drift, and no credit given for polish.",
"files": [],
"expectations": [
"Grades the page row by row against a per-endpoint rubric rather than giving impressionistic feedback",
"Flags the absence of any documented response code beyond 200, requiring at least one 4xx, at blocker severity",
"Requires the 402 insufficient_funds and 422 currency_mismatch errors to be documented with both cause and fix, not just their code values",
"Scores the error surface as its own rubric row, never folded into the examples row",
"Flags the missing idempotency and retry note as a finding, since the page never states the Idempotency-Key support",
"Flags the missing invocation-ordering constraint (source account must clear KYC verification before transfers) as a finding",
"Flags the four paragraphs of double-entry architecture prose as quadrant drift, explanation content on a lookup-shaped reference page, despite the user's fondness for it",
"Gives the architecture prose no credit toward completeness",
"Flags the missing version statement as a finding",
"Flags the missing authentication or scope requirements as a finding, since nothing states what permissions the call needs",
"Flags parameter-depth gaps, such as destination_account_id lacking a description or example and currency listing no allowed values",
"Grades the missing idempotency note and ordering constraint at major severity, distinct from the blocker-level undocumented error surface",
"The verdict states the page is not done, refusing the most-polished-page framing on the strength of the failed rubric rows"
]
},
{
"id": 7,
"prompt": "Platform team at Quillon here - HR-tech API, 74 endpoints. Our docs regenerate from the OpenAPI spec on every merge and CI blocks the merge if the docs site build fails, so we've considered docs solved for a while. Two things came up though. First, an enterprise prospect's AI coding assistant keeps calling Quillon endpoints that don't exist. Second, someone pointed out our spec has never actually been tested against the running API - we always assumed generation keeps everything honest. I used Optic for breaking-change checks at my last job and was about to set it up here. We have a dedicated staging environment. What should we actually be verifying, and with what?",
"expected_output": "A plan that rejects the site-build check as a gate, adds a lint gate plus contract tests against staging with concrete failure criteria and realistic first-run expectations, replaces the archived Optic with a maintained differ, probes for spec-vs-live drift, and routes the coding-agent problem to its own discipline.",
"files": [],
"expectations": [
"Rejects the docs-site-builds-green check as a quality gate, because a site rebuilds cleanly from a spec that lies about the API",
"States that a spec with no contract-test coverage is unverified (for example, unverified prose with better formatting) and that the generated reference inherits every lie in it",
"Recommends property-based contract testing against the staging environment, naming Schemathesis or a category peer",
"Sets the contract gate criteria as CI failing on any generated-input 500, any schema-violating response, and any undiffed breaking change",
"Sets the expectation that the first contract-test run typically surfaces 5 to 15 issues, framed as normal",
"Does not recommend Optic, noting it was archived, and recommends a maintained breaking-change differ such as oasdiff instead",
"Routes the AI-coding-agent hallucination problem to a dedicated coding-agent docs-optimization discipline or skill rather than folding spec-annotation rules into this audit",
"Recommends probing whether the live API carries undocumented-but-valid routes or parameters despite the spec existing, since spec-inference testing has found such surface in production APIs",
"Adds a spec lint gate for completeness alongside the contract tests for accuracy",
"Distinguishes the two gates' roles: the linter verifies the spec describes itself, the contract tests verify it tells the truth",
"Selects the lint-plus-contract-tests default rung, since docs already render from the spec and a safely-callable staging environment exists"
]
}
],
"trigger_queries": [
{ "query": "Audit our API reference against this OpenAPI spec - I think a bunch of our error codes were never documented", "should_trigger": true },
{ "query": "How complete is our REST API documentation, really? I want a number, not a feeling", "should_trigger": true },
{ "query": "Integrators keep opening tickets about request parameters that aren't documented anywhere", "should_trigger": true },
{ "query": "Our API docs keep drifting out of sync with the actual API - how do we stop that for good?", "should_trigger": true },
{ "query": "Set up CI so a PR can't merge if the OpenAPI spec is missing descriptions or examples", "should_trigger": true },
{ "query": "We hand-write our reference pages in a CMS and they're always stale - is that the root cause?", "should_trigger": true },
{ "query": "Are there endpoints live in production that our docs never mention?", "should_trigger": true },
{ "query": "our api docs are bad enough that customers read our source on github to figure out the calls", "should_trigger": true },
{ "query": "The reference still describes three endpoints we deleted last year", "should_trigger": true },
{ "query": "Measure our reference coverage: operations, parameters, response codes, errors, all of it", "should_trigger": true },
{ "query": "Should we move from hand-written API docs to spec-generated ones, and what would it take?", "should_trigger": true },
{ "query": "Add a Spectral or Redocly lint gate to our docs pipeline", "should_trigger": true },
{ "query": "Every code tab in our reference is the same curl call relabeled as Python and Ruby - audit the samples", "should_trigger": true },
{ "query": "I want working per-language SDK examples on every endpoint like Stripe has", "should_trigger": true },
{ "query": "Our OpenAPI spec and the published reference disagree - how do I reconcile them and keep them reconciled?", "should_trigger": true },
{ "query": "Set up contract tests so our spec can't lie about the API", "should_trigger": true },
{ "query": "We need a try-it console in the API reference with test credentials pre-filled", "should_trigger": true },
{ "query": "Check that every endpoint page documents the 4xx responses it can actually return", "should_trigger": true },
{ "query": "What gates should our docs CI run so reference quality can't regress again?", "should_trigger": true },
{ "query": "Score our public API docs against Stripe and Twilio", "should_trigger": true },
{ "query": "A new customer spent a full day guessing what one of our endpoints returns because the page doesn't say", "should_trigger": true },
{ "query": "Do a completeness audit of our developer reference against the swagger file", "should_trigger": true },
{ "query": "how do teams keep api reference docs accurate long term without a docs police", "should_trigger": true },
{ "query": "Undocumented error codes are our top support-ticket driver - quantify the gap", "should_trigger": true },
{ "query": "The docs site rebuilds green on every merge but the content is wrong half the time", "should_trigger": true },
{ "query": "Run a documentation quality check on our OpenAPI spec", "should_trigger": true },
{ "query": "Which of our 60 endpoints are missing request and response examples?", "should_trigger": true },
{ "query": "Enterprise devs churn during integration and I think it's the reference docs - build me the evidence", "should_trigger": true },
{ "query": "We ship Python, Node and Go SDKs but the docs only show raw HTTP examples", "should_trigger": true },
{ "query": "Our endpoint pages mix in long tutorial prose - is that padding or a real problem?", "should_trigger": true },
{ "query": "Developers say they stopped trusting our docs after hitting one wrong page - fix the trust problem", "should_trigger": true },
{ "query": "Our docs CI should fail when an endpoint ships without a description or a 4xx response documented", "should_trigger": true },
{ "query": "audit whether every parameter in our reference has a type, constraints and an example", "should_trigger": true },
{ "query": "What's a defensible completeness bar for API reference docs before we GA?", "should_trigger": true },
{ "query": "Our reference is generated from the spec but nobody lints the spec - what are we missing?", "should_trigger": true },
{ "query": "I inherited an API with no OpenAPI spec and docs of unknown accuracy - where does an audit even start?", "should_trigger": true },
{ "query": "New integrators keep emailing support asking what fields POST /orders accepts", "should_trigger": true },
{ "query": "Is there a tool that fails CI when someone adds an endpoint without documenting it?", "should_trigger": true },
{ "query": "Our support team spends half its time answering questions the reference should answer", "should_trigger": true },
{ "query": "Verify our published API reference matches what the gateway actually serves", "should_trigger": true },
{ "query": "Design an error-code taxonomy for our public API", "should_trigger": false },
{ "query": "Should we adopt RFC 9457 problem details for our error responses?", "should_trigger": false },
{ "query": "Add idempotency keys so clients can retry POST requests safely", "should_trigger": false },
{ "query": "What retry backoff and jitter guidance should we design for API consumers?", "should_trigger": false },
{ "query": "Pick a versioning scheme for our API - URI path or header?", "should_trigger": false },
{ "query": "How long should the deprecation notice window be before we sunset v1?", "should_trigger": false },
{ "query": "Review our REST resource naming and status-code usage before launch", "should_trigger": false },
{ "query": "Is our cursor pagination design consistent with best practice?", "should_trigger": false },
{ "query": "Redesign our developer portal navigation and onboarding flow", "should_trigger": false },
{ "query": "Reduce time-to-first-call on the portal's signup-to-API-key path", "should_trigger": false },
{ "query": "Design a sandbox test mode with magic test values like Stripe's test cards", "should_trigger": false },
{ "query": "What webhook retry policy and HMAC signing scheme should we ship?", "should_trigger": false },
{ "query": "Which languages should get official SDKs first?", "should_trigger": false },
{ "query": "Should our SDKs be generated with Speakeasy or handwritten per language?", "should_trigger": false },
{ "query": "Design our API key format, prefixes and rotation policy", "should_trigger": false },
{ "query": "Which rate-limit headers should we return - X-RateLimit or the IETF fields?", "should_trigger": false },
{ "query": "Set up a public status page and incident communication for our API", "should_trigger": false },
{ "query": "Should we build an MCP server for our SaaS, and which tools should it expose?", "should_trigger": false },
{ "query": "Design the schema conventions for our public GraphQL API", "should_trigger": false },
{ "query": "Should we expose gRPC to external developers or keep it internal?", "should_trigger": false },
{ "query": "Design OAuth scopes and the consent screen for third-party apps", "should_trigger": false },
{ "query": "Audit the information architecture of our whole docs site - guides, tutorials, landing pages", "should_trigger": false },
{ "query": "Optimize our documentation so AI coding agents stop hallucinating endpoints", "should_trigger": false },
{ "query": "Write standards for authoring and testing the code samples in our tutorials", "should_trigger": false },
{ "query": "Turn our documentation into a marketing channel that ranks in search", "should_trigger": false },
{ "query": "Write a getting-started tutorial for our payments API", "should_trigger": false },
{ "query": "Draft the reference page for our new /v2/refunds endpoint", "should_trigger": false },
{ "query": "Which ETL platforms should we ship a source connector for?", "should_trigger": false },
{ "query": "Give partners a dashboard of their failed API calls and webhook deliveries", "should_trigger": false },
{ "query": "Design the review process for third-party apps on our marketplace", "should_trigger": false },
{ "query": "Improve our listing on the Salesforce AppExchange", "should_trigger": false },
{ "query": "Decide between REST, GraphQL and webhooks for our first public surface", "should_trigger": false },
{ "query": "Design bulk Parquet exports to customers' S3 buckets", "should_trigger": false },
{ "query": "Plan partner-developer onboarding from signup to first submitted app", "should_trigger": false },
{ "query": "I'm hiring an API product manager - what should the interview loop test?", "should_trigger": false },
{ "query": "Set up customer-facing SQL access over JDBC", "should_trigger": false },
{ "query": "Our OpenAPI YAML fails validation in the editor - fix the syntax errors", "should_trigger": false },
{ "query": "Set up Backstage TechDocs for our internal developer platform", "should_trigger": false },
{ "query": "Write the release notes and changelog for our v2.3 API update", "should_trigger": false },
{ "query": "Schemathesis found 500s in our API - help me debug the server crashes", "should_trigger": false }
]
}
references/audit-rubric.md›
# Audit rubric, templates, and benchmark checklist
## Severity tiers
- **Blocker** - an integrator cannot complete a correct call without guessing or reading source: undocumented endpoint, undocumented required parameter, undocumented error code with no cause/fix, documented endpoint that no longer exists, no spec at all.
- **Major** - the call succeeds but reliability or self-service suffers: missing response example, missing 4xx documentation, missing parameter constraints or example, snippet absent for a shipped language, no version statement, retry/idempotency safety unstated.
- **Minor** - polish gaps: stale prose, quadrant drift (tutorial/explanation content on a reference page), inconsistent terminology between endpoints, broken anchors.
## Census template (step 1)
| Operation | In spec? | In published reference? | In SDKs? | Discrepancy |
| ------------------------------ | -------- | ----------------------- | -------- | ------------------------------------------ |
| `POST /v1/payments` | yes | yes | yes | - |
| `GET /v1/payments/{id}/events` | yes | **no** | yes | live + SDK-exposed, undocumented - blocker |
| `DELETE /v1/drafts/{id}` | no | yes | no | documented but removed - blocker |
Both directions matter:
- Live-but-undocumented surface forces guessing.
- Documented-but-removed endpoints destroy trust in every page around them.
Probe beyond the spec where feasible (gateway logs, SDK internals, support tickets naming unknown endpoints) - production APIs have been shown to carry undocumented-but-valid routes even when a spec exists.
## Per-endpoint rubric (step 2)
Grade every operation on each row; every "no" becomes a finding at the listed severity.
| # | Row | Pass condition | Severity if failed |
| --- | ---------------------- | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| 1 | Description | Operation has a purpose statement in the reference (what it does, not how the product works) | Blocker |
| 2 | Parameters - presence | Every spec parameter appears: name, location (path/query/header/body), type | Blocker |
| 3 | Parameters - depth | Each carries required/optional, constraints (min/max, format, enum values, disallowed values), and a sample value | Major |
| 4 | Request example | At least one complete, valid request body/call shown | Major |
| 5 | Response example | At least one realistic success response body shown | Major |
| 6 | Response codes | Every status the endpoint can return is listed, including at least one 4xx | Blocker |
| 7 | Error codes | Every error code documented with its cause AND its fix, not just its value | Blocker |
| 8 | Per-language snippets | One sample per shipped SDK language, exercising the SDK - not relabeled curl | Major |
| 9 | Try-it affordance | The endpoint is executable from the page (console, runnable collection), ideally with pre-filled test credentials | Minor (Major if the platform already offers one and it's broken) |
| 10 | Version statement | The page states which API version it documents and links the migration guide | Major |
| 11 | Idempotency/retry note | States whether the operation is idempotency-safe and how retries behave | Major |
| 12 | Ordering constraints | Any invocation-ordering requirement ("create X before Y") is stated | Major |
| 13 | Auth requirements | Required scopes/permissions for this operation are stated | Major |
| 14 | Quadrant shape | Page is lookup-shaped; worked examples fine, explanatory digressions flagged | Minor |
| 15 | Currency | No references to removed fields, old versions, renamed products | Minor |
Rows 7, 11, and 12 are the ones auditors habitually skip - error surface, retry safety, and ordering constraints are exactly the omissions research finds most neglected.
## Report template (step 3)
```markdown
# API Reference Audit - {API name} - {date}
## Verdict
{One paragraph: coverage headline numbers, the drift diagnosis
(spec-generated vs hand-maintained - if hand-maintained, state that every
finding below will recur until stage 1 of the rollout is done), and the
recommended remediation tier.}
## Coverage scores
| Dimension | Score | Gate |
| ------------------------------------------------------- | ----------- | ---- |
| Operations documented | 41/44 (93%) | 100% |
| Parameters fully documented | 78% | 100% |
| Operations with all responses documented (incl. ≥1 4xx) | 61% | 100% |
| Error codes with cause+fix | 34% | 100% |
| Snippet parity (operations x languages) | 55% | 100% |
## Findings by severity
### Blockers
- {endpoint} - {rubric row} - {what's missing, and what an integrator does instead}
### Major
...
### Minor
...
## Benchmark comparison
{Where the reference stands against the Stripe/Twilio checkable mechanisms
- see checklist below. Name what they do that this reference doesn't.}
## Remediation
{The chosen tier, the rollout stages it implies, in order, each with its
pass threshold.}
```
## Stripe/Twilio checkable-mechanism checklist
Cite Stripe and Twilio by name as the paired benchmark - but compare against their checkable mechanisms, not their reputation. Attribution is well-corroborated practitioner consensus, not a controlled study.
Stripe:
- Reference content generated from the spec, so it cannot drift from the contract.
- Working, personalized test API keys embedded directly in code samples for logged-in users.
- Persistent language switcher; prose paired with a runnable sample in a multi-pane layout.
- Typed, self-describing resource-ID prefixes (`ch_`, `cus_`, `pi_`, `sub_`) so every example teaches the ID scheme.
- Documentation quality written into engineering career ladders and enforced in review.
Twilio:
- Unusually complete per-endpoint error tables.
- A public, CI-tested snippet repository (`TwilioDevEd/api-snippets`): one repo, snippets versioned per helper-library release, a testing harness with a fake API server validating every snippet before merge. This mechanism - not editorial diligence - is what produces per-language parity that never ships broken samples.
- Task-oriented guides layered over (not replacing) the complete endpoint list.
A reference that scores 100% on the rubric but has none of these mechanisms will not stay at 100%; the mechanisms are what the rollout stages install.
references/measured-evidence.md›
# Measured evidence, ranked by evidence quality
Every figure this skill cites, ranked by evidence quality. Cite the evidence-quality label alongside the number - a peer-reviewed corpus measurement, a survey self-report, and a vendor case study license different claims.
## Completeness gaps (the audit's motivation)
| Finding | Source | Evidence quality | What it licenses you to claim |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Mean documentation quality of 67.11% across 2,529 public OpenAPI specs; description/example quality drops from 42.18% to 24.00% as APIs grow; parameter examples absent from 86.95% of specs; quality scored across five dimensions (format, version, metadata, servers, descriptions/examples) | OASQuali - Decrop, Vandeloise, Heymans, Perrouin (Univ. of Namur), ICWE 2026, Springer LNCS 16625, DOI 10.1007/978-3-032-29372-5_7 | Peer-reviewed; every figure and the five-dimension methodology confirmed directly against the accepted manuscript, hosted on the University of Namur institutional repository | Incompleteness is the measured norm; missing parameter examples are the single most common gap. |
| 39% of developers call inconsistent docs their biggest roadblock; 44% dig through source code to understand APIs; documentation a top-four problem five consecutive years | Postman, 2024 State of the API Report (6th edition, 5,600+ respondents) | Large-n survey self-report, not spec measurement | Perceived cost of bad docs at industry scale; "dig through source" is the behavioral symptom the audit's gates target. |
| Lack of documentation was the #1 obstacle to consuming APIs (54.3%) | Postman, 2020 State of the API (reaffirmed 2023) | Survey self-report | Longitudinal persistence of the problem. |
| Standardization is the top API challenge at 52% | SmartBear, State of Software Quality / API, 2023 (1,100+ practitioners) | Survey self-report | Consistency (same format everywhere) is a first-order complaint, not polish. |
| Only 76.6% (59 of 77) of real-world APIs achieved fully successful automated tool generation from their specs; failures were API-wide contract deficiencies (missing auth schemes, malformed base URLs, undocumented runtime headers), not per-endpoint issues | "From REST to MCP", arXiv 2507.16044 (2025); 3,784 deduplicated OpenAPI contracts | Preprint, empirical corpus study | A few structural spec defects can silently break machine consumption of an otherwise mostly-complete reference. |
| Spec-inference testing found undocumented-but-valid GET routes and query parameters (returning 2xx) in shipped production APIs | RESTSpecIT - Decrop et al., arXiv 2402.05102 (2024) | Preprint, tool-based measurement, true positives | "The spec says X" never guarantees "the spec says everything" - census beyond the spec where feasible. |
| Almost 10% of undocumented exceptions findable in the Android platform API manifested in real crashes (901,274 crash records, 3,539 apps) | Kechagia, Fragkoulis, Louridas, Spinellis, Journal of Systems and Software, 2018 | Peer-reviewed; Android platform-API study | Undocumented failure surface has a measurable downstream cost. Never cite as a REST/HTTP error-code statistic. |
| Broad practitioner support for documenting parameters, response bodies, errors, per-endpoint verb meaning - but disallowed parameter values and invocation-ordering constraints are commonly neglected | Tufts qualitative study of REST API design practices, VL/HCC 2023 | Peer-reviewed, qualitative | Justifies rubric rows 3 and 12 - the gaps checklists miss. |
| 1,275,568 valid OpenAPI specs with pre-computed quality metrics | APIstic - Serbout & Pautasso, MSR 2024 | Peer-reviewed dataset paper | Corpus-scale analysis is feasible. Not a quality verdict - cite no number from it. |
## Thresholds (stage gates)
| Figure | Source | Evidence quality | Use |
| ----------------------------------------------------------------------------- | ---------------------------------------- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| 80/100 documentation/completeness score as the default passing bar | RateMyOpenAPI (Zuplo) | Vendor tool default | Stage-2 lint-gate threshold when using a scored linter. A per-spec mechanism and a tool default, not an industry average; never present it as one. |
| First contract-test run on a production schema typically surfaces 5-15 issues | Practitioner reports around Schemathesis | Practitioner consensus | Set expectations for stage 3 - findings are normal, not a sign the tool is broken. |
| First successful API call within 5-10 minutes of opening the docs | Mintlify's stated target | Vendor-stated practitioner target | The "under 10 minutes" time-to-first-call trend bound. A borrowed bound, not a measured industry statistic. |
## Try-it / time-to-first-call effects - vendor-produced, not isolated
| Figure | Source | Evidence quality | Caveats to state when citing |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Forking a ready-to-run collection cut time-to-first-call from 17 to 10 minutes (1.7x) for one publisher; across the full sample, developers were 1.7x-56x faster; "20x" is the headline aggregate | Joyce Lin, "Improve Your Time to First API Call by 20x", Postman Blog, April 2023 | Vendor experiment (Postman has a direct commercial interest in the collection/console story) | Not a controlled, independent result; measures a ready-to-run collection, not an embedded try-it console in isolation. |
| "At least 10x, maybe more" faster time-to-first-call and a 225% rise in developer registrations after launching interactive docs with sandbox testing | Postman, Moneris customer story, 2025 | Vendor customer story | Conflates the console with sandbox testing and better onboarding generally. Cite as a named case study only. |
Both figures above bundle the try-it console with other changes, so neither isolates its effect. Recommend try-it on the strength of the Stripe/GitBook practitioner consensus (GitBook grades interactive try-it as the differentiator between good and great reference docs), and measure your own before/after time-to-first-call rather than reusing any multiplier above.
## Benchmark attribution
Stripe and Twilio as the paired gold standard is well-corroborated practitioner consensus (independently named by Mintlify, DEV Community, Apidog, and others) - not a controlled comparison against unnamed competitors. Compare against their checkable mechanisms (see the audit-rubric file's checklist), never their reputation.
references/tooling-catalog.md›
# Tooling catalog per rollout stage
Tools are named as instances of their category. Recommend the category first, then whichever instance fits the team's stack. Every tool below is spec-driven - a machine-readable spec (OpenAPI/AsyncAPI/GraphQL SDL) is the prerequisite for all of them.
## Stage 1 - source-of-truth generation platforms
Render the reference from the spec so a page can never be edited independently of the contract.
- **Redocly, Stoplight, SwaggerHub, ReadMe, Bump.sh, Mintlify** - hosted docs platforms consuming the spec. Mintlify ties docs to Git with bi-directional sync so a spec change and its rendered-reference change land in the same PR review.
- **Zuplo** - the strongest "one spec, every surface" instance found: a single OpenAPI document drives gateway routing, request validation, interactive docs, the developer portal, and API-key scoping simultaneously.
- Open-source renderers: **Swagger UI** (original, basic try-it), **Redoc** (three-panel layout; try-it gated to the commercial tier), **Stoplight Elements** (React components).
## Stage 2 - spec linters (completeness gate)
Lint on every commit touching the spec, pre-commit and in CI, before any other check - a spec failing its own lint cannot produce a trustworthy reference regardless of renderer.
- **Spectral** (Stoplight, open-source) - given/then JSONPath rules with severities; `spectral lint --fail-severity` gates CI; built-in `oas` ruleset flags missing descriptions.
- **Redocly CLI**
- Built-in rules including `operation-description`, `operation-operationId`, `operation-operationId-unique`, `no-invalid-media-type-examples`.
- CI-friendly output formats.
- `.redocly.lint-ignore.yaml` as a legacy escape hatch.
- **Vacuum** (open-source, Go) - Spectral-ruleset-compatible and much faster; adds example-based rules checking that schemas/parameters/operations actually carry examples - the exact gap OASQuali measured as most common.
- **RateMyOpenAPI** (Zuplo) - grades a spec 0-100 across categories including documentation and completeness; 80/100 is its default passing bar. Cite the per-spec scoring mechanism only; the bar is a tool default, not an industry average.
Rule set to enforce at minimum:
- Per-operation description and operationId.
- Documented responses including a 4xx.
- Examples on parameters and schemas.
Start at warn severity, flip to fail-on-warn once clean.
## Stage 3 - contract testing (accuracy gate)
The spec is only a source of truth as long as it contains truth. A spec with no contract-test coverage is unverified prose with better formatting.
- **Schemathesis** (Python, property-based, open-source) - generates test cases straight from the OpenAPI/GraphQL schema; asserts the server never 500s on generated input and every response matches the documented schema. Used by Netflix, SAP, Red Hat, IBM, JetBrains. Practitioners report a first run on a production schema typically surfaces 5-15 issues - the tool for catching undocumented behavior.
- **Portman** (Apideck, open-source) - converts the spec into a Postman collection with injected contract tests (status, content-type, schema compliance), variation tests, and chained-CRUD tests; runs via Newman in CI.
- **Dredd** (open-source) - validates live responses against the documented description. Carry Phil Sturgeon's caveat when recommending it: Dredd "is testing your documentation is accurate using the API, it is not for testing your API" - he now recommends OpenAPI-3.x contract assertions folded into the existing test suite instead of a separate Dredd pass.
- **Prism** (Stoplight, open-source) - turns any spec into a mock server plus validation proxy; consumer-side contract testing before the real API exists.
- **oasdiff / ApiNotes** - spec-to-spec breaking-change diffing in CI. (Optic, the prior standard recommendation for this job, was archived January 2026 - do not recommend it.)
Gate merges on lint + contract-test pass, never on the docs site merely rebuilding - a site rebuilds cleanly from a spec that lies.
## Stage 4 - per-language snippet parity and try-it
The mechanism is the OpenAPI `x-codeSamples` extension: a per-operation array of language-tagged samples the docs platform renders in the reference panel, replacing generic HTTP snippets that don't correspond to real SDK usage.
Two ways to keep parity in sync:
- **SDK-generator overlays**
- **Speakeasy** - generates idiomatic SDKs across ~10 languages and pushes `x-codeSamples` overlays for all operations. Production users include Vercel, Cloudflare, Mistral.
- **APIMatic**, **Fern** (publishes to npm/PyPI/Maven Central/NuGet/RubyGems; acquired by Postman), **Stainless** - peer options.
- **OpenAPI Generator** - the free option (50+ targets), with the known caveat that its output is "technically correct but rarely production ready".
- **Hand-maintained, CI-tested snippet repository** - the Twilio model: one repo, one testing harness, a fake API server validating every snippet before merge. Higher editorial effort, full idiomatic control.
Platform notes:
- ReadMe defaults to HTTP-based snippets and needs a Stainless/APIMatic integration for SDK-based ones.
- Mintlify renders multi-language examples from the spec and integrates with Speakeasy.
Try-it: embed a console with pre-filled test credentials (the Stripe model) or ship a ready-to-run collection. Effect-size numbers for either are vendor-produced - see the measured-evidence file before citing any multiplier.
## Stage 5 - institutionalization
No tool - process:
- Same-PR spec/docs changes enforced in review.
- Documentation expectations in the engineering ladder.
- Quarterly re-audit against support-ticket and docs-search analytics.
SKILL.md›
---
name: api-reference-quality
description: Audit a published API reference at the endpoint level against its spec surface - every operation, parameter, response code, and error documented, request/response examples present, per-language code snippets in parity, try-it affordances working - and install the source-of-truth gates that stop reference drift (spec-driven generation, OpenAPI lint, contract tests, snippet parity in CI). Use whenever the user mentions API reference docs, OpenAPI docs completeness, undocumented endpoints or error codes, stale or drifting docs, or docs CI gates - even if they never say "reference quality". Endpoint-level reference only. Do NOT use for portal IA and onboarding - use samber/developer-platform-skills@developer-portal-design instead.
license: MIT
metadata:
author: Samuel Berthe
version: "1.0.1"
---
# API Reference Quality
You are an API reference auditor. Judge whether a published API reference documents the API's full contract surface completely enough that an integrator never has to guess, read source code, or open a ticket - then close the gaps through process, not one-off edits.
The frame is Fern's: reference documentation is a projection of the API contract; written by hand, it becomes a second copy kept in sync only by human effort. Diátaxis endorses the same move - reference material "can be generated automatically by the software it describes, which is a powerful way of ensuring that it remains faithfully accurate to the code." Every finding this audit produces is either a completeness gap or a drift mechanism, and the fix for drift is always structural.
The gaps are the industry norm, not the exception. Expect findings.
- OASQuali (ICWE 2026, 2,529 public OpenAPI specs) measured a mean documentation quality of 67.11%, with parameter examples absent from 86.95% of specs.
- Postman's 2024 State of the API survey (5,600+ respondents) found 39% call inconsistent docs their biggest roadblock and 44% dig through source code to understand APIs.
## Clarifying questions
**Memory (advised):** When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.
Ask before auditing. Each answer changes a later step. Batch them - this is a tactical audit, not a strategy interview.
1. Is there a machine-readable spec (OpenAPI, AsyncAPI, GraphQL SDL)? Request it. No spec means step 1 starts from routes and gateway config instead.
2. Is the published reference rendered from that spec, or hand-maintained prose? (decides whether drift is structural - see step 4)
3. Which languages do you ship SDKs or snippets for, and how are snippets produced - generated from the spec, or hand-written per page?
4. What CI gates currently touch the spec or docs: lint, contract tests, breaking-change diff, none?
5. Who reads the reference - third-party integrators, first-party developers, coding agents? (routes the agent side; see next section)
6. Remediation ceiling: by when must the reference be trustworthy, is this a one-off cleanup or a standing gate the platform keeps for years, and how much effort can you spend (hours, a team on the hook, appetite for a docs-platform migration)? (re-ranks the tier choice - see step 4)
## Reader types
The reference reader is a developer or a program, and the completeness bar is identical for both. The split that changes the work is human reader vs coding-agent reader:
- **Human readers** are this skill's scope: readable per-endpoint prose, per-language snippets they can paste, a try-it console, error tables with cause and fix.
- **Coding-agent readers** fail differently - they invent API calls when spec annotations are ambiguous. Route that side to `samber/developer-relations-skills@coding-agent-docs-optimization`; do not duplicate its spec-annotation rules (operationId, enum closure, description split) here. The same spec feeds both readers, so stages 1-3 below serve both.
## Workflow
1. Census the surface - build the endpoint inventory and diff it against the published reference, both directions.
2. Audit every endpoint against the rubric.
3. Score, tier the findings, and deliver the report.
4. Choose the remediation tier - a ranked choice, below.
5. Run the staged rollout up to the chosen tier.
6. Measure, gate, and re-audit on a schedule.
Each step has a section below, in order.
## 1. Census the surface
- Take the spec as the census source when one exists; enumerate every operation, parameter, response code, and declared error. Without a spec, enumerate from route definitions, gateway config, or SDK method lists - and record "no spec" as the audit's first blocker finding.
- Diff spec against published reference in both directions. Documented-but-removed endpoints destroy trust ("if an endpoint is described but no longer exists, trust in documentation drops" - qa-checklist.dev); live-but-undocumented surface makes integrators guess.
- Do not assume the spec is the whole truth: spec-inference testing (RESTSpecIT, 2024) found undocumented-but-valid routes and parameters in shipped production APIs. Where feasible, probe for undocumented surface (gateway logs, SDK internals, support tickets naming endpoints the docs lack).
- Deliver the census as a table: operation, in spec?, in published reference?, in SDKs?, discrepancy.
## 2. Audit every endpoint
Grade each operation against [references/audit-rubric.md](references/audit-rubric.md). Record severity per finding (blocker / major / minor, defined in the rubric). Each operation needs:
- A description.
- Every parameter with type, required-ness, constraints, and example.
- Every response code, including at least one 4xx.
- Every error code with cause and fix.
- Request and response examples.
- Per-language snippets.
- A version statement.
- An idempotency-safety note.
Three checks auditors habitually skip; run them explicitly:
- **Error surface.** Every error code the endpoint can return needs its cause and its fix documented, not just its numeric value - score it as its own rubric row, never folded into "examples". Undocumented failure surface has a measured downstream cost: almost 10% of undocumented exceptions findable in the Android platform API manifested in real crashes (Kechagia et al., 2018 - an Android platform study; never cite it as a REST statistic).
- **Constraints most checklists miss.** Disallowed parameter values and invocation-ordering requirements ("create X before calling Y") are the omissions a Tufts qualitative study (VL/HCC 2023) found commonly neglected even where parameters and errors are documented.
- **Quadrant drift.** A reference page is lookup-shaped (Diátaxis): organized around the product's structure, serving fact lookup. A worked example is in-quadrant; a "why this design" digression or embedded tutorial is content drift - flag it, because prose padding masquerades as completeness.
Count only real per-language parity: a snippet per shipped SDK language on every operation, exercising the SDK, not a generic HTTP/curl block relabeled per tab.
## 3. Score and report
Deliver an audit report, not a fixed list:
- Coverage percentages against the census: operations documented, parameters fully documented, responses documented (incl. ≥1 4xx per operation), error codes documented with cause+fix, snippet parity per language.
- Per-endpoint findings table with severity tiers.
- The drift diagnosis from clarifying question 2: hand-maintained reference means every finding will recur; say so in the report's first paragraph.
- The benchmark comparison, using checkable mechanisms, not reputation. Cite both by name; the report template in the rubric file shows where:
- Stripe: reference generated from the spec, live personalized test keys embedded in samples, typed resource-ID prefixes like `ch_`/`cus_`, docs quality written into engineering ladders.
- Twilio: per-endpoint error tables, a public CI-tested snippet repository where every sample runs against a fake API server before merge.
## 4. Choose the remediation tier
Three tiers for closing the gaps and keeping them closed. Ranked:
- effort: `full generated-docs migration > lint gate + contract tests > lint gate only`
- value: `full generated-docs migration > lint gate + contract tests > lint gate only`
- efficiency: `lint gate + contract tests > lint gate only > full generated-docs migration`
- **Default rung: lint gate + contract tests** (stages 2-3 below). A spec linter enforces completeness mechanically on every PR; contract tests keep the spec honest against the live API - a spec with no contract-test coverage is unverified prose with better formatting. Days of setup, owned by one team, and it converts the audit from a one-off into a standing gate.
- **Step down to lint gate only** when no safely-callable test environment exists yet. Near-zero effort - a linter in CI in an afternoon - but it only verifies the spec describes itself, not that it tells the truth. Promote back up the moment a sandbox exists.
- **Deleted, not demoted: contract tests with no safely-callable environment.** Where question 4 confirms there is nowhere property-based tests can fire real requests, the contract-test rung leaves this session's menu entirely - it is not a slower option; it is one nothing can execute. Not parked at the bottom, because "we'll add contract tests too" reappears in every plan and quietly never ships. Re-promotion trigger: a sandbox or staging surface a test suite may call.
- **Promote to the full generated-docs migration** - source-of-truth generation platform plus SDK-generated snippets (stages 1+4). This is the starved option: highest value and highest effort (a quarter-scale platform move), so it loses every efficiency round. Two conditions promote it anyway: the reference is hand-maintained (drift is then structural - no gate on the spec protects pages nobody regenerates), or a docs-platform or SDK-generation change is already scheduled, making the marginal cost small.
- This ranking is a default, not a law. Re-rank against clarifying question 6 and what you know of the team, and say which answer moved which option:
- A hard near-term deadline promotes the lint-only rung as a first increment.
- A standing-gate mandate promotes the migration; a one-off cleanup demotes it further.
- An in-house docs-platform team makes the migration nearly free and flips the winner outright.
Tool options per tier live in [references/tooling-catalog.md](references/tooling-catalog.md).
## 5. Run the staged rollout
Five stages, strictly ordered - each assumes the earlier ones hold. Never recommend stage 3 to a team that has not done stage 1. The stage reached doubles as the maturity score.
1. **Adopt the source of truth.** Reference pages render from the spec. Endpoint tables stop being hand-edited. Pass: no reference page is ever edited independently of the spec.
2. **Gate completeness in CI.** A spec linter runs on every PR touching the spec, requiring per-operation description and operationId, documented responses including a 4xx, and examples on parameters and schemas. Start at warn severity. Flip to fail-on-warn once clean. Pass: zero lint errors at the chosen ruleset, or ≥80/100 on a RateMyOpenAPI-style score (80 is Zuplo's own default passing bar - a per-spec mechanism and a tool default, not an industry average, so never present it as one).
3. **Gate accuracy with contract tests.** Property-based testing against the schema plus documented-contract conformance plus a breaking-change diff. Expect findings: practitioners report a first Schemathesis run on a production schema typically surfaces 5-15 issues. Pass: CI fails on any generated-input 500, any schema-violating response, any undiffed breaking change.
4. **Achieve snippet parity and interactivity.** Per-language samples via `x-codeSamples` from an SDK generator, or a Twilio-style CI-tested snippet repository. Add a try-it console with pre-filled test credentials (the Stripe model). Pass: every shipped language has a working sample on every operation. `samber/developer-relations-skills@docs-code-sample-standards` owns sample governance and testing once snippets exist - this skill's parity gate feeds that skill's corpus.
Try-it effect sizes are vendor-produced and not cleanly isolated:
- Postman's own experiment reported developers 1.7x-56x faster with a ready-to-run collection.
- Postman's Moneris customer story claims ~10x faster time-to-first-call plus 225% more registrations.
Cite both as named vendor case studies, never as controlled results, and note they conflate the console with better onboarding generally.
5. **Institutionalize.**
- Every API-changing PR carries its spec/docs change.
- Documentation expectations enter PR review and the engineering ladder (the Stripe practice).
- Re-audit quarterly against support tickets and docs-search analytics.
Escalation trigger: persistent "inconsistent docs" complaints or "dig through source" behavior in surveys means a dedicated docs owner and stricter gates, not another audit pass.
## 6. Measure, gate, and re-audit
Gates - iterate until all pass:
Scope the gates to the remediation tier step 4 chose, then iterate until every gate in scope passes. A gate for a stage the chosen tier does not reach is not a failure - holding a lint-only team to a contract gate rejects the rung this skill itself prescribed for them, and the honest report says "out of scope at this tier", never "failed".
- **Census coverage: 100%** - in scope at every tier.
- Every spec operation appears in the published reference.
- Every parameter carries type, required-ness, description, and example.
- Every response code is documented with at least one 4xx per operation.
- Every observable error code is documented with cause and fix.
- Zero documented-but-removed endpoints.
- **Lint gate: zero errors** at the chosen ruleset (stage 2 threshold above) - in scope at every tier.
- **Contract gate: zero violations** (stage 3 threshold above).
- In scope from the default rung up.
- Out of scope at the lint-only rung, where no callable environment exists - it returns the moment one does.
- **Snippet parity: 100%** of operations x shipped languages.
- In scope only at the full generated-docs migration (stage 4).
- Below that tier, record parity as a measured percentage and a finding, not a failed gate.
Trends to watch, never gate:
- Time-to-first-call (target under 10 minutes, the practitioner consensus bound). Measure your own before/after rather than reusing vendor multipliers.
- Share of support tickets caused by missing or wrong reference content.
- Docs-search queries returning no result.
Published-vs-self-set:
- The 80/100 lint bar, the 5-15 first-run contract findings, and the survey and spec-corpus figures are published numbers (details in [references/measured-evidence.md](references/measured-evidence.md)).
- The 100% census-coverage gates are this skill's own definition of done - completeness is binary for the integrator who hits the one undocumented endpoint.
## Failure modes
- Auditing the rendered pages against themselves instead of against the spec and live surface - misses undocumented endpoints entirely, the worst gap class.
- Treating an unlinted, contract-untested spec as ground truth; the census inherits every lie in it.
- Accepting "the docs site rebuilds green" as a gate - a site rebuilds cleanly from a spec that lies about the API.
- Counting relabeled curl as per-language parity.
- Documenting only the 2xx path and calling the endpoint covered.
- Padding thin pages with tutorial prose instead of facts - quadrant drift scored as completeness.
- Shipping the audit with no CI gate behind it; the same findings return next quarter.
- Recommending a later stage to a team missing an earlier one.
- Citing vendor try-it multipliers or the OASQuali percentages without their evidence-quality labels.
## Invocation examples
- "Audit our API reference against this OpenAPI spec - I think half the error codes aren't documented."
- "Integrators keep opening tickets about parameters that aren't in the docs. Measure how complete our reference actually is and tell me what to fix first."
- "Set up CI gates so our API docs can't drift from the spec again - we have a sandbox environment available."
- "We hand-write our reference pages in the CMS. Is that why they're always stale, and what would moving to spec-generated docs take?"
## References
- [./references/audit-rubric.md](./references/audit-rubric.md) - per-endpoint rubric rows with severity tiers, the census and report templates, and the Stripe/Twilio checkable-mechanism checklist.
- [./references/tooling-catalog.md](./references/tooling-catalog.md) - spec linters, contract-test tools, snippet-parity generators, and docs platforms per rollout stage, with known caveats.
- [./references/measured-evidence.md](./references/measured-evidence.md) - every figure this skill cites, with its source and evidence-quality label (peer-reviewed, survey self-report, vendor case study, practitioner consensus).
- samber/developer-platform-skills@api-error-design
- samber/developer-platform-skills@api-versioning-policy
- samber/developer-platform-skills@api-idempotency-retry
- samber/developer-platform-skills@sdk-portfolio-strategy
- samber/developer-platform-skills@public-api-design-review
- samber/developer-relations-skills@developer-docs-structure-audit