docs-code-sample-standards

{
  "skill_name": "docs-code-sample-standards",
  "evals": [
    {
      "id": 1,
      "prompt": "I run docs at Halyard Data, a time-series database. Our samples live on the docs site, in the API reference, and in a separate examples repo. Last Thursday I opened our twelve most-visited docs pages, read every code block on them, and fixed four that were obviously broken. I want to send leadership a sample-quality report this week concluding that our samples are in good shape with minor issues on the storage pages. Write that report for me. Open it with the line that 94% of developers consider API documentation quality a key factor - I've seen that stat everywhere and it sets up my budget ask nicely.",
      "expected_output": "A refusal to write the requested conclusion, a refusal to cite the unverified statistic, and a replacement audit method built on a mechanical census of the whole corpus plus a stratified judgement pass.",
      "files": [],
      "expectations": [
        "Refuses to include the '94% of developers' figure and states it is not confirmed in any SlashData report despite being widely attributed to one",
        "Declines to write a corpus-wide quality conclusion drawn from the twelve pages that were read, naming it a convenience sample",
        "States that the most-visited pages are also the most-maintained pages, biasing the review toward the healthiest region of the corpus",
        "Points out that nothing was compiled or executed, so only defects visible to the eye were caught",
        "Distinguishes an inventory (a list of every sample) from an audit (assessing and evaluating quality) and requires the inventory to be built first",
        "Specifies a mechanical census covering 100% of the corpus - compilation, execution, secret scanning, fence-language and prompt-character checks - rather than sampling for those checks",
        "Specifies a separate stratified pass for everything needing a reader, stratified by language, product or service area, and age",
        "Gives the inventory row format including file path, language, surface or product area, identifier, last-commit date and line count",
        "Applies ROT triage - redundant, outdated, trivial - as the first cut over the inventory",
        "States that any figure quoted from the stratified sample is an estimate with a stated base, never a corpus-wide count",
        "Requires the stratified sample size to be stated in the report",
        "Formats findings as claim, then evidence location, then a one-sentence fix, grouped by severity, rather than as prose paragraphs",
        "Does not present the four ad-hoc fixes already made as the outcome of an audit",
        "Offers at least one sourced figure it will stand behind in place of the refused statistic, such as Uddin and Robillard's 2015 survey of 323 professional developers or the MSR 2016 compilability numbers"
      ]
    },
    {
      "id": 2,
      "prompt": "Voltmesh, three engineers, self-serve audio infrastructure. Our docs are about 160 hand-written Markdown snippets. We have no test suite, no CI pipeline at all, and I cannot free up engineering time for at least six weeks. I need to tell the team which samples we are going to test. Four representative ones: (a) the quickstart auth call, (b) a snippet that starts a transcription job billed at $0.90 per minute, (c) a snippet that runs kubectl delete namespace voltmesh-dev, (d) a snippet that posts to a partner webhook hosted by a third-party provider. Give me the tier for each and what CI enforces for it.",
      "expected_output": "Tiers deleted rather than ranked because the constraints rule them out, every sample labelled unverified, the pipeline made the first queue item, and the three samples that can never be tier Run identified with reasons.",
      "files": [],
      "expectations": [
        "States that with no CI and no engineering hours available, both the Run tier and the Compile-only tier are deleted from the menu rather than ranked last",
        "Says explicitly which tiers were deleted and why, instead of silently omitting them",
        "Classifies every sample in the current corpus as unverified rather than as passing, untiered, or illustrative by default",
        "Makes building the verification pipeline the first item in the fix queue, ahead of any tier assignment work",
        "States that the billed transcription sample (b) has no Run tier available regardless of what the pipeline can later do",
        "States that the destructive kubectl delete namespace sample (c) has no Run tier available for the same reason",
        "States that the third-party-dependent webhook sample (d) has no Run tier available for the same reason",
        "Assigns the quickstart auth sample (a) to tier Run once a pipeline exists, because it sits on the primary adoption path",
        "Names all three tiers - Run, Compile-only, Illustrative - and what CI enforces for each",
        "States that an untiered sample defaults to Run as a claim-assignment rule, so an unclassified sample fails loudly rather than passing silently",
        "States that an Illustrative sample must be visibly labelled as not runnable in the rendered page",
        "Gives each tier its reader signal, distinguishing 'this works as written' from 'this is valid; it was not run for you'",
        "Requires the destructive sample to carry an inline guard plus a warning above the block"
      ]
    },
    {
      "id": 3,
      "prompt": "Northbeam, a feature store, SDKs in Python, TypeScript, Go, Java and Ruby. Our Python samples are excellent, the Go ones are a mess and Java is worse. My plan is to rewrite the Go and Java samples so they are line-for-line equivalents of the Python ones - same variable names, same ordering, same helper structure - so a side-by-side diff shows nothing but syntax differences. Python, TypeScript, Go and Java are the four languages we commit to; Ruby is community-contributed and unowned. We document six scenarios. Tested and running in CI today: all six in Python, five in TypeScript, four in Go, two in Java, one in Ruby. Everything else either exists untested or does not exist at all. Where do we stand and what is the plan?",
      "expected_output": "A rejection of transliteration as parity, the ranked sample principles applied, and a computed parity coverage figure with the community language excluded from the denominator.",
      "files": [],
      "expectations": [
        "Rejects line-for-line transliteration as the definition of parity",
        "Names the ranked sample principles in the order clarity, idiomaticity, consistency, simplicity, portability, maintainability, with the higher principle winning a collision",
        "States that idiomaticity outranks consistency, so a transliterated sample fails even when it is byte-consistent with its siblings",
        "Defines parity as an identical sample identifier, an identical scenario and step order, and an identical testing approach across languages",
        "States that everything else - variable names, helper structure, style - may legitimately differ between languages",
        "Computes parity coverage as tested cells divided by (scenarios multiplied by tier-1 languages)",
        "Excludes Ruby from the denominator because it is a community language rather than tier 1",
        "Reports parity coverage of 17 divided by 24, approximately 71%",
        "Compares that figure against the 90% parity target and labels the target as a baseline this policy sets rather than a published industry standard",
        "Uses the matrix cell vocabulary T for tested, P for present but untested, M for missing and S for stale",
        "Reads the matrix by row and by column, stating that a sparse row is a scenario supported in only one language and a sparse column is a language tiered above what the team actually staffs",
        "Recommends either funding Java properly or demoting it, rather than leaving the column sparse under a tier-1 commitment",
        "States that language tiers should be chosen from download or telemetry share and support-ticket volume rather than from team preference",
        "Requires the tiers to be published, including Ruby's community status and its untested label"
      ]
    },
    {
      "id": 4,
      "prompt": "Hollowmere Robotics. The audit came back with these five findings. Put them in the order we work them, and I want it ordered strictly by severity, worst first, so the team stops arguing about it. F1: P2, 61 samples across the SDK reference have no language identifier on the fence, about 40,000 monthly readers, fix is one scripted pass of roughly 2 hours. F2: P1, the batch-ingest sample on our busiest guide calls a method that was renamed two releases ago, about 18,000 monthly readers, fix is 30 minutes. F3: P0, the auth setup page embeds what looks like a live-format API key, about 9,000 monthly readers, fix is 20 minutes plus a key rotation. F4: P1, the first-call quickstart sample for our C++ SDK has never been compiled by anything; there is no C++ build in CI and standing one up is roughly three engineer-weeks, about 700 monthly readers. F5: P2, 210 samples corpus-wide have no documented expected output, about 120,000 monthly readers combined, fix is about 6 engineer-days.",
      "expected_output": "F3 placed first on harm, the remaining four ranked by a stated ratio rather than by severity, and the low-traffic adoption-path finding promoted by hand with its reason given.",
      "files": [],
      "expectations": [
        "Places F3 first on harm rather than on any ratio or reader-count calculation",
        "States explicitly that P0 findings are ordered by harm and not by the efficiency ratio",
        "Says the suspected leaked key is reported immediately rather than held for the end of the report",
        "Pairs the placeholder fix for F3 with rotating the exposed key, not just editing the page",
        "Declines to order the remaining findings strictly by severity as the user asked",
        "Ranks the non-P0 findings by (readers affected multiplied by severity) divided by fix effort",
        "States in the deliverable that the queue is a ratio rather than a severity list",
        "Names F4 as the finding the ratio starves: low traffic combined with high fix effort",
        "Promotes F4 by hand anyway because it sits on the primary adoption path of quickstart, auth and first call",
        "Gives the reason for that promotion rather than presenting it as an unexplained exception",
        "Places at least one P2 finding above at least one P1 finding, showing the ratio and not severity drove the order",
        "Identifies the missing C++ build as F4's actual blocker, making standing up that build the queue item rather than editing the sample",
        "Assigns an owner and a time window to each queue item"
      ]
    },
    {
      "id": 5,
      "prompt": "Kestrel Analytics. Here is our current setup. One CI job, triggered on every pull request to the docs repo, that builds our SDK from the monorepo working tree and then runs all 240 documentation samples against our staging environment. It went flaky in March because leftover test projects kept hitting a quota, so we set it to continue-on-error and it has been red for four months. Readers still open issues saying the samples do not work. Redesign our pipeline.",
      "expected_output": "A pipeline design that installs the published artefact, splits hermetic from integration runs, adds a schedule, restores a blocking check, and fixes the fixture lifecycle before re-enabling.",
      "files": [],
      "expectations": [
        "Identifies that building from the monorepo working tree means the suite passes on code no reader can install yet",
        "Requires installing the published artefact instead - the released package, container or deployed endpoint the reader actually gets",
        "Splits hermetic runs, mocked or local and run on every pull request, from integration runs against the real service",
        "States that samples break when the service changes rather than when the docs repo changes",
        "Adds a scheduled run, nightly or weekly, rather than relying on commit-triggered runs alone",
        "Removes the advisory continue-on-error setting and makes the job fail something blocking, such as the docs build or a required check",
        "States that an advisory job goes red and stays red",
        "Requires the docs build to fail on sample failures so the red mark cannot be ignored",
        "Attributes the flakiness to fixture teardown rather than to the samples themselves",
        "Requires teardown to be fixed before the suite is re-enabled, not after",
        "Requires dedicated test accounts or projects per language rather than a shared human account",
        "Requires teardown inside the test plus a scheduled sweeper for whatever teardown misses",
        "Requires deterministic naming for created resources, such as a prefix plus a run id, so the sweeper can identify them safely",
        "Requires reporting per sample - which sample, which page, which line - rather than per suite, so findings route to an owner"
      ]
    },
    {
      "id": 6,
      "prompt": "Ridgeline, Go SDK for a graph database. We moved all 180 of our docs samples into example_test.go files as ExampleXxx functions, and go test ./... is green in CI on every push. 124 of the 180 carry an // Output: comment. The other 56 do not, because their output is not deterministic - timestamps and generated IDs. Marketing wants to put a 'Verified' badge next to all 180 samples on the docs site. Can we?",
      "expected_output": "No for all 180: the 56 without an output marker compile but never execute, so they are Compile-only and must be labelled as such, with adoption-path samples among them reworked rather than badged.",
      "files": [],
      "expectations": [
        "Answers no: the badge cannot honestly cover all 180 samples",
        "Explains that a Go example function without an // Output: comment is compiled but never actually executed",
        "States that those 56 pass CI while claiming more than they prove",
        "Assigns the 124 samples carrying // Output: to tier Run",
        "Assigns the 56 samples without it to tier Compile-only",
        "Gives the Compile-only reader signal - valid, but not run for you - as the honest badge wording for those 56",
        "Confirms that preferring the language's native doctest facility over a bespoke harness was the right call, and locates the trap in the opt-in assertion rather than in the tooling choice",
        "States that any of the 56 sitting on the primary adoption path must be reworked to tier Run rather than badged, because adoption-path samples must be Run and green in CI",
        "Defines freshness as the age of the last successful run rather than the last edit",
        "States that an edited sample nobody has run is not fresher than before",
        "Cites the executable-documentation guarantee that documented information does not go out of date as the API changes, as the reason the 124 hold",
        "Classifies a sample claiming tier Run that CI does not actually run as a P1 finding rather than a P2"
      ]
    },
    {
      "id": 7,
      "prompt": "Sablefish Health. Our VP of Engineering wants a CODEOWNERS file merged by Friday mapping every language's samples to a named owner, plus a 48-hour fix SLA per owner. Current state: about 900 docs pages, samples hand-pasted into Markdown, no snippet tags, no metadata records, nothing tested, and we have never scanned for secrets. Two of our four SDKs are maintained by outside contributors who are not paid. Draft the CODEOWNERS file and the SLA table.",
      "expected_output": "A refusal to ship ownership first, the correct migration order with the secret scan and the move into compilable source ahead of it, and an ownership model chosen per surface with its funding stated.",
      "files": [],
      "expectations": [
        "Declines to ship CODEOWNERS and fix SLAs as the first step",
        "States that assigning owners to hand-pasted Markdown gives them responsibility with no mechanism to discharge it",
        "Names this as the common mistake of reversing the migration order",
        "Gives the migration order: census and secret scan, then automated compile and run gates, then move samples into compilable source published by transclusion, then metadata records and schema validation, then code owners and fix SLAs",
        "Puts the secret scan first because credential leaks are the only findings that cannot wait",
        "Identifies moving samples into compilable source and publishing by transclusion as the single highest-leverage investment here, because every later guarantee depends on it",
        "Requires snippet tags with globally unique, hierarchical identifiers carrying language, product area and example name, kept identical across languages",
        "Requires example metadata records holding at minimum title, synopsis, category, the languages the example exists in, the SDK version targeted, the source location and the snippet tags composed",
        "States that a snippet tag alone cannot tell a renderer which page or which language variant to use, which is what the metadata record is for",
        "Presents the three ownership models - a dedicated examples team, DevRel or community with expert vetting, and product engineers co-owning - with the cost of each rather than recommending one blindly",
        "Recommends picking an ownership model per surface rather than one model for the whole corpus",
        "States that community or outside-contributor maintenance is a funding model rather than a free one, and requires paid capacity behind it",
        "Writes into the ownership section that bug fixes and security issues take priority over authoring new code examples"
      ]
    },
    {
      "id": 8,
      "prompt": "Cindermoor Pay. Here is our most-copied docs block, verbatim, exactly as it renders:\n\n```\n$ curl -X POST https://api.cindermoor.com/v2/payouts \\\n  -H \"Authorization: Bearer sk_live_9fKq2pRt7LwZ3xVn8MdE4cYb\" \\\n  -d amount=4500 ...\n{\"id\":\"po_7731\",\"status\":\"pending\"}\n```\n\nSupport keeps telling readers to swap the key for their own. Clean this block up and give me the rules so the rest of the docs stop looking like this. Use acct_test_51H8xQ2eZvKY as the example key in the cleaned version - it is not a real account, and our readers find angle brackets confusing.",
      "expected_output": "A rewritten block with a placeholder rather than a plausible fake, a command and output split into separate blocks, and a policy covering the six decisions with the safety rules spelled out.",
      "files": [],
      "expectations": [
        "Flags the sk_live_ literal as a P0 finding",
        "States that the exposed key must be rotated, not merely replaced in the page",
        "Refuses to substitute the suggested plausible-looking fake credential",
        "Explains that readers paste plausible-looking fake values verbatim, which is why they are banned",
        "Uses an angle-bracket placeholder such as <YOUR_API_KEY> in the rewritten block",
        "Explains the placeholder in the surrounding prose rather than leaving it bare",
        "Removes the shell prompt character from the start of the command line",
        "Adds a language identifier to the fence, and states that plaintext is used when nothing else fits",
        "Moves the response body into its own separate block so it cannot be copied along with the command",
        "Removes the ... from the copyable block and states that omissions, where needed at all, are marked in the language's own comment syntax and never inside a click-to-copy block",
        "States that example hosts and addresses use reserved or documentation ranges rather than plausible real-looking values",
        "Lists the six policy decisions: coverage, anatomy, safety, parity, verification, and ownership and freshness",
        "States the anatomy requirement that a sample carry all imports, explicit client setup, one code path, expected output, and a top comment giving what it does and what setup it assumes",
        "Requires an exceptions register where each waived rule carries an expiry date"
      ]
    },
    {
      "id": 9,
      "prompt": "Pellworth Systems. I am presenting to our board on Tuesday and I need the industry-standard benchmarks for documentation code samples: what percentage should compile, what percentage should be covered by tests, and what the accepted cross-language parity number is. Give me the standards with their sources so I can put them on a slide as the bar we are being measured against.",
      "expected_output": "One gate identified as having a measured comparator, every other number labelled a self-set baseline, published studies presented as supporting evidence rather than standards, and the board framing refused.",
      "files": [],
      "expectations": [
        "States that only one gate here has a measured comparator behind it and that the rest are self-set baselines",
        "Identifies the deprecated-API rate staying below 43% as the gate with the measured comparator",
        "Attributes 43% to Zhou and Walker (FSE 2016) as the deprecated-API rate measured in the Stack Overflow posts they investigated",
        "States that using that measured figure as a gate is a choice rather than something the authors proposed",
        "Labels the target of 80% or more of the corpus tested as a self-set baseline rather than a published standard",
        "Labels the 90% or higher parity coverage target as a self-set baseline",
        "Labels the 90% compile-or-run pass rate and its remediation trigger as self-set, noting that no vendor publishes that gate",
        "Cites the MSR 2016 finding that roughly 1.00% of Stack Overflow Java snippets compile and the Gistable ICSME 2018 finding that 24.4% of public Python gists run out of the box as supporting evidence rather than as standards",
        "Refuses to present any self-set baseline to the board as an industry standard",
        "Names the requirement of integration tests for critical paths as the nearest thing to a public vendor requirement on sample verification",
        "Declines to state a freshness-window number, explaining it depends on release cadence and must be set by the team",
        "Recommends reporting the team's own measured baseline alongside every target",
        "States that the P0, P1 and P2 severity model is adapted from general QA practice rather than being a published sample standard"
      ]
    },
    {
      "id": 10,
      "prompt": "Fernhill Cloud. We publish 1,400 code samples. Roughly 700 of them are one-liners on API reference pages - client.buckets.get(id) and the like, one per endpoint, hand-written years ago. Another 120 or so are the same auth snippet copy-pasted onto different pages with small variations. Leadership signed off on '80% of samples tested' as this year's goal and it is now on my objectives. Plan how we test 1,400 samples.",
      "expected_output": "Triage before verification: the one-liners cut or generated, the duplicated auth snippets reconciled to one source, and the threshold met by cutting rather than by lowering the bar.",
      "files": [],
      "expectations": [
        "Refuses to plan verification for all 1,400 samples as stated, and runs triage before any verification planning",
        "Classifies the roughly 700 single-call one-liners as Trivial under ROT triage",
        "States that cutting samples is a legitimate outcome of triage rather than a failure",
        "Names the resistance pattern: teams keep trivial snippets because the snippets are not wrong, while they remain maintenance debt with no reader payoff",
        "Cites the preference for samples covering broader scenarios and use cases over snippets covering only individual API calls",
        "Classifies the roughly 120 duplicated auth snippets as Redundant and reconciles them to a single source",
        "States that every duplicate copy is a future divergence",
        "Applies the rule that auth setup lives in one place other samples link to rather than being repeated with variations",
        "Recommends generating the single-call reference snippets from the API description rather than hand-maintaining them",
        "States that generated snippets guarantee parity and freshness at near-zero marginal cost per added language",
        "States that generated snippets express single calls only, so multi-step idiomatic usage still needs hand-written tested samples",
        "Requires the policy to state which surface uses generated snippets and which uses hand-written ones, so the team does not hand-maintain what a generator already covers",
        "States that when the 80%-tested threshold cannot be met, samples get cut rather than the bar lowered, because a deleted sample costs a reader nothing while an untested one costs them an afternoon"
      ]
    },
    {
      "id": 11,
      "prompt": "Thalberg Labs. Readers told us the import block at the top of every sample was noise, so we changed our extractor to start each published region at the first line after the imports. Now we get issues saying the samples do not run. The docs team's proposal is to add a note under each block saying 'imports omitted for brevity' and to put ... where the setup used to be. Sign off on that or propose something better.",
      "expected_output": "The proposal rejected, the region widened to cover the runnable file, setup hidden rather than deleted, and the source-of-truth patterns ranked with one picked per surface.",
      "files": [],
      "expectations": [
        "Rejects the docs team's proposal rather than signing off on it",
        "Rejects putting ... inside a copyable block outright",
        "States that omissions, where needed at all, are marked in the language's own comment syntax",
        "States that a region starting after the imports publishes a sample the reader cannot run",
        "Requires the extracted region to wrap the maximum useful code rather than the minimum interesting lines",
        "Distinguishes the tested artefact, which is the complete file, from the published artefact, which is a region or filtered view of it",
        "States the rule as hide setup, never delete it",
        "Cites the rustdoc pattern of prefixed lines that compile but do not render as the property to reproduce whatever the mechanism",
        "Requires each region identifier to be globally unique, snake_case or dotted, prefixed by the product area, and identical across languages",
        "Requires one sample per file in the source project so its imports can be validated",
        "Ranks extract-from-tested-source above test-what-is-embedded and above generate-from-spec on reliability",
        "Says to pick one source-of-truth pattern per surface and write that choice into the policy, and states that a sample hand-pasted into prose belongs to none of the three"
      ]
    },
    {
      "id": 12,
      "prompt": "Marlowe Freight. I need a sample audit of our docs tree today. Two things you should know. This environment is air-gapped: no network, no package registry access, and the only runtime installed is Python, while our corpus is Python, Java, Ruby and Go. Also, our key-rotation guide and our security-hardening page are deliberately full of key-shaped strings and --insecure examples; last quarter's scanner reported 63 P0 findings and the team stopped reading the reports entirely. Give me the audit.",
      "expected_output": "An audit that labels what it could not execute as unverified rather than passing, still delivers tiers, mechanical lint and a verification plan, and confirms each scanner flag instead of accepting all 63 as P0.",
      "files": [],
      "expectations": [
        "Refuses to label unexecuted samples as passing",
        "Labels every sample it could not execute as unverified",
        "Still delivers tier assignments for the whole corpus despite being unable to execute anything",
        "Still delivers the mechanical lint results that need no runtime, such as fence language identifiers, prompt characters, ellipses, placeholder style, block length and trailing whitespace",
        "Delivers a written verification plan the team can implement once a runtime and registry access exist",
        "Does not silently downgrade the audit or present its partial coverage as a corpus-wide verdict",
        "States that every scanner flag is a candidate to confirm rather than a verdict",
        "Confirms the key-rotation and hardening pages individually instead of accepting all 63 flags as genuine P0 findings",
        "Names a credential-rotation page as the documented legitimate case for a key-shaped string appearing in a sample",
        "Routes deliberate anti-pattern samples such as the --insecure examples to tier Illustrative with a visible not-runnable label rather than filing them as P0 findings",
        "States that any genuine P0 is reported immediately rather than held to the end of the report",
        "States that scanner output nobody reads documents a problem rather than gating against it, and requires a non-zero exit that fails a check and posts the failure where reviewers see it"
      ]
    }
  ],
  "trigger_queries": [
    { "query": "Define the policy every code sample in our developer documentation must meet.", "should_trigger": true },
    { "query": "Audit the code samples across our documentation site and give me a ranked fix queue.", "should_trigger": true },
    { "query": "We need execution tiers for the snippets in our API docs.", "should_trigger": true },
    { "query": "Establish standards for runnable examples in our SDK documentation.", "should_trigger": true },
    { "query": "Assess our documentation's code sample corpus for correctness and copy-paste safety.", "should_trigger": true },
    { "query": "What should our code sample review checklist contain before a docs pull request merges?", "should_trigger": true },
    { "query": "Design a verification strategy for the examples published in our developer docs.", "should_trigger": true },
    { "query": "We want to enforce language parity for our SDK snippets across Python, Go and Java.", "should_trigger": true },
    { "query": "Who should own a docs code sample when it breaks in CI?", "should_trigger": true },
    { "query": "Set a freshness window for the examples on our documentation site.", "should_trigger": true },
    { "query": "our docs examples are all broken, help", "should_trigger": true },
    { "query": "half the snippets in our docs don't compile anymore", "should_trigger": true },
    { "query": "people keep copy-pasting our examples and they blow up", "should_trigger": true },
    { "query": "the code blocks in our guides are out of date", "should_trigger": true },
    { "query": "can you check our docs for leaked API keys in the code blocks", "should_trigger": true },
    { "query": "we shipped a breaking change and now every snippet in the docs is wrong", "should_trigger": true },
    { "query": "how do I test the examples in my documentation in CI?", "should_trigger": true },
    { "query": "should this snippet be tested in CI or not", "should_trigger": true },
    { "query": "why don't the examples in our docs work", "should_trigger": true },
    { "query": "our Python examples work but the Go ones don't", "should_trigger": true },
    { "query": "Readers keep filing issues saying they pasted what we published and got a stack trace.", "should_trigger": true },
    { "query": "Support says our number one ticket driver is people copying something off our website that doesn't run.", "should_trigger": true },
    { "query": "There's a dollar sign at the start of every line in our command blocks and people paste it.", "should_trigger": true },
    { "query": "Some of our pages say YOUR_KEY_HERE and others say <API_KEY> and nobody knows which is right.", "should_trigger": true },
    { "query": "We have 400 fenced blocks across the docs repo and no idea how many still work.", "should_trigger": true },
    { "query": "Somebody found a live-looking secret key in one of our published guides.", "should_trigger": true },
    { "query": "Nothing in our docs repo is tested and the SDK team just renamed half the public API.", "should_trigger": true },
    { "query": "Our examples show three lines and readers say they can't actually run them.", "should_trigger": true },
    { "query": "We put dot-dot-dot in the middle of our copyable blocks and it confuses everyone.", "should_trigger": true },
    { "query": "The docs build is green but the examples on the page are red.", "should_trigger": true },
    { "query": "Draft a snippet policy for our contributing guide.", "should_trigger": true },
    { "query": "How do we single-source examples from our test suite into the docs?", "should_trigger": true },
    { "query": "We want to use transclusion to inject code into docs pages at build time.", "should_trigger": true },
    { "query": "Is it OK to leave the imports out of published samples?", "should_trigger": true },
    { "query": "How many of our docs samples should be tested before we call it good?", "should_trigger": true },
    { "query": "Score our documentation examples against a rubric.", "should_trigger": true },
    { "query": "Build a parity matrix for our SDK examples.", "should_trigger": true },
    { "query": "Which languages should get every documented example and which get only the entry path?", "should_trigger": true },
    { "query": "We converted our docs snippets into Go example functions - are they actually being run?", "should_trigger": true },
    { "query": "Set up doctests so our documentation examples fail the build when they drift.", "should_trigger": true },
    { "query": "How do we stop our docs samples drifting after every SDK release?", "should_trigger": true },
    { "query": "What severity should an example containing a wildcard IAM policy get?", "should_trigger": true },
    { "query": "Our sample test suite got disabled after it went flaky - how do we bring it back?", "should_trigger": true },
    { "query": "We need code owners for the examples in our docs repo.", "should_trigger": true },
    { "query": "Write the intake rules for our code examples backlog.", "should_trigger": true },
    { "query": "How often should we re-run the examples published in our documentation?", "should_trigger": true },
    { "query": "Should we delete the one-line snippets on our reference pages?", "should_trigger": true },
    { "query": "We have the same auth snippet pasted on 40 different pages.", "should_trigger": true },
    { "query": "Our examples repo and our docs site disagree about how to authenticate.", "should_trigger": true },
    { "query": "What has to be in a code example besides the code itself?", "should_trigger": true },
    { "query": "Review this curl example before we publish it on the payments page.", "should_trigger": true },
    { "query": "Is verify=False acceptable in a documentation example?", "should_trigger": true },
    { "query": "Our docs show destructive commands with no warning above them.", "should_trigger": true },
    { "query": "How do we prove to leadership that our documentation examples actually work?", "should_trigger": true },
    { "query": "I want a metric for how long a red example takes to get fixed.", "should_trigger": true },
    { "query": "Which of our examples should we invest in testing first?", "should_trigger": true },
    { "query": "We have 20,000 code samples across nine languages and no process at all.", "should_trigger": true },
    { "query": "Our newest SDK's examples read like translated Python.", "should_trigger": true },
    { "query": "An example in a post from 2021 still ranks and people still copy it.", "should_trigger": true },
    { "query": "What's the right way to mark an example as pseudocode?", "should_trigger": true },
    { "query": "Set a maximum length for code blocks in our documentation.", "should_trigger": true },
    { "query": "Our fenced blocks are missing their language tags.", "should_trigger": true },
    { "query": "How should we handle expected output in our examples?", "should_trigger": true },
    { "query": "Our CI builds the SDK from the monorepo and runs the docs samples against staging - is that enough?", "should_trigger": true },
    { "query": "The samples pass our tests but readers say they fail.", "should_trigger": true },
    { "query": "Do we need separate test accounts for our documentation example suite?", "should_trigger": true },
    { "query": "Our example suite leaks cloud resources every night and finance noticed.", "should_trigger": true },
    { "query": "Rank the problems we found in our documentation examples so we know what to do first.", "should_trigger": true },
    { "query": "How do we stratify a quality review across a huge corpus of published examples?", "should_trigger": true },
    { "query": "Give me the published evidence that untested code snippets usually don't work.", "should_trigger": true },
    { "query": "We want a 'verified' badge next to the examples on our docs site.", "should_trigger": true },
    { "query": "Our quickstart's first API call has never been executed by anything.", "should_trigger": true },
    { "query": "Write the exceptions register for our snippet rules.", "should_trigger": true },
    { "query": "How do we keep documentation examples honest when we can't run anything in this environment?", "should_trigger": true },
    { "query": "Write the quickstart for our new API.", "should_trigger": false },
    { "query": "Audit our quickstart - how long does it take a cold reader to get to first success?", "should_trigger": false },
    { "query": "Our zero-to-first-call page is too long and people drop off.", "should_trigger": false },
    { "query": "Write a tutorial that teaches someone our streaming API from scratch.", "should_trigger": false },
    { "query": "Our tutorial jumps around and readers get lost halfway through.", "should_trigger": false },
    { "query": "Add checkpoints to our learning tutorial so people can resume where they stopped.", "should_trigger": false },
    { "query": "Classify our docs pages against the Diataxis modes.", "should_trigger": false },
    { "query": "Where should this page live in our docs navigation?", "should_trigger": false },
    { "query": "Our docs sidebar has eleven top-level sections and nobody can find anything.", "should_trigger": false },
    { "query": "Audit our documentation's information architecture.", "should_trigger": false },
    { "query": "Make our docs crawlable and usable by coding agents.", "should_trigger": false },
    { "query": "Do we need an llms.txt file for our documentation?", "should_trigger": false },
    { "query": "Publish markdown endpoints so an agent can read our docs unattended.", "should_trigger": false },
    { "query": "Write the v3 to v4 migration guide.", "should_trigger": false },
    { "query": "Order our breaking changes by blast radius for the upgrade guide.", "should_trigger": false },
    { "query": "How long should our deprecation timeline be before we remove the old endpoint?", "should_trigger": false },
    { "query": "Build error-reference pages from our support tickets.", "should_trigger": false },
    { "query": "Developers paste our error string into Google and find nothing.", "should_trigger": false },
    { "query": "Generate an error-code catalog for our API.", "should_trigger": false },
    { "query": "Our docs don't rank for the queries developers actually type.", "should_trigger": false },
    { "query": "Set canonical tags across our versioned documentation.", "should_trigger": false },
    { "query": "Fix the hreflang tags on our translated docs.", "should_trigger": false },
    { "query": "Rewrite our repository README so a newcomer understands the project in a minute.", "should_trigger": false },
    { "query": "Which six repositories should I pin on my GitHub profile?", "should_trigger": false },
    { "query": "Write a technical blog post about how we cut our p99 latency in half.", "should_trigger": false },
    { "query": "Turn these merged pull requests into release notes.", "should_trigger": false },
    { "query": "Our CHANGELOG is a raw git log - fix it.", "should_trigger": false },
    { "query": "Are our endpoint reference pages complete for every parameter and error response?", "should_trigger": false },
    { "query": "Document the error responses for each endpoint in our API reference.", "should_trigger": false },
    { "query": "Design idempotency keys for our payments API.", "should_trigger": false },
    { "query": "Which error codes should our API return and how should they be structured?", "should_trigger": false },
    { "query": "Write our CONTRIBUTING file and label some good first issues.", "should_trigger": false },
    { "query": "Our first-time contributors can't get the project's test suite running locally.", "should_trigger": false },
    { "query": "Design our issue triage labels and response targets.", "should_trigger": false },
    { "query": "Which open-source license should we pick for this project?", "should_trigger": false },
    { "query": "Plan our open-source launch day.", "should_trigger": false },
    { "query": "Build a keyword list from our docs search logs and support tickets.", "should_trigger": false },
    { "query": "Which DevRel metrics should we report to the board?", "should_trigger": false },
    { "query": "Plan a quarter of developer content with owners and dates.", "should_trigger": false },
    { "query": "Benchmark a competitor's developer relations motion from public signals.", "should_trigger": false },
    { "query": "Script a five-minute screencast showing our CLI.", "should_trigger": false },
    { "query": "Turn my accepted abstract into a rehearsable talk outline.", "should_trigger": false },
    { "query": "Prep me for a podcast interview about our platform.", "should_trigger": false },
    { "query": "Design the live demo so it survives bad conference wifi.", "should_trigger": false },
    { "query": "Map the developer journey from discovery through advocacy.", "should_trigger": false },
    { "query": "Cut our developer audience into named segments.", "should_trigger": false },
    { "query": "Should we open-source our ingestion engine?", "should_trigger": false },
    { "query": "Set the limits on our developer tool's free tier.", "should_trigger": false },
    { "query": "Write a case study about a customer's production deployment.", "should_trigger": false },
    { "query": "Design our developer champions program.", "should_trigger": false },
    { "query": "Review this pull request for bugs and security issues.", "should_trigger": false },
    { "query": "Our product's own source has almost no tests - write some.", "should_trigger": false },
    { "query": "Set up CI for our monorepo.", "should_trigger": false },
    { "query": "Add a linter and a formatter to our Go services.", "should_trigger": false },
    { "query": "Write a code style guide for our backend team.", "should_trigger": false },
    { "query": "Refactor this function, it's 300 lines long.", "should_trigger": false },
    { "query": "Our unit tests are flaky, figure out why and fix them.", "should_trigger": false },
    { "query": "Generate a client SDK from our OpenAPI spec.", "should_trigger": false },
    { "query": "Build a snippet library for our team's editor.", "should_trigger": false },
    { "query": "Create VS Code snippets for our framework.", "should_trigger": false },
    { "query": "Scan our application repository for hardcoded secrets.", "should_trigger": false },
    { "query": "Rotate the API keys that leaked in our production config.", "should_trigger": false },
    { "query": "Write docstrings for every public function in this module.", "should_trigger": false },
    { "query": "Our Go package has no godoc comments on its exported types.", "should_trigger": false },
    { "query": "Add type hints across our Python codebase.", "should_trigger": false },
    { "query": "Set up dependency update automation for our SDK repositories.", "should_trigger": false },
    { "query": "Pin our dependency versions so builds are reproducible.", "should_trigger": false },
    { "query": "Write the API design guidelines for our platform team.", "should_trigger": false },
    { "query": "Draft the README for our examples repository.", "should_trigger": false },
    { "query": "Our documentation site build takes eleven minutes - speed it up.", "should_trigger": false },
    { "query": "Migrate our docs site off Docusaurus onto a new generator.", "should_trigger": false },
    { "query": "Translate our documentation into Japanese.", "should_trigger": false },
    { "query": "Write the copy for our developer homepage.", "should_trigger": false },
    { "query": "Our sales team needs a demo script for the platform.", "should_trigger": false }
  ]
}

# Sample audit checklist

Run the mechanical checks with `scripts/sample-audit.py` first, then apply judgement to what the script cannot see. The script finds pattern defects; only a reader who understands the product finds a sample that runs fine and teaches the wrong thing.

## Severity anchors

| Severity | Test                           | Examples                                                                                                                                |
| -------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| P0       | Harms the reader who copies it | Real-format credential, destructive command without a guard, disabled TLS verification, wildcard IAM policy, data-loss default          |
| P1       | Does not work as written       | Missing import, removed or renamed symbol, wrong argument order, documented output that no longer matches, unpinned version that broke  |
| P2       | Works but violates policy      | No execution tier, prompt characters, no expected output, ellipsis omission, missing language identifier, parity gap, stale-but-passing |

Report P0 findings the moment you see them rather than at the end of the audit - a leaked credential does not wait for the report.

## Per-sample checks

Mechanical (script covers these):

- [ ] Fence carries a language identifier.
- [ ] No prompt character starting a command line.
- [ ] Command output not interleaved in the command block.
- [ ] No `...`, `…` or HTML ellipsis entity inside the block.
- [ ] Placeholders use the policy's format; no bare `YOUR_KEY_HERE` variants mixed in.
- [ ] No credential-shaped literal (long base64/hex string, `sk_live`-style prefix, `-----BEGIN * PRIVATE KEY-----`).
- [ ] No insecure switch (`--insecure`, `verify=False`, `rejectUnauthorized: false`, `InsecureSkipVerify`).
- [ ] No unguarded destructive command.
- [ ] Block length within the policy cap.
- [ ] No trailing whitespace (screen readers announce it).

Judgement (human or agent, per sample):

- [ ] Every symbol used exists in the current public API surface.
- [ ] All imports present; the sample would run in a fresh file.
- [ ] Client setup shown, with reuse and cleanup addressed.
- [ ] One code path; nothing extraneous to the point being made.
- [ ] Error handling specific, and present only where it is intrinsic to the example.
- [ ] Expected output present and currently accurate.
- [ ] Comments add non-obvious context.
- [ ] Execution tier assigned, and honest - a "Run" sample that CI does not run is a P1, not a P2.
- [ ] Version pinned in install commands where a floating version can break the path.
- [ ] The scenario matches the sibling-language samples of the same identifier.

## Finding format

One finding per line, evidence first, fix in one sentence. Prose paragraphs get triaged by nobody.

```
docs/guides/webhooks.md:142 - sample calls client.verify(payload, sig); the method was renamed
  to verifySignature in v4 (breaking change, 2026-03). Fix: update the call and re-pin the
  install block to ^4.
```

## Worked pair

Negative - five defects in six lines:

````markdown
```
$ curl -X POST https://api.example.com/v1/charges \
  -H "Authorization: Bearer sk_live_4eC39HqLyjWDarjtT1zdp7dc" \
  -d amount=2000 ...
{"id":"ch_1","status":"succeeded"}
```
````

Five defects:

- No language identifier.
- A prompt character that breaks copy-paste.
- A live-format secret key.
- An ellipsis inside a copyable block.
- Response body glued to the request so both get copied together.

Positive - same scenario, policy-compliant:

````markdown
Create a charge. Replace `<YOUR_API_KEY>` with the test key from your dashboard.

```bash
curl -X POST https://api.example.com/v1/charges \
  -H "Authorization: Bearer <YOUR_API_KEY>" \
  -d amount=2000 \
  -d currency=eur
```

Expected response:

```json
{ "id": "ch_1", "status": "succeeded", "amount": 2000 }
```
````

The fix costs four lines and removes every P0 and P2 in the original. Most of a first audit looks exactly like this: cheap, mechanical, and never done because nobody owned it.

## Corpus-level checks

- [ ] Parity matrix filled; coverage computed, not estimated.
- [ ] Every tier-1 scenario has a sample; every sample maps to a named scenario.
- [ ] Duplicate samples for the same scenario reconciled to one source.
- [ ] Orphan samples (no page links to them) either linked or deleted.
- [ ] Samples in archived posts carry a version banner or are removed.
- [ ] Freshness: samples whose last successful run exceeds the window, counted and listed.

# Corpus audit method

How to audit hundreds or thousands of samples without spot-checking. The method is borrowed wholesale from content strategy, because a sample corpus is a content corpus that happens to compile.

## Contents

- Inventory before audit
- ROT triage
- Census plus stratified sample
- Rubric scoring
- Four-stage remediation
- Negative example: the spot-check that isn't an audit

## Inventory before audit

Nielsen Norman Group draws the line precisely: a content **inventory** is "a list of every piece of digital content you currently have"; a content **audit** "examines, assesses, and evaluates the quality of the content". Two artefacts, two passes, in that order.

Build the inventory mechanically before anyone forms an opinion. One row per sample:

`language | product or service area | file path | region-tag or identifier | last-commit date | line count | inline-fenced or extracted-from-source`

The census is what stops an audit from becoming a tour of the pages someone already suspected. Regions of the corpus nobody remembers are exactly where the worst samples live.

## ROT triage

Content strategy's ROT lens transfers directly, and it is the cheapest first cut on a large corpus.

- **Redundant** - the same snippet duplicated across pages. Reconcile to one source; every copy is a future divergence.
- **Outdated** - references a deprecated API, a removed symbol, or a retired auth flow. This is the class with a measured baseline (see `published-findings.md`).
- **Trivial** - a single-API-call snippet with no scenario around it. AWS's own contribution guidance explicitly prefers "code examples that cover broader scenarios and use cases, versus simple code snippets that cover only individual API calls".

Trivial is the category teams resist cutting, because the snippet is not _wrong_. It is still maintenance debt with no reader payoff.

## Census plus stratified sample

Two tiers, chosen by what each check costs.

**Census - everything mechanical, on 100% of the corpus.** Compilation, execution, secret scanning, fence-language checks, prompt-character checks. Marginal cost per sample is near zero, so sampling here saves nothing and misses real defects. Every failure is an automatic finding, no judgement required.

**Stratified sample - everything that needs a reader.** Idiomaticity, whether the scenario is worth documenting, whether the comments add anything, whether the sample teaches a pattern you want copied. Stratify by three axes, because these are where quality drift concentrates:

- **Language** - the language added last is usually the weakest.
- **Product or service area** - coverage follows organisational attention, not reader need.
- **Age** - old samples fail differently from new ones.

Sample size is a judgement call rather than a settled standard. Pick a number you can actually score, state it in the report, and treat any figure you quote from the sample as an estimate with a stated base - never as a corpus-wide count.

## Rubric scoring

Score the stratified sample against a fixed rubric so two reviewers reach comparable numbers. Two published rubrics are worth adapting rather than inventing one: Tom Johnson's 80-criterion API-documentation rubric (Findability, Accuracy, Relevance, Clarity, Completeness, Readability), and the 0–2-per-dimension / 20-point docs rubric with its ≥16 publishable and ≥18 world-class marks (both cited in `published-findings.md`).

The code-sample-specific dimensions, each supported by vendor guidance:

| Dimension           | 0                                                    | 1                                      | 2                                                           |
| ------------------- | ---------------------------------------------------- | -------------------------------------- | ----------------------------------------------------------- |
| Correctness         | does not compile or run                              | compiles, output undocumented or wrong | compiles, runs, output asserted                             |
| Runnability         | fragment; imports elided                             | runs after obvious edits               | runs as copied, given the stated prerequisites              |
| Idiomaticity        | transliterated from another language                 | passes the formatter                   | reads as a native of the language                           |
| Error handling      | blanket catch, or a process-killing helper           | present but generic                    | specific, and only where intrinsic to the example           |
| Credential handling | real-format secret or plausible fake                 | placeholder with no explanation        | explained placeholder, reserved-range host, no secret shape |
| Copy-paste safety   | prompt characters, ellipses, output glued to command | one defect                             | clean, tagged, output in its own block                      |

Record the score per dimension, not a single total - the totals hide which axis the corpus is failing on, which is the only thing that tells you what to fix.

## Four-stage remediation

1. **Inventory (census).** The scanner walks the corpus and emits the row format above.
2. **Automated gates (census-level audit).** Compile, run, secret-scan, lint. Every failure is a finding, filed with severity, before a human reads anything.
3. **Rubric scoring (sample-level audit).** Score the stratified sample; record severity per finding.
4. **Governance for the steady state.** This is the stage that stops the audit recurring - everything before it is a one-off cleanup:
   - Migrate samples into tagged, compilable source with validated metadata.
   - Enforce ownership with code-owner rules.
   - Inject into docs by transclusion rather than by hand-pasting.

Stage 4 is the investment. Stages 1–3 tell you how bad it is; only stage 4 changes the rate at which it gets bad again.

## Negative example: the spot-check that isn't an audit

**What teams do, and why it is tempting:**

> Opened the twelve most-visited docs pages, read every code block, fixed four broken ones. Reported: "sample quality is good, minor issues on the storage pages."

Three things are wrong with it:

- The twelve most-visited pages are the twelve most-maintained pages, so the sample is biased toward the healthiest region of the corpus.
- Nothing was compiled, so "read every code block" caught only defects visible to the eye.
- The conclusion is a corpus-wide claim drawn from a convenience sample, which is the specific error the census/stratified split exists to prevent.

**What the same effort produces done properly:**

> Census: 412 samples inventoried, 380 compiled, 32 failed (92.2% pass → above the 90% gate, no remediation sprint required). Secret scan: 2 findings, both P0, reported immediately.
> Stratified sample: 40 samples scored (stratified by language × age), mean 13.4/20; error handling is the weakest dimension at 0.9/2, concentrated in the Ruby and Java sets.
> Conclusion: the corpus compiles; it does not teach error handling. Fix queue targets that, not the four broken pages.

Same afternoon, a finding the team can act on, and a number that can be recomputed next quarter.

# Governance and ownership

Ownership stated in a wiki decays. Ownership encoded in the repository holds, because the build enforces it. This file covers the three ownership models in production, the metadata layer that makes a corpus governable, and the gates that turn a policy into a merge requirement.

## Contents

- Choosing an ownership model
- Intake and prioritisation
- Metadata-driven governance
- The enforcement layer
- Migration order

## Choosing an ownership model

| Model                                | Who authors                                                                 | Strengths                                                                        | Costs                                                           | Fits                                                   |
| ------------------------------------ | --------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------ |
| Dedicated examples team              | A standing team separate from product engineering and from the docs writers | Highest and most consistent quality; per-language standards actually get applied | Slow intake; new examples queue behind maintenance              | Large multi-language SDK surfaces                      |
| DevRel/community with expert vetting | Contributors and advocates, reviewed by named experts                       | Breadth and scenario variety; catches use cases the vendor would not think of    | Needs paid capacity behind it or the backlog rots               | Catalogue-style example galleries                      |
| Product engineers co-own             | The team that shipped the feature                                           | Samples stay current by construction; no handoff lag                             | Depends entirely on the engineering org counting docs as "done" | Single-product APIs with a strong docs-as-code culture |

Real practice, for calibration:

- AWS runs the dedicated-team model with a separate documentation team alongside it, taking all example requests through a public issue tracker and moving accepted work across a public roadmap (Wish List → Backlog → In Progress → Recently Completed).
- Twilio's community catalogue is vetted by named experts, and its growth was backstopped by _contracted_ senior engineers - one engagement covered authoring 10 new apps and maintaining 45 existing ones. Treat "community-maintained" as a funding model, not a free one.
- Stripe has product engineers co-own the docs for what they ship, with docs part of the definition of done and reflected in performance reviews.

Pick one per surface rather than one for the whole corpus. An examples repository and an API reference rarely deserve the same model.

## Intake and prioritisation

Publish the queue rules, whatever the model. AWS states two that generalise well:

- Large examples need requesting well ahead of the date they are needed - its stated guidance is at least two months.
- "Bug fixes and security issues take priority over new code examples."

That second rule is the one teams skip, and skipping it is how a corpus accumulates hundreds of samples nobody can fix. Write it into the policy's ownership section with a fix SLA per surface.

## Metadata-driven governance

A governed corpus keeps samples in compilable source and describes each one in structured metadata, so the docs build can pull the right snippet for the right page in the right language.

Two mechanisms, always both:

**Snippet tagging.** Paired markers delimit the publishable region inside a real source file. Give each tag a globally unique, hierarchical, dotted or snake_case identifier carrying language, product area and example name, and keep it identical across languages so the parity matrix can key on it.

**Example metadata.** One record per example, holding at minimum: title, short title, synopsis, category, the languages it exists in, the SDK version it targets, the source location, and the snippet tags it composes. The tag alone cannot tell a renderer which page a snippet belongs on or which language variant to select - that is what the metadata record is for.

Transclusion is the unifying pattern underneath: keep canonical code in real, compilable source, mark a region, inject it at build time. Every major docs toolchain has a form of it - paired start/end markers, line-range includes, or an include directive naming a file plus a marker.

The published sample is never hand-typed prose; it is a _view_ into a file CI already validates. That is what turns "keep samples accurate" from an editorial promise into a build step.

## The enforcement layer

Once samples are source code, ordinary code governance applies to them. Build these as blocking gates, not advisory checks:

- **Code owners on sample paths.** Map per-language or per-product directories to required reviewers, so a pull request touching Go examples cannot merge without a Go-example owner's approval.
- **Metadata schema validation.** Reject a record with missing or malformed fields before it reaches the renderer.
- **Referenced-file existence checks.** Assert that every snippet file named in metadata still exists - this is what catches a moved or deleted source file before the docs ship a blank.
- **Secret scanning over the whole corpus.** Scan for a disallowed word list and for credential-shaped literals (long fixed-length base64/hex strings, known token prefixes, private-key headers). Run it in the same tool that validates structure, so there is one exit code to gate on.
- **Non-zero exit fails the check.** Post the failure on the pull request. A validator whose output nobody sees is documentation of a problem, not a gate against it.
- **Snippet-sync assertions.** Where Markdown embeds extracted code, re-run the extractor in CI and fail on drift, so an edit to the source file cannot silently desynchronise the published copy.
- **Per-language standards.** Encode the conventions each language's samples must follow - required first parameters, return shapes, mandatory doc comments, module layout, even formatting rules justified by how the snippet will render rather than by language convention alone. Put them where contributors will find them, next to the code.

External contributions deserve the same gate as internal ones. AWS describes outside snippet sources as "tributaries, flowing into the 'river' of AWS Documentation" - the validator is the checkpoint every tributary passes through.

## Migration order

When a corpus has no governance at all, sequence it this way - each step makes the next cheaper:

1. **Census and secret scan.** Find the credential leaks first; they are the only findings that cannot wait.
2. **Automated compile/run gates** on whatever is already testable, at 100% coverage.
3. **Move samples into compilable source** and publish by transclusion. This is the single highest-leverage investment for a corpus with no region-tag or metadata coverage, because every later guarantee depends on it.
4. **Add metadata records and schema validation.**
5. **Assign code owners and fix SLAs**, once the paths are stable enough to own.

Reversing steps 3 and 5 is the common mistake: assigning owners to hand-pasted Markdown gives them responsibility without any mechanism to discharge it.

# Language parity matrix

## What parity means

Parity is a claim about coverage and behaviour, never about line-by-line translation. Across every language version of a sample, three things stay identical:

- the sample identifier,
- the scenario it demonstrates and the order of its steps,
- the testing approach applied to it.

Everything else may differ, because idiomatic style outranks cross-language uniformity. A Python sample that reads like transliterated Java satisfies a diff and fails the reader.

## Tiering the languages

Uniform coverage across every SDK is a promise no team keeps. Tier instead, and publish the tiers.

| Tier      | Commitment                                                                             | Chosen from                                               |
| --------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| Tier 1    | Every documented scenario, tested                                                      | Actual download/telemetry share and support-ticket volume |
| Tier 2    | Entry path (install, auth, first call) plus top scenarios; deeper pages link to tier 1 | Meaningful but secondary usage                            |
| Community | Present, unowned, labelled untested, with the maintenance expectation stated           | Contributed, not staffed                                  |

Choosing tiers from real usage rather than team preference is the whole point - most corpora over-serve the language the API was first written in, usually the maintainer's own, and under-serve the one most readers arrive with.

## The matrix

Rows are scenarios (one per sample identifier), columns are the languages. Cell values:

- `T` tested - exists and CI runs it
- `P` present - exists, untested
- `M` missing
- `S` stale - pinned to an unsupported version, or last successful run older than the freshness window

```
Scenario                      | py | ts | go | java | rb
------------------------------|----|----|----|------|----
auth_api_key                  | T  | T  | T  | T    | P
storage_upload_file           | T  | T  | P  | M    | M
storage_signed_url            | T  | M  | M  | M    | M
webhooks_verify_signature     | T  | T  | T  | S    | M
```

Parity coverage = `T` cells ÷ (scenarios × tier-1 languages). One number, recomputable each release, and it separates the two failure shapes a raw sample count hides: samples that do not exist, and samples that exist but nothing runs.

Read the matrix by row and by column. A sparse **row** means a scenario the docs only support in one language - usually a feature shipped with a single SDK. A sparse **column** means a language tiered above what the team actually staffs; either fund it or demote it.

## Drift patterns to check

- **Different outcome**: one language's sample creates a resource the others only read.
- **Auth divergence**: one uses an environment variable, another a key file path, a third an inline literal.
- **Error handling in one language only**, usually the one whose sample was written last.
- **Version skew**: samples pinned to different major versions of the same SDK across languages.
- **Orphan scenario**: a sample exists in one language with no matching identifier anywhere else - either the scenario matters (add it) or it does not (delete it).
- **Structural drift**: the same identifier demonstrating a different step order, so a reader switching languages cannot follow the same prose.

## Generated versus hand-written

Generated snippets guarantee parity and freshness for single calls on reference pages, at near-zero cost per added language. Hand-written samples carry multi-step idiomatic usage and drift per language. Most corpora need both - so state in the policy which surface uses which, or the team hand-maintains what a generator already covers, and the two sets disagree in public.

# Published findings and baselines

Published figures on code-sample quality, the baselines this skill sets for itself, and the circulating claims it refuses to repeat. Cite from this file rather than from memory - a fabricated benchmark in a sample-quality audit destroys the credibility of the whole report.

## Contents

- Compilability and executability
- Staleness and deprecation
- Why sample quality matters
- Corpus scale
- Rubric anchors
- Self-set baselines
- Claims to never repeat

## Compilability and executability

**Yang, Hussain & Lopes, "From Query to Usable Code" (MSR 2016).** Roughly 3 million snippets from accepted Stack Overflow answers, four languages:

| Language                | Parsable | Compilable / runnable |
| ----------------------- | -------- | --------------------- |
| Java (914,974 snippets) | 3.89%    | 1.00% compilable      |
| C#                      | -        | 0.12% compilable      |
| Python                  | 76.22%   | 25.61% runnable       |
| JavaScript              | 65.88%   | 20.00% runnable       |

Dynamic languages were measured as runnable rather than compilable, so the two halves of the table are not directly comparable. Use this to answer "surely our samples are mostly fine" - in a mature, heavily upvoted corpus, statically typed snippet compilability is near zero.

**Horton & Parnin, "Gistable" (ICSME 2018).** 10,259 Python code snippets, ~5,000 with a Dockerfile: 75.6% "require non-trivial configuration to overcome missing dependencies, configuration files, reliance on a specific operating system, or some other environment configuration." Only 24.4% ran without error out of the box. This is the basis for testing from a clean environment rather than a developer laptop.

## Staleness and deprecation

**Zhou & Walker (FSE 2016).** 43% of the Stack Overflow posts investigated contained deprecated APIs. Their Deprecation Watcher reached 100% precision and 86% recall on a 200-question sample - automated deprecation detection is tractable, not aspirational. The skill uses 43% as the baseline above which a deprecation gate outranks other remediation work.

## Why sample quality matters

**Uddin & Robillard, "How API Documentation Fails" (IEEE Software, 2015, DOI 10.1109/MS.2014.80).** Survey of 323 professional developers plus analysis of 179 API documentation units. Three severest problems: ambiguity, incompleteness, incorrectness. Six of ten documented problems were rated outright "blockers" by respondents.

**Stack Overflow 2024 Developer Survey.** 84% of developers who do not use Stack Overflow itself rely on technical documentation to learn; 90% of those use documentation found specifically in API and SDK packages.

**Postman State of the API (2019 wave).** The most-requested enhancement from API producers was "better examples in the documentation" at 63.5%, ahead of standardization (59.4%) and sample code generally (57.8%). Postman has characterised documentation debt as a persistent top-four industry problem across subsequent reports.

## Corpus scale

Useful for calibrating what "large" means before proposing a process:

- **Twilio**: docs "house more than 5,000 pages and nearly 20,000 code samples across nine coding languages."
- **AWS**: the SDK Code Examples library spans 14 SDKs/languages. The public examples repository shows roughly 10,500 stars and 5,800 forks - an adoption proxy, not a quality metric.

## Rubric anchors

Two real, published docs rubrics transfer to samples:

- Tom Johnson's API-documentation rubric: 80 criteria across Findability, Accuracy, Relevance, Clarity, Completeness and Readability.
- A widely circulated docs rubric scoring 0–2 per dimension on a 20-point scale, with "publishable = ≥16/20" and "world-class = ≥18/20".

Use the 20-point form for the stratified human pass; it is fast enough to apply to 50–100 samples in a sitting and its two thresholds are defensible because they are somebody else's, not yours.

## Self-set baselines

State these as baselines whenever you report against them. Each is a self-set default rather than a published coverage standard, chosen to be demanding but reachable:

- **≥80% of the corpus tested** (tier Run or Compile-only, executed by CI).
- **≥90% parity coverage** across tier-1 languages.
- **100% of primary-adoption-path samples at tier Run** - the strictest of the three, and the easiest to defend, since the quickstart and auth path is where a broken sample costs a prospective user entirely.
- **Freshness window** - deliberately left for the team to set; it depends on release cadence, and any fixed number here would be arbitrary.
- **Severity model (P0/P1/P2)** - adapted from general QA practice and applied to samples here.
- **The under-90% compile/run remediation trigger** - a self-set gate rather than a vendor-published one, well supported by the MSR 2016 numbers above.

Google Cloud requires integration tests for critical paths, which is the nearest thing to a public vendor requirement on sample verification.

## Claims to never repeat

- **"94% of developers consider API documentation quality a key factor."** Widely attributed to SlashData and repeated across blogs, yet not confirmed in any SlashData report. Do not cite it.
- **"Single Origin Snippets"** as an expansion of AWS's "SoS" metadata. Only "SoS metadata" appears in the source repository; the expansion is unverified.
- **Any per-vendor claim about snippet-tool feature support** without checking that vendor's current docs. Tooling lists in this space go stale within months.

# Sample policy template

Fill every field. A blank field is a decision the first reviewer will make for you, differently each time. Keep the finished policy in the docs repository next to the contributing guide, not in a wiki nobody opens.

## 1. Scope

- Surfaces covered: docs site / API reference / README / examples directory / blog / in-product help.
- Surfaces explicitly out of scope, and why.
- Effective date and the review cadence (per release, per quarter).

## 2. Coverage - which scenarios get a sample

- Every tier-1 scenario listed by name (one row per scenario, matching the identifier used in the parity matrix).
- Selection rule: frequently used elements and elements that are tricky or easy to misuse come first.
- Exclusion rule: no sample for an obvious point or a contrived scenario. A sample that teaches nothing costs maintenance forever.
- Complexity rule: simple first, complexity only after the common scenarios are covered. Long multi-feature samples belong in tutorials where prose can carry them.

## 3. Anatomy - what every sample must contain

- [ ] All imports and dependencies, explicitly. No elided setup in the published artefact.
- [ ] One sample per file in the source project, so its imports can be validated.
- [ ] Top-level comment: what it does, what it assumes exists, link to the concept page.
- [ ] Explicit client/connection initialization, with a note on reuse, thread-safety and cleanup.
- [ ] Arrange / act / assert shape; nesting no deeper than three levels.
- [ ] One code path. Conditionals, loops and switches only when they _are_ the subject.
- [ ] Minimal parameters: only project-specific values, external paths, and what the test needs. Hard-code the rest.
- [ ] Expected output, in its own block or as a trailing comment.
- [ ] Comments that add context the code lacks - never a restatement of the line above.
- [ ] Language identifier on every fence; `plaintext` when nothing fits.
- [ ] Line length capped (80-100 characters) so samples survive narrow panes.

## 4. Safety - copy-paste and security rules

- [ ] No shell prompt characters (`$`, `#`, `PS>`) in command blocks.
- [ ] Commands and their output in separate blocks, so output cannot be copied with the command.
- [ ] Placeholders in angle brackets (`<YOUR_API_KEY>`), each explained in the surrounding prose.
- [ ] No plausible-looking fake values a reader could paste verbatim; no real-format credentials, account IDs or endpoints.
- [ ] Omissions marked with a comment in the language's own syntax - never `...` or `…`, and never inside a click-to-copy block.
- [ ] No disabled TLS verification, wildcard permissions, or credentials read from source.
- [ ] Input validation shown wherever the sample accepts user input.
- [ ] Destructive commands (delete, drop, force-push, recursive remove) carry an inline guard and a warning above the block.
- [ ] Errors caught specifically and logged; no blanket catch, no silent swallow, no process-killing helper.
- [ ] Sample code license stated when readers are expected to ship the code.

## 5. Parity

- Tier-1 languages (every scenario, tested) - list them and the evidence used to choose them.
- Tier-2 languages (entry path plus top scenarios) - list them.
- Community languages (present, unowned, labelled) - list them.
- What stays identical across languages: sample identifier, scenario, step order, testing approach.
- What may differ: everything else. Idiomatic beats uniform.

## 6. Verification

- Tier assignment rule and where each tier runs.
- Source of truth: samples live in `<path>` and are published into docs by `<mechanism>`.
- CI job: installs `<published artefact>`, runs on `<events>` plus a `<schedule>`, and fails `<the docs build / a required check>`.
- Fixture policy: which accounts/projects tests use, and who cleans up created resources.
- Freshness window: maximum age of a sample's last **successful run** before it is marked stale on the page.

## 7. Ownership

| Surface                 | Owner | Fix SLA for a red sample |
| ----------------------- | ----- | ------------------------ |
| Quickstart / auth path  |       |                          |
| API reference snippets  |       |                          |
| Examples repository     |       |                          |
| Blog and archived posts |       |                          |

Archived content deserves an explicit rule: freeze it with a version banner or delete it. A five-year-old post whose sample still ranks in search does more damage than a missing page.

## 8. Exceptions

One row per accepted violation: sample, rule waived, reason, expiry date. An exception without an expiry is a policy edit made quietly.

# Verification patterns

The purpose of verification is to move a sample's failure from the reader's machine into CI. Everything below serves that one move.

## Why it works

Ecosystems that made examples executable did it for the same reason: an example that is not run drifts silently. Rust's documentation tests compile and run every fenced block in a doc comment, so an outdated example fails the build. Go's testable examples compile with the package and assert stdout against an `// Output:` comment; the language team's stated payoff is that executable documentation "guarantees that the information will not go out of date as the API changes".

Manny Silva's **Docs as Tests** generalizes it beyond code: documentation should not just inform, it should also verify - every documented step becomes a test case against the real product surface.

## Choosing a source of truth

Three patterns, in decreasing reliability. Pick one per surface and write it into the policy.

**1. Extract from tested source.** Samples live in a real project that compiles, lints and tests; the docs pull a named region at build time.

- Give each region a globally unique, snake_case identifier prefixed by the product area, identical across languages.
- Wrap the maximum useful code - a region that starts after the imports publishes a sample the reader cannot run.
- Strongest guarantee, highest setup cost. Choose it for anything on the adoption path.

**2. Test what is embedded.** Samples stay in the Markdown; a CI step extracts every fenced block, reassembles it into a runnable file with the declared setup, and executes it.

- Cheapest to adopt on an existing corpus and the only option when docs are contributed by non-engineers.
- Needs a per-block metadata convention (tier, setup fixture, expected output) carried in the fence info string or an adjacent comment.
- Weak at multi-block sequences; be explicit about which blocks share state.

**3. Generate from a spec.** Per-language request/response snippets generated from the API description.

- Guarantees parity and freshness for reference pages at near-zero marginal cost per language.
- Expresses single calls only. Multi-step, idiomatic usage still needs hand-written, tested samples.

A sample pasted into prose by hand belongs to none of these. That is the default state of most corpora and the thing the policy exists to eliminate.

## Hiding setup without breaking runnability

Readers want the three interesting lines; the test needs the whole file. Every workable scheme separates the two: the tested artefact is the complete file, the published artefact is a region or a filtered view of it.

Rustdoc does this with `#`-prefixed lines that compile but do not render. Reproduce the property, whatever the mechanism - never publish a fragment that was made runnable by deleting nothing and hoping.

## CI shape

- **Install the published artefact.** Registry package, released container, deployed endpoint - whatever the reader will have. A suite that builds from the working tree passes on code no reader can install yet.
- **Split hermetic from integration.** Hermetic (mocked or local) runs on every pull request for speed. Integration runs against the real service, because that is where the breakage actually originates.
- **Run on a schedule, not only on commit.** Samples break when the _service_ changes, not when the docs repo changes. A nightly or weekly run is what catches server-side drift.
- **Fail something that blocks.** The docs build or a required check. An advisory job goes red and stays red.
- **Report per sample, not per suite.** The output the maintainer needs is "which sample, which page, which line", so the finding can be routed to its owner.

## Fixtures

Untended fixtures are the usual reason a sample suite gets disabled six months in.

- Dedicated test accounts/projects per language, never a shared human account.
- Teardown in the test, plus a scheduled sweeper for what teardown missed - a suite that leaks resources eventually costs money and gets switched off.
- Deterministic naming for created resources (prefix + run id) so the sweeper can identify them safely.
- Secrets from the CI secret store, never from the sample. The sample shows a placeholder; the harness injects the value.

## Freshness signals

Track these per sample and expose the worst of them on the page:

- Age of the last **successful run**, not the last edit - an edited sample nobody ran is not fresher.
- Version pinned in the sample versus the current released version.
- Symbols referenced that no longer exist in the public API surface.
- Issues or tickets quoting the sample verbatim.

## Degrading gracefully

When you cannot execute anything in the current environment - no CI access, no runtime for that language - do not silently downgrade the audit. Produce the tier assignments, the mechanical lint results, and a written verification plan the team can implement, and label every unexecuted sample's status as _unverified_ rather than _passing_.

#!/usr/bin/env python3
"""Extract every fenced code sample in a docs tree and lint it mechanically.

Walks a documentation directory, emits one row per fenced block with the facts a
sample audit needs, and flags the pattern defects a policy check can decide
without understanding the product: missing language identifier, shell prompt
characters, ellipsis omissions, credential-shaped literals, insecure switches,
unguarded destructive commands, placeholder style and block length.

Usage:
    sample-audit.py <docs-dir> [--format tsv|json|summary] [--ext .md,.mdx]
                    [--max-lines N] [--placeholder-style angle|brace|upper]

Columns: path, line, lang, lines, chars, tier, flags.

`tier` is read from the fence info string when the corpus already carries one
(```python tier=run), otherwise reported as `unset` — the audit treats unset as
tier Run, so silence is never an excuse for an untested sample.

Judgement checks (does the symbol still exist, is the documented output current,
does the sibling-language sample do the same thing) are deliberately out of
scope: they need the product, not a regex. Treat every flag as a candidate to
confirm, not a verdict — a docs page about credential rotation legitimately
contains a key-shaped string.
"""

import argparse
import json
import os
import re
import sys

FENCE_OPEN = re.compile(r"^(\s*)(`{3,}|~{3,})\s*([^\s`]*)\s*(.*)$")
TIER_INFO = re.compile(r"\btier\s*=\s*(compile-only|compile|run|illustrative)\b", re.I)

SHELL_LANGS = {"bash", "sh", "shell", "zsh", "console", "terminal", "powershell", "ps", "cmd"}
PROMPT = re.compile(r"^\s*(\$|#|>|PS[^>]*>)\s+\S")
ELLIPSIS = re.compile(r"(^|\s)(\.\.\.|…)(\s|$)|&hellip;|&#8230;")
SECRET = re.compile(
    r"(sk_live_|pk_live_|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{20,}|xox[baprs]-|"
    r"-----BEGIN [A-Z ]*PRIVATE KEY-----|eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,})"
)
LONG_LITERAL = re.compile(
    r"(?i)\b(api[_-]?key|secret|token|password|passwd|client[_-]?secret)\b\s*[:=]\s*"
    r"[\"']([A-Za-z0-9/+_-]{16,})[\"']"
)
INSECURE = re.compile(
    r"(--insecure\b|--no-check-certificate\b|verify\s*=\s*False|rejectUnauthorized\s*:\s*false|"
    r"InsecureSkipVerify\s*:\s*true|curl\s+(-k|\S*\s-k)\b|NODE_TLS_REJECT_UNAUTHORIZED)"
)
DESTRUCTIVE = re.compile(
    r"(rm\s+-[a-z]*[rf][a-z]*\s+/|\bDROP\s+(TABLE|DATABASE|SCHEMA)\b|\bTRUNCATE\s+TABLE\b|"
    r"\bDELETE\s+FROM\s+\w+\s*;|git\s+push\s+--force\b|kubectl\s+delete\s+(ns|namespace)\b|"
    r"\bmkfs\b|\bdd\s+if=.*of=/dev/)",
    re.I,
)
PLACEHOLDER_STYLES = {
    "angle": re.compile(r"<[A-Za-z0-9_ -]{2,40}>"),
    "brace": re.compile(r"\{\{?[A-Za-z0-9_ -]{2,40}\}?\}"),
    "upper": re.compile(r"\b(YOUR|MY|EXAMPLE)_[A-Z0-9_]{2,40}\b"),
}
TRAILING_WS = re.compile(r"[ \t]+$")
OUTPUT_HINT = re.compile(r"^\s*[\[{]|^\s*(HTTP/|\w+:\s|\|)|^\s*(Error|Traceback|Exception)\b")


def blocks(path, text):
    """Yield (start_line, lang, info, body_lines) for each fenced block."""
    lines = text.splitlines()
    index = 0
    while index < len(lines):
        opened = FENCE_OPEN.match(lines[index])
        # A fence only opens when the marker is at the start of the trimmed line.
        if not opened or not lines[index].lstrip().startswith(("```", "~~~")):
            index += 1
            continue
        indent, marker, lang, info = opened.groups()
        closer = re.compile(r"^\s*" + re.escape(marker[0]) + "{" + str(len(marker)) + r",}\s*$")
        body = []
        cursor = index + 1
        while cursor < len(lines) and not closer.match(lines[cursor]):
            body.append(lines[cursor])
            cursor += 1
        yield index + 1, lang, info, body
        index = cursor + 1


def lint(lang, info, body, args):
    flags = []
    text = "\n".join(body)
    lowered = (lang or "").lower()

    if not lang:
        flags.append("no-language-id")
    if lowered in SHELL_LANGS or not lang:
        prompted = [line for line in body if PROMPT.match(line)]
        if prompted:
            flags.append("prompt-char")
        # Output glued under a command in the same block defeats copy-paste.
        if prompted and any(OUTPUT_HINT.match(line) for line in body):
            flags.append("output-in-command-block")
    if ELLIPSIS.search(text):
        flags.append("ellipsis-omission")
    if SECRET.search(text) or LONG_LITERAL.search(text):
        flags.append("credential-shaped-literal")
    if INSECURE.search(text):
        flags.append("insecure-switch")
    if DESTRUCTIVE.search(text):
        flags.append("destructive-command")
    if any(TRAILING_WS.search(line) for line in body):
        flags.append("trailing-whitespace")
    if len(body) > args.max_lines:
        flags.append("over-length")

    expected = PLACEHOLDER_STYLES[args.placeholder_style]
    others = [name for name, rx in PLACEHOLDER_STYLES.items()
              if name != args.placeholder_style and rx.search(text)]
    if others and not expected.search(text):
        flags.append("placeholder-style:" + ",".join(others))

    tier = TIER_INFO.search(info or "")
    return (tier.group(1).lower() if tier else "unset"), flags


def main():
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("docs_dir")
    parser.add_argument("--format", choices=["tsv", "json", "summary"], default="tsv")
    parser.add_argument("--ext", default=".md,.mdx,.markdown")
    parser.add_argument("--max-lines", type=int, default=40)
    parser.add_argument("--placeholder-style", choices=sorted(PLACEHOLDER_STYLES),
                        default="angle")
    args = parser.parse_args()

    if not os.path.isdir(args.docs_dir):
        sys.exit(f"not a directory: {args.docs_dir}")

    extensions = tuple(e if e.startswith(".") else "." + e for e in args.ext.split(","))
    rows = []
    for root, dirs, files in os.walk(args.docs_dir):
        dirs[:] = [d for d in dirs if not d.startswith(".") and d != "node_modules"]
        for name in sorted(files):
            if not name.endswith(extensions):
                continue
            path = os.path.join(root, name)
            with open(path, "r", encoding="utf-8", errors="replace") as handle:
                text = handle.read()
            for line, lang, info, body in blocks(path, text):
                if not body:
                    continue
                tier, flags = lint(lang, info, body, args)
                rows.append({
                    "path": os.path.relpath(path, args.docs_dir),
                    "line": line,
                    "lang": lang or "",
                    "lines": len(body),
                    "chars": sum(len(l) for l in body),
                    "tier": tier,
                    "flags": flags,
                })

    if args.format == "json":
        json.dump(rows, sys.stdout, indent=2)
        sys.stdout.write("\n")
        return

    if args.format == "tsv":
        print("path\tline\tlang\tlines\tchars\ttier\tflags")
        for row in rows:
            print("\t".join([row["path"], str(row["line"]), row["lang"], str(row["lines"]),
                             str(row["chars"]), row["tier"], ",".join(row["flags"]) or "-"]))
        return

    counts = {}
    for row in rows:
        for flag in row["flags"] or ["clean"]:
            counts[flag.split(":")[0]] = counts.get(flag.split(":")[0], 0) + 1
    languages = {}
    for row in rows:
        languages[row["lang"] or "(none)"] = languages.get(row["lang"] or "(none)", 0) + 1
    print(f"samples: {len(rows)}  files: {len({r['path'] for r in rows})}")
    print("languages: " + ", ".join(f"{k}={v}" for k, v in sorted(
        languages.items(), key=lambda kv: -kv[1])))
    print("tiers: " + ", ".join(f"{k}={v}" for k, v in sorted(
        {r["tier"]: sum(1 for x in rows if x["tier"] == r["tier"]) for r in rows}.items())))
    print("flags:")
    for flag, count in sorted(counts.items(), key=lambda kv: -kv[1]):
        print(f"  {flag}: {count}")


if __name__ == "__main__":
    main()

---
name: docs-code-sample-standards
description: Defines the policy every code sample in developer documentation must meet and audits an existing sample corpus against it, returning a ranked fix queue. Use whenever the user mentions docs code samples, code snippets in documentation, runnable examples, examples that no longer compile, copy-paste failures, snippet drift after an API change, testing docs examples in CI, SDK snippet parity across languages, sample maintenance ownership, or asks why the examples in their docs do not work - even if they only say the docs are broken. Covers sample anatomy, copy-paste and security safety, execution tiers, single-sourcing from tested code, and language parity. Not for writing a quickstart page - use samber/developer-relations-skills@developer-quickstart-guide.
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.0.1"
---

# Docs Code Sample Standards

You are a developer-documentation engineer. You produce two artefacts: a **sample policy** (the rules every published code sample must satisfy) and a **sample audit** that scores the existing corpus against that policy and ranks what to fix.

A code sample is the only part of the docs a reader executes. Prose that is slightly wrong wastes a minute; a sample that is slightly wrong ships into their codebase.

Uddin and Robillard's 2015 survey of 323 professional developers found ambiguity, incompleteness, and incorrectness the severest API-documentation problems, with six of the ten problem types they documented rated outright blockers. The survey covers API documentation as a whole, and samples are where all three defects get executed rather than merely read. Treat samples as a tested artefact with an owner, not as decoration inside a page.

## Scope check

Confirm the task is really about the sample corpus before starting. Route elsewhere when:

- The task is writing one zero-to-first-success page → quickstart skill.
- The task is a teaching tutorial rather than the samples inside it → tutorial skill.
- The problem is _where content lives_ across the docs site → docs information-architecture skill.
- The task is endpoint-by-endpoint completeness of API reference pages → API reference skill.
- The target reader is a coding agent, not a human → agent-documentation skill.
- The code under review is the product's own source, not a published sample → this is a code review, not a docs task.

See the References section for the exact skill identifiers.

## Interview

Ask these one at a time, offering options where you can. Stop as soon as you can name the surfaces, the languages and the tier-1 scenarios; do not run the whole list mechanically.

**Memory: Store a few info about the current task or project context. When memory lives in a file, use `devrel-context.md`; if a different memory system is in use, rely on that instead. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**
1. Where do samples live today: docs site pages, API reference, README, an examples repository/directory, blog posts, in-product help? Which of those are in scope?
2. What is the product class: hosted API, SDK/library, CLI, framework, infrastructure component?
3. Which languages are published, and which one do readers actually arrive with most often?
4. Are any samples executed by anything today (a test suite, a docs build, a manual pass) or are they all hand-maintained prose?
5. Who owns sample correctness when it breaks: docs team, the SDK owners, support, nobody?
6. What triggers a sample to break here: SDK releases, service-side API changes, auth changes, dependency drift?
7. Is this a greenfield policy, an audit of an existing corpus, or both?
8. What can you spend: can samples move into a tested project, how many engineering hours are available for a CI harness, or must the policy work with samples staying in Markdown?
9. What date does this have to show a result by?
10. Do you want a one-off cleanup of the corpus, or a standard the team keeps enforcing every release?

Questions 4, 8, 9 and 10 decide the tier ranking. Ask them before classifying anything.

- **Q9:** a near date promotes compile-only, which ships in a week.
- **Q10:** a standing-enforcement mandate promotes Run on the adoption path.
- **Q4 and Q8:** no CI and no hours to build one deletes both Run and Compile-only from the menu.
- **Q8:** an existing test suite the samples can live inside promotes Run outright.

## Method

Three named methods carry this work. Name them in the deliverable so the team can read the source material instead of taking your word for it. The first two are one discipline (verify mechanically before judging) applied at two scales; the third resolves rule conflicts.

**Inventory before audit** (Nielsen Norman Group) applies the discipline at corpus scale. An inventory is "a list of every piece of digital content you currently have"; an audit "examines, assesses, and evaluates the quality". Build the census mechanically first, then judge.

Layer content strategy's **ROT triage** (redundant, outdated, trivial) over the inventory as the first cut.

**Docs as Tests** (Manny Silva) applies it at page scale. Documentation should not only inform, it should verify: each documented step becomes a test case run against the real product. Language ecosystems already encode this; the Go blog states that executable documentation "guarantees that the information will not go out of date as the API changes".

**Google Cloud's ranked sample principles** settle any collision between two rules: **clarity → idiomaticity → consistency → simplicity → portability → maintainability** (the higher principle wins). That ordering ends the recurring argument about cross-language uniformity: a sample that reads like a transliteration fails idiomaticity even when it is byte-consistent with its siblings.

## Workflow

1. **Inventory the corpus.** Enumerate every published sample with its file, line, language, surface and last-commit date. Run `scripts/sample-audit.py <docs-dir>` if you can execute scripts; otherwise walk the tree and build the same table by hand. Nobody can set a policy for a corpus whose size they are guessing at.
2. **Triage with ROT.** Mark redundant duplicates, outdated samples referencing removed APIs, and trivial single-call snippets with no scenario around them. Cutting is a legitimate outcome here.
3. **Classify each sample by execution tier** (table below). This is the highest-leverage decision in the whole policy: it decides what CI must enforce and what stays honest prose.
4. **Draft the policy** using [./references/sample-policy-template.md](./references/sample-policy-template.md). Fill every decision; an unfilled one becomes an argument during the first review.
5. **Set language tiers and build the parity matrix** per [./references/language-parity-matrix.md](./references/language-parity-matrix.md). Publish the tiers; an unstated tier reads as neglect rather than as a decision.
6. **Audit in two passes** per [./references/audit-method.md](./references/audit-method.md): mechanical checks on 100% of the corpus, then rubric scoring on a sample stratified by language, product area and age. Apply [./references/audit-checklist.md](./references/audit-checklist.md) per sample. Report findings as _claim → evidence location → one-sentence fix_, grouped by severity, never as prose paragraphs.
7. **Design verification** per [./references/verification-patterns.md](./references/verification-patterns.md): which tier runs where, how samples are single-sourced from tested code, what the CI job installs, and how fixtures get cleaned up.
8. **Assign ownership and governance** per [./references/governance-and-ownership.md](./references/governance-and-ownership.md). Answer one question concretely: when a sample goes red in CI on a Friday afternoon, whose queue does it land in? Then encode that answer as code-owner rules and blocking checks rather than as a paragraph.
9. **Rank the fix queue.** P0 goes first on harm, not on efficiency (a credential or a destructive command outranks any ratio). Order everything below it by (readers affected × severity) ÷ fix effort, and say in the report that the queue is a ratio rather than a severity list. That ratio starves the low-traffic sample that needs a whole harness built to fix it; promote it by hand when it sits on the adoption path.
10. **Re-measure against the pass threshold** below. Iterate until it holds; report the score honestly if it does not, and label which numbers are baselines rather than benchmarks.
11. **Record the decisions.** If your environment has persistent memory, store the tier assignments, language tiers, ownership and rejected scope, so later docs work does not relitigate them.

## Execution tiers

Every sample gets exactly one tier. The tier is a claim to the reader as much as an instruction to CI.

| Tier         | Meaning                                                                              | CI enforces                  | Engineering and upkeep                                                                     | Reader signal                           |
| ------------ | ------------------------------------------------------------------------------------ | ---------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------- |
| Run          | Executes; documented output is asserted                                              | Full execution + output diff | a standing job (fixtures, credentials, teardown, a red build every time the service moves) | "This works as written"                 |
| Compile-only | Type-checks or builds, never executed (billing, destructive, third-party dependency) | Build/lint only              | a week to stand up, then near-zero per sample                                              | "This is valid; it was not run for you" |
| Illustrative | Pseudocode, a fragment, or a deliberate anti-pattern                                 | Excluded                     | near-zero (a visible label)                                                                | Labelled on the page as not runnable    |

- efficiency: compile-only > run > illustrative
- value: run > compile-only > illustrative
- effort: run > compile-only > illustrative
- compliance cost: run > compile-only == illustrative

Compile-only leads because one build job catches the whole P1 class (removed symbols, missing imports, code that never compiled) for no fixtures and no credentials. Compile-only and illustrative tie on compliance cost because neither one holds a credential or calls anyone else's service; Run does both, so it drags in a secret-handling review and, where the sample calls a third party, that provider's terms for automated traffic.

Run is what this order starves: the only tier whose claim cannot rot, and the one that costs a standing job to keep. Promote it anyway on the primary adoption path (quickstart, auth, first call) and on whatever the copy-button counts say readers actually paste. Those are the samples where being wrong costs a reader their afternoon.

Delete tiers the constraints rule out rather than ranking them last, and say which you deleted. A team with no CI has no Run tier and no Compile-only tier: every sample in that corpus is _unverified_ until a pipeline exists, and building the pipeline is the first queue item rather than the tiering. A sample that bills, destroys or depends on a third party has no Run tier whatever the pipeline can do.

This ordering is a default that shifts with who executes it. An existing integration test suite the samples can be single-sourced into makes Run nearly free and promotes it to the top. A team that already runs a doctest facility has paid most of the compile-only setup already.

Two rules keep this honest:

- Untiered samples default to Run (a claim-assignment rule, not an investment recommendation), so that silence never becomes an excuse and an unclassified sample fails loudly.
- Illustrative must be visibly labelled in the rendered page; an unlabelled illustrative sample is indistinguishable from a broken one.

Prefer a language's native doctest facility over a bespoke harness where one exists; it gives you Run and Compile-only for free. Watch the matching trap: in ecosystems where the assertion is opt-in, an example without its expected-output marker still compiles and is never actually run, so it passes CI while claiming more than it proves.

## The six policy decisions

Every argument about samples reduces to one of these. Decide each explicitly:

1. **Coverage** - which scenarios deserve a sample. Frequently used and tricky-to-use elements first; never a sample that illustrates an obvious point. Prefer multi-step scenarios over single-API-call snippets.
2. **Anatomy** - what a sample must contain: all imports, explicit client setup, one code path, expected output, and a top comment stating what it does and what setup it assumes.
3. **Safety** - the copy-paste and security rules: no prompt characters, angle-bracket placeholders, reserved-range hosts and addresses, no real-looking credentials, no disabled TLS, no destructive command without a guard.
4. **Parity** - which languages are tier 1, and what "the same sample" means across them.
5. **Verification** - tiers, where samples are sourced from, what CI runs and on what schedule.
6. **Ownership and freshness** - who fixes a red sample, and the maximum age of a sample's last _successful run_ before it is treated as stale.

Two rules quietly remove whole classes of defect:

- Show one failure path for any API that fails in normal operation.
- Keep auth setup in one place that other samples link to, rather than repeating it with variations.

## Invocation examples

| The user asks                                    | You run                                                                                        | You return                                                   |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| "Audit the code samples in our docs"             | `scripts/sample-audit.py docs/ --format summary` for the census, then the checklist per sample | Sample audit report (shape below) plus the fix queue         |
| "Write us a code sample policy"                  | Interview, then fill `references/sample-policy-template.md`                                    | A completed policy document, every field decided             |
| "Our Python examples work but the Go ones don't" | `scripts/sample-audit.py docs/ --format tsv`, then the parity matrix                           | Filled matrix, parity coverage figure, per-language gap list |
| "Should this snippet be tested in CI?"           | Tier classification only                                                                       | One tier, the reason, and what CI must enforce for it        |
| "Check the samples for leaked keys"              | `scripts/sample-audit.py docs/ --format json`, filter for the `credential-shaped-literal` flag | P0 list, reported immediately rather than at the end         |

Common invocations of the script:

```
scripts/sample-audit.py docs/ --format summary
scripts/sample-audit.py docs/ --format json --ext .md,.mdx > inventory.json
scripts/sample-audit.py docs/ --max-lines 30 --placeholder-style angle
```

Treat every flag it raises as a candidate to confirm, not a verdict; a page about credential rotation legitimately contains a key-shaped string.

## Audit output

Produce a short report, not a wall of findings:

```
## Sample audit - <corpus>, <date>
Corpus: 412 samples across 6 languages, 138 pages.
Census: 380 compiled, 32 failed (92.2% pass). Secret scan: 2 findings.
Rubric: 40 samples scored (stratified by language x age), mean 13.4/20.
Health: 34% tested (skill baseline is 80%, not an industry standard).
Findings: 11 P0, 47 P1, 190 P2.

### P0 - ship-blocking (11)
docs/auth/setup.md:88 - sample embeds a live-format API key.
  Fix: replace with <YOUR_API_KEY> and rotate the leaked key.
...

### Fix queue
1. P0 credential leaks (11) - owner: SDK team - this week
2. Untested quickstart path samples (9) - owner: docs - sprint
...
```

Severity anchors:

- **P0**: harms the reader who copies it (credential, destructive command, insecure default, data loss).
- **P1**: does not work as written (compile error, removed symbol, missing import, wrong output).
- **P2**: works but violates policy (missing tier, prompt characters, no expected output, parity gap).

## Pass threshold

One gate has a measured comparator behind it; the rest are baselines this skill sets. Say which is which every time you report them; [./references/published-findings.md](./references/published-findings.md) holds the citations and the circulating claims this skill refuses to repeat.

**Gate with a measured comparator:**

1. **Deprecated-API rate below 43%.** Above it, build a deprecation-detection gate before anything else. 43% is the deprecated-API rate Zhou and Walker (FSE 2016) measured in the Stack Overflow posts they investigated; the figure is theirs, using it as a gate is this skill's choice. A governed corpus doing worse than public Q&A has a detection problem, not an authoring problem.

**Skill baselines - override with the team's own data:**

2. Zero P0 findings, and 100% of samples on the primary adoption path (quickstart, auth, first call) tier Run and green in CI.
3. Compile/run pass rate ≥90%; below that, stop new authoring and run a remediation sprint first. No vendor publishes this gate; it is a recommendation, well supported by how badly ungoverned snippet corpora do (1.00% of Stack Overflow's Java snippets compile, MSR 2016; 24.4% of public Python gists run out of the box, Gistable, ICSME 2018) but not a standard.
4. ≥80% of the whole corpus is tier Run or Compile-only and executed by CI.
5. Parity coverage ≥90% across tier-1 languages (tested cells ÷ scenarios × tier-1 languages), and no sample's last _successful run_ older than the policy's freshness window.

Re-run the audit after each fix batch rather than declaring victory from the fix list. When threshold 4 cannot be met, cut samples instead of lowering the bar; a deleted sample costs a reader nothing, an untested one costs them an afternoon.

## KPIs

- Tested-sample share and parity coverage, tracked per release rather than per audit.
- Mean time from a red CI sample to a merged fix; the real measure of whether ownership exists.
- Support tickets or issues quoting a sample verbatim.
- Copy-button clicks per sample, where the docs platform exposes them: high-copy samples earn tier Run first.
- Sample age distribution: count of samples whose last successful run exceeds the freshness window.
- Rubric score per dimension over time, not the total; the total hides which axis is failing.

## Failure modes

| Symptom                                        | Fix                                                                                                             |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Samples pass CI but fail for readers           | CI installs from the working tree; install the published artefact instead                                       |
| Sample suite disabled after flaky failures     | Split hermetic (per-PR) from integration (scheduled); fix fixture teardown before re-enabling                   |
| Fragment cannot run because imports are elided | Single-source from a real file and publish a marked region; hide setup, never delete it                         |
| Reader pastes a placeholder verbatim           | Use angle brackets and reserved-range hosts, never a plausible fake value; explain each placeholder             |
| Same scenario behaves differently per language | Fix parity by scenario and outcome, not by translating one language's code                                      |
| Audit was a read-through of the top pages      | Census the whole corpus mechanically; stratify the human sample - see the negative example in `audit-method.md` |
| Policy written, corpus never re-audited        | Put the audit in the release checklist; a policy nobody measures is a wish                                      |
| Samples rewritten for style, not correctness   | Delegate style to the ecosystem's formatter and linter; spend review on behaviour                               |
| Docs build stays green while samples are red   | Fail the docs build on sample failures, or the red mark gets ignored                                            |
| Owners assigned to hand-pasted Markdown        | Move samples into compilable source first; ownership without a mechanism is a name on a wiki page               |
| A quoted benchmark nobody can source           | Check `published-findings.md`; it lists the circulating claims this skill refuses                               |

## References

- See [./references/sample-policy-template.md](./references/sample-policy-template.md) for the policy document to fill in.
- See [./references/audit-method.md](./references/audit-method.md) for the census/stratified method, the rubric and a negative example.
- See [./references/audit-checklist.md](./references/audit-checklist.md) for per-sample checks, severity anchors and a good/bad worked pair.
- See [./references/verification-patterns.md](./references/verification-patterns.md) for execution tiers, single-sourcing and the CI shape.
- See [./references/language-parity-matrix.md](./references/language-parity-matrix.md) for language tiering, the matrix and drift patterns.
- See [./references/governance-and-ownership.md](./references/governance-and-ownership.md) for the three ownership models, metadata-driven governance, blocking CI gates and the migration order.
- See [./references/published-findings.md](./references/published-findings.md) for sourced figures, self-set baselines and claims to refuse.
- Run [./scripts/sample-audit.py](./scripts/sample-audit.py) to extract and mechanically lint every fenced sample in a docs tree.
- See samber/developer-relations-skills@developer-tutorial for teaching tutorials, a primary consumer of these samples.
- See samber/developer-relations-skills@developer-docs-structure-audit for where content belongs across a docs site.
- See samber/developer-relations-skills@version-migration-guide for before/after samples in breaking-change guides.
- See samber/developer-relations-skills@developer-troubleshooting-docs for the error pages a failing sample should link to.
- See samber/developer-relations-skills@coding-agent-docs-optimization when the sample's consumer is an agent, not a human.
- See samber/developer-platform-skills@api-reference-quality for per-endpoint reference completeness; this skill owns sample policy, testing and language parity, that one owns endpoint-level coverage.