samber/developer-relations-skillsチェック済み
SKILL DETAIL
developer-troubleshooting-docs
samber/developer-relations-skills/developer-troubleshooting-docs
Turns support tickets, issue history and error telemetry into troubleshooting and error-reference pages a developer finds by pasting the error string - symptom, conditions, cause, fix and verification entries, an error-code catalog generated from one source of truth, and fixes wired back into the product's own error output. Use whenever the user mentions troubleshooting docs, documenting error codes, error message pages, a known-issues page, building an FAQ from support tickets, or "we answer the same question every week" - even if they only say "users keep hitting this error". Not for debugging a live incident. Do NOT use for docs search ranking - use samber/developer-relations-skills@docs-seo.
インストール · 417出典を見る
Installation
npx skills add https://github.com/samber/developer-relations-skills --skill developer-troubleshooting-docs
スキルファイル
SKILL.md
最終同期 · 2026/10/23
evals/evals.json›
{
"skill_name": "developer-troubleshooting-docs",
"evals": [
{
"id": 1,
"prompt": "I run docs at Halyard, we make a self-hosted CI runner. Our head of support wants a troubleshooting section live before our 4.0 launch on the 30th, he's asked for about 15 pages covering the errors people hit. Problem: we don't have a ticket system (support is a shared inbox nobody tags), no error telemetry, and our docs site search doesn't log queries. He said just use your knowledge of common CI and Docker failures and write the 15 pages, we'll correct them later. Can you draft them?",
"expected_output": "A refusal to draft pages from imagination, an explanation of why evidence-free troubleshooting entries fail, a list of signal sources Halyard could actually stand up, and a proposal to install a capture habit and revisit after a cycle rather than shipping 15 invented pages before the 30th.",
"files": [],
"expectations": [
"Declines to draft the 15 pages from general knowledge of CI or Docker failures",
"States that writing troubleshooting pages without a failure signal produces entries for failures that do not happen while the real ones stay undocumented",
"Produces no troubleshooting entry for an invented error string",
"Names at least one reachable signal source despite the absence of a ticket system, such as shared-inbox threads, the issue tracker, community threads, error telemetry, docs search logs, or an assistant's unanswered-query log",
"Proposes a standing capture habit as the next step rather than a one-off writing pass",
"Names KCS or Knowledge-Centered Service explicitly",
"Describes capture happening during resolution rather than reconstructed from memory afterwards",
"Proposes revisiting after a cycle of capture rather than delivering before the 30th",
"Does not promise that the 4.0 launch date will be met with published troubleshooting pages",
"Recommends at least one unreported source such as telemetry, search logs, or an unanswered-query log alongside any reported source, rather than reported sources only",
"Does not propose deriving the page list from a competitor's documentation or from general web search results"
]
},
{
"id": 2,
"prompt": "Bitlathe, we sell a managed Postgres branching service. I pulled 340 support tickets from the last two quarters and tallied the recurring ones:\n\n- 84 tickets: `FATAL: remaining connection slots are reserved` - always the same fix, bump the pool setting, support spends ~4 minutes each\n- 61 tickets: `could not connect to server: Connection refused` - same fix, they used the wrong port, ~3 minutes\n- 57 tickets: `connection to server failed: Connection refused` - wrong port again, different wording from an older client\n- 22 tickets: branch creation hangs forever on first use during signup; support spends ~35 minutes each and roughly half the people never come back\n- 14 tickets: `ERROR: could not extend file` on WAL exhaustion for customers on dedicated clusters - takes an engineer 2-3 days per case, one of them was a contract customer whose incident review we ran last month\n- 11 tickets: `ERROR: permission denied for schema public` - we're changing the default grants in 5.0 next month and it goes away\n- 9 tickets: account suspended for non-payment, they get a generic error\n- 7 tickets: all filed in the two weeks after 4.6 shipped, `panic: send on closed channel` during failover\n\nGive me the writing order for my one technical writer.",
"expected_output": "A merged, ranked queue with the two Connection refused clusters combined, the regression and the non-self-serviceable clusters removed rather than demoted, the signup blocker promoted, the WAL cluster promoted on contract and incident-review grounds, and the whole queue presented for agreement before writing.",
"files": [],
"expectations": [
"Merges the 61-ticket and 57-ticket Connection refused clusters into one cluster on the grounds that they are one failure worded two ways",
"States the ranking criterion as reader-hours unblocked per hour of writing and verification rather than raw ticket frequency",
"Does not present a queue ordered purely by ticket count",
"Places the connection-slots cluster and the merged wrong-port cluster early as frequent, reproducible work",
"Promotes the 22-ticket signup-time branch-creation hang above at least one higher-count cluster because it blocks first use",
"States that a reader who abandons on the first-use path never files the ticket that would have ranked the cluster",
"Identifies the 7-ticket `panic: send on closed channel` cluster as a regression to file against 4.6 rather than a page to write",
"Removes the 11-ticket `permission denied for schema public` cluster from the queue entirely because 5.0 eliminates it, rather than ranking it last",
"Removes the 9-ticket account-suspension cluster from the queue because the reader cannot act on the answer, rather than ranking it last",
"States that such clusters are deleted rather than demoted, because a demoted cluster silently returns as scope",
"Promotes the 14-ticket WAL-exhaustion cluster despite its low count, citing the contract customer or the incident review",
"States that the efficiency order starves the rare and expensive failure and names the condition that overrides it",
"Presents the ranked queue for the user's agreement before any entry is written"
]
},
{
"id": 3,
"prompt": "Nomsk, we ship an observability agent. Five clusters survived our last mining pass and my VP wants a page for each by end of month:\n\nA. `agent: operation failed (code 7)` - 41 records. The message is literally that string; support has to ask for logs every single time to work out which of six things went wrong.\nB. Collector silently drops spans when the customer sits behind a TLS-terminating proxy - 19 records. Fix is: edit `agent.yaml`, then regenerate the cert bundle, then edit `collector.toml`, then restart, in that exact order, and getting the order wrong corrupts the queue.\nC. `403 ingest rejected: account not provisioned` - 33 records. Every one of them is a trial that expired or a plan downgrade.\nD. `E_SHARD_UNAVAILABLE` - 28 records, all in the 11 days since 5.2 shipped, none before.\nE. Memory ballooning under high cardinality - 12 records, but it's the named cause in four of our last seven incident reviews.\n\nWrite me the five pages.",
"expected_output": "A routing decision per cluster instead of five pages: A to a product message rewrite, B to a DX change or single command, C to a product surface or macro, D filed as a 5.2 regression, E escalated to a scheduled product fix with any interim page named as tolerated debt, each with an owner.",
"files": [],
"expectations": [
"Does not produce five troubleshooting pages",
"Routes cluster A to a product-side error-message rewrite, stating that a docs page cannot be pasted into the reader's terminal",
"States that rewriting the emitted string is a product change owned by an engineer, not a docs task",
"Routes cluster B to a product or DX change, or to a single command that performs the steps, rather than documenting a three-file ordered edit as the permanent answer",
"Routes cluster C to a product surface or a support macro rather than a public page, because the cause is account state",
"Identifies cluster D as a regression introduced by 5.2 and says to file it rather than document it",
"States that documenting cluster D would document a bug into permanence",
"Escalates cluster E to a scheduled product fix because the same failure class recurs across several postmortems",
"States out loud that any interim page for cluster E is an admission that a defect is being tolerated",
"Names an owner for each cluster routed away from docs",
"States that deciding not to document an error is a legitimate output, referencing that not all diagnostics warrant a durable explanation",
"Names Google SRE postmortem practice, or its rule that action items must be preventative rather than merely mitigative, as the basis for the escalation gate"
]
},
{
"id": 4,
"prompt": "Tessellate, we do a build cache for monorepos. Write the docs entry for this one. The error our CLI prints is exactly:\n\n```\ntessellate: error: cache upload rejected (HTTP 413): object 4f2a91c0-8b3d-4e21-9f77-0c2a1d5e8b44 exceeds the per-object limit of 512 MiB for plan 'team'\n```\n\nThree things cause it, from what support tells me:\n- Someone is on the `team` plan and genuinely has an artifact over 512 MiB. This is the cause about 70% of the time. Diagnosing it means running `tessellate cache inspect --largest 10`, which takes around 3 minutes on a big repo.\n- They set `TESSELLATE_MAX_OBJECT` in CI above their plan limit, which the client doesn't validate. Maybe 25%. Checking is instant, one `echo $TESSELLATE_MAX_OBJECT`.\n- Rare, 5%, the upload got chunked wrong by a corporate proxy and the size header is bogus. Checking needs a `curl -v` against our endpoint with `--trace-ascii`, about 10 minutes.\n\nIt happens on CLI 2.1 through current (2.8) on Linux and macOS, not on Windows where we cap differently. Applies to `team` and `starter` plans; `enterprise` has no per-object limit. Make it good, our current draft just says 'Simply reduce your artifact size, this is an easy fix.'",
"expected_output": "A six-part entry, message through escape hatch, with the string fenced verbatim and only the object ID placeholdered, branched cause blocks ordered by how often each check is the actual cause per minute of reader time, and no blaming or minimising language.",
"files": [],
"expectations": [
"The entry carries all six parts: message, applies-to, cause, fix, confirmation, escape hatch",
"The six parts appear in that order",
"The error text appears verbatim inside a fenced code block",
"The object UUID is replaced by an obvious placeholder while the rest of the string, including casing and punctuation, is preserved",
"The title or first heading contains the emitted error text or its identifying substring, not a topic label such as 'Upload problems'",
"An applies-to section names CLI 2.1 through 2.8, Linux and macOS, and the `team` and `starter` plans, and states that `enterprise` is unaffected",
"The fix branches into separate per-cause blocks rather than one flat list of every check",
"The 3-minute plan-limit check is ordered ahead of the instant `TESSELLATE_MAX_OBJECT` check",
"The ordering rationale is stated as how often that check is the actual cause per minute of the reader's time, explicitly not cheapest-check-first",
"The proxy-chunking cause is placed last as the successive fall-back",
"The words 'simply', 'easy' and 'just' do not appear describing the fix",
"No sentence attributes the failure to the reader forgetting or doing something wrong",
"The confirmation part names a specific observable output or state rather than the absence of the error",
"The escape hatch names the specific artifacts to attach and a place to report, rather than a bare 'contact support'"
]
},
{
"id": 5,
"prompt": "Quillrock, auth-as-a-service. Our staff writer proposed this structure for the troubleshooting rewrite and I want a second opinion before we commit:\n\n- One page, `/docs/troubleshooting`, with all 28 of our OAuth errors as H3 sections on it.\n- Titles by topic so the reader isn't scared: 'Authentication problems', 'Token problems', 'Redirect problems'.\n- Nine of the 28 errors share the same root explanation about clock skew. Rather than repeat ourselves nine times, we write it once in an 'Understanding token validity' section at the top and each of the nine links up to it. DRY, same as we'd do in code.\n- Error strings go in a collapsed 'Technical details' accordion at the bottom of each section so the page reads cleanly.\n\nSanity-check this for me.",
"expected_output": "A rejection of all four decisions: split into addressable targets, verbatim strings as titles, the clock-skew explanation repeated on each of the nine entries under ARID rather than linked, and the string surfaced in the first screen rather than hidden in an accordion.",
"files": [],
"expectations": [
"Rejects the single-page structure holding all 28 errors",
"States that only one of the 28 errors can win a given query on a single page",
"Recommends one addressable target per error: its own page, or a stable anchor on a codes page",
"Rejects the topic-based titles and requires the emitted error string verbatim in the title and first heading",
"Rejects the DRY consolidation of the clock-skew explanation and requires that content repeated on each of the nine entries",
"Names the ARID principle, Accept (some) Repetition In Documentation, as the basis for that repetition",
"States that the string match matters more than the duplication",
"Rejects putting the error string in a collapsed accordion at the bottom of a section",
"Requires the message, cause and first fix to appear in the first screen before any feature preamble",
"States that a block depending on the paragraph above it retrieves as a fragment when chunked into a search or assistant index",
"Notes that anchors on a mega page get lost in caches, or that such a page becomes unmaintainable",
"Does not recommend the shared 'Understanding token validity' hub as the primary structure"
]
},
{
"id": 6,
"prompt": "Saltbox, dev tool for edge functions. Marketing pulled up a competitor and told us to copy them: put FAQPage structured data on all 34 of our troubleshooting entries to get the expanded results in Google, and rewrite the headings as questions, 'Why is my function timing out?', 'Why am I getting a 502?', because that's how people search. Our SEO contractor also wants a whole-site crawl audit, canonical cleanup and internal linking redesign folded into the same ticket. How do we sequence this?",
"expected_output": "A refusal of the FAQPage markup and the question headings, with the exact-string findability rules substituted, the site-wide SEO work routed elsewhere, and product-side wiring named as the higher-value alternative.",
"files": [],
"expectations": [
"Recommends against adding FAQPage structured data to the troubleshooting entries",
"States that the FAQ rich result no longer appears in search results, so the markup buys nothing",
"Does not present FAQPage markup as a low-cost hedge worth doing anyway",
"Rejects question-shaped headings in favour of the emitted error string",
"States that readers arrive by pasting an exact string rather than by browsing or asking a question",
"Requires the error string verbatim in the title, the first heading and the body",
"Declines the site-wide crawl, canonical and internal-linking work as outside this job",
"Routes the site-wide SEO work to a documentation SEO skill rather than performing it",
"Does not claim that FAQ markup improves AI assistant or LLM citation odds as a substitute justification",
"Names at least one product-side wiring that outperforms the markup, such as shipping a documentation URL in the error payload or an explain path in the CLI"
]
},
{
"id": 7,
"prompt": "Verdant Systems. I have to get half a headcount approved to write error documentation and our CFO wants one page. Here's what I was going to put in it: a support ticket costs us somewhere between $15 and $50, I've seen $28 quoted a lot, and good self-service docs deflect around 60-70% of contacts, so at our 1,400 tickets a quarter this pays for itself in a month. Objective: drive tickets for the top 20 errors to zero by Q3. Tighten this up for me.",
"expected_output": "A business case with both folklore figures refused, the company's own ticket cost substituted or declared unknown, the zero-tickets objective replaced by a negotiated share-resolved number bounded by the published self-service ceiling, and the loyalty research supplied as the sourced argument.",
"files": [],
"expectations": [
"Refuses to use the $15-$50 or $28 per-ticket cost figure",
"States that the per-ticket cost figure circulates without traceable methodology",
"Refuses the 60-70% deflection figure on the same grounds",
"Instructs the user to substitute Verdant's own finance figures for ticket cost, or to state plainly that the number is unknown",
"Rejects 'drive tickets to zero' as a defensible objective",
"Cites the finding that only 14% of customer service issues are fully resolved in self-service",
"Cites the 36% figure for issues customers rated 'very simple'",
"Attributes those figures to a Gartner survey of 5,728 customers",
"Reframes the objective as raising the share of contacts resolved without a ticket, stated as a number agreed with the user",
"Offers the Dixon, Freeman and Toman loyalty research as the sourced argument, including the 96% versus 9% high-effort versus low-effort disloyalty contrast",
"States that per-error deflection is measurable while a site-wide deflection rate is not",
"Recommends resolution rate and cost per resolution over deflection rate, because repeat contacts hide inside deflection",
"Does not substitute a different unattributed industry benchmark in place of the rejected ones"
]
},
{
"id": 8,
"prompt": "Orrery, we ship a stream-processing runtime. We have 140-ish error codes. They're declared in three places: a `const` block in `internal/errs/codes.go`, a TypeScript `enum` in our JS client, and a table on our Confluence wiki that I'm fairly sure hasn't been touched since we were at 90 codes. Plan was: assign the 140 pages across my two writers, two weeks each, publish as a codes reference. Also our docs site is on an ancient Hugo theme and I've been meaning to migrate to Docusaurus, might as well do both at once. Tell me how to scope this.",
"expected_output": "A recommendation to generate the catalog from a single machine-readable source before writing any page, scoped as a small engineering task in the product repo with a named owning team, backed by published precedents and a per-code schema, with the Docusaurus migration explicitly unbundled.",
"files": [],
"expectations": [
"Recommends generating the catalog from one machine-readable source of truth before any per-code page is written",
"States that a hand-synced catalog drifts by default and that generation removes the failure mode rather than policing it",
"Identifies the three declaration sites as evidence that the drift already exists",
"Recommends diffing the documented wiki table against the codes the product can actually emit, and using the size of that diff as the business case",
"Cites at least one published implementation by name, such as PostgreSQL's errcodes file, Rust's rustc_error_codes, Twilio's error dictionary, or Stripe's error-code catalog",
"Recommends that the source file carry, per code, a description, an erroneous example, an explanation and a fix",
"Attributes that four-field schema to Rust RFC 1567",
"States that the work lands in the product repository, not the docs repository",
"Names identifying the owning engineering team as a prerequisite to proposing it",
"Frames the change as a one-off engineering task rather than a docs task",
"Explicitly rejects bundling the catalog generation with the Hugo-to-Docusaurus migration",
"Does not assign 140 hand-written pages across the two writers as the plan",
"Recommends carrying a documentation URL or deep link from the emitted error itself, in the payload or via an explain command"
]
},
{
"id": 9,
"prompt": "Pilcrow, managed Kafka. I've drafted 9 entries straight from our support team's resolution notes. Status: 6 I could reproduce on 3.4, which is what I run locally and is our latest; 3 I couldn't, one needs an air-gapped install, one only happens behind a specific F5 load balancer a customer has, one is intermittent and I saw it once. My manager wants all 9 merged Monday; her view is that a fix that worked for a real customer is verified by definition. We support 3.0 through 3.4. One of the entries includes a curl example I copied from a ticket that still has the customer's bearer token and their org ID in it.",
"expected_output": "A verification protocol applied to all nine: execute each fix from the reader's state, capture the post-fix signal, check both version edges, cold-read by someone else, ship the three unreproduced entries only with an explicit not-reproduced note, and strip the token and org ID.",
"files": [],
"expectations": [
"Requires each fix to be executed as written before publication, not inferred from resolution notes",
"Rejects the claim that a fix which worked in a support case is verified for publication",
"Requires running the fix from the reader's starting state rather than from an already-configured machine",
"Requires capturing the post-fix signal from that run as the confirmation line",
"Requires checking the oldest supported version, 3.0, as well as the current 3.4",
"Requires a reviewer who did not write the entry to follow it cold",
"States that the cold reader's first hesitation marks a step carrying an unstated assumption",
"Permits the three unreproduced entries to ship only with an explicit note that they are reported under stated conditions and not reproduced",
"Does not treat withholding all three unreproduced entries as the only available option",
"States that incorrect documentation is worse than missing documentation",
"Requires the bearer token and the customer org ID to be removed from the curl example",
"States that publishing a full token teaches readers to paste their own into public issues or tickets",
"Requires each command to state its prerequisites and what to do when that command itself fails",
"States that all six entry parts must be present and that five out of six is a draft rather than a publishable entry"
]
},
{
"id": 10,
"prompt": "Kestrel Data. We run a docs assistant on our site, a RAG bot over the docs, rebuilt whenever we remember. We're about to merge 20 new troubleshooting entries and rewrite 6 existing ones; that's maybe a fifth of our docs corpus by word count. After the merge we're done and I close the epic, right? Separately, someone on the team keeps pushing us to add llms.txt and llms-full.txt and treats it like a W3C thing we're out of compliance with, how much should I care? And our bot currently makes up plausible-sounding fixes for errors we've never documented, which we noticed last week.",
"expected_output": "Publishing redefined to end at re-index with the three re-embedding triggers named and this change identified as crossing one, llms.txt deflated to an optional proposal, an explicit refusal guardrail for the bot, and chunk-safe entry structure required.",
"files": [],
"expectations": [
"States that publishing ends at re-index, not at merge",
"Names the three re-embedding triggers: the embedding model meaningfully improving, roughly 10-15% of the content changing, and the domain shifting",
"Identifies the roughly one-fifth corpus change as crossing the content-change trigger",
"Labels the 10-15% figure as practitioner guidance rather than a measured or standardized threshold",
"Assigns the re-index the same owner as the docs release, as a booked maintenance trigger",
"States that a corrected page keeps being answered from the stale index until re-embedding happens",
"States that llms.txt is a proposal, not a ratified standard, with no W3C or IETF working group behind it",
"Advises treating llms.txt as cheap and optional, worth adding only if the generator supports it and never worth a project",
"Recommends an explicit 'I don't know' guardrail so the bot refuses rather than inventing a fix for an undocumented error",
"States that an honest refusal costs less trust than an invented fix",
"Requires each cause-remedy pair to stand alone so it survives chunking, rather than depending on the paragraph above it",
"Requires explicit field headings such as message, applies-to, cause, fix and confirmation as retrieval anchors",
"Recommends feeding the assistant's unanswered-query log back in as a signal source for the next clustering pass"
]
},
{
"id": 11,
"prompt": "Marlowe Interchange. We sell an EDI gateway; every customer is a signed enterprise contract, average $180k a year, 60 accounts. Our plan for sourcing troubleshooting content: mine the public GitHub issues on our open-source client library, since that's where we get the most volume, about 400 issues. We've also got around 40 Zendesk macros the support team maintains privately, which we were going to leave alone since they're internal. And we have error telemetry but nobody's ever exported it. Where do we start?",
"expected_output": "A corrected sourcing mix that pairs the private enterprise tickets with the unused telemetry, reconciles the 40 macros with public entries instead of ignoring them, anticipates environment-specific enterprise causes, and gates publication on what may legally be published.",
"files": [],
"expectations": [
"Rejects mining the public GitHub issues as the sole signal source",
"States that for contract-bound customers the loudest failures sit in private tickets and never reach the public tracker",
"States that public-only mining under-serves exactly the accounts that pay",
"Requires at least one reported source and one unreported source",
"Classifies the private tickets or macros as the reported source and the error telemetry as the unreported source",
"Recommends exporting the error telemetry rather than leaving it unused",
"States that the gap between reported and unreported sources is where readers silently give up",
"Requires the 40 support macros to be reconciled with the public entries rather than left untouched, so the two never disagree",
"States that where both exist, one is either a link to the other or a superset of it",
"Anticipates environment-specific causes typical of enterprise deployments, naming at least two of proxies, SSO, air-gapped installs or IP allowlists",
"Requires those environment facts to be carried in the entry's applies-to or conditions block",
"Requires checking whether a cause or fix may be published at all before writing it, covering security-sensitive detail or customer-identifiable information",
"Requires genericizing captured content: customer identifiers stripped, environment facts that determine applicability kept"
]
},
{
"id": 12,
"prompt": "Fenwick Labs. This keeps happening: we do a docs sprint, ship 20 troubleshooting pages, six months later we're back where we started. I want it to be a standing habit instead. We're on Zendesk. Support is 12 people, one technical writer (me), and engineering won't give us much. Who writes what, and how do I make it stick? Also I assumed Zendesk's ticket-to-article button basically does this for us.",
"expected_output": "A named capture loop with the resolver writing at resolution time, a structured ticket template plus a recurring promotion pass, a distributed-authoring/centralized-editing role split with a staffing ratio, and a correction of what the Zendesk button actually does.",
"files": [],
"expectations": [
"Names KCS or Knowledge-Centered Service as the operating model",
"Attributes KCS to the Consortium for Service Innovation",
"Distinguishes the solve loop, capture while resolving, from the evolve loop, restructure for reuse",
"Requires the resolving support engineer to capture the raw material at resolution time, while the error text and environment are still in front of them",
"States that entries reconstructed from memory are the ones that turn out wrong",
"Recommends a structured template field in the ticket tool capturing error text, environment, cause and resolution",
"Recommends a recurring pass that promotes repeat resolutions into entries",
"Describes the model as distributed authoring with centralized editing rather than a single owning role",
"Assigns editing, standardization and information architecture to the technical writer, and the error string itself to engineers",
"Corrects the Zendesk assumption: it creates an article from the ticket being worked but does not pre-populate the draft, so transcription time must be budgeted",
"States that no promote-to-article tool clusters by root cause first, so the clustering step stays manual",
"Warns that skipping clustering produces a page restating one customer's ticket instead of covering the failure",
"Gives a coach-to-knowledge-worker ratio as a staffing input, roughly 1:5 to 1:8 at start-up moving toward about 1:50 at maturity"
]
},
{
"id": 13,
"prompt": "Hexline. Three months ago we published 22 troubleshooting entries. Quarterly review numbers: ticket volume on those errors is basically flat, down 4%, within noise. Docs search still shows 190 zero-result searches last month for strings we definitely have pages for. My writer's proposal for next quarter is 30 more entries. Two other things on my list: six of our known-issues entries are for bugs we fixed in 5.0 back in June, and every entry ends with 'Ask in #support-legacy on Slack', that channel was archived in July. What should next quarter look like?",
"expected_output": "A findability-first quarter: re-run the clustering pass, diagnose the flat volume and zero-result searches as a discoverability failure rather than a coverage one, retire the six shipped known-issues entries, replace the dead escape hatch everywhere, and install staleness stamps and maintenance triggers.",
"files": [],
"expectations": [
"Rejects writing 30 more entries as the primary next-quarter action",
"Requires re-running the clustering pass and comparing it against the pre-publication pass",
"States that the same clusters reappearing unchanged indicates a discoverability problem, not a volume problem",
"Reads the 190 zero-result searches for covered strings as evidence that the error string is not verbatim on the page",
"States that zero-result searches for covered strings should reach zero",
"Requires the six known-issues entries for bugs fixed in 5.0 to be retired, moved into the changelog, or folded into the error entry with a version note",
"States that a known-issues list that only grows becomes a graveyard nobody reads",
"Requires the dead #support-legacy escape hatch to be replaced everywhere it appears, not only on the entries being edited",
"Names a watched channel and the attachments to include as the replacement escape hatch",
"States that a dead escape hatch converts a helped reader into an angry one",
"Recommends stamping each page with a last-verified date and an owner so staleness is visible without reading the page",
"Names entries touched since the last release as a staleness metric, and notes that publishing volume without it is vanity",
"Recommends archiving an entry whose cluster disappears from the mining pass for two cycles",
"Recommends persisting the durable decisions, such as signal sources and export paths, the agreed cluster ranking, the negotiated coverage target, and the clusters routed away from docs with their reasons, so the next pass is a diff rather than a re-derivation"
]
}
],
"trigger_queries": [
{ "query": "turn our support tickets into troubleshooting docs", "should_trigger": true },
{ "query": "we need a page for every error code", "should_trigger": true },
{ "query": "the same three errors flood our Discord every week", "should_trigger": true },
{ "query": "our error docs exist but nobody finds them", "should_trigger": true },
{ "query": "write a known-issues page for the 2.0 release", "should_trigger": true },
{ "query": "users keep hitting ERR_STREAM_PREMATURE_CLOSE and we have nothing written about it", "should_trigger": true },
{ "query": "how do I build an error reference out of our issue tracker", "should_trigger": true },
{ "query": "we answer the same question every week and it never reaches the docs", "should_trigger": true },
{ "query": "can you help me document our error codes", "should_trigger": true },
{ "query": "I want a troubleshooting section for the SDK", "should_trigger": true },
{ "query": "our support team is drowning in the same five tickets", "should_trigger": true },
{ "query": "should every error code get its own page or one big table", "should_trigger": true },
{ "query": "build an FAQ out of last quarter's support tickets", "should_trigger": true },
{ "query": "we have 200 error codes in an enum and nothing documented", "should_trigger": true },
{ "query": "customers paste our error string into Google and land nowhere", "should_trigger": true },
{ "query": "write up the workaround for the bug we haven't fixed yet", "should_trigger": true },
{ "query": "how should a troubleshooting page be structured", "should_trigger": true },
{ "query": "our troubleshooting page is one giant wall of 30 issues", "should_trigger": true },
{ "query": "I need to bring support volume down with better documentation", "should_trigger": true },
{ "query": "the docs search log is full of error strings returning zero results", "should_trigger": true },
{ "query": "help me cluster our tickets into pages worth writing", "should_trigger": true },
{ "query": "which of these recurring failures actually deserve documentation", "should_trigger": true },
{ "query": "we keep documenting around bugs instead of fixing them", "should_trigger": true },
{ "query": "our error table in the docs disagrees with the constants in the code", "should_trigger": true },
{ "query": "generate the error catalog from a single source of truth", "should_trigger": true },
{ "query": "readers can't tell whether our fix actually worked", "should_trigger": true },
{ "query": "put a docs link inside our API error responses", "should_trigger": true },
{ "query": "what should go in a 'still stuck' section", "should_trigger": true },
{ "query": "audit our troubleshooting pages", "should_trigger": true },
{ "query": "our known-issues list has entries for bugs fixed a year ago", "should_trigger": true },
{ "query": "rank which errors to document first", "should_trigger": true },
{ "query": "should we document this error or fix the message", "should_trigger": true },
{ "query": "write the entry for invalid_grant", "should_trigger": true },
{ "query": "we're getting the same Stack Overflow question over and over", "should_trigger": true },
{ "query": "how do I stop answering the same GitHub issue every week", "should_trigger": true },
{ "query": "make our error pages easier for the docs bot to answer from", "should_trigger": true },
{ "query": "our AI assistant still gives the old answer after we fixed the page", "should_trigger": true },
{ "query": "I want to prove the error documentation reduced tickets", "should_trigger": true },
{ "query": "what's a realistic target for self-service resolution", "should_trigger": true },
{ "query": "turn these 40 resolution notes into pages", "should_trigger": true },
{ "query": "we have telemetry on error codes but nothing documented", "should_trigger": true },
{ "query": "our escalation link points at a Slack channel we archived", "should_trigger": true },
{ "query": "help me write the exit-code documentation for our CLI", "should_trigger": true },
{ "query": "document why the build hangs at 90%", "should_trigger": true },
{ "query": "we need pages people find by pasting the error", "should_trigger": true },
{ "query": "set up a habit where support writes the article while resolving", "should_trigger": true },
{ "query": "who should own writing our error documentation", "should_trigger": true },
{ "query": "how many of our top failures should be covered this quarter", "should_trigger": true },
{ "query": "our entries say 'simply reinstall' and customers hate it", "should_trigger": true },
{ "query": "review this error page draft", "should_trigger": true },
{ "query": "our webhooks never arrive and three customers filed tickets about it", "should_trigger": true },
{ "query": "should I write one page per ticket", "should_trigger": true },
{ "query": "what goes in the 'applies to' part of an error entry", "should_trigger": true },
{ "query": "we can't reproduce half of the fixes we documented", "should_trigger": true },
{ "query": "our troubleshooting steps assume a working machine", "should_trigger": true },
{ "query": "add verification steps to our fix instructions", "should_trigger": true },
{ "query": "we want an --explain flag like rustc for our compiler errors", "should_trigger": true },
{ "query": "do we need FAQ schema on our troubleshooting pages", "should_trigger": true },
{ "query": "convert our Zendesk macros into public documentation", "should_trigger": true },
{ "query": "our enterprise customers hit failures that never show up in public issues", "should_trigger": true },
{ "query": "what metrics show troubleshooting docs are working", "should_trigger": true },
{ "query": "how much does a support ticket cost us and what do docs save", "should_trigger": true },
{ "query": "the CFO wants a business case for writing error documentation", "should_trigger": true },
{ "query": "our error message says 'operation failed' and nothing else", "should_trigger": true },
{ "query": "what's the right order for the checks in a troubleshooting how-to", "should_trigger": true },
{ "query": "we published the pages but the same tickets keep coming", "should_trigger": true },
{ "query": "help me decide what not to document", "should_trigger": true },
{ "query": "our docs and our support macro give different answers", "should_trigger": true },
{ "query": "make a triage page for the symptom that maps to several codes", "should_trigger": true },
{ "query": "how do I word a 'reported but not reproduced' disclaimer", "should_trigger": true },
{ "query": "our error strings changed in 3.0 and the docs still show the old text", "should_trigger": true },
{ "query": "stamp our troubleshooting pages so we can see which are stale", "should_trigger": true },
{ "query": "we're launching a new version and want the known-issues list ready", "should_trigger": true },
{ "query": "how do I keep the code enum and the docs table in sync", "should_trigger": true },
{ "query": "I need to document 137 exit statuses from our container runtime", "should_trigger": true },
{ "query": "should the fix explanation live in the docs or in the product", "should_trigger": true },
{ "query": "we get 60 tickets a month about the same timeout and it's always the same fix", "should_trigger": true },
{ "query": "the same configuration mistake generates 30 issues a month and I want a page that catches it", "should_trigger": true },
{ "query": "audit our docs structure against Diataxis", "should_trigger": false },
{ "query": "which of our docs pages are tutorials pretending to be reference", "should_trigger": false },
{ "query": "our navigation is a mess, reorganize the documentation site", "should_trigger": false },
{ "query": "write a quickstart that gets someone to their first API call", "should_trigger": false },
{ "query": "our getting-started page takes 40 minutes, cut it down", "should_trigger": false },
{ "query": "add fail branches to our quickstart steps", "should_trigger": false },
{ "query": "write the migration guide for our 3.0 breaking changes", "should_trigger": false },
{ "query": "document what breaks when upgrading from v2 to v3", "should_trigger": false },
{ "query": "our codemod only covers 70% of the breaking changes, how do I document that boundary", "should_trigger": false },
{ "query": "turn this week's merged PRs into release notes", "should_trigger": false },
{ "query": "write the changelog entry saying this bug is fixed in 4.1", "should_trigger": false },
{ "query": "our CHANGELOG.md is just raw commit messages", "should_trigger": false },
{ "query": "find the keywords developers search when integrating a payments API", "should_trigger": false },
{ "query": "build a prioritized keyword list from our docs search logs", "should_trigger": false },
{ "query": "our docs pages aren't indexed by Google, fix the canonicals", "should_trigger": false },
{ "query": "set up hreflang across our translated docs versions", "should_trigger": false },
{ "query": "our versioned docs are competing with each other in search results", "should_trigger": false },
{ "query": "design the label taxonomy for our GitHub issue triage", "should_trigger": false },
{ "query": "set response-time targets for our open issue queue", "should_trigger": false },
{ "query": "route security reports away from the public tracker", "should_trigger": false },
{ "query": "define the policy every code sample in our docs must meet", "should_trigger": false },
{ "query": "our code snippets don't compile, audit the sample corpus", "should_trigger": false },
{ "query": "make our API docs usable by a coding agent unattended", "should_trigger": false },
{ "query": "publish OpenAPI and markdown endpoints for our documentation", "should_trigger": false },
{ "query": "write a tutorial that teaches someone our query language", "should_trigger": false },
{ "query": "add checkpoints the learner can resume from", "should_trigger": false },
{ "query": "write an engineering blog post about our rate limiter rewrite", "should_trigger": false },
{ "query": "rewrite our README so a stranger can run the first command in a minute", "should_trigger": false },
{ "query": "which DevRel metrics should we report to the board", "should_trigger": false },
{ "query": "build the tracking plan for our docs and repository events", "should_trigger": false },
{ "query": "write our code of conduct and the moderation enforcement ladder", "should_trigger": false },
{ "query": "design the path to a stranger's first merged contribution", "should_trigger": false },
{ "query": "map the developer journey from discovery to advocacy", "should_trigger": false },
{ "query": "cut our developer audience into named, sized segments", "should_trigger": false },
{ "query": "plan a quarter of developer content with owners and reviewers", "should_trigger": false },
{ "query": "benchmark a competitor's developer relations motion", "should_trigger": false },
{ "query": "how do I land a developer advocate job", "should_trigger": false },
{ "query": "write the job posting and scorecard for a developer advocate", "should_trigger": false },
{ "query": "choose a license for our open-source project", "should_trigger": false },
{ "query": "design our open-source governance model", "should_trigger": false },
{ "query": "which conferences should we sponsor next year", "should_trigger": false },
{ "query": "prep me for a technical podcast interview next week", "should_trigger": false },
{ "query": "outline my accepted conference talk", "should_trigger": false },
{ "query": "write a CFP submission for a platform engineering conference", "should_trigger": false },
{ "query": "script a 10-minute screencast about our CLI", "should_trigger": false },
{ "query": "design our live demo so it survives the stage", "should_trigger": false },
{ "query": "plan the launch of our open-source project", "should_trigger": false },
{ "query": "pick the pricing tiers for our developer tool", "should_trigger": false },
{ "query": "should we open source our scheduler", "should_trigger": false },
{ "query": "which open-source projects should our company sponsor", "should_trigger": false },
{ "query": "production is down with a 502 loop, help me debug it", "should_trigger": false },
{ "query": "our deploy is failing right now, what's wrong", "should_trigger": false },
{ "query": "I'm getting ERR_MODULE_NOT_FOUND in my build, fix it", "should_trigger": false },
{ "query": "why does this Python traceback happen", "should_trigger": false },
{ "query": "walk me through debugging this memory leak", "should_trigger": false },
{ "query": "write the postmortem for last night's outage", "should_trigger": false },
{ "query": "design the on-call runbook for our payments service", "should_trigger": false },
{ "query": "set up an incident response process with severity levels", "should_trigger": false },
{ "query": "our error rate spiked 20 minutes ago, triage it", "should_trigger": false },
{ "query": "rewrite the error messages our product emits so they're clearer", "should_trigger": false },
{ "query": "improve error handling in our Go service", "should_trigger": false },
{ "query": "add structured logging to our API", "should_trigger": false },
{ "query": "review this code for security vulnerabilities", "should_trigger": false },
{ "query": "design our API's error response format", "should_trigger": false },
{ "query": "what HTTP status codes should our API return for validation failures", "should_trigger": false },
{ "query": "define our rate limit policy and the headers we return", "should_trigger": false },
{ "query": "design idempotency keys for our payments endpoint", "should_trigger": false },
{ "query": "write our API status page communication policy", "should_trigger": false },
{ "query": "how should we version our public API", "should_trigger": false },
{ "query": "write a support macro for password reset requests", "should_trigger": false },
{ "query": "set up our Zendesk ticket routing rules", "should_trigger": false },
{ "query": "help me hire a support engineer", "should_trigger": false },
{ "query": "reduce our first response time in support", "should_trigger": false },
{ "query": "build a customer health score for our accounts", "should_trigger": false },
{ "query": "write the FAQ section for our pricing page", "should_trigger": false },
{ "query": "optimize our blog posts to get cited by AI search engines", "should_trigger": false },
{ "query": "add schema markup to our marketing landing pages", "should_trigger": false },
{ "query": "translate our documentation into German and Japanese", "should_trigger": false }
]
}
references/discoverability-and-metrics.md›
# Discoverability, verification and measurement
Contents: how readers arrive · page-level findability · wiring the product back into docs · what no longer works · verification protocol · metrics · sourced benchmarks and numbers to refuse · freshness · pass threshold · maintenance triggers.
## How readers arrive
A developer hitting an error pastes the string somewhere: a search engine, an assistant, the docs search box, or the issue tracker. Nobody browses a troubleshooting section speculatively. Every findability decision follows from that.
Assistants now sit between many readers and the docs - they retrieve and paraphrase whichever page states the error, the cause and the steps most explicitly. The structure that helps a human scanner (labelled fields, one cause per block, fenced error text) is the same structure that survives retrieval, so there is no separate optimization to do.
## Page-level findability
- Put the error string verbatim in the title, the first heading and the body. "Troubleshooting authentication" never matches a search for `invalid_grant`.
- Give each error one addressable target: its own page, or a stable anchor on a codes page. A code buried mid-paragraph cannot be linked from the product.
- Answer in the first screen. Message, cause, first fix - before any preamble about the feature.
- Keep the page indexable: no login wall, no client-side-only rendering, stable URL, redirect if it moves.
- Name the versions the entry applies to on the page, not just in a sidebar; readers land on old copies.
- Cross-link the entry from the page where the error is produced (the auth guide, the deploy step), not only the reverse.
## Wiring the product back into the docs
The strongest referrer is the failure itself.
- Ship a documentation URL in the error payload. Stripe's API errors carry a `doc_url` attribute pointing at that code's entry, so the integration surfaces the link at failure time.
- Give the CLI an explain path: `rustc --explain E0382` prints the same content as the web page, in the terminal where the reader already is.
- Keep identifiers stable across releases. Renaming an error code invalidates every link, bookmark and cached answer pointing at it.
- Make the emitted string and the entry title identical strings; two spellings of one error halve the match rate.
## What no longer works
- **FAQ structured data**: eligibility was cut to well-known government and health sites in 2023, and Google Search Central announced the rich result stopped appearing entirely on 7 May 2026, removing the documentation in June 2026. Marking troubleshooting content as `FAQPage` for search real estate now buys nothing.
- **Question-shaped headings written for search rather than for readers** ("Why does my build fail?") when the reader is searching an exact string.
- **A single mega "Troubleshooting" page** holding thirty issues: only one of them can win the query, anchors get lost in caches, and the page becomes unmaintainable.
## Verification protocol
Publish nothing that has not been run.
1. Reproduce the error on a supported version, and capture the emitted text from that run.
2. Execute the fix exactly as written, from the state the reader is in - not from your already-configured machine.
3. Record the post-fix signal as the "confirm it worked" line.
4. Check applicability at both edges: the oldest supported version and the current one. Note where behaviour differs.
5. Have someone who did not write the entry follow it cold. Their first hesitation marks the step with an unstated assumption.
6. Re-check every command and placeholder for leaked credentials and customer identifiers.
Entries that cannot be reproduced (environment-specific, third-party, intermittent) get published only with an explicit "reported under these conditions, not reproduced here" note - an unverified fix stated confidently costs more trust than an honest gap. Write the Docs' documentation principles state the underlying rule directly: "Consider incorrect documentation to be worse than missing documentation."
The Good Docs Project applies the same bar as a shipping gate: publish a troubleshooting page only if the team can test the fixes on every supported platform and keep them current. An untested page burns the reader's remaining patience at the exact moment they have least of it.
## Metrics
| Metric | How to read it |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Cluster coverage | Share of the top-N clusters with a published entry. The headline number. |
| Ticket volume per covered error | Compare fixed windows before and after publication. Per-error deflection is measurable; a site-wide "deflection rate" is not. |
| Zero-result docs searches for covered strings | Should reach zero; a nonzero value usually means the string is not verbatim on the page. |
| Entry as first search result for its string | Check per entry on the real search surface, not in an SEO tool. |
| Time-to-resolve on tickets linking the entry | Falls when the entry is good; rises when it is stale and readers try it first. |
| Entries touched since the last release | Staleness proxy - publishing volume without this number is vanity. |
| Escape-hatch quality | Share of escalations that arrive with the attachments the entry asked for. |
Deep instrumentation across devrel surfaces belongs to the analytics skill; the numbers above only need docs search, ticket counts and page analytics.
## Sourced benchmarks, and the numbers to refuse
Set expectations against published data rather than ambition.
- **The self-service ceiling.** Gartner's December 2023 survey of 5,728 customers found only 14% of customer service and support issues fully resolved in self-service, and only 36% even for issues customers rated "very simple", despite 73% using self-service at some point. Gartner Senior Director Eric Keller: "while 73% of customers use self-service at some point in their customer service journey, it's concerning to see that so few fully resolve there." An earlier Gartner survey (n=8,398) reported 9%. Best-in-class self-service does not fully resolve most contacts; write the objective accordingly.
- **The loyalty case.** CEB/Gartner research published as Dixon, Freeman and Toman, "Stop Trying to Delight Your Customers" (_Harvard Business Review_, 2010) and expanded in _The Effortless Experience_ (2013): 96% of customers who experience a high-effort service interaction become more disloyal, against 9% for a low-effort interaction. This is the argument for the work that survives a stakeholder who does not believe in ticket deflection.
- **Resolution over deflection.** Practitioner analysis argues resolution rate matters more than deflection rate, and that cost per resolution is the true efficiency metric because it accounts for the repeat contacts that cost-per-contact hides.
- **Refuse the folklore.** The widely repeated deflection percentages and the "$15-$50 per support ticket" figure circulate without traceable methodology. Treat any unattributed deflection or per-ticket cost number as marketing; use the team's own finance figures or state that the number is unknown.
## Freshness
No published, docs-specific freshness SLA from a named vendor could be verified for troubleshooting content; the commonly cited cadence guidance is SEO-oriented (a substantive update changing roughly 20-30% of textual content, with date-only changes counting against quality). Two workable rules in its absence:
- Refresh whenever the product changes - the trigger is the release, not the calendar.
- If a competent reader could point at a provably stale sentence, the page is overdue.
## Pass threshold
Every entry ships with all six parts present:
- verbatim message
- applicability
- cause
- executed fix
- confirmation signal
- escape hatch
Five out of six is a draft. That gate is structural and non-negotiable.
The coverage number is not. **80% of the top 20 clusters within the current cycle is this skill's self-set baseline, not a published benchmark** - negotiate it with the user against real capacity, write down the number you agreed, and re-run the clustering pass afterwards to compare. The same top clusters reappearing unchanged is the signal that the entries are not being found, not that more entries are needed.
## Maintenance triggers
- A release changes an error's text, code or cause → update or retire the entry in the same release.
- The underlying bug ships a fix → move the known-issues entry into the changelog and delete it.
- An entry's cluster disappears from the mining pass for two cycles → archive it rather than keeping it in the navigation.
- The escape hatch's channel closes or moves → fix every entry pointing at it; a dead escape hatch converts a helped reader into an angry one.
references/entry-templates.md›
# Entry templates and worked examples
Contents: standards the block comes from · page shapes · error-reference template · troubleshooting how-to template · known-issues template · worked example (weak vs. strong) · quoting rules · microcopy style guides.
## Table of Contents
- [Where the block comes from](#where-the-block-comes-from)
- [Page shapes](#page-shapes)
- [Error reference entry](#error-reference-entry)
- [Message](#message)
- [Applies to](#applies-to)
- [Cause](#cause)
- [Fix](#fix)
- [Confirm it worked](#confirm-it-worked)
- [Still stuck](#still-stuck)
- [Troubleshooting how-to](#troubleshooting-how-to)
- [Symptoms](#symptoms)
- [Before you start](#before-you-start)
- [Check 1 - <cheapest, most common cause>](#check-1-cheapest-most-common-cause)
- [Check 2 - <next cause>](#check-2-next-cause)
- [Confirm it worked](#confirm-it-worked)
- [Still stuck](#still-stuck)
- [Known-issues entry](#known-issues-entry)
- [Worked example](#worked-example)
- [Authentication problems](#authentication-problems)
- [Message](#message)
- [Applies to](#applies-to)
- [Cause](#cause)
- [Fix](#fix)
- [Confirm it worked](#confirm-it-worked)
- [Still stuck](#still-stuck)
- [Microcopy: the string versus the page](#microcopy-the-string-versus-the-page)
- [Quoting the error text](#quoting-the-error-text)
## Where the block comes from
The six-part entry is not invented here. It is the intersection of three published models, which is why it is worth defending when a team wants to shorten it.
- **OASIS DITA 1.3** (Part 2, Technical Content) defines a `<troubleshooting>` topic type whose `<troublebody>` runs condition/symptom → cause → remedy, with several cause-remedy pairs supported explicitly as "successive fall-backs for resolving a condition". That is the multi-cause fix section, standardized.
- **KCS** article fields are issue, environment, cause, resolution - the environment field is the "Applies to" block, and it is the part teams drop first and regret first.
- **Rust RFC 1567** mandates, per error code: description, erroneous code example, explanation, how to fix. It is the strictest published per-entry template and a good bar for a reference catalog.
Two quality standards apply to the prose inside the block. **ISO 24495-1:2023**, the first international plain-language standard, defines success as readers being able to "find what they need, understand what they find, and use that information". **Nielsen heuristic 9** requires error content to use plain language, precisely identify the problem, offer a constructive solution, and never blame the user.
## Page shapes
| Shape | Choose when | Reader arrives with | Lives in the docs as |
| ---------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------------------------------------------------------------- |
| Error reference entry | The product emits a stable identifier (`ERR_INVALID_ARG_TYPE`, `E0382`, `invalid_grant`, exit code 137) | An exact string | One page per code, or one anchored row per code on a codes page |
| Troubleshooting how-to | The failure has symptoms but no code ("webhooks never arrive", "build hangs at 90%") | A description of misbehaviour | One page per symptom, under the feature it belongs to |
| Known-issues entry | Cause confirmed, fix not shipped | A suspicion that it is not their fault | A dated list per product area, pruned on release |
A coded error written as a how-to loses the match; an uncoded symptom forced into a code table loses the diagnosis. When a symptom has several possible codes, write the how-to as the triage page and link out to each code entry.
## Error reference entry
````markdown
# `ERROR_IDENTIFIER`
<one line: what the product was trying to do when it emitted this>
## Message
```text
<the error text, byte-for-byte as emitted, variable parts in <ANGLE_BRACKETS>>
```
````
## Applies to
<versions / platforms / plans / configurations where this occurs>
## Cause
<one or two sentences, in the product's own model of the world>
## Fix
1. <check or command>
2. <check or command>
<repeat as "### If <condition>" blocks when there are several distinct causes,
most-often-the-cause per minute of reader time first, not cheapest first>
## Confirm it worked
<the output, log line, or state the reader should now see>
## Still stuck
<where to report it, and what to attach: version, config excerpt, full trace>
---
Last verified: <YYYY-MM-DD> on <version> · Owner: <team or handle>
````
The last-verified stamp and owner are what make staleness visible without reading the page. A troubleshooting entry with no date is indistinguishable from a correct one until a reader runs a dead command.
## Troubleshooting how-to
```markdown
# <Symptom, in the reader's words>
## Symptoms
- Does <observable A> happen?
- Do you see <observable B> in the logs?
## Before you start
<what to have open: log access, a reproducible request, admin rights>
## Check 1 - <cheapest, most common cause>
<how to check, what a healthy result looks like, what to do if it is not healthy>
## Check 2 - <next cause>
…
## Confirm it worked
## Still stuck
````
Order checks by `probability ÷ cost`, not by architectural layer. A reader in an incident abandons the page at the first step that takes ten minutes.
Give each command its own prerequisite and failure branch. The reader is on a broken machine, not the healthy one the steps were written on:
```markdown
3. Check the service is reachable: `curl -sf https://<HOST>/healthz`
- Requires: network access to <HOST>, no proxy interception.
- If this fails with a TLS error: your proxy is terminating TLS - see <entry>.
- Expected: HTTP 200 with body `ok`.
```
## Known-issues entry
```markdown
### <Short title> - affects <versions>, reported <YYYY-MM-DD>
**Symptom**: <what the reader sees, error text included>
**Cause**: <confirmed cause, or "under investigation">
**Workaround**: <steps, or "none - avoid <operation> until fixed">
**Tracking**: <issue link> **Fixed in**: <version, or "unreleased">
```
Delete the entry when the fix ships, or move it into the error entry with a version note. A known-issues list that only grows becomes a graveyard nobody reads.
## Worked example
The same failure, written twice.
### Weak
```markdown
## Authentication problems
If you have trouble authenticating, make sure your credentials are correct and
that your token has not expired. Check the auth documentation for more details.
If the problem persists, contact support.
```
Nothing here is searchable (the error string is absent), nothing is checkable (no command, no expected output), and the only action is "contact support" - which is the outcome the page exists to prevent.
### Strong
````markdown
# `invalid_grant`
The token endpoint rejected the refresh token you presented.
## Message
```text
{"error":"invalid_grant","error_description":"Token is expired or revoked"}
```
````
## Applies to
All API versions. Refresh tokens issued before <VERSION> expire after 30 days;
tokens issued after it expire after 90 days.
## Cause
The refresh token has expired, was already exchanged (refresh tokens are
single-use), or was revoked when the user changed their password.
## Fix
### If the token is older than its lifetime
1. Read the issue date: `curl -s https://api.example.com/v1/token/info -H "Authorization: Bearer <TOKEN>"`.
2. Re-run the authorization flow to obtain a new refresh token.
3. Store the new refresh token returned in the same response - the old one is now dead.
### If two processes refresh concurrently
Both receive the same token, one exchange wins, the loser gets `invalid_grant`.
Serialize refreshes behind a single owner, or retry once after re-reading the
stored token.
## Confirm it worked
`token/info` returns `"status": "active"` and a new `expires_at` in the future.
## Still stuck
Open an issue with the token's `jti`, the timestamp of the failed request, and
your client ID. Never paste the token itself.
````
Every claim is checkable, the fix branches on cause instead of listing everything, and the escape hatch says what to attach - and what not to.
### Also weak, and more tempting
The trap is not only the empty page above. It is the entry that looks complete and still fails the reader:
```markdown
# Token error
Simply refresh your token - this is an easy fix. If you forgot to store the new
refresh token, you'll get this error again. Error 401.
````
Four defects, each common:
- The title is not the emitted string, so nobody finds it.
- "simply" and "easy" read as blame to someone the fix is not working for.
- "you forgot" blames the reader outright (Nielsen heuristic 9).
- There is no verification step, so the reader cannot tell a real fix from a second failure masking the first.
Being wrong in a confident, tidy voice is worse than being visibly incomplete.
## Microcopy: the string versus the page
These are two surfaces with different budgets, and conflating them is the most common structural mistake. Microsoft's style guide draws the line explicitly: "avoid long how-to or instructional guidance in errors - that content lives in your documentation." The emitted string stays short and points at the entry; the entry carries the diagnostic depth.
| Guide | Blame | Codes and jargon | Notable rule |
| ---------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Microsoft Writing Style Guide | "Avoid phrasing that blames the user"; passive voice acceptable to avoid blame | Avoid the word "error"; hide codes under a details affordance | Reserve "please" for when the software is at fault; "sorry" only for serious problems |
| Microsoft Win32 error message guidelines | - | No technical jargon; no "error" in the title bar | "Write a separate error message for each known cause" - one generic message serves nobody |
| Nielsen Norman Group (heuristic 9) | Never blame the user | Plain language, no bare "Error 502" | Make errors visible, reduce the work to fix them, educate along the way |
| Go Code Review Comments | - | - | Error strings are not capitalized and do not end with punctuation, because they compose into larger messages |
| PostgreSQL error message style guide | Readers "aren't expected to know the details" | Avoid implementation details | Separate primary message, detail and hint |
| Rust RFC 1567 | - | Code plus `--explain` entry | Explanations help users "understand why their code cannot be accepted", not just paste a fix |
Elm is the origin of the non-adversarial tone this table keeps circling. Evan Czaplicki's stated goal was to "get the compiler to a point where people feel like it is actually helping them learn Elm syntax", and Elm maintains a public error-message catalog repository for confusing messages. Its influence on Rust's diagnostics is widely noted.
Rewriting the emitted string itself is a product change, not a docs task. Hand it to the owning engineer and record it in the queue.
## Quoting the error text
- Copy the string from a real run, not from the source code: the emitted text often differs from the format template.
- Keep casing, punctuation and line breaks. Readers scan-match against their terminal.
- Replace only what genuinely varies, with a placeholder that cannot be mistaken for content: `<PATH>`, `<REQUEST_ID>`, `<VERSION>`.
- Put it in a fenced block so copy-paste is clean and search engines index it as text.
- When the product emits several variants of one failure, list them all in the same block; readers search on whichever they got.
- Redact secrets in every example, and say so - a page that shows a full token teaches readers to paste theirs into an issue.
references/error-catalog-and-ai-surfaces.md›
# Generated error catalogs and AI-assistant surfaces
Contents: why generate the catalog · published implementations · how to propose it · retrieval-friendly page shape · re-indexing triggers · machine-readable surfaces · the unanswered-query loop.
## Why generate rather than hand-maintain
A catalog written twice - once as a constant in code, once as a row in the docs - drifts by default. Nobody plans the drift; it arrives with the release that adds a code and forgets the docs. Generation removes the failure mode instead of policing it: one file feeds both artefacts, so the constant and the entry cannot disagree.
Raise this before writing a page per code. It is a small engineering task with a permanent payoff, and it changes the maintenance cost of everything downstream in this skill.
## Published implementations
| Project | Source of truth | What it generates |
| ---------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| PostgreSQL | `src/backend/utils/errcodes.txt` | The C header `errcodes.h` and `doc/src/sgml/errcodes-list.sgml`, the documentation's error-code table |
| Rust | `compiler/rustc_error_codes` | The online error index and the `rustc --explain <code>` terminal output, from Markdown explanations linked in the `rustc_error_codes` crate |
| Twilio | One error dictionary | The web reference page plus a downloadable JSON list of all REST API error codes |
| Stripe | One error-code catalog | The reference table, plus the `doc_url` attribute carried inside every API error object, deep-linking to that code's entry |
Rust's RFC 1567 also standardizes what each generated entry must contain:
- error description
- erroneous code example
- explanation
- how to fix
Adopt that as the schema of the source file, not just of the rendered page.
Analogous published accounts outside error catalogs: Sidero Labs' Talos describes wanting "a single source of truth both for markdown and for YAML comments", and Istio moved its reference docs onto a proto/Go-struct source of truth. A specific "we generated our error site from protobuf enums" case study could not be located - treat the protobuf-enum route as a general technique rather than a citable playbook.
## How to propose it
1. Ask where error identifiers are declared today (question 3 of the interview). Several declaration sites means the drift already exists; find it before proposing anything.
2. Diff the documented catalog against the codes the product can actually emit. The size of that diff is the business case.
3. Propose the smallest version: one file, one generator, both outputs checked in. Do not bundle it with a docs-site migration.
4. Name the owning team. This lands in the product repository, not the docs repository, and a docs-owned proposal to change compiler source goes nowhere.
## Serving these pages to assistants
Increasingly the reader is a retrieval system paraphrasing the page rather than a human scanning it. The good news is that there is no separate optimization: the structure that helps a scanner is the structure that survives retrieval.
- **Write blocks that stand alone.** Chunking is one of the most critical steps in a retrieval pipeline, and naive fixed-token splitting damages retrieval when it cuts a sentence, paragraph or logical idea in half. A cause-remedy pair that depends on the paragraph above it retrieves as a fragment.
- **Label the fields.** Symptom, conditions, cause, fix, verification as explicit headings gives the retriever anchors and gives the answer somewhere to point.
- **Document the negative cases.** If information is not explicitly written down it does not exist in the assistant's knowledge base; an undocumented workaround cannot be answered, only guessed at.
- **Prefer an honest refusal to a guess.** Where the assistant is configurable, an explicit "I don't know" guardrail is worth more than coverage: a bot that invents a fix for an undocumented error costs more trust than one that says no documented fix exists yet.
## Re-indexing is part of publishing
Embedding rot is a maintenance hazard distinct from ordinary staleness: the published page is current and the retrieval index still serves the old text. Practitioner guidance from vendors building docs-grounded assistants is to re-embed the corpus when the embedding model meaningfully improves, when roughly 10-15% of the content has changed, or when the domain itself shifts.
Add the trigger to the maintenance plan, with the same owner as the docs release. A troubleshooting workflow that edits pages without ever re-triggering an index rebuild silently serves stale answers to the readers most likely to be mid-incident.
## Machine-readable surfaces
`llms.txt` - an H1 project name, a blockquote summary and H2-organized links, with an optional `llms-full.txt` inlining the full text - is a proposal published by Jeremy Howard of Answer.AI in September 2024. It is explicitly a proposal, not a ratified standard, with no W3C or IETF working group behind it. Treat it as cheap and optional: worth adding if the docs generator supports it, never worth a project.
The more durable version of the same idea is serving clean Markdown for any page on request, as GitHub does through its Markdown API.
Designing these machine-facing entry points across a whole SDK or API surface is a different job from making troubleshooting entries retrievable, and it belongs to samber/developer-relations-skills@coding-agent-docs-optimization. Stop here at the page level.
## The unanswered-query loop
An assistant's failure log is the newest form of the zero-result-search signal, and the strongest one, because it captures the question in the reader's own words rather than as keywords. Feed it back into step 1: unanswered or badly-answered queries become clusters, clusters become entries, and the entries reduce the next cycle's failures. Ask in the interview whether that log is accessible - many teams have it and never look at it.
references/signal-mining.md›
# Mining failure signals into a page queue
Contents: signal sources · self-serve vs. contract-bound audiences · running the clustering script · queue cut-off · what to route away from docs · KCS capture loop and role ladder · promote-to-article tooling · feedback signals.
## Table of Contents
- [Signal sources](#signal-sources)
- [Audience differences](#audience-differences)
- [Running the clustering script](#running-the-clustering-script)
- [Ranking rubric](#ranking-rubric)
- [Route away from docs](#route-away-from-docs)
- [Capture loop (KCS)](#capture-loop-kcs)
- [Who writes, who edits](#who-writes-who-edits)
- [Promote-to-article tooling (integration note)](#promote-to-article-tooling-integration-note)
- [Feedback and search signals](#feedback-and-search-signals)
## Signal sources
| Source | What to export | What it over-represents |
| --------------------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------- |
| Support tickets | Subject, first message, resolution note, tags, dates | Paying customers; failures worth complaining about |
| Issue tracker | Titles and bodies of issues labelled question/support, plus issues closed as configuration | Vocal, technical users |
| Docs site search | Queries, result counts, click-through | People who already trust the docs |
| Community channels and public Q&A | Recurring question threads | Beginners and integration-time failures |
| Error telemetry | Counts per error code and per version | Everything - including failures nobody reports |
| Sales / solutions engineers | Their private FAQ list | Evaluation-time blockers (limits, proxies, SSO) |
| Reviews of the last N incidents | Customer-visible symptoms | Rare but expensive failures |
Take at least one reported source and one unreported source. Tickets alone show what people complain about; telemetry alone shows what breaks. The gap between them is where silent abandonment lives.
## Audience differences
The page anatomy is identical for every audience - the sourcing is not.
- **Self-serve / individual developers (including OSS)**: no ticket system. Issues, discussions, community channels and search logs carry the whole signal, and the reader has no escalation path, so the escape hatch must be a real, watched channel.
- **Contract-bound / enterprise**: the loudest failures sit in private tickets and never reach the public tracker. Mining only public sources under-serves exactly the accounts that pay. Expect environment-specific causes (proxies, SSO, air-gapped installs, IP allowlists) that need their own conditions block, and check whether a fix may be published at all before writing it.
- **Both**: a support macro and a public entry should never disagree. When one exists, the other is either a link to it or a superset of it.
## Running the clustering script
```bash
python3 scripts/error-cluster.py tickets.csv issues.jsonl --min-count 3 --top 40
python3 scripts/error-cluster.py export.jsonl --format json > clusters.json
python3 scripts/error-cluster.py tickets.csv --key-length 40 # merge over-split clusters
cat support-notes.txt | python3 scripts/error-cluster.py -
```
The script reads three input formats:
- CSV/TSV, where a row is a record.
- JSON/JSONL, where an object is a record.
- Plain text, where a line is a record.
From each record it extracts error-like fragments, replaces the parts that vary (paths, UUIDs, numbers, quoted values, URLs) with placeholders, and ranks clusters by how many records they appear in, with first/last-seen dates when the export carries a date field.
Read the output as a shortlist, not as a verdict:
- Merge clusters that are the same failure under different wording; lower `--key-length` when one failure keeps splitting.
- Split a cluster whose members have different causes - the same string can come from two places.
- A cluster whose `first_seen` sits just after a release is a regression to report, not a page to write.
- Records with no error text at all still matter; skim them for symptom-only failures, which become how-to pages.
Without file access or a scriptable environment, do the same pass by hand on the top 50 tickets: strip the variable parts, tally, and keep one raw example per cluster.
## Ranking rubric
The factors, the ratio they feed and the resulting order are in `SKILL.md` Step 2, where the queue is actually built. One consequence of that ranking belongs here: stop when the marginal cluster is under a handful of occurrences. Ten excellent entries beat sixty stubs, because every stub still costs a maintenance pass on every release.
## Route away from docs
Not every recurring failure is a documentation gap. Say so explicitly and hand it to the right owner:
| Symptom of the cluster | Real fix |
| ------------------------------------------------ | -------------------------------------------------------------------------------- |
| Error text does not name what to do | Rewrite the message in the product (a doc page cannot be pasted into a terminal) |
| Fix is "edit three config files in order" | Product/DX change, or a command that does it |
| Cause is account state (quota, plan, suspension) | Product surface or a support macro, not a public page |
| Recent spike, single version | Regression - file it; a page would document a bug into permanence |
| Fix requires credentials the reader lacks | Escalation path, documented as such |
| Same failure class across several postmortems | Scheduled product fix; the page is the interim measure only |
Documenting around a defect makes the defect permanent, and the page outlives everyone's memory of why it exists. Google's SRE practice states the same rule for incidents: postmortem action items must be _preventative_, not merely _mitigative_, because unaddressed outages "tend to regularly resurface and accumulate over time". Runbooks and workaround pages "can speed up incident response significantly" but are explicitly interim; the PagerDuty case study in the SRE workbook shows the postmortem, not the runbook, driving the permanent fix.
Rust runs the same gate at the other end of the scale: "not all diagnostics have a code", so only errors judged worth a durable, educational explanation get an `--explain` entry. Deciding _not_ to document an error is a legitimate output of this step.
## Capture loop (KCS)
Knowledge-Centered Service, published by the Consortium for Service Innovation, is the operating model behind this: capture knowledge _in_ the resolution workflow (solve loop), then restructure it for reuse (evolve loop). Its article fields - issue, environment, cause, resolution - are the entry block this skill writes. Two consequences for a docs team:
- The resolver writes the raw material at resolution time, while the exact error text and environment are still in front of them. Reconstructed-from-memory entries are the ones that turn out to be wrong.
- Genericize before publishing: strip the customer's identifiers, keep the environment facts that determine whether the entry applies.
Gartner endorsed the same pattern independently in August 2024 guidance: scale self-service content by "expanding content creation responsibilities to reps, enabling them to create knowledge as part of the issue resolution workflow, rather than as a separate process".
Propose the loop to the user as a standing habit, not a one-off: a template field in the ticket tool ("error text, environment, cause, resolution") plus a weekly pass that promotes repeat resolutions into entries keeps the queue from rebuilding.
## Who writes, who edits
The recurring model is distributed authoring with centralized editing, not a single owning role.
| Role | Output | When |
| ------------------------------------------ | ------------------------------------------------------------------- | ---------------------------------- |
| Support/success engineer (KCS Contributor) | First-draft article captured during resolution | Solve loop |
| Technical writer | Editing, standardization, information architecture, UI-text review | After the draft, then continuously |
| Engineer | The error string itself, and long-form explanations shipped in code | At code-authoring time |
| KCS Coach / domain expert | Dedup, structure, currency of the content base | Continuous |
GitLab's public handbook is the clearest published example of the engineering half. Engineers "typically write the first draft of documentation for the new features they create", and technical writers review and maintain it. GitLab also names "user interface: any user-facing text, such as UI text and error messages" as its own technical-writing work type, rather than something outside docs.
The KCS licensing model names four author roles - Candidate (work-in-progress only), Contributor (validated articles without coach review), Publisher (externally facing content), Coach (develops others' competency). The KCS v6 Practices Guide (Technique 7.1) gives the one published staffing number in this area: a start-up program runs a coach-to-knowledge-worker ratio of 1:5 to 1:8, evolving toward roughly 1:50 once mature. Use it as a planning input when the user asks what staffing a capture program costs.
## Promote-to-article tooling (integration note)
Every major helpdesk ships a promote-a-ticket-to-an-article path, so check what the user already has before proposing a manual process. None of them clusters by root cause first - that step stays yours, and skipping it produces a page that restates one customer's ticket instead of covering the failure.
- **Zendesk** - Knowledge in the Agent Workspace creates a help-center article directly from the ticket being worked. It does not pre-populate the draft from the ticket, so budget the transcription time.
- **Microsoft Dynamics 365** - "Convert To > To Knowledge Article" auto-populates the draft with the case description, and for a resolved case the case resolution and resolution description too. The cleanest capture path of the group.
- **Freshdesk** - Email-to-KBase converts an agent's ticket reply into a solution article by CC'ing the knowledge base.
- **Salesforce** - Lightning Knowledge attaches articles to cases, including at case close so a reopened case shows what the customer already tried. Only _published_ articles can be attached, so a draft entry cannot be referenced during the resolution it came from.
- **Oracle Fusion Service** - creates draft articles directly from service requests.
## Feedback and search signals
- "Was this helpful?" widgets are cheap but low-resolution: a bare thumbs-down cannot distinguish "the page is wrong" from "my problem was never self-servable". Pair the rating with an optional free-text "why" field, and expect response bias plus widget blindness when the control looks like page furniture.
- Zero-result and no-answer search queries are the strongest low-effort gap signal. In the AI-assistant era the assistant's own unanswered-query log is the same signal on a new surface: a question it could not answer is a page that does not exist yet.
scripts/error-cluster.py›
#!/usr/bin/env python3
"""Cluster raw support/issue text into candidate troubleshooting pages.
Reads exported tickets, issues, log lines or search queries, extracts the
error-like strings from each record, normalizes the parts that vary between
occurrences (paths, IDs, numbers, quoted values), and ranks the resulting
clusters by how many records they appear in.
One cluster is one candidate page. The raw examples are kept so the verbatim
error text can be copied into the entry.
Usage:
python3 scripts/error-cluster.py tickets.csv issues.jsonl notes.txt
python3 scripts/error-cluster.py export.csv --min-count 3 --top 40
python3 scripts/error-cluster.py export.jsonl --format json > clusters.json
cat log.txt | python3 scripts/error-cluster.py -
Input formats are detected per file: .csv/.tsv (all cells of a row form one
record), .json/.jsonl (all string values of an object form one record),
anything else (one line = one record).
"""
import argparse
import csv
import json
import re
import sys
from collections import defaultdict
# Patterns that identify an error-like fragment inside free text. Ordered from
# most specific to most generic; every match is kept, so one record can produce
# several candidate strings.
ERROR_PATTERNS = [
(r"\bERR_[A-Z0-9_]+\b", 0), # Node-style codes
(r"\b[A-Z]{2,}\d{3,5}\b", 0), # SQLSTATE / E0382 / ORA-style
(r"\bE[A-Z]{3,}\b", 0), # ECONNREFUSED, EACCES
(r"\b[A-Za-z_][A-Za-z0-9_.]*(?:Error|Exception)\b[^\n]{0,120}", 0),
(r"(?:^|[\s\"'])(?:error|fatal|panic|warning)\s*:\s*[^\n]{0,120}", re.IGNORECASE),
(r"\bHTTP\s*(?:status\s*)?[45]\d{2}\b[^\n]{0,80}", re.IGNORECASE),
(r"\b[a-z][a-z0-9]*(?:_[a-z0-9]+)*_(?:error|failed|failure|invalid|denied|declined"
r"|required|unsupported|expired|missing|exceeded|timeout)\b", 0),
(r"\b(?:invalid|missing|unknown|unsupported|expired|denied)_[a-z0-9_]+\b", 0),
]
# Variable parts replaced before counting, so two occurrences of the same
# failure with different IDs land in one cluster.
NORMALIZERS = [
(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "<UUID>"),
(r"https?://[^\s'\"]+", "<URL>"),
(r"[\w.+-]+@[\w-]+\.[\w.]+", "<EMAIL>"),
(r"(?:/[\w.\-@]+){2,}/?", "<PATH>"),
(r"[A-Za-z]:\\[^\s'\"]+", "<PATH>"),
(r"\b[0-9a-fA-F]{7,}\b", "<HEX>"),
(r"\b\d+(?:\.\d+){2,}\b", "<VERSION>"),
(r"'[^']*'", "'<VALUE>'"),
(r'"[^"]*"', '"<VALUE>"'),
(r"`[^`]*`", "`<VALUE>`"),
(r"\b\d+\b", "<N>"),
]
DATE_KEYS = ("created_at", "createdat", "date", "created", "opened_at", "timestamp")
# How many characters of the normalized error identify the cluster.
KEY_LENGTH = 60
def records_from_file(path):
"""Yield (text, date_or_None) per record."""
if path == "-":
for line in sys.stdin:
if line.strip():
yield line.strip(), None
return
lower = path.lower()
if lower.endswith((".csv", ".tsv")):
delimiter = "\t" if lower.endswith(".tsv") else ","
with open(path, newline="", encoding="utf-8", errors="replace") as handle:
for row in csv.DictReader(handle, delimiter=delimiter):
cells = [str(v) for v in row.values() if v]
date = next((str(row[k]) for k in row if k and k.lower() in DATE_KEYS and row[k]), None)
if cells:
yield " ".join(cells), date
return
if lower.endswith((".json", ".jsonl", ".ndjson")):
with open(path, encoding="utf-8", errors="replace") as handle:
content = handle.read().strip()
objects = []
if content.startswith("["):
objects = json.loads(content)
else:
objects = [json.loads(line) for line in content.splitlines() if line.strip()]
for obj in objects:
if isinstance(obj, dict):
strings = [str(v) for v in obj.values() if isinstance(v, (str, int, float)) and str(v)]
date = next((str(obj[k]) for k in obj if k.lower() in DATE_KEYS and obj[k]), None)
yield " ".join(strings), date
else:
yield str(obj), None
return
with open(path, encoding="utf-8", errors="replace") as handle:
for line in handle:
if line.strip():
yield line.strip(), None
def extract(text):
found = []
for pattern, flags in ERROR_PATTERNS:
for match in re.finditer(pattern, text, flags):
fragment = match.group(0).strip().lstrip("\"'")
if len(fragment) >= 4:
found.append(re.sub(r"\s+", " ", fragment))
# A bare code (ENOENT) found inside a fuller message ("Error: ENOENT: no such
# file…") is the same failure: keep the richest fragment so the cluster label
# carries the text a reader would actually paste into a search box.
return [f for f in found if not any(f != other and f in other for other in found)]
def normalize(fragment):
normalized = fragment
for pattern, replacement in NORMALIZERS:
normalized = re.sub(pattern, replacement, normalized)
return normalized.strip().rstrip(".,;:")
def cluster_key(normalized, key_length=KEY_LENGTH):
"""Group on the head of the message.
Tickets append prose to the error ("… every time", "… when refreshing the
token"), which would otherwise split one failure across several clusters.
The first characters carry the identity; the tail carries the reporter's
mood.
"""
return normalized[:key_length]
def main():
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("files", nargs="+", help="input files, or - for stdin")
parser.add_argument("--min-count", type=int, default=2, help="drop clusters seen fewer times (default 2)")
parser.add_argument("--top", type=int, default=50, help="how many clusters to print (default 50)")
parser.add_argument("--format", choices=["tsv", "json"], default="tsv")
parser.add_argument(
"--key-length",
type=int,
default=KEY_LENGTH,
help=f"characters of the normalized error used to group (default {KEY_LENGTH}; "
"lower it when one failure still splits across clusters)",
)
args = parser.parse_args()
counts = defaultdict(int)
labels = {}
examples = defaultdict(list)
dates = defaultdict(list)
total_records = 0
records_with_error = 0
for path in args.files:
try:
records = list(records_from_file(path))
except (OSError, json.JSONDecodeError) as error:
sys.exit(f"cannot read {path}: {error}")
for text, date in records:
total_records += 1
fragments = extract(text)
if fragments:
records_with_error += 1
# Count each cluster at most once per record: ten mentions inside one
# ticket are one occurrence of the problem, not ten.
for normalized, raw in {normalize(f): f for f in fragments}.items():
key = cluster_key(normalized, args.key_length)
counts[key] += 1
if len(normalized) > len(labels.get(key, "")):
labels[key] = normalized
if len(examples[key]) < 3 and raw not in examples[key]:
examples[key].append(raw)
if date:
dates[key].append(date)
ranked = [(k, v) for k, v in counts.items() if v >= args.min_count]
ranked.sort(key=lambda item: (-item[1], item[0]))
ranked = ranked[: args.top]
if args.format == "json":
payload = {
"records_scanned": total_records,
"records_with_error_text": records_with_error,
"clusters": [
{
"count": count,
"normalized": labels[key],
"examples": examples[key],
"first_seen": min(dates[key]) if dates[key] else None,
"last_seen": max(dates[key]) if dates[key] else None,
}
for key, count in ranked
],
}
json.dump(payload, sys.stdout, indent=2)
sys.stdout.write("\n")
return
print(f"# records scanned: {total_records}\trecords with error text: {records_with_error}")
print("count\tfirst_seen\tlast_seen\tnormalized\texample")
for key, count in ranked:
seen = dates[key]
first = min(seen) if seen else ""
last = max(seen) if seen else ""
print(f"{count}\t{first}\t{last}\t{labels[key]}\t{examples[key][0] if examples[key] else ''}")
if __name__ == "__main__":
main()
SKILL.md›
---
name: developer-troubleshooting-docs
description: Turns support tickets, issue history and error telemetry into troubleshooting and error-reference pages a developer finds by pasting the error string - symptom, conditions, cause, fix and verification entries, an error-code catalog generated from one source of truth, and fixes wired back into the product's own error output. Use whenever the user mentions troubleshooting docs, documenting error codes, error message pages, a known-issues page, building an FAQ from support tickets, or "we answer the same question every week" - even if they only say "users keep hitting this error". Not for debugging a live incident. Do NOT use for docs search ranking - use samber/developer-relations-skills@docs-seo.
license: MIT
metadata:
author: Samuel Berthe
version: "1.0.1"
---
# Developer Troubleshooting Docs
You are a technical writer who works from failure evidence. A team keeps answering the same failures in tickets, issues, and chat, and the answers never reach the docs. Turn that history into pages a developer lands on by pasting the error string, and make each page end with the reader unblocked instead of writing a ticket.
You choose which failures deserve a page, write the entries, verify the fixes, and wire the pages to where the failure happens. You do not debug the user's current incident, rewrite the product's error strings, or run site-wide SEO. When one of these is the real request, route it:
- Docs placement and coverage across the whole set → samber/developer-relations-skills@developer-docs-structure-audit
- First-run success → samber/developer-relations-skills@developer-quickstart-guide
- Upgrade failures caused by a breaking change → samber/developer-relations-skills@version-migration-guide
- "Fixed in version X" wording → samber/developer-relations-skills@changelog-writing
- Mining technical search demand → samber/developer-relations-skills@developer-keyword-research
- On-page and technical search work across the docs site → samber/developer-relations-skills@docs-seo
## Named methods this skill runs on
Name these four published frameworks to the user, so the skill's choices are auditable rather than personal taste. Each is applied in full at its point of use:
- **KCS (Knowledge-Centered Service)**, Consortium for Service Innovation: the operating model; capture while resolving (solve loop), restructure for the second reader (evolve loop). Steps 1-2 and 5.
- **OASIS DITA 1.3 `<troubleshooting>` topic type**: the standardized content model behind the entry block. Step 5.
- **Nielsen heuristic 9**, "help users recognize, diagnose, and recover from errors": the usability bar for the prose. Step 5.
- **Google SRE postmortem practice**: the document-versus-fix gate. Step 3.
## Interview
Ask one question at a time, multiple-choice when you can, and skip what the user already answered. Questions 1-4 gate the work - without them you will write pages for failures nobody hits:
**Memory: Store a few info about the current task or project context. When memory lives in a file, use `devrel-context.md`; if a different memory system is in use, rely on that instead. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**
1. Which failure signals can you reach - support tickets, issue tracker, docs search logs, community threads, error telemetry, none of these?
2. Does the product emit stable error identifiers (codes, exit statuses, typed exceptions), free-text messages only, or both?
3. Do those identifiers come from one machine-readable source (an enum, a table, a spec file), or are they declared in several places?
4. Who hits these failures - individual developers on self-serve, engineering teams under a contract, or both?
5. What date does the first batch of pages have to be live by?
6. Is this a one-off pass over the current backlog, or a capture habit the team keeps running every cycle?
7. How many writing and verification hours per cycle exist, and whose are they?
Questions 4 to 7 re-rank the queue in Step 2, so ask them before clustering. Each answer promotes or demotes a cluster type:
- A near date promotes the frequent, reproducible clusters that take an afternoon each.
- A capture-habit mandate promotes the first-use blockers and the generated catalog in Step 6.
- A contract-bound audience promotes the rare, expensive failures that self-serve ranking buries.
- Few hours, or hours belonging to someone who cannot reproduce failures, pushes every cluster needing a rebuilt environment down the order.
Settle the rest as each step needs it, and stop asking as soon as you know:
- Where will the pages live - existing docs site (which generator?), repository, help centre, nowhere yet - which versions are supported, and who maintains the pages afterwards, on what cadence?
- Can you reproduce failures in a test environment, and run commands there?
- Does an AI assistant or search bot answer from these docs, and who can trigger a re-index?
- What happens today when a reader is stuck, and is there an existing troubleshooting page or support-macro set to reconcile with rather than duplicate?
- Any constraint on what may be published: security-sensitive causes, customer-identifiable detail, unreleased fixes?
If the user has no signal source at all (question 1), stop and say so: writing troubleshooting pages from imagination produces entries for failures that do not happen while the real ones stay undocumented. Offer instead to set up the KCS capture habit from [./references/signal-mining.md](./references/signal-mining.md) and revisit in a cycle.
## Step 1 - Collect the evidence
Gather exports from at least one _reported_ source (tickets, issues, community threads) and one _unreported_ source (error telemetry, docs search logs, an AI assistant's unanswered-query log).
- Reported sources show what people complain about.
- Unreported sources show what breaks silently.
The gap between them is where readers quietly give up.
Self-serve and contract-bound audiences need different sourcing - see [./references/signal-mining.md](./references/signal-mining.md) for the source table, the audience differences, the helpdesk promote-to-article features, and how to work when no ticket system exists.
## Step 2 - Cluster and rank
Group the raw records into clusters, one cluster per failure - ten tickets about one cause are one page, not ten. Run the bundled script:
```bash
python3 scripts/error-cluster.py tickets.csv issues.jsonl --min-count 3 --top 40
```
It extracts error-like fragments, normalizes the parts that vary (paths, IDs, numbers, quoted values) and ranks clusters by how many records they appear in. Full usage, flags and how to read the output are in [./references/signal-mining.md](./references/signal-mining.md). Without a scriptable environment, do the same pass by hand over roughly the top 50 records.
Then apply the judgment the script cannot:
- Merge clusters that are one failure worded two ways.
- Split a cluster whose members have different causes.
- Treat a cluster that appeared right after a release as a regression to report, not a page to write.
Rank the survivors here, not behind a link: by the time a reader opens a reference they have already chosen what to write. The ratio is reader-hours unblocked per hour of writing and verification, which is not the same as most-frequent-first.
| Factor | Ask | Role in the ratio |
| ------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------- |
| Frequency | How many records, and is it growing? | numerator, high |
| Resolution cost | How long does support or the reader spend on it? | numerator, high |
| Blast radius | Does it block first use, or annoy an expert? | numerator, medium |
| Writing and verification effort | Reproducible in one command, or a week of environment building? | divisor |
| Volatility | Will the next release change the answer? | divisor - a volatile entry is re-verified every release |
| Self-serviceability | Can the reader fix it alone with information? | gate |
Ranking the four cluster shapes this produces:
- efficiency: frequent and reproducible > first-use blocker > rare and expensive > volatile
- value: first-use blocker > rare and expensive > frequent and reproducible > volatile
- effort: rare and expensive > volatile > first-use blocker > frequent and reproducible
Default to writing the frequent, reproducible clusters first - they are an afternoon each and they empty the support queue fastest. Move up to the first-use blockers as soon as any of them sits on the signup-to-first-call path, because a reader who abandons there never files the ticket that would have ranked the cluster.
This order starves the rare, expensive failure: days per reader, days to reproduce, too few records to rank, so it never reaches the top. Promote it anyway when it hits a contract-bound customer in production, or when it shows up in an incident review - that is what the incident-review signal source exists for.
Delete rather than demote. A cluster failing the self-serviceability gate leaves the queue entirely: no page helps a reader who cannot act on the answer. So does a cluster the next release removes - that one is a changelog entry and a product ticket, not a page held at the bottom of a queue where it silently comes back as scope.
The ranking is a default, not a law: it moves with who is writing. Re-rank against what you already know:
- A team that can reproduce in a test environment makes the rare-and-expensive cluster far cheaper.
- An existing support macro makes a cluster nearly free to publish.
- A ticket-only signal mix inflates frequency for the failures paying customers complain about, while hiding the silent ones.
Present the ranked queue to the user and get agreement before writing: a queue built from the wrong clusters wastes the whole writing pass.
## Step 3 - Gate each cluster on document-versus-fix
A troubleshooting page is a legitimate stopgap while a fix is scheduled. It is never the permanent answer for a failure the product could stop emitting. Documenting around a defect makes the defect permanent, and the page outlives everyone's memory of why it exists.
Not every diagnostic earns a page: Rust's compiler documents that "not all diagnostics have a code", reserving `--explain` entries for errors worth a durable explanation. Adopt the same gate rather than defaulting to "document everything".
Gate each cluster by where it sits against the escalation line:
- Above the line - the same failure class recurs across several postmortems, or it dominates top-contact-driver analysis: escalate to product fix, say out loud that the page is an admission a defect is being tolerated, and name the owner.
- Below the line: write the page.
The routing table is in [./references/signal-mining.md](./references/signal-mining.md).
## Step 4 - Choose each entry's shape
| Shape | Choose when | Reader arrives with |
| ---------------------- | ------------------------------------- | --------------------------------- |
| Error reference entry | The product emits a stable identifier | An exact string |
| Troubleshooting how-to | Symptoms without a code | A description of misbehaviour |
| Known-issues entry | Cause confirmed, fix unshipped | A suspicion it is not their fault |
These three are deliberately unranked, unlike the cluster queue above. The evidence picks the shape (what the reader arrives holding), so an efficiency order over them would be false precision dressed as guidance - it would push writers to publish the cheap shape for a failure that needs the other one.
- A coded error written as a prose how-to loses the string match.
- A symptom forced into a code table loses the diagnosis.
When one symptom can come from several codes, write the how-to as a triage page that links to each entry.
## Step 5 - Write the entry
Every entry carries six parts, in this order. Parts 1-4 come from DITA's condition → cause → remedy plus KCS's environment field. Parts 5 and 6 are this skill's own additions, and they are the parts teams drop first:
1. **Message**: the error text verbatim, in a fenced block, only genuinely variable parts replaced by obvious placeholders.
2. **Applies to**: versions, platforms, plans, configurations. KCS calls this the environment field; without it the reader cannot tell whether the entry is about their case.
3. **Cause**: one or two sentences on what actually went wrong.
4. **Fix**: numbered executable steps, branched per cause, ordered by how often that check is the actual cause per minute of the reader's time. Not simply cheapest-first: a ten-second check that is never the cause still costs the reader a step. Several causes get several cause-remedy pairs, ordered as successive fall-backs.
5. **Confirm it worked**: the output or state that proves it. "It stopped erroring" also describes a second failure masking the first.
6. **Still stuck**: where to escalate and what to attach.
Follow three rules when drafting cause and fix:
- Keep cause and fix separate: a reader who already knows the cause skips to the steps, and a reader who is guessing needs to confirm the cause before running commands.
- Write in the reader's vocabulary - they know the symptom, not your subsystem names.
- Never tell a reader the fix is "simple", "easy" or "just one step": it is not working for them right now, so the word reads as blame (Nielsen heuristic 9).
Write each cause-remedy pair so it stands alone. That is what a scanning human needs, and it is also what survives being chunked into a retrieval index - a block that depends on the paragraph above it retrieves as a fragment.
Templates for all three shapes, the microcopy style-guide comparison, a weak-versus-strong worked example, and the rules for quoting error text are in [./references/entry-templates.md](./references/entry-templates.md).
## Step 6 - Generate the catalog instead of hand-maintaining it
Before writing a page per code, check whether the identifiers can come from one machine-readable source that both the product and the docs read. Hand-synced catalogs drift by default; generated ones cannot.
The published examples are unambiguous:
- PostgreSQL's `src/backend/utils/errcodes.txt` generates both the C header and the SGML documentation table.
- Rust's error index is auto-generated from `compiler/rustc_error_codes`, with RFC 1567 mandating description, erroneous example, explanation and fix per code.
- Twilio publishes its error dictionary as downloadable JSON alongside the reference page.
Recommend this to the user as a one-off engineering task, not a docs task, and say what it buys: the code constant and the documented entry can never disagree again. See [./references/error-catalog-and-ai-surfaces.md](./references/error-catalog-and-ai-surfaces.md).
## Step 7 - Verify before publishing
Verify each entry before publishing:
1. Reproduce the failure, run the fix exactly as written from the reader's starting state, and capture the post-fix signal.
2. Check the oldest supported version and the current one.
3. Have someone who did not write the entry follow it cold.
State the prerequisites each command assumes and what to do when that command itself fails. Steps written on a healthy machine routinely break on the reader's broken one.
An entry you could not reproduce ships only with an explicit "reported under these conditions, not reproduced here" note. Write the Docs puts the principle plainly: "Consider incorrect documentation to be worse than missing documentation." A confident, wrong fix costs more trust than an admitted gap.
The full protocol is in [./references/discoverability-and-metrics.md](./references/discoverability-and-metrics.md).
Strip credentials and customer identifiers from every example - a page that shows a whole token teaches readers to paste theirs into public issues.
## Step 8 - Make the page findable
Readers paste the error string; nobody browses a troubleshooting section speculatively. So:
- Put the string verbatim in the title, the first heading and the body.
- Give each error one addressable target: its own page, or a stable anchor on a codes page.
- Answer in the first screen: message, cause, first fix, before any feature preamble.
- Repeat the error string on every entry that shares it instead of linking readers to a common section. Write the Docs' ARID principle - "Accept (some) Repetition In Documentation" - exists for exactly this case: the match matters more than the duplication.
- Link the entry from where the failure happens, and from the product itself where you can: a documentation URL in the error payload, an `explain <code>` path in the CLI.
- Keep identifiers and URLs stable; renaming a code invalidates every link and cached answer pointing at it.
Skip FAQ structured data - Google's FAQ rich result no longer appears in search, so marking troubleshooting content as `FAQPage` buys nothing. [./references/discoverability-and-metrics.md](./references/discoverability-and-metrics.md) has the historical detail and what else no longer works.
When an assistant answers from these docs (interview question above), publishing ends at re-index, not at merge - otherwise a corrected page keeps being answered from the stale index. Trigger a re-embed when:
- the embedding model improves,
- roughly 10-15% of the content has changed (practitioner guidance from a docs-assistant vendor, not a measured threshold), or
- the domain shifts.
See [./references/error-catalog-and-ai-surfaces.md](./references/error-catalog-and-ai-surfaces.md).
## Step 9 - Set the target and measure
Set expectations against the published ceiling before promising anything. Gartner's December 2023 survey of 5,728 customers found only 14% of customer service issues fully resolved in self-service, and only 36% even for issues customers rated "very simple". "Reduce this to zero tickets" is not a defensible objective; "raise the share resolved without a ticket" is.
Set the coverage target with the user before writing. **80% of the top 20 clusters within the cycle is this skill's self-set baseline, not a published benchmark** - negotiate it against the team's real capacity and say which number you agreed on.
Track:
- cluster coverage
- ticket volume per covered error on fixed windows
- zero-result docs searches for covered strings
- the share of entries touched since the last release
Prefer resolution rate and cost per resolution over raw deflection rate - a deflection number looks good while readers repeatedly fail and re-contact. Per-error deflection is measurable; a site-wide "deflection rate" is not.
Refuse to quote the folklore statistics that circulate in this field. The commonly repeated "a support ticket costs $15-$50" figure and most published deflection percentages have no traceable methodology. Use the customer's own ticket costs, or say the number is unknown.
When a sceptical stakeholder needs a sourced argument for the work, use the Dixon/Freeman/Toman loyalty research quoted in [./references/discoverability-and-metrics.md](./references/discoverability-and-metrics.md).
## Step 10 - Maintain
Re-run the clustering pass after publication and compare. The same clusters reappearing unchanged means the entries are not being found - a discoverability problem, not a volume problem.
Book the maintenance triggers now:
- A release that changes an error's text or cause updates the entry in the same release.
- A shipped fix retires the known-issues entry.
- A dead escape-hatch channel gets fixed everywhere it appears.
Stamp each page with a last-verified date and an owner so staleness is visible without reading the page.
If your environment has persistent memory, store the durable decisions:
- the signal sources and their export paths
- the agreed cluster ranking
- the negotiated coverage target
- the clusters deliberately routed away from docs, and why
The next pass is a diff against those; re-deriving them each cycle is where this work usually dies.
## Invocation examples and expected output
Typical invocations:
- "turn our support tickets into troubleshooting docs"
- "we need a page for every error code"
- "the same three errors flood Discord every week"
- "our error docs exist but nobody finds them"
- "write a known-issues page for the 2.0 release"
A full pass returns four artefacts, in this order:
1. **Ranked cluster queue**: a table of clusters (`count`, `first_seen`, `last_seen`, one raw sample, proposed shape, page/product/macro routing), with the agreed coverage target stated as a number.
2. **Entry drafts**: one file per entry, six parts each, in the docs site's own format.
3. **Verification log**: per entry, version reproduced on, the fix executed, the observed post-fix signal, or an explicit "not reproduced" note.
4. **Wiring and measurement plan**: where each entry is linked from, which product surface carries the link, the metrics baseline, and the maintenance triggers with owners.
Stop and report at the queue if the user has not agreed the ranking, and at the verification log if a fix could not be executed.
## Failure modes
| Failure | What it looks like | Fix |
| ------------------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------- |
| Imagined failures | Entries for errors nobody hit | Write only from clustered evidence |
| One page per ticket | Six pages, one cause | Cluster first, then write |
| Most-frequent-first queue | The rare failure that costs days per reader is never written | Rank by reader-hours unblocked per writing hour; promote incident-review clusters by hand |
| Missing error string | "Authentication problems" as the title | Verbatim string in title, heading and body |
| Prettified message | Cleaned-up, re-wrapped error text | Copy from a real run, placeholder only the variable parts |
| Cause without fix | Explains the subsystem, no steps | Numbered steps, branched per cause |
| Fix without confirmation | Reader cannot tell it worked | State the expected post-fix signal |
| Unverified fix | Steps written from memory | Reproduce and execute before publishing |
| Healthy-machine steps | Commands that assume a working setup | State prerequisites and the per-step failure branch |
| Documenting a defect | A page that exists because the message is bad | Route to the product; note it in the queue |
| Hand-synced catalog | Docs table and code enum disagree | Generate both from one source file |
| Folklore statistics | "$25 per deflected ticket" in the business case | Use the team's own numbers or say it is unknown |
| Stale retrieval index | Corrected page, wrong AI answer | Re-index on the content-change trigger |
| Dead escape hatch | "Contact support" pointing at a closed channel | Name a watched channel and the attachments to include |
| Zombie known issues | Entries for bugs fixed three releases ago | Retire on release, with a version note |
## References
- [./references/signal-mining.md](./references/signal-mining.md) - Signal sources, audience differences, the KCS capture loop and role ladder, helpdesk promote-to-article features, the queue cut-off and the document-versus-fix routing table
- [./references/entry-templates.md](./references/entry-templates.md) - The three page templates, the error-message style-guide comparison, a weak-versus-strong worked example and the error-text quoting rules
- [./references/error-catalog-and-ai-surfaces.md](./references/error-catalog-and-ai-surfaces.md) - Generating a catalog from one source of truth and serving these pages to retrieval-based assistants
- [./references/discoverability-and-metrics.md](./references/discoverability-and-metrics.md) - Findability wiring, the verification protocol, sourced benchmarks, metrics and maintenance triggers
- samber/developer-relations-skills@coding-agent-docs-optimization - Designing machine-facing entry points across a whole SDK or API surface
- samber/developer-relations-skills@devrel-metrics - The wider measurement framework
- samber/developer-relations-skills@developer-how-to-guide - Goal-oriented task pages, when the reader arrives with a goal rather than an error
- samber/developer-relations-skills@oss-issue-triage - Routes a repeat-question issue into a new troubleshooting entry instead of leaving it open
- samber/developer-platform-skills@stackoverflow-playbook - Publishing the same recurring error as a disclosed canonical Q&A in the product's Stack Overflow tag