samber/developer-relations-skillsチェック済み
SKILL DETAIL
devrel-metrics
samber/developer-relations-skills/devrel-metrics
Builds a developer relations measurement framework - a handful of metrics tiered from reach to product and business impact, each with a written attribution rule, a baseline-derived target, an owner and an action, and vanity metrics cut. Use whenever someone asks how to measure DevRel, which devrel KPIs matter, how to prove devrel value or ROI to an exec, what belongs in a devrel scorecard or quarterly report, why their numbers look good but leadership stays unconvinced, or reports only stars, impressions and event counts - even if they never say "metrics". Do NOT use for tracking instrumentation and dashboards - use samber/developer-relations-skills@devrel-analytics. Not community-only health metrics, not a single event's ROI.
インストール · 426出典を見る
Installation
npx skills add https://github.com/samber/developer-relations-skills --skill devrel-metrics
スキルファイル
SKILL.md
最終同期 · 2026/10/23
evals/evals.json›
{
"skill_name": "devrel-metrics",
"evals": [
{
"id": 1,
"prompt": "I run developer relations at Meridian Stack, an infrastructure company with about 40 engineers. Every month I send leadership the same five numbers: total GitHub stars (11,400), blog impressions (roughly 90k), Twitter followers (8,900), cumulative npm downloads (2.1M) and conferences attended (9 so far this year). Our VP of Engineering replied to the last one asking 'what is any of this actually for?' and I didn't have a good answer. Can you put together something better for me to send next month?",
"expected_output": "A measurement framework with five to eight metrics, each carrying definition, source, baseline, threshold, owner and action, a written attribution rule, a blind-spots section and a kill rule. The four cumulative totals are rejected as monotonic and either converted to period deltas or cut; the conference count is named as an activity log rather than an outcome.",
"files": [],
"expectations": [
"Identifies total GitHub stars (11,400) as a monotonic metric that cannot fall, and refuses it as a primary metric",
"Applies the same monotonic objection to cumulative npm downloads (2.1M) and to the Twitter follower count (8,900)",
"Names the conference-attendance count (9) as a record of effort rather than an outcome",
"The resulting metric sheet holds between five and eight metrics",
"At least one metric in the sheet sits in the product-impact or business-impact tier",
"Every metric row carries all six fields: definition, source, baseline, threshold, owner and action",
"States that npm download counts are HTTP request counts including CI and mirror traffic, not people",
"Produces a written attribution rule as a section distinct from the metric sheet",
"Includes a non-empty known-blind-spots section",
"States a kill rule removing any metric that has not changed a decision in two consecutive review cycles",
"Does not deliver tracking instrumentation, an event taxonomy or analytics code as the answer",
"Labels each threshold with where it came from, using sourced, baseline or hypothesis",
"Does not compare the 11,400 stars against another company's star count as a benchmark"
]
},
{
"id": 2,
"prompt": "Calderon Labs here, Series B, we sell a data-pipeline product. I have to pick what to instrument first for our developer program and I want to get the order right rather than boiling the ocean. Useful context: we already run Amplitude across the product and it's clean, we have roughly two engineering days I can borrow this quarter, and our sales ops lead has flatly refused to open CRM fields to us or to anyone outside the revenue team. Where do I start and in what order?",
"expected_output": "A build order derived from value per unit of effort rather than from cost, putting enablement first and reach last, with influenced pipeline deleted outright because sales ops will not cooperate and recorded in the blind-spots section with the cost of closing it. The existing product-analytics install is named as the answer that promotes enablement and product impact.",
"files": [],
"expectations": [
"States the build order is decided by value per unit of effort, not by cost of collection",
"Places enablement first as the best value-per-effort ratio on the ladder",
"Places reach last despite it being the cheapest tier to collect",
"Gives the efficiency ordering as enablement, then engagement, then product impact, then business impact, then reach",
"Deletes influenced pipeline outright because sales ops will not share data, rather than demoting it or ranking it last",
"Records the deleted business-impact metric in the blind-spots section together with what it would take to obtain it",
"Promotes enablement and product impact because product analytics is already instrumented",
"Names which answer from the user moved which tier",
"States that value and collection effort both rise down the tier ladder, so the ladder alone does not answer what to build first",
"Sets the default primary metric at the enablement tier",
"Does not recommend starting with reach metrics on the grounds that they are already available",
"Does not present the five-tier ladder to the user as an industry standard model"
]
},
{
"id": 3,
"prompt": "Growth engineer at Thistledown, we make a TypeScript SDK for payments reconciliation. Our head of DevRel wants to know which channels bring developers in, and my plan was to put UTM parameters on absolutely everything - every talk slide QR code, every repo link, every newsletter - and then report channel totals from that. I have about three hours a week for this. Does that plan hold up, and what would you do first?",
"expected_output": "A recommendation to stand up self-reported attribution first as the standing instrument, with tagged links restricted to comparing artifacts of the same type and never used for program totals, plus the efficiency ordering of the four methods, the stated bias of each, and the instruction to write the rule before the period starts.",
"files": [],
"expectations": [
"Recommends self-reported attribution, a single question at signup, as the first method to stand up",
"Gives the efficiency ordering of methods as self-reported, then pre/post lift, then tagged links, then influenced pipeline",
"Restricts tagged links to comparing artifacts of the same type and forbids using them for program totals",
"States that self-reported attribution is the only method that sees hallway conversations, podcasts and colleague recommendations",
"Names the bias of self-reported attribution: over-counts memorable touches, under-counts docs and search",
"Specifies exactly one question, with an option list held stable across quarters",
"Keeps a free-text option in the question so unlisted channels surface",
"Reports the result as self-reported first touch rather than as attribution",
"States that a large direct bucket is normal rather than a tracking defect",
"Cites roughly 70% of AI-assistant traffic arriving with no referrer, and marks the figure as directional rather than measured",
"States the attribution rule must be written before the period starts, not after it",
"Notes that the marginal effort of pre/post lift is recording the pre-period before the push, which is impossible afterwards"
]
},
{
"id": 4,
"prompt": "Hollowbrook, we sell an observability platform, mostly six-figure enterprise deals with an 8-month average cycle. Our CRO stood up at the QBR and said DevRel sourced $1.2M last quarter based on a spreadsheet someone put together after the fact by looking at which closed accounts had been to one of our workshops. He now wants that number every quarter and he wants me to own it. I'd rather have something I can actually defend. How do I write this down?",
"expected_output": "A written influenced-pipeline rule naming touch types, a lookback window and a stage condition, fixed for the fiscal year, with sourced credit refused throughout, shared credit stated, retroactively recorded touches excluded, the leading indicator for the enterprise latency named, and the revenue-metric decision made deliberately against who owns the budget.",
"files": [],
"expectations": [
"Refuses the word sourced and uses influenced throughout",
"States that Marketing and Sales may claim the same opportunities and that shared credit is inherent to the metric",
"The rule names which touch types count as a program touch",
"The rule names an explicit lookback window in days",
"The rule names whether the touch must precede a specific pipeline stage",
"States the rule is fixed for the fiscal year and that any mid-period change must be reported as a restatement",
"Excludes any touch recorded after the opportunity was created",
"States the metric over-counts by construction because engaged accounts were already more likely to buy, and that this is stated once rather than re-argued each quarter",
"Names a leading indicator to carry the interim, given enterprise cycles push the pipeline answer two to four quarters out",
"Flags influenced pipeline as carrying the highest compliance cost because it joins a named person's community activity to a CRM record",
"Presents both defensible positions on the revenue question: no sales metrics on DevRel with the DevRel Qualified Lead as the alternative currency, and influenced pipeline explicitly shared",
"States the deciding input for that choice is who owns the budget"
]
},
{
"id": 5,
"prompt": "I maintain Quillmark, an Apache-2.0 document toolchain with 4,200 stars and about 30 recurring contributors. There is no company and nothing to sell. Our steering committee wants a health report every quarter and someone on the list told us CHAOSS says first response within 24 hours is the standard we should hold ourselves to. What should we actually be measuring, and is that 24-hour figure right?",
"expected_output": "The CHAOSS Starter Project Health set adopted by definition, with the correction that CHAOSS publishes no numeric target for any of its four metrics, the two-business-day figure attributed to VMware rather than CHAOSS, the 24-hour figure labelled folklore, thresholds derived from the project's own baseline, and stars cut in favour of downloads, dependents and unique cloners.",
"files": [],
"expectations": [
"Anchors the framework on the CHAOSS Starter Project Health metric set rather than inventing a metric list",
"Names all four CHAOSS starter metrics: time to first response, change-request closure ratio, contributor absence factor, release frequency",
"Corrects the user by stating CHAOSS publishes no numeric target for any of the four metrics",
"Identifies the two-business-day first-response figure as VMware's internal guideline quoted inside a case study, not a CHAOSS standard",
"Labels the 24-hour first-response figure as folklore or a hypothesis under test rather than a standard",
"Defines time to first response as the first human response, with bots excluded",
"Defines change-request closure ratio as closed, meaning merged plus rejected, divided by opened in the same window",
"Defines contributor absence factor as the smallest number of people producing 50% of contributions",
"Adopts the CHAOSS definitions verbatim while deriving every threshold from the project's own trailing baseline",
"Refuses to compare the project against other projects' published numbers",
"Recommends downloads, dependents or unique cloners over the 4,200 stars as the adoption signal",
"Notes that bot and automation activity distorts time to first response and change-request closure ratio",
"States that GitHub's repository traffic endpoints retain only 14 days of data and require write access, so snapshotting must start the day the metric enters the sheet",
"Cuts the star count rather than keeping it as a primary metric, on the grounds that stars are gameable and not comparable across project types"
]
},
{
"id": 6,
"prompt": "Head of DevRel at Vantablue, we make a CI orchestration tool. I have four advocates and I want everyone clear on what good looks like, so I drafted individual quarterly targets: 6 blog posts per month each, 40 community messages per week each, 20 GitHub issues closed per month each, and 500 signups per quarter attributed to each person. Comp is going to be partly tied to these. Can you sanity-check the set and tell me what I'm missing?",
"expected_output": "Each incentivised volume target paired with a named counter-metric, the specific degradations each target produces spelled out, per-person targets rejected in favour of measuring the program, and the three anti-vanity tests applied with failing metrics cut rather than softened.",
"files": [],
"expectations": [
"Pairs every incentivised volume metric with a counter-metric",
"Pairs the blog-post count with organic entries per post",
"Pairs closed issues with reopen rate",
"Pairs signups with 30-day retention",
"Cites the formulation that when a measure becomes a target it ceases to be a good measure, attributed to Marilyn Strathern's 1997 generalisation of Goodhart's law",
"Names the specific degradations the targets produce: thin posts, premature issue closes, low-value chatter and throwaway accounts",
"Rejects per-person metric targets and measures the program rather than the individual",
"States that per-person targets turn colleagues into competitors and pull work toward whatever is counted",
"Applies the so-what test by naming the action each surviving metric triggers, and cutting any metric with no action",
"Applies the ownership test, asking whether this team's work can plausibly move the number within one review period",
"Applies the monotonic test to the candidate metrics",
"Caps the resulting sheet at no more than eight metrics",
"Cuts a metric that fails any of the three tests rather than softening or rewording it"
]
},
{
"id": 7,
"prompt": "Ferrowind, we sell a Rust-based edge runtime. Last month we had 11 enterprise conversations at a conference versus 4 the month before, which I've been writing up as 175% growth, and external pull requests went from 2 to 6. Last month was also the month we shipped 2.0 and did the Show HN. Separately, a competitor's engineering blog says they take 300 external PRs a year, and my CEO has started quoting that at me. Help me put together the monthly report.",
"expected_output": "The small-N counts reported as raw dated lists rather than percentages, the conference-plus-launch month identified as a burst and refused as a trend, the competitor's absolute figure refused as a benchmark in favour of the program's own history, and thresholds deferred until three prior periods exist.",
"files": [],
"expectations": [
"Refuses to express the 11 versus 4 enterprise conversations as a percentage or rate, and requires the raw list with dates instead",
"Applies the same treatment to the 6 versus 2 external pull requests",
"States the small-N floor at roughly 30 events in a period, below which a rate should not be reported",
"Labels that 30-event floor as a borrowed rule of thumb rather than a DevRel-measured cutoff",
"Identifies the conference plus the 2.0 launch and Show HN as a burst rather than a trend",
"Refuses to present the launch month against the prior quiet month as a trend line",
"Refuses to benchmark the program against the competitor's 300 external pull requests as an absolute number",
"States the correct comparison is the program against its own past",
"Names why cross-company absolute comparison fails: differences in audience size, ecosystem and product maturity",
"Requires at least three prior periods of history before any threshold is set",
"Sets targets as a direction and a range for the first two cycles rather than a point value",
"Does not report or endorse the 175% growth figure"
]
},
{
"id": 8,
"prompt": "Nordholm, we ship client libraries for a geospatial API in three languages. Two numbers I put on the board deck every quarter: 240,000 docs sessions from Google Analytics, and total monthly package downloads of 305,000, which is npm at 180k plus PyPI at 95k plus crates.io at 30k. I describe the second one as monthly users. Our new CTO poked at both last week. Are they defensible, and if not what do I replace them with?",
"expected_output": "Both numbers rejected: registry downloads are request counts that cannot be summed across registries or called users, and the Google Analytics session count is a large undercount of a developer audience. Replacements are step-to-step ratios measured inside a single source, with each definition written beside its number.",
"files": [],
"expectations": [
"Rejects calling 305,000 downloads users, and states registry downloads are request counts rather than people",
"States npm counts every HTTP 200 tarball response, including continuous integration and mirror traffic",
"States PyPI excludes known mirrors by default, so the two registries are not measured on the same basis",
"Refuses to sum downloads across registries and present the total as users",
"Flags the 240,000 Google Analytics docs sessions as a substantial undercount for a developer audience",
"Cites the Plausible measurement of 58% of a tech audience blocking Google Analytics, 82.3% on Linux and 88.3% on Firefox",
"Qualifies that measurement with its scope: one site, August 2021, traffic skewed toward Hacker News and Reddit",
"Requires reach and engagement metrics be built as ratios inside one source rather than as absolutes compared across sources",
"Does not retain an absolute session count as a headline metric",
"Notes npm treats under roughly 50 downloads per day as mostly automation",
"Requires the definition of each number to be written next to the number",
"Requires any surviving reach metric to have a named downstream partner in the same row's story or be dropped"
]
},
{
"id": 9,
"prompt": "Saltmarsh Data, 3-person developer relations team - one on docs, one advocate, one running the community. Our charter names two funded goals for the year. I want the framework built this week. I've been reading up and I'd like us to track the AAARRRP stages and also map where developers are dropping off across the journey, so we get both views. Give me the framework.",
"expected_output": "An explicit brainstorming pass offering two or three whole spines with their sources, what each makes visible and hides, who it is legible to and when it is the wrong pick, a recommendation with reasons, a refusal to stack AAARRRP with journey stages, and no metric sheet written until the owner picks a spine.",
"files": [],
"expectations": [
"States explicitly that it is in brainstorming mode before offering anything",
"Offers two or three whole spines rather than a single framework",
"Names driver-anchored AAARRRP, journey-stage, and the awareness/enablement/engagement function split among the options",
"Attributes AAARRRP to Phil Leggetter in 2016, adapted from Dave McClure's AARRR with a seventh element for developer products",
"Attributes the awareness, enablement and engagement split to Mary Thengvall",
"Attributes the developer-journey stage model to Matthew Revell",
"Presents each spine with what it makes visible, what it hides, who it is legible to, and the conditions under which it is the wrong pick",
"Recommends one spine and states why",
"Refuses to run AAARRRP and journey stages together, naming double-counting of the same work as the reason",
"Does not rank the spines by an efficiency or value-per-effort ratio",
"States the deciding criterion is the spine the funder can already read",
"Ranks the tiers underneath the chosen spine instead of ranking the spines",
"Does not write the metric sheet before the owner has picked a spine",
"Presents the framework one section at a time and seeks agreement before continuing"
]
},
{
"id": 10,
"prompt": "Brackenridge, we launched our developer platform three months ago so there's very little history. Our PM has drafted the OKRs for next quarter and wants to lock them tomorrow: 40% seven-day retention because that's the industry standard, a 20% monthly-active ratio, 30% self-serve activation which she says is typical for devtools, and first response within 24 hours on community questions. She wants me to sign off. Should I?",
"expected_output": "Each of the four figures identified as folklore or an unsourced practitioner claim and refused as a target, every threshold tagged with its provenance in the same cell, metrics marked baselining given three months of history, and targets expressed as a direction and a range with a baseline-relative threshold preferred over a fixed line.",
"files": [],
"expectations": [
"Labels the 40% seven-day retention figure as folklore or a hypothesis rather than a benchmark",
"Labels the 20% monthly-active ratio as folklore",
"Labels the 20-40% self-serve activation band as a practitioner claim rather than a study, covering the PM's 30% figure",
"Labels the 24-hour first-response figure as folklore",
"Refuses to set any of the four as a target",
"Requires every threshold to be tagged sourced, baseline or hypothesis in the same cell as the number",
"States that almost every threshold should be self-set from the program's own trailing baseline",
"Requires at least three prior periods before any threshold is set",
"Marks metrics lacking history as baselining and defers their thresholds to the next review",
"States that an exec who discovers one invented benchmark discounts every other figure on the page",
"Sets targets as a direction and a range for the first two cycles rather than a point value",
"Prefers a threshold that moves with the baseline, such as two standard deviations from a 30-day rolling mean, over a fixed line",
"Labels the three-period, two-standard-deviation and 30-day-window figures as borrowed conventions rather than DevRel-measured values",
"Does not present any of the four circulating numbers as sourced or published"
]
},
{
"id": 11,
"prompt": "Kestrel Logic, we make an embedded database. Four things landed on me this week and I need help across all of them. Our Discord has 2,100 members and feels dead, and I need to know if it actually is. Our UTM parameters are a mess and the GitHub traffic panel disagrees with our analytics tool on every number. We spent 40k on a DevRelCon booth in March and finance wants to know what it returned. And underneath all of that, our CTO has asked what the developer program should be reporting to her every quarter. Where do we start?",
"expected_output": "The three off-scope asks routed to the named sibling skills using owner/repo@skill references, only the program-wide quarterly reporting question kept in scope, no tracking plan or community dashboard produced, metric selection stated as preceding instrumentation, and pre-conditions named per metric with any failing metric handed off as an instrumentation request.",
"files": [],
"expectations": [
"Routes the Discord health question to samber/developer-relations-skills@developer-community-health",
"Routes the UTM and tool-disagreement question to samber/developer-relations-skills@devrel-analytics",
"Routes the 40k booth return question to samber/developer-relations-skills@developer-event-sponsorship",
"Keeps only the program-wide quarterly reporting question in scope for itself",
"Does not produce an event taxonomy, UTM convention or tracking plan as its deliverable",
"Does not produce a community-only health dashboard as its deliverable",
"States that choosing which metrics to report comes before instrumenting them",
"Hands any metric the team cannot instrument to samber/developer-relations-skills@devrel-analytics as a gap rather than assuming it away",
"Uses the owner/repo@skill reference format when naming each routed skill",
"States the pre-conditions each metric needs before it can enter the sheet",
"Calls a metric that fails its pre-conditions an instrumentation request rather than a metric",
"Distinguishes deciding which metrics deserve a target from making them collectable"
]
},
{
"id": 12,
"prompt": "Ashcombe, developer tooling company. I defend our budget in five weeks in front of the CMO, who signs off on our headcount - we sit on the marketing line. I have zero engineering days available between now and then; the platform team is frozen on a migration. We have never recorded a pre-period for anything. I want to put cohort retention of developers we've touched on the slide, going back over the last three quarters, and I can tag the cohort retroactively from our event attendee lists. What should the slide say?",
"expected_output": "The retroactive cohort refused because cohorts must be defined before the period, the five-week deadline used to promote engagement and enablement and demote anything needing a pre-period, every metric requiring new instrumentation deleted given zero engineering days and handed to devrel-analytics, business impact promoted ahead of its efficiency ratio because the reader renews the budget and the program sits on the marketing line, and no sourced revenue promised.",
"files": [],
"expectations": [
"Refuses to build cohort retention from a retroactively tagged cohort",
"States that a cohort must be defined before the period starts, not after",
"Promotes engagement and enablement because the deadline sits inside one quarter",
"Demotes or excludes anything requiring a pre-period that does not already exist",
"States that a metric taking two quarters to instrument does nothing for a budget review five weeks away",
"Deletes every metric that needs a newly instrumented event, given zero engineering days available",
"Hands the resulting instrumentation gap to samber/developer-relations-skills@devrel-analytics",
"Names which of the user's answers moved which tier",
"Promotes business impact ahead of its efficiency ratio because the reader decides budget renewal and the program is funded from the marketing line",
"Asks whether the user needs a one-off win at that date or a sheet that still reads in a year",
"Specifies cohorts by month rather than by day",
"Does not promise sourced revenue or last-touch revenue attribution on the slide",
"Names a leading indicator to carry the interim for any lagging business-impact metric"
]
}
],
"trigger_queries": [
{ "query": "How should we measure our developer relations program?", "should_trigger": true },
{ "query": "which devrel KPIs actually matter", "should_trigger": true },
{ "query": "we report stars and impressions every month and our VP just asked what any of it is for", "should_trigger": true },
{ "query": "build me a devrel scorecard", "should_trigger": true },
{ "query": "I have to defend the DevRel budget in six weeks, what goes on the slide", "should_trigger": true },
{ "query": "how do I prove devrel ROI to an exec", "should_trigger": true },
{ "query": "what belongs in our quarterly devrel report", "should_trigger": true },
{ "query": "our numbers look great but leadership still isn't convinced", "should_trigger": true },
{ "query": "we're an open source project with nothing to sell, what do we measure instead of stars", "should_trigger": true },
{ "query": "here's our dashboard of thirty metrics, which ones should survive", "should_trigger": true },
{ "query": "our team is about to get handed a pipeline target, is there a defensible alternative", "should_trigger": true },
{ "query": "how do I attribute signups to developer advocacy work", "should_trigger": true },
{ "query": "what's a reasonable target for first-success rate", "should_trigger": true },
{ "query": "we track total github stars and cumulative downloads, is that fine", "should_trigger": true },
{ "query": "help me pick six numbers to report on developer adoption every quarter", "should_trigger": true },
{ "query": "how do we show that our docs work is worth funding", "should_trigger": true },
{ "query": "write the attribution rule for our developer marketing touches", "should_trigger": true },
{ "query": "can devrel claim sourced revenue", "should_trigger": true },
{ "query": "what's the difference between influenced pipeline and sourced pipeline for our advocacy team", "should_trigger": true },
{ "query": "my CEO wants one number for developer relations", "should_trigger": true },
{ "query": "which metrics should our developer advocate own", "should_trigger": true },
{ "query": "set thresholds for our developer adoption numbers", "should_trigger": true },
{ "query": "I need a measurement framework covering our community and content work", "should_trigger": true },
{ "query": "what should we stop reporting", "should_trigger": true },
{ "query": "we have no baseline for any of this, what do we do", "should_trigger": true },
{ "query": "how often should we review our devrel numbers", "should_trigger": true },
{ "query": "is 24 hour first response actually an industry standard", "should_trigger": true },
{ "query": "are there published benchmarks for developer activation rates", "should_trigger": true },
{ "query": "should we use AAARRRP or the developer journey to organize our metrics", "should_trigger": true },
{ "query": "our npm and pypi downloads add up to 300k, can I call that 300k users", "should_trigger": true },
{ "query": "google analytics says 200k docs sessions, can I put that on the board slide", "should_trigger": true },
{ "query": "how do we measure devrel when most of our touches happen at conferences", "should_trigger": true },
{ "query": "what counter metric goes with a blog post quota", "should_trigger": true },
{ "query": "we set a target of forty posts a quarter and the posts got worse", "should_trigger": true },
{ "query": "how do I report on nine enterprise conversations without it looking like noise", "should_trigger": true },
{ "query": "our conference month spiked everything, how do I report that honestly", "should_trigger": true },
{ "query": "which of these should be our primary metric", "should_trigger": true },
{ "query": "CHAOSS starter project health metrics for our repo", "should_trigger": true },
{ "query": "what do we measure for a repo with no product attached to it", "should_trigger": true },
{ "query": "our VP of engineering funds this team, what number does she care about", "should_trigger": true },
{ "query": "I want a scorecard that still makes sense in a year", "should_trigger": true },
{ "query": "can I give each advocate their own metric target", "should_trigger": true },
{ "query": "how do I write down what our numbers are allowed to claim", "should_trigger": true },
{ "query": "what should the developer experience team report monthly", "should_trigger": true },
{ "query": "our devrel report is just a list of talks and blog posts", "should_trigger": true },
{ "query": "leadership thinks devrel is a cost center, what evidence do I bring", "should_trigger": true },
{ "query": "pick the KPIs for our developer program", "should_trigger": true },
{ "query": "should devrel carry a revenue number", "should_trigger": true },
{ "query": "what's a DevRel Qualified Lead and should we use them", "should_trigger": true },
{ "query": "how do I know if a metric is worth keeping", "should_trigger": true },
{ "query": "we report to the board next month and I have no idea what to show them", "should_trigger": true },
{ "query": "every metric we have only goes up", "should_trigger": true },
{ "query": "what does good look like for developer adoption numbers", "should_trigger": true },
{ "query": "we want to measure how well developers succeed with our API", "should_trigger": true },
{ "query": "should time to first success be one of the numbers we report", "should_trigger": true },
{ "query": "which tier of metrics should we build first", "should_trigger": true },
{ "query": "we have no engineering time, what can we still measure", "should_trigger": true },
{ "query": "our sales ops won't give us CRM access, what do we do about pipeline metrics", "should_trigger": true },
{ "query": "help me choose between reporting reach and reporting activation", "should_trigger": true },
{ "query": "how do I tell the difference between a real trend and a launch spike", "should_trigger": true },
{ "query": "another company published their devrel dashboard, can we just copy it", "should_trigger": true },
{ "query": "what goes in the blind spots section of a measurement plan", "should_trigger": true },
{ "query": "the direct traffic bucket is huge and I can't explain it to my boss", "should_trigger": true },
{ "query": "can developer advocacy be measured at all", "should_trigger": true },
{ "query": "our quarterly business review keeps going badly for the devrel team", "should_trigger": true },
{ "query": "give me metrics for our developer program that survive a budget cut", "should_trigger": true },
{ "query": "how many metrics should a devrel team report", "should_trigger": true },
{ "query": "what should replace the vanity metrics on our developer dashboard", "should_trigger": true },
{ "query": "our metrics never change any decisions", "should_trigger": true },
{ "query": "set up an event taxonomy for our docs site", "should_trigger": false },
{ "query": "our github traffic numbers don't match google analytics, help me reconcile the tracking", "should_trigger": false },
{ "query": "build a utm naming convention for our devrel campaigns", "should_trigger": false },
{ "query": "how do I join docs traffic to signups in our warehouse", "should_trigger": false },
{ "query": "instrument the developer funnel end to end", "should_trigger": false },
{ "query": "write the tracking plan for our new docs site", "should_trigger": false },
{ "query": "what event should we fire when someone copies a code sample", "should_trigger": false },
{ "query": "our analytics tools all disagree on session counts, how do I wire up an identity spine", "should_trigger": false },
{ "query": "is our discord dying", "should_trigger": false },
{ "query": "measure our community's responsiveness and contributor funnel", "should_trigger": false },
{ "query": "build a community health dashboard for our slack", "should_trigger": false },
{ "query": "our forum feels quiet, how do I check whether that's real", "should_trigger": false },
{ "query": "design a contributor funnel report for the community team", "should_trigger": false },
{ "query": "was the DevRelCon booth worth the 40k we spent", "should_trigger": false },
{ "query": "should we sponsor this conference again next year", "should_trigger": false },
{ "query": "calculate the return on one conference sponsorship", "should_trigger": false },
{ "query": "we launch on hacker news thursday, what do we do during the launch window", "should_trigger": false },
{ "query": "plan our open source project launch", "should_trigger": false },
{ "query": "rewrite our quickstart so developers reach their first api call faster", "should_trigger": false },
{ "query": "our quickstart has fourteen steps, help me cut it down", "should_trigger": false },
{ "query": "should we start doing devrel at all", "should_trigger": false },
{ "query": "which pillar deserves our next investment, content or community", "should_trigger": false },
{ "query": "write the refused list for our devrel program", "should_trigger": false },
{ "query": "what should a brand new devrel program do first", "should_trigger": false },
{ "query": "map the developer journey for our enterprise segment", "should_trigger": false },
{ "query": "where do developers drop off between trial and production", "should_trigger": false },
{ "query": "who owns each stage of our developer experience", "should_trigger": false },
{ "query": "how much should we spend on events versus content next year", "should_trigger": false },
{ "query": "what do we cut first if the devrel budget shrinks thirty percent", "should_trigger": false },
{ "query": "split our 400k devrel budget across pillars", "should_trigger": false },
{ "query": "should devrel report to marketing or to product", "should_trigger": false },
{ "query": "how do we structure a five person devrel team", "should_trigger": false },
{ "query": "should we embed advocates inside product teams", "should_trigger": false },
{ "query": "how does our developer experience compare to our closest competitor's", "should_trigger": false },
{ "query": "what is our biggest competitor doing in devrel right now", "should_trigger": false },
{ "query": "benchmark our docs and quickstart against our rivals", "should_trigger": false },
{ "query": "our signups are high but paid accounts are flat", "should_trigger": false },
{ "query": "should we go bottom-up or hire a sales team", "should_trigger": false },
{ "query": "when should we contact a free user", "should_trigger": false },
{ "query": "segment our developer audience", "should_trigger": false },
{ "query": "which developer persona should we prioritize this year", "should_trigger": false },
{ "query": "what should I put on my developer advocate resume", "should_trigger": false },
{ "query": "write a job description for a senior developer advocate", "should_trigger": false },
{ "query": "design the interview loop for a devrel hire", "should_trigger": false },
{ "query": "build our devrel content calendar for q3", "should_trigger": false },
{ "query": "how often should we publish engineering blog posts", "should_trigger": false },
{ "query": "should we build a certification program for our developers", "should_trigger": false },
{ "query": "design the curriculum for our developer academy", "should_trigger": false },
{ "query": "why doesn't our docs site rank for our own product name", "should_trigger": false },
{ "query": "find the keywords developers search before they find a tool like ours", "should_trigger": false },
{ "query": "rewrite our repo readme so it converts", "should_trigger": false },
{ "query": "optimize our github org profile page", "should_trigger": false },
{ "query": "set up a triage rotation for our github issues", "should_trigger": false },
{ "query": "write a governance model for our open source project", "should_trigger": false },
{ "query": "should we publish our revenue numbers publicly every month", "should_trigger": false },
{ "query": "start a weekly devlog for our project", "should_trigger": false },
{ "query": "what should we charge for our api tier", "should_trigger": false },
{ "query": "open core or hosted, which business model fits us", "should_trigger": false },
{ "query": "calculate our SaaS net revenue retention and CAC payback", "should_trigger": false },
{ "query": "what are the standard startup metrics a seed stage company reports to investors", "should_trigger": false },
{ "query": "design a grafana dashboard for our kubernetes cluster", "should_trigger": false },
{ "query": "create a launchdarkly metric for this feature flag experiment", "should_trigger": false },
{ "query": "pull the copilot usage metrics for our engineering org", "should_trigger": false },
{ "query": "query prometheus for p99 latency on the checkout service", "should_trigger": false },
{ "query": "build a KPI dashboard layout in figma", "should_trigger": false },
{ "query": "compute portfolio risk metrics for our trading book", "should_trigger": false },
{ "query": "set up google analytics 4 on our marketing site", "should_trigger": false },
{ "query": "review our sprint velocity and cycle time metrics", "should_trigger": false },
{ "query": "add custom metrics instrumentation to our python service", "should_trigger": false }
]
}
references/attribution-methods.md›
# Attribution methods
How to decide what a DevRel program is allowed to claim, and how to write the rule down before the period starts.
The method numbering below is exposition order, not a ranking. Stand them up in the efficiency order decided in SKILL.md § Attribution realism: `self-reported > pre/post lift > tagged links > influenced pipeline`.
## Contents
- [Why DevRel attribution breaks](#why-devrel-attribution-breaks)
- [Method 1 - self-reported attribution](#method-1--self-reported-attribution)
- [Method 2 - tagged links and last-touch analytics](#method-2--tagged-links-and-last-touch-analytics)
- [Method 3 - influenced pipeline](#method-3--influenced-pipeline)
- [Method 4 - pre/post lift and holdouts](#method-4--prepost-lift-and-holdouts)
- [Writing the attribution rule](#writing-the-attribution-rule)
- [The revenue-metric decision](#the-revenue-metric-decision)
## Why DevRel attribution breaks
Three structural reasons, none of them fixable with better tracking:
1. **Touches happen off-web.** A hallway conversation, a podcast listened to while driving, a colleague's recommendation. No pixel exists for any of them.
2. **Referrers disappear.** Aggregators, chat clients, privacy browsers and PDF or slide links strip the referrer. Roughly 70% of AI-assistant-driven traffic arrives with no referrer header and is classified as "direct". A large direct bucket during a launch week is the normal case.
3. **Every source is biased toward itself.**
- Ad platforms count their own conversions inside their own windows.
- Web analytics anchors on last non-direct click and dumps the unknown into direct.
- CRM reflects whatever a rep typed.
Reconcile with a stated method; never average them.
The consequence is not "measurement is impossible". It is that the method must be chosen, stated and held constant, so a change in the number means a change in the world rather than a change in the counting.
## Method 1 - self-reported attribution
Ask the human. A short question at signup, at first sales conversation, or in an onboarding email.
**Bias:** over-counts memorable touches (a talk, a podcast, a specific person), under-counts boring-but-real ones (search, docs, retargeting). Accept it: DevRel's touches are disproportionately the memorable kind, so this is the one method that sees the program at all.
**Question set that works.** One required question, free text optional:
```
How did you first hear about <product>?
( ) Search engine or AI assistant
( ) A colleague or friend
( ) A conference talk, meetup or workshop
( ) A podcast, video or livestream
( ) A blog post or newsletter - ours or someone else's
( ) An open-source project or repository
( ) Social media
( ) Other: ______
```
Rules that keep it usable:
- Keep the option list stable across quarters - a changed list breaks the trend.
- Always keep the free-text box - it surfaces channels you did not list.
- Never make it more than one question.
- Read the free text quarterly rather than only the counts.
**Reading it honestly:** report it as "self-reported first touch", not as attribution. Response rates and recall both bias it. Its value is direction and discovery, not precision.
## Method 2 - tagged links and last-touch analytics
Tag the links you control, one campaign or event per tag, never reused.
**Bias:** sees only clicks from taggable surfaces. Everything printed, spoken, screenshotted or shared in a chat client vanishes.
**Use for:** comparing two artifacts of the same type (which post, which event landing page, which docs entry point). Never for program totals - the untaggable share is too large and varies by channel, so totals compare tracking coverage rather than performance.
Pair every event, talk or campaign with a dedicated short URL or code, and record the pre-period baseline for the destination, or the lift cannot be claimed at all.
## Method 3 - influenced pipeline
The B2B answer, and the one most likely to be demanded. It requires a written rule, because "influenced" without a rule means whatever the presenter needs it to mean.
**Worked rule (adapt the specifics, keep the shape):**
> An opportunity is **DevRel-influenced** when, in the 180 days before it reached stage 2, at least one contact on the account recorded one of: attendance at a program-run event or workshop; a question answered by the team in a public community venue; a self-reported first touch naming a program surface; or a documented conversation logged by a program team member.
>
> Influence is claimed as _influenced pipeline value_, never as sourced revenue. Marketing and Sales may claim the same opportunities. The rule is fixed for the fiscal year; a mid-year change invalidates the trend and must be reported as a restatement.
Things the rule must always name:
- Which touch types count.
- The lookback window.
- Whether the touch must precede a specific stage.
Things it must never do:
- Change mid-period.
- Claim sourcing.
- Count a touch that was recorded after the opportunity was created.
**Bias:** over-counts by construction. Accounts that engage are already more likely to buy; the metric cannot separate influence from selection. State this once in the framework and stop re-litigating it.
**Latency:** enterprise cycles push the answer two to four quarters out. Name the leading indicator carrying the interim - typically qualified conversations or evaluations that survived review.
## Method 4 - pre/post lift and holdouts
The only family with a causal claim. In advertising, randomised tests routinely show observational attribution overstating true lift by a large multiple, which is the argument for doing this at all. The same literature also shows how much volume a properly powered test needs, which is why DevRel rarely gets a clean one.
**What is affordable at DevRel volume:**
- **Pre/post against a recorded baseline** for one bounded push (a docs rewrite, a launch, a regional event series). Record the pre-period before the push, not after.
- **A regional or segment holdout**: run the push in some regions or ecosystems and not others, compare the difference in differences. Works for event series and localized content.
- **A staggered rollout**: ship the change to one docs section or one SDK first.
**What to claim:** direction and rough magnitude. Never a point estimate with a decimal place. A "+12% lift" whose interval spans zero is not a finding, and DevRel-sized samples produce wide intervals by default.
## Writing the attribution rule
The rule is one short section of the framework document. It names:
1. Which methods are in use, and for what.
2. What each method may and may not claim, in one sentence each.
3. The lookback window and any stage condition.
4. What happens to the trend if the rule changes (a restatement, announced).
5. The known blind spots - the touches no method sees.
Point five is what makes the rest credible. An exec who discovers an unstated gap discounts the whole sheet; one who was told about it up front reads the remaining numbers as honest.
## The revenue-metric decision
Two defensible positions. Pick one deliberately; the deciding input is who owns the budget.
**No sales metrics on DevRel.** Mary Thengvall's position (14 December 2019): a quota "muddies our work too much and changes what should be a genuine relationship into one that revolves around money". The proposed currency instead is the **DevRel Qualified Lead** - a connection passed to whichever team can act on it, logged and reviewed monthly or quarterly for patterns:
- Marketing for a case study
- Product for feedback or a beta
- Engineering for a bug
- BizDev for a partnership
- Recruiting for a candidate
- Sales for a customer
It borrows the qualified-lead vocabulary the business already speaks without pretending every relationship ends in a sale.
**Influenced pipeline, explicitly shared.** A program funded from a sales or marketing line will be measured on a revenue-adjacent number whether it proposes one or not. Proposing the honest version - influenced, rule-bound, shared credit, with the leading indicator named - beats having a dishonest one assigned by someone with no view of the work.
Write the choice into the framework with its reason. A team that never made the choice ends up doing both badly: chasing pipeline informally while reporting reach formally.
references/measurement-plan-template.md›
# Measurement framework template and worked example
## Contents
- [Template](#template)
- [Worked example](#worked-example)
- [Weak versus strong, section by section](#weak-versus-strong-section-by-section)
## Template
```markdown
# DevRel measurement framework - <company>, <horizon>
## Driver and goals
Funded driver: <one of the seven, in the funder's words> - owner: <name, role>
Goals this horizon: <at most three, each traceable to a drop-off or a business need>
## Metric sheet
| # | Metric | Tier | Definition | Source | Baseline | Threshold | Owner | Action if breached |
| --- | ------ | ---- | ---------- | ------ | -------- | --------- | ----- | ------------------ |
Primary metric per goal is marked **P**. Total rows: 5-8.
Every threshold carries a source tag in its own cell: `(published: <who published it>)`,
`(baseline)` for a line derived from this program's own history, or `(hypothesis)` for a
circulating number being tested. A cell with no tag is not ready to ship.
## Attribution rule
Methods in use and what each may claim. Lookback window. Stage condition.
What a rule change does to the trend. Blind spots this rule cannot see.
## Reporting
Cadence: <monthly operational / quarterly decision / annual reset>
Audience and format per cadence.
The five-sentence summary the funder can repeat without you.
## Known blind spots
<what this framework cannot observe, and what it would take to observe it>
## Review
Next review: <date>. Kill-rule status per metric (cycles since it last changed a decision).
```
## Worked example
Fictional but realistic: a Series A API company, self-serve entry with a sales-assisted enterprise tier, one advocate plus one docs engineer.
```markdown
# DevRel measurement framework - Northwind API, H2
## Driver and goals
Funded driver: developer enablement - "developers sign up and never get their first call working"
- owner: VP Product.
Goals: (1) raise first-success rate on the self-serve path; (2) cut the support load new
accounts create; (3) keep an honest read on whether enablement work reaches enterprise deals.
## Metric sheet
| # | Metric | Tier | Definition | Source | Baseline | Threshold | Owner | Action if breached |
| ------- | --------------------------------------------------- | --------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------ | -------------------------------- | ------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------- |
| 1 **P** | First-success rate, 7 days | Enablement | share of new accounts making a successful authenticated API call within 7 days of signup | product analytics, `first_successful_call` event | 22% (Q1 21%, Q2 23%, Q3 22%) | below 20% for two consecutive months (baseline) | docs engineer | run a cold-run test of the quickstart and fix the top step failure before new content ships |
| 2 | Median time to first success | Enablement | signup → `first_successful_call`, median | product analytics | 41 min | above 60 min (baseline) | docs engineer | instrument step drop-off and locate the slow step |
| 3 | Step-level drop-off, quickstart | Enablement | share abandoning at each documented step | docs + product analytics | baselining | - | docs engineer | set threshold at next review |
| 4 | Support tickets per 100 new accounts, first 30 days | Enablement | tickets opened ÷ new accounts × 100 | support system | 14 | above 18 (baseline) | support lead | mine the top three ticket topics into troubleshooting pages |
| 5 **P** | 30-day retention of first-success cohort | Product impact | still making calls 30 days after first success, by monthly cohort | product analytics | 58% | below 50% (baseline) | advocate | interview five lapsed accounts before proposing anything |
| 6 | Self-reported first touch | Engagement | signup question, stable option list | signup survey | 61% response rate | response rate below 40% (hypothesis - a circulating figure, not a measured benchmark) | advocate | shorten or reposition the question |
| 7 **P** | Influenced pipeline | Business impact | see attribution rule | CRM | €0 recorded (rule new this half) | reported, not targeted, for two quarters | VP Product | none - baselining |
| 8 | Verbatim evidence | Qualitative | quotes and friction findings with source, date and the fix shipped | manual log | 4 per quarter | fewer than 2 (baseline) | advocate | schedule two user conversations |
## Attribution rule
Self-reported first touch is the standing instrument, reported as "self-reported", never as
attribution. Tagged links are used per campaign for artifact comparison only, never for totals.
An opportunity is DevRel-influenced when, in the 180 days before it reached stage 2, a contact on
the account attended a program event, had a public question answered by the team, or named a
program surface as first touch. Influenced value is never presented as sourced; Marketing may
claim the same opportunities. The rule is fixed for the fiscal year; any change is announced as a
restatement.
Blind spots: word-of-mouth inside a company, docs read without signing up, AI assistants
answering from our docs without a visit, and anything reaching an enterprise buyer through a
partner.
## Reporting
Monthly: metrics 1-4 and 6 to the product team, one page, trend lines against baseline.
Quarterly: full sheet plus the qualitative evidence to the VP Product and the exec team.
Annual: re-derive the sheet from the driver; assume half the metrics change.
Five-sentence summary: "New developers get their first API call working 22% of the time within a
week, and we are moving that to the high twenties this half. The step data will tell us which
part of the quickstart is losing them. Accounts that reach first success stay: 58% are still
calling at 30 days. We report enterprise influence, not sourced revenue, and the rule is written
down. Everything else we publish is an input to those numbers, not a result."
## Known blind spots
No visibility into docs readers who never sign up. No identity link between community members and
accounts, so community-to-expansion cannot be measured this half; it needs an identity mapping in
the CRM, roughly two engineering days.
## Review
Next review: end of Q4. Kill-rule status: metric 6 has not changed a decision in one cycle.
```
## Weak versus strong, section by section
**Driver**
- Weak: "Grow developer adoption and community engagement while supporting sales."
- Strong: "Developer enablement - developers sign up and never get their first call working. Owner: VP Product."
The weak version names three drivers, so every candidate metric scores equally and the sheet fills with everything measurable.
**A metric row**
- Weak: `Docs traffic | up 15% | marketing dashboard`
- Strong: `First-success rate, 7 days | Enablement | share of new accounts making a successful authenticated API call within 7 days | product analytics, first_successful_call event | 22% | below 20% two months running | docs engineer | cold-run the quickstart, fix the top step failure`
The weak row cannot be acted on, cannot be audited, and cannot fall for a reason anyone can name.
**A threshold**
- Weak: `first response within 24 hours - industry standard`
- Strong: `first response within 24 hours (self-set hypothesis - widely repeated, unpublished; our trailing median is 31 hours, revisit after three periods)`. Confirmed via search: CHAOSS's own guidance explicitly avoids hardcoding a universal number, offering "two business days" only as a worked example of a self-selected benchmark - not 24 hours, and not a target CHAOSS itself endorses.
The weak version invites exactly one question - "whose standard?" - and it has no answer. The strong version survives that question, and it also tells the reader what the team is actually doing about the gap. The same applies to any figure lifted from a conference talk. CHAOSS, the only vendor-neutral publisher of repository health definitions, deliberately leaves the target for each of its four starter metrics to the project, so a repository threshold presented as an external standard is almost certainly folklore.
**Attribution**
- Weak: "DevRel drove €400K in pipeline this quarter."
- Strong: "€400K of pipeline is DevRel-influenced under the written rule; Marketing claims the same opportunities. Sourced revenue is not something this team measures."
The weak version wins one meeting and loses the next one, when someone asks Sales for the same number.
**Reporting**
- Weak: a 30-row dashboard, refreshed when someone remembers.
- Strong: eight rows, a named owner each, monthly operational read and quarterly decision read, with a kill rule that removes anything which has not changed a decision in two cycles.
**Blind spots**
- Weak: section omitted.
- Strong: three named gaps with the cost of closing one of them.
Publishing the gaps is what makes the rest of the sheet believable. An exec who finds an unstated gap discounts everything else on the page.
references/metric-catalog.md›
# Metric catalog
Candidate metrics per tier, plus the published spines they hang off. Pick from here; do not ship the list.
Tiers below run reach → business impact, which is the order value and collection effort both rise in - not the order to build in. Build order is efficiency, decided in SKILL.md § The tier ladder: `enablement > engagement > product impact > business impact > reach`.
## Contents
- [Named spines and their sources](#named-spines-and-their-sources)
- [Tier 1 - reach](#tier-1--reach)
- [Tier 2 - engagement](#tier-2--engagement)
- [Tier 3 - enablement](#tier-3--enablement)
- [Tier 4 - product impact](#tier-4--product-impact)
- [Tier 5 - business impact](#tier-5--business-impact)
- [Open-source variants](#open-source-variants)
- [Qualitative evidence that belongs in the sheet](#qualitative-evidence-that-belongs-in-the-sheet)
## Named spines and their sources
| Spine | Source | Shape | Choose when |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| AAARRRP | Phil Leggetter, 2016, adapted from Dave McClure's AARRR with a seventh element for developer products | awareness, acquisition, activation, retention, referral, revenue, product | a charter already names which of the seven the business funds |
| Developer journey | Matthew Revell, 2016 | initial awareness → deep awareness → evaluation and validation → commitment → standardization → community and advocacy | the question is where developers leak, not which goal to fund |
| Awareness / enablement / engagement | Mary Thengvall, 2021 | three functions mapped to advocate, developer-experience and community-management roles | several people each need a number they own |
| Orbit levels | Orbit Model, published 2019, no longer actively developed | explorer → participant → contributor → advocate, gravity = love × reach | community depth specifically; not a program-wide spine |
| CHAOSS Starter Project Health | CHAOSS, `chaoss.community/kb/metrics-model-starter-project-health/` | time to first response, change-request closure ratio, contributor absence factor, release frequency | the product is a repository |
Revell's journey guide closes by deferring per-stage measurement to a future guide that was never published. That is a fair statement of the field: the stage vocabulary is settled, the per-stage numbers are not, and inventing them for a specific product is legitimate work - just label them as yours.
## Tier 1 - reach
| Metric | Definition | Source | Caveat |
| ---------------------- | ----------------------------------------------------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Content sessions | sessions on posts, docs pages or landing pages published by the program | web analytics | attribution to a channel is directional at best, and a third-party script misses most of a developer audience - report step-to-step ratios inside one source, not absolutes |
| Video and stream views | views over a fixed window after publication, not lifetime | platform analytics | definitions of a "view" differ per platform; never sum across platforms |
| Talk audience | in-room attendance plus recording views at 30 days | organiser numbers, platform analytics | organiser counts are usually registrations, not attendance |
| Newsletter reach | delivered and opened, by issue | mail platform | open rates are inflated by privacy proxies; use clicks or replies instead where possible |
| Mentions | third-party mentions of the product in developer venues over a period | listening tooling or manual search | volume says nothing about sentiment; pair with a read of the actual threads |
Never make one of these a program goal. They exist to explain a downstream movement, not to be moved.
## Tier 2 - engagement
| Metric | Definition | Source | Caveat |
| ------------------------------------ | -------------------------------------------------------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------------- |
| Questions asked by non-staff | questions opened in community venues or the forge by people outside the team, per period | forge API, community export | needs a staff/bot flag to mean anything |
| Repository clones and unique cloners | per period, against a trailing baseline | repository traffic panel | short retention windows; export monthly or lose the history |
| Docs search terms with no result | count and top terms per period | docs search log | the highest-yield engagement metric for a docs team: it names the gap |
| Qualified conversations | conversations with someone who has the problem, influences the decision, and agreed to a next step | conversation log with a fixed shape | define the bar before the event, or the count is not comparable |
| Sample-repo forks and template uses | per period | forge API, template telemetry | forks include vanity forks; pair with clones |
## Tier 3 - enablement
The tier a DevRel program most directly owns. Prefer these as primaries when the driver is developer enablement or adoption.
| Metric | Definition | Source | Caveat |
| --------------------------------- | ---------------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------ |
| First-success rate | share of new accounts reaching the defined success event within N days | product analytics | the success event must be one specific, instrumented action, agreed in writing |
| Median time to first success | signup (or docs entry) → success event | product analytics | medians only; the mean is destroyed by long-tail sessions |
| Step-level drop-off | share abandoning at each documented step | product analytics plus docs analytics | needs the quickstart instrumented per step; often the first real analytics ask |
| Support load per 100 new accounts | tickets or issues opened per 100 new accounts in their first 30 days | support system | falling load can also mean falling adoption; read alongside signups |
| Docs task success | share of testers completing a fixed task set on a cold environment | moderated or unmoderated test runs | small-N by nature; report the run list, not a percentage |
| Cold-run defect count | failures found running the documented path on a clean machine | scheduled cold run or docs-as-tests job | zero defects usually means the run was not cold |
Self-serve activation figures in the 20-40% band circulate widely for developer products. Treat them as order-of-magnitude context, not a benchmark - the sourcing is practitioner claim, not study.
## Tier 4 - product impact
| Metric | Definition | Source | Caveat |
| ----------------------------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------- | ---------------------------------------------------------------- |
| Cohort activation | activation rate of accounts whose first touch was a program surface, versus all accounts | product analytics plus a touch tag | needs the cohort defined before the period, not after |
| Cohort retention at 30/60/90 days | still active at each mark, by join cohort | product analytics | one cohort per period; do not average cohorts of different sizes |
| Second-project rate | share of developers starting a second project or second integration | product analytics | the strongest single signal that first success was real |
| Feature adoption after an enablement push | adoption of the targeted feature before versus after, against a control feature | product analytics | pick the control feature in advance |
| Contributor conversion | users who become contributors within a period | forge API | see the open-source variants below |
## Tier 5 - business impact
| Metric | Definition | Source | Caveat |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------- |
| Influenced pipeline | value of opportunities whose account had at least one recorded program touch before the stage | CRM plus a touch record | over-counts by construction; never present without the word "influenced" |
| DevRel Qualified Leads routed | connections passed to another team with the receiving team named, per period | a maintained log | a count of relationships, not of revenue; review for patterns quarterly |
| Support deflection | tickets avoided, estimated from community answers to questions that would otherwise be tickets | support plus community data | an estimate with a stated method, always; label it as one |
| Expansion in accounts with community presence | net expansion in accounts with at least one active community member, versus those without | CRM plus community identity mapping | selection bias runs both ways; state it |
| Hires sourced | candidates entering the hiring funnel from program surfaces | applicant tracking | keep separate from product metrics; different consent and different owner |
| Ecosystem integrations shipped | third-party integrations, plugins or SDKs released in the period | forge, registry, partner list | a slow metric; annual reading, not quarterly |
## Open-source variants
When the product is a repository and there is no purchase, anchor on the CHAOSS starter set rather than inventing metrics:
- **Time to first response** - median elapsed time from an issue or change request opening to a first _human_ response, bots excluded. The two-business-day figure widely quoted as the CHAOSS target is not one: it appears only inside VMware's case study on the starter-model page, as that company's internal guideline. Cite it that way.
- **Change-request closure ratio** - closed (merged plus rejected) ÷ opened in the same window. CHAOSS states the expectation qualitatively: projects should keep up with incoming change requests and resolve them in a timely manner. A ratio holding near or above 1 is a reasonable self-set line, not a published standard.
- **Contributor absence factor** - smallest number of people producing 50% of contributions. CHAOSS states the direction only: lower means higher dependency on fewer contributors, and therefore higher risk.
- **Release frequency** - releases per period including point releases. CHAOSS keeps this one deliberately open, stating that "no two projects have the same needs", and notes that delayed security releases leave users without an easy upgrade path.
**Treat every cadence or ratio cutoff as yours to set.** Any cutoff in a repository health report is either one company's internal figure (name the company), a self-set baseline (say so), or folklore. Adopt CHAOSS's definitions verbatim so the numbers stay comparable, and derive the thresholds yourself.
CHAOSS states its own caveats:
- "automation and bot activity can influence the usefulness of several metrics in this model, in particular Time to First Response and Change Request Closure Ratio".
- The absence factor ignores knowledge specialisation.
- Release frequency is meaningless for docs-only repositories and may be counted off-forge.
- "it's important not just to look at the numbers, but also at the trends".
- It also refuses cross-project comparison outright: "we do not compare projects against each other, but instead, we expect teams to use their metrics to make improvements".
Downloads, dependents and unique cloners beat stars for adoption. An issue opened by a stranger who read the docs is worth more than a hundred stars.
When the artifact ships to a hub rather than to a package registry - a model or dataset directory, a connector or plugin marketplace - add two metrics and keep them apart:
- **Hub downloads or installs** - a dependency signal. It accrues slowly to whatever is wired into someone's scheduled pipeline, so it barely moves at launch and keeps moving for years afterwards.
- **Hub likes or favourites** - an attention signal. It spikes on release and decays, and it concentrates on different artifacts entirely.
Reporting one as if it were the other is the standard failure on these surfaces; the hubs' own published analyses say so. Neither is deduplicated by user unless the hub's paid analytics tier is bought, so both are request counts, not people. Definitions differ per hub and sometimes per surface _within_ a hub - write the definition next to the number, and never sum across hubs.
## Qualitative evidence that belongs in the sheet
Numbers alone lose the argument when the counts are small, which is most of the time in DevRel. Reserve two rows of the framework for evidence that is not a number:
- **Verbatim quotes** from developers, with source and date, on what the program changed for them.
- **Friction findings** from cold runs and support threads, with the fix shipped and the date.
Both are auditable and both answer "so what". A framework of pure counts under-reports exactly the work that developers value most.
Two rules keep qualitative rows from becoming anecdote:
- **Tag each finding with a confidence level** - high, medium or low - set by how many independent sources back it, and show the tag next to the finding.
- **Hold a minimum sample.** Below roughly five to ten independent data points for a given segment, present the finding as a hypothesis, not a conclusion. This is the qualitative twin of the small-N floor applied to counts.
SKILL.md›
---
name: devrel-metrics
description: Builds a developer relations measurement framework - a handful of metrics tiered from reach to product and business impact, each with a written attribution rule, a baseline-derived target, an owner and an action, and vanity metrics cut. Use whenever someone asks how to measure DevRel, which devrel KPIs matter, how to prove devrel value or ROI to an exec, what belongs in a devrel scorecard or quarterly report, why their numbers look good but leadership stays unconvinced, or reports only stars, impressions and event counts - even if they never say "metrics". Do NOT use for tracking instrumentation and dashboards - use samber/developer-relations-skills@devrel-analytics. Not community-only health metrics, not a single event's ROI.
license: MIT
metadata:
author: Samuel Berthe
version: "1.0.1"
---
# DevRel Metrics
You are a developer relations measurement strategist. You decide which numbers a DevRel program reports, what each one is allowed to claim, and what happens when one moves.
The output is a **measurement framework**: five to eight metrics, each with a definition, a source, a baseline, a threshold, an owner and an action, plus one written attribution rule. Every DevRel team can produce numbers; almost none can produce numbers that survive the question "so what would you do differently?". Closing that gap is the whole job.
Two facts shape everything below:
- DevRel touches sit early and often off-web, so honest attribution is a stated rule rather than a tracking feature.
- DevRel effort compounds slowly, so a framework that changes every quarter measures noise.
## Route before measuring
Name which skill fits, and stop if it is not this one.
- Instrumenting the tracking - UTM discipline, event taxonomy, funnel views across docs, blog, repos → `samber/developer-relations-skills@devrel-analytics`. This skill decides _which_ metrics; that one makes them collectable.
- Measuring a community only - activity, responsiveness, contributor funnel, sentiment → `samber/developer-relations-skills@developer-community-health`.
- One event's return → `samber/developer-relations-skills@developer-event-sponsorship`.
- A launch window's numbers → `samber/developer-relations-skills@oss-launch`.
- Whether a quickstart converts → `samber/developer-relations-skills@developer-quickstart-guide`.
- Choosing the program's driver, goals and pillar mix → `samber/developer-relations-skills@devrel-strategy`. That decision is this skill's **input**; if it does not exist yet, go there first and come back.
- Mapping the journey stages themselves → `samber/developer-relations-skills@developer-journey-map`.
## Interview
Ask one question at a time, multiple-choice where the options are knowable. Stop once you can name the funded driver, the reader, and what data exists - confirm the rest as you go.
**Memory: Store a few info about the current task or project context. When memory lives in a file, use `devrel-context.md`; if a different memory system is in use, rely on that instead. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**
1. Why does the company fund this work: developer adoption, sales enablement, developer enablement, product input, ecosystem and partnerships, contributor community, or employer brand? Which would the person holding the budget name first?
2. Who reads these numbers, and what decision do they make with them - renew the budget, move headcount, kill a channel, nothing?
3. Is adoption individual (a developer decides alone) or company-led (a developer evaluates, someone else signs)? Or an open-source project with no purchase at all?
4. What can a developer actually do with the product: sign up self-serve, call an API, install a package, clone a repo, request a trial through sales?
5. What is reported today, and has any of it ever changed a decision?
6. Which data exists without new engineering work: product analytics, CRM, forge/repository data, docs analytics, community exports, a signup survey?
7. What is the review cadence, and what is the hard date the first numbers have to land by?
8. Do you need a one-off win at that date, or a sheet that still reads in a year?
9. Has a number already been promised to leadership that you now have to live with?
10. What is the effort ceiling - engineering days available for instrumentation, standing hours per month and whose, and which other team (product analytics, sales ops, privacy) has actually agreed to cooperate?
If the answer to 1 is "all of them", the program has a sponsor problem before it has a measurement problem. Send it to `samber/developer-relations-skills@devrel-strategy` rather than building a sheet that hedges across seven drivers.
Re-rank the tier ladder against 7, 8 and 10 before shortlisting anything, and say which answer moved which tier:
- A deadline inside one quarter promotes engagement and enablement - the tiers already collectable - and demotes anything needing a pre-period. A metric taking two quarters to instrument does nothing for a budget review next month.
- A one-off win promotes whatever is readable now; a compounding mandate promotes product impact, whose cohorts only pay off after several periods.
- An effort ceiling with no engineering days deletes every metric needing a new event and turns it into a request to `samber/developer-relations-skills@devrel-analytics`.
- No cooperation from sales ops deletes influenced pipeline outright. Name it as deleted; never park it at the bottom of the sheet.
## Workflow
1. Route, then interview until the driver, the reader and the data access are clear.
2. Restate the funded driver and the program's goals in one sentence each. Everything below hangs off them.
3. Brainstorm metric spines - see below. Do not write a sheet yet.
4. Build the shortlist against the tier ladder, in its efficiency order: one primary metric per goal, 5-8 total, at least one from product or business impact.
5. Reality-check each candidate against the data the team actually has. Delete anything the team cannot instrument, name it as deleted in the blind-spots section, and hand the gap to `samber/developer-relations-skills@devrel-analytics` rather than assuming it away or ranking it last.
6. Write the attribution rule before touching targets. It determines what half the metrics are even allowed to say.
7. Capture baselines: at least three prior periods per metric. Without history, mark the metric "baselining" and set its threshold at the next review.
8. Set targets as a direction and a range, not a point, for the first two cycles.
9. Attach an owner and a named action to every threshold. A metric with no action attached gets cut, not kept.
10. Set the cadence, write the framework document, and present it section by section for validation.
## Brainstorm the spine before choosing metrics
Say explicitly that you are in brainstorming mode. Offer **two or three whole spines**, not one, and let the owner pick before you detail anything - the spine determines every metric under it, so a wrong one is expensive to unwind later.
The three that fit most programs:
- **Driver-anchored.** One primary metric per AAARRRP goal the strategy funded. Best when a charter already names the goals. Gives up legibility to teams organised differently.
- **Journey-stage.** One metric per stage where developers currently drop off. Best when the complaint is "we do not know where they leak". Gives up the direct line from metric to budget line.
- **Function-split (awareness / enablement / engagement).** One metric per function, matching how the team is staffed. Best when several people each need a number they own. Gives up cross-team causality.
Present each with:
- what it makes visible
- what it hides
- who it is legible to
- the conditions under which it is the wrong pick
Recommend one and say why. Named sources and how these chain: [./references/metric-catalog.md](./references/metric-catalog.md).
Do not rank the spines by efficiency - that would be false precision here. A spine the funder cannot read is worth nothing whatever it cost, so every ordering collapses to "the one your funder already speaks". Rank the tiers underneath the chosen spine instead.
Never stack two spines. A framework carrying both AAARRRP goals and journey stages double-counts the same work and doubles the sheet.
## The tier ladder
These five tiers consolidate what the field measures in practice rather than reproducing a named industry model. Cost of collection rises down the list, and so does what a number is worth in a budget review. Never present the ladder to an exec as an industry standard.
"Enablement" here names a measurement tier - one of three unrelated senses the word carries in this collection, alongside the enablement pillar in `samber/developer-relations-skills@devrel-strategy`'s pillar mix and the enablement stage in Mary Thengvall's awareness → enablement → engagement mapping used by `samber/developer-relations-skills@devrel-team-structure`. None of the three should be assumed equivalent to another.
| Tier | Question | Examples | Honest limit |
| --------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| Reach | did anyone see it | impressions, views, attendees, followers | volume only; the easiest tier to inflate, never a program goal |
| Engagement | did anyone do something | questions asked, repo clones, doc searches, qualified conversations, newsletter replies | attention with intent; still no product contact |
| Enablement | did they succeed | signup → first-success rate, median time to first success, step drop-off, tickets per 100 new users | the tier DevRel most directly owns and can most directly move |
| Product impact | did behaviour change | activation and 30/60/90-day retention of a touched cohort, second-project rate, feature adoption after a push | needs a named cohort and a pre-period, or it is a coincidence |
| Business impact | did money or risk move | influenced pipeline, expansion in accounts with active members, support deflection, hires sourced, integrations shipped | always shared credit; say so every time |
### Which tier to build first
Value and effort rise together the whole way down this ladder, so the ladder alone never answers "what do I instrument first". Compute the ratio. It reorders the list non-monotonically, and the winner is neither end.
- **effort** - instrumentation, cross-team coordination, standing upkeep; cost of collection _is_ the effort here, so one line covers both: `business impact > product impact > enablement > engagement > reach`
- **value** - what the number is allowed to claim in front of the funder: `business impact > product impact > enablement > engagement > reach`
- **compliance cost** - the review it triggers and what it costs to unbuild: `business impact > product impact > enablement > engagement == reach`
- **efficiency** - value per unit effort: `enablement > engagement > product impact > business impact > reach`
Orders of magnitude behind that ratio, tier by tier:
- **Reach.** A dashboard already open, but a zero numerator stays zero however small the denominator - which is why the cheapest tier ranks last and not first.
- **Engagement.** Mostly reads what the forge and the docs search log already keep (hours), and one of its metrics names a fix on its own.
- **Enablement.** Costs one instrumented success event - engineering days once, near-zero after - and buys the number this team can most directly move: the best ratio on the ladder.
- **Product impact.** Adds a cohort tag and a pre-period that must exist _before_ the period starts, so a quarter of coordination plus a wait.
- **Business impact.** Adds CRM fields, sales-ops cooperation and a rule held for a fiscal year, then answers two to four quarters late.
`engagement > product impact` is a break, not a tie:
- Engagement wins only because its data is already collected, so flip it wherever product analytics are already instrumented.
- The single `==` holds because engagement and reach both read public or aggregate data the team already has - neither adds a new link between a person and an account, so neither triggers a new review.
**Default rung.** Put the primary at enablement. Promote business impact ahead of its ratio when the reader of these numbers decides whether to renew the budget, or when the program is funded from a sales or marketing line.
That promotion is what the "at least one metric from product or business impact" rule below enforces. High value plus high effort loses every round to a ratio, so a program measuring by efficiency alone reports reach and engagement forever and never earns its budget - and business impact is the only tier that survives a budget review.
This ordering is a default, not a law; it shifts with context and with who executes it. Re-rank it against what you already know about this team:
- An existing product-analytics install or clean self-serve telemetry collapses the effort on enablement and product impact and promotes both.
- A sales team that will not share pipeline data does not demote business impact, it deletes it.
Delete what the team cannot instrument, and name it as deleted in the framework's blind-spots section, with what it would take to get it. A ruled-out metric parked at the bottom of the sheet reappears next quarter as scope.
Rules that keep the sheet usable:
- One primary metric per goal. Several supporting metrics are fine; several primaries are a tie nobody breaks.
- At least one metric from product or business impact. A framework that stops at engagement cannot answer the funder's question.
- At most eight metrics total. A sheet of five that gets refreshed beats twenty that rot by month three. Dashboard-design practice outside DevRel converges on the same range - 5-7 headline KPIs - as a convention, not a study.
- Every reach metric needs a downstream partner in the same row's story, or it does not ship.
Candidate metrics per tier, with definitions, sources and caveats - including the open-source and B2B variants: [./references/metric-catalog.md](./references/metric-catalog.md).
## Attribution realism
Write the rule down before the period starts. A rule invented after the quarter is a narrative.
Four methods, each biased in a known direction. Pick deliberately and hold them constant:
| Method | Bias | Use for |
| ------------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Self-reported attribution ("how did you hear about us") | over-counts memorable touches, under-counts docs and search | the standing instrument - the only one that sees hallway, podcast and word-of-mouth touches |
| Tagged links and last-touch analytics | under-counts anything not clicked from a taggable surface; inflates "direct" | comparing artifacts, never program totals |
| Influenced pipeline against a written touch rule | over-counts by construction; shared credit | B2B, only when the CRM can carry the touch and the word "influenced" never drops |
| Pre/post lift, or a holdout on one region or segment | closest to causal, usually under-powered at DevRel volume | one bounded push with a recorded pre-period |
Which one to stand up first, when the hours only cover one or two:
- **effort** - setup plus standing upkeep: `influenced pipeline > pre/post lift > tagged links > self-reported`
- **value** - what it lets the program claim: `influenced pipeline > pre/post lift > self-reported > tagged links`
- **compliance cost** - the review it triggers: `influenced pipeline > tagged links > pre/post lift == self-reported`
- **efficiency**: `self-reported > pre/post lift > tagged links > influenced pipeline`
Self-reported wins by a wide margin:
- One question at signup, near-zero standing cost, and the only method that sees the hallway, the podcast and the colleague's recommendation at all.
- Pre/post lift's marginal effort is recording the pre-period before the push - near-zero on the day, impossible afterwards - and it buys the only causal claim available.
- Tagged links cost a naming convention plus per-campaign discipline and may only ever compare artifacts.
The `==` is real:
- Pre/post lift and self-reported both read aggregates or a volunteered answer, so neither triggers a new privacy review.
- Influenced pipeline tops the compliance line because it joins a named person's public community activity to a CRM record, and that identity map is hard to unbuild once it exists.
Influenced pipeline is what this order starves:
- Last on efficiency, and the only method a budget-holder reads.
- Promote it ahead of its ratio when the budget comes from a sales or marketing line, which is the same decision as the revenue question below.
- A method whose data the team cannot get - no CRM touch field, no signup form to add a question to - leaves the framework by name into the blind-spots section, rather than sitting at the bottom of the list.
Three things to say out loud in every report:
- A large "direct" bucket is normal, not a bug. Aggregators, chat clients, privacy browsers, and AI assistants strip referrers; roughly 70% of AI-assistant traffic arrives with no referrer and lands in "direct" (Demand Curve newsletter #331 - one newsletter's figure, so treat the number as directional).
- Enterprise cycles push any pipeline answer two to four quarters out. Name the leading indicator carrying the interim.
- Two teams must never claim the same revenue as sourced. Influenced is shared by definition.
**Decide the revenue question explicitly.** Both positions below are defensible, and the framework has to pick one - the deciding input is who owns the budget.
- **No sales metrics on DevRel.** Mary Thengvall's position (December 2019) is that DevRel should never carry sales metrics, because a quota "changes what should be a genuine relationship into one that revolves around money"; the non-revenue currency is the DevRel Qualified Lead, a connection routed to whichever team can use it.
- **Influenced pipeline, explicitly shared.** A team funded from a sales or marketing line will be measured on a revenue-adjacent number whether it proposes one or not, so proposing the honest version beats having a dishonest one assigned.
Method details and worked rules: [./references/attribution-methods.md](./references/attribution-methods.md).
## Anti-vanity guardrails
Run every candidate metric through three tests. A metric failing any of them is cut, not softened.
1. **Monotonic test.** Can the number only go up? Total stars, cumulative followers, all-time downloads and total members cannot fall when the program stops working, so they carry no information. Convert to a period delta or a rate, or drop it - stars in particular are gameable and non-comparable across project types, "the Monopoly money of open-source metrics" (Ashley Hathaway, developerrelations.com, 2017).
2. **"So what" test.** If it moved 20% next month, would anyone do anything differently? Name the action or cut the metric.
3. **Ownership test.** Can this team's work plausibly move it inside one review period? Company ARR fails for most programs; activation of a cohort the team touched usually passes.
Then protect the survivors:
- **Pair anything incentivised with a counter-metric.** "When a measure becomes a target, it ceases to be a good measure" (Marilyn Strathern's 1997 generalisation of Goodhart's law) shows up as thin posts from a post-count target, premature closes from an issue-closure target, chatter from a message-volume target, and throwaway accounts from a signup target. The fix is pairing volume with quality:
- posts with organic entries per post
- closed issues with reopen rate
- signups with 30-day retention
- **Respect the small-N floor.** Below roughly 30 events in a period - a borrowed rule of thumb, not a DevRel-measured cutoff - report the raw list with dates instead of a rate. Conference talks, enterprise conversations and external pull requests are all naturally low-count, and small-N percentages swing wildly enough to trigger bad decisions.
- **Never benchmark against another company's absolute numbers.** Compare the program to its own past. Audience size, ecosystem and product maturity change what normal looks like.
- **Separate a burst from a trend.** Launches, conferences and incidents spike everything. Comparing a launch month against a quiet month is this field's most common false trend.
- **Treat circulating cutoffs as hypotheses.** Thresholds like the 24-hour response, the 20% monthly-active ratio and the 40% seven-day retention are folklore rather than measured benchmarks. Say so when you use one, and replace it with the program's own trailing median after three periods.
## B2B versus individual adoption versus open source
The tier ladder holds for all three - the units and the readable metrics change:
- **Company-led adoption (B2B).** Count accounts alongside people: accounts with at least one active developer, evaluations that survived security or architecture review, expansion in accounts with community presence. Volume is low, so lean on the raw list and named-account conversations rather than rates. Pipeline answers arrive two to four quarters late, so the framework needs an explicit leading indicator per lagging metric.
- **Individual adoption.** Volume makes funnels readable: signup → first success, cohort retention, second-project rate, self-reported attribution at signup with enough responses to segment. This is where enablement-tier metrics do the most work, and where a docs or quickstart fix shows up fastest.
- **Open source with no purchase.** The product is the repository and the ecosystem. Anchor on the CHAOSS Starter Project Health set rather than inventing one: time to first response, change-request closure ratio, contributor absence factor, release frequency. CHAOSS defines all four and sets a target for none - the thresholds are yours to derive, and the definitions are worth adopting verbatim so the numbers stay comparable to other projects' published ones. Downloads and dependents beat stars: an issue opened by a stranger who read the docs beats a hundred stars.
- **Same for all three.** The attribution rule, baseline-derived thresholds, an action per metric, and the ban on monotonic primaries.
## Sourced, self-set, or folklore
Tag every threshold in the sheet with where it came from, in the same cell as the number: `sourced`, `baseline` (self-set), or `hypothesis` (folklore under test). An exec who discovers that one "benchmark" was invented discounts every other figure on the page, including the honest ones.
- **Sourced.** Traceable to a named, dated publication. Very few DevRel numbers qualify, and almost none of them tell you what good looks like. The one that comes closest sets no bar at all: CHAOSS's Starter Project Health model defines four repository metrics and publishes **no numeric target for any of them**. The two-business-day first-response figure circulating as a CHAOSS standard is VMware's internal guideline, quoted inside a case study on that page - attribute it to VMware or leave it out.
- **Self-set.** Derived from this program's own trailing baseline. Almost every threshold in the sheet should come from here. Saying so is a strength: it is the only kind of target that accounts for this audience, this product's maturity and this team's size.
- **Folklore.** A round number in circulation that no measurement backs: 24-hour response, 20% monthly-active ratio, 40% seven-day retention, "20-40% self-serve activation". Treat each as a hypothesis to test against the baseline. Never set one as a target, and label it as folklore whenever you repeat it.
Most sourced facts instead constrain what you may measure. Three that reshape almost every sheet:
- Client-side analytics under-counts developers badly. Plausible measured 58% of a tech audience blocking Google Analytics - 82.3% on Linux, 88.3% on Firefox (August 2021, one site whose traffic came heavily from Hacker News and Reddit, measured against Plausible's own proxied script). Absolute sessions from a third-party script are wrong by a factor that varies per page, so build reach and engagement metrics as ratios inside one source, never as absolutes compared across sources.
- Registry downloads are request counts, not people - by the registries' own documentation. npm counts every HTTP 200 tarball response, CI and mirrors included, and treats under roughly 50/day as mostly automation; PyPI excludes known mirrors by default. Never sum two registries and call the result users.
- Sources expire on their own schedule. GitHub's repository traffic endpoints keep 14 days, and only for repos you have write access to (its REST docs); package-registry time series last months, not years; chat venues keep whatever the plan allows. A metric defined today may be uncomputable for last quarter - start snapshotting the day it enters the sheet.
## Set the thresholds
- Baseline first: three prior periods before any threshold. Without history, ship the metric marked "baselining" and set its threshold at the next review.
- Set targets as a direction and a range for the first two cycles ("first-success rate from 22% into the 28-33% band"); switch to a point target only once the baseline is stable.
- Prefer a threshold that moves with the baseline over a fixed line: flag when the value leaves its own trailing window, for example two standard deviations from a 30-day rolling mean. Static lines drift out of date and produce alerts nobody reads by the third month.
- Cohort by month, never by day. Daily cohorts at DevRel volumes are single-digit, and every retention curve drawn from them looks flat regardless of what happened.
The specific numbers in these rules - three periods, two standard deviations, the 30-day window, like the small-N floor above - are conventions borrowed from adjacent dashboard and survey practice, not values measured in a DevRel context. Use them, and label them as conventions if anyone asks where they came from.
## Write it down, run it, prune it
- Give every metric six fields: definition, source, baseline, threshold, owner, action. No exceptions.
- State each metric's **pre-conditions** before it enters the sheet - what has to already be true for the number to exist at all (an instrumented success event, editorial control of the blog, write access to the repository whose traffic you want). A metric that fails this is an instrumentation request, not a metric.
- Give any metric that needs explaining its own page, in the shape the Developer Relations Foundation's Metrics Index publishes: definition, why it is impactful, pre-conditions, formula, worked example, analysis, guardrails, obstacles, automation. Adopt the shape only - the repository is still open, not archived, but has shipped one worked metric page (blog post publish rate) since it launched, so there is no catalogue there yet to look up.
- Read monthly for operations, quarterly for decisions, annually to reset the framework itself. Anything faster than monthly reads noise at DevRel volumes.
- **Kill rule.** Remove any metric that has not changed a decision in two consecutive review cycles. State the rule in the document so removal is a scheduled event rather than an argument.
## Invocation examples
- "We report stars, blog views and event count every month, and our VP just asked what any of it is for. Build us a metrics framework."
- "I defend the DevRel budget in six weeks. What belongs on that slide, and what can I honestly claim?"
- "Our team is about to be handed a pipeline target. Is there a defensible alternative, and how do I write it down?"
- "We are an open-source project with nothing to sell. What do we measure instead of stars?"
- "Here is our current dashboard - which of these thirty metrics should survive?"
Each produces the same artifact: the framework document below, built section by section with the owner approving each one. Not a dashboard, not tracking code, not an exhaustive metric list.
## Output
Produce a measurement framework document with six sections: driver and goals, metric sheet, attribution rule, reporting, known blind spots, review. Load the full template, a worked example and a weak-versus-strong comparison before drafting: [./references/measurement-plan-template.md](./references/measurement-plan-template.md).
Present one section at a time and get agreement before writing the next. A wrong driver caught at the first heading costs a paragraph; caught at the last, it costs the document.
## Pass threshold
The framework is done when it scores 100% on this check. Iterate until it does, and report the check explicitly.
- Five to eight metrics, no more.
- Every metric has all six fields: definition, source, baseline (or an explicit "baselining" mark), threshold, owner, action.
- At least one metric in the product-impact or business-impact tier.
- No monotonic metric anywhere as a primary.
- Every metric passes the "so what" and ownership tests.
- The attribution rule is written, names its methods and its lookback window, and uses "influenced" wherever credit is shared.
- Every incentivised volume metric has a counter-metric.
- Every threshold is labelled `sourced`, `baseline` or `hypothesis`, and no `hypothesis` threshold is presented as a target.
- The blind-spots section is non-empty.
A framework failing the count line does not need discipline. It has one goal too many.
## Common failure modes
- **Activity log as a report.** "12 talks, 40 posts, 6 events" counts effort, not outcome. It survives exactly one budget review.
- **One number to rule them all.** No single metric covers a program spanning docs, content, community and events - developer communities are "not one single number to rule them all" (Hathaway, 2017). Use a small set with one primary per goal.
- **Instrumenting before deciding.** Building tracking for everything measurable produces a dashboard nobody reads. Choose the metrics, then hand the gap to `samber/developer-relations-skills@devrel-analytics`.
- **Reach relabelled as impact.** Renaming impressions to "developer reach" changes nothing. The tier is set by what the number observes, not what it is called.
- **Promising last-touch revenue.** DevRel rarely owns the last click. Promising sourced revenue creates a debt that comes due at the next review.
- **Copying another company's dashboard.** Their metrics encode their driver, stage and instrumentation. Derive from yours.
- **Borrowed benchmark.** Quoting a round number from a conference talk or a blog post as the industry standard. It gets checked exactly once, and everything else on the sheet is doubted afterwards. Label it as a hypothesis and calibrate against your own trailing baseline instead.
- **Absolute traffic numbers from a blocked source.** Reporting "docs sessions" from a third-party analytics script as a headline number, when most of a developer audience never executes it. Report the ratio between two steps measured by the same source.
- **Launch month as the trend.** A launch or conference spike presented as a trend line destroys credibility the following month.
- **A dashboard with no owner.** Unowned metrics stop being refreshed by the third month. The fix is fewer metrics, not more automation.
- **Measuring the team instead of the program.** Per-person metric targets turn colleagues into competitors and pull the work toward whatever is counted.
- **Silence on the blind spots.** An exec who finds an unstated gap distrusts the whole sheet. Publishing the gaps is what makes the rest believable.
## Memory
If your harness has persistent memory, store the funded driver, the metric sheet with its baselines and thresholds, and the attribution rule once the owner validates them. The next quarter then starts from a recorded baseline instead of re-deriving one, and a threshold change becomes a visible decision rather than a silent edit.
## References
- [./references/metric-catalog.md](./references/metric-catalog.md) - candidate metrics per tier with definitions, sources and caveats; the named spines and their published origins.
- [./references/attribution-methods.md](./references/attribution-methods.md) - the four methods in depth, a worked influenced-pipeline rule, a self-reported attribution question set.
- [./references/measurement-plan-template.md](./references/measurement-plan-template.md) - the framework template, a full worked example, and a weak-versus-strong comparison of every section.
- CHAOSS publishes the Starter Project Health metric definitions at `chaoss.community`.
- The Developer Relations Foundation publishes per-metric page schema and contribution routes at `github.com/DevRel-Foundation/metrics-index`.
- samber/developer-relations-skills@devrel-team-structure - the org shape and reporting line a metrics framework has to fit.
- `samber/developer-relations-skills@developer-first-gtm` - picks the few numbers that prove the chosen adoption motion works, before this framework tiers the rest.
- samber/developer-relations-skills@developer-segmentation - the segment cut a per-segment metric sheet is built against.
- samber/developer-relations-skills@developer-education-strategy - the cohort, completion and lab-pass metrics a certification or academy program reports.
- samber/developer-relations-skills@devrel-budget-allocation - the spend a program's measurement-readability gate checks before it grows a line.
- samber/developer-relations-skills@devrel-competitor-analysis - compares the program outward; this skill instruments it inward.