samber/developer-relations-skills已通过检查
SKILL DETAIL
developer-community-health
samber/developer-relations-skills/developer-community-health
Designs and runs a developer community health measurement framework: activity, responsiveness, contributor-funnel and sentiment metrics, honest instrumentation, baseline-derived thresholds, and a report that ends in decisions. Use whenever someone asks how to measure their developer community, which community health metrics to track, whether their Discord, Slack or forum is dying, why the community feels quiet, or wants a community health dashboard, contributor funnel, community KPI set or engagement report - even if they only say "is our community doing OK". Do NOT use for launching a community (samber/developer-relations-skills@developer-community-launch) or company-wide devrel KPIs (samber/developer-relations-skills@devrel-metrics).
安装量 · 410查看来源
Installation
npx skills add https://github.com/samber/developer-relations-skills --skill developer-community-health
技能文件
SKILL.md
最近同步 · 2026年10月23日
evals/evals.json›
{
"skill_name": "developer-community-health",
"evals": [
{
"id": 1,
"prompt": "I run community for Tessellate, an open-source charting library. Our Discord has 4,213 members, up from 3,050 in January. Our welcome bot posts a templated \"thanks for asking, someone will be with you soon\" reply in every new #help thread within about 5 seconds, and the analytics panel shows a median first-response time of 9 seconds, which looks fantastic. Leadership wants a monthly community health dashboard starting next month. I've been told the industry standards are under 24 hours for support response, over 20% monthly active ratio, and under 30% of messages coming from staff. Can you build me the dashboard and tell me where we stand against those benchmarks?",
"expected_output": "A metric sheet of at most 8 metrics covering all four pillars, with bots excluded from responsiveness, the three quoted benchmarks labelled as unsourced folklore, member count refused as a headline, and an owner plus action on every threshold.",
"files": [],
"expectations": [
"Excludes bot accounts from the time-to-first-response metric and states that the 9-second median is an artifact of the welcome bot rather than human responsiveness",
"Defines time to first response as the time until the first response from a human, with bots filtered at query time",
"Recommends keeping an explicit bot list or bot account flag used when querying",
"Labels the 24-hour response time, the 20% monthly active ratio and the 30% staff-share figures explicitly as unsourced folklore rather than industry standards",
"States that those folklore figures may be quoted only as starting hypotheses and must be replaced by Tessellate's own trailing median after at least three periods",
"Refuses to use total member count (4,213, or the growth from 3,050) as a headline health metric and explains that total membership only ever goes up",
"Proposes active participants, defined as distinct non-bot people posting in the period, in place of total membership",
"Limits the metric sheet to at most 8 metrics",
"Covers all four pillars - responsiveness, contributor funnel, activity and sentiment - with at least one metric each",
"Writes each metric's formula beside its name in the sheet itself",
"Attaches a named owner and a named action to every threshold breach",
"Labels the origin of every number in the sheet as published and attributable, a self-set working convention, or unsourced folklore"
]
},
{
"id": 2,
"prompt": "Quick sanity check on Kelvinstack, our open-source stream processor. Over the last 12 months the commit counts by person were: Ana 812, Bruno 640, Chen 455, Dagny 210, Eli 168, Farid 95, Gita 60, Hana 40. Ana, Chen and Hana all work at Northwind Systems. Bruno and Dagny both work at Solano Labs. Eli, Farid and Gita are independents with no shared employer. A customer's procurement team is asking how exposed the project is if key people leave. What do I tell them?",
"expected_output": "Contributor absence factor computed as 2 and elephant factor computed as 1, with the employer-level concentration named as the worse of the two, no cross-project comparison, and an action attached.",
"files": [],
"expectations": [
"Defines the contributor absence factor as the smallest number of people producing 50% of contributions",
"States the total contribution count as 2,480 and the 50% threshold as 1,240",
"Reports the contributor absence factor as 2, because Ana and Bruno together account for 1,452 commits",
"Computes the elephant factor by grouping contributions by employer rather than by individual person",
"Reports the elephant factor as 1, because Northwind Systems alone accounts for 1,307 commits, above the 1,240 threshold",
"States explicitly that the employer-level concentration is worse than the person-level concentration for this project",
"States that a higher absence factor is safer and that a lower one signals dependency on fewer contributors",
"Notes that neither factor accounts for knowledge distribution, so two contributors with disjoint expertise are not interchangeable",
"Does not compare Kelvinstack's figures against other open-source projects or against a claimed industry-standard cutoff",
"States that no published numeric threshold exists for either factor",
"Recommends an identity mapping across commit emails and handles before trusting the contributor counts",
"Attaches a named action to the concentration risk, such as opening the reviewer or maintainer rung to the next contributors"
]
},
{
"id": 3,
"prompt": "I'm at Pelago, we sell an API testing tool, and our community is a 2,600-member Discord plus a small Discourse forum. I ran all of last quarter's Discord messages through a sentiment model and it scored 78% positive, which I'm putting in the QBR as our community sentiment number. I also sent a one-off survey asking \"how likely are you to recommend Pelago to other developers\" and got 9 responses: 4 promoters, 3 passives, 2 detractors. I calculated that as detractors minus promoters, so -22. Does that all hold up?",
"expected_output": "The classifier score rejected in favour of survey instruments, the NPS-style subtraction corrected to promoters minus detractors (+22), the 9-response sample treated as qualitative, and a lapsed-member instrument added.",
"files": [],
"expectations": [
"Rejects the 78% automated sentiment score as a community health metric because classifiers misread blunt bug reports and dry developer humour as negative",
"States that sentiment must come from asking people directly through a survey rather than from classifying existing messages",
"Corrects the direction of the subtraction to promoters minus detractors, giving +22 rather than -22",
"Requires the direction of the subtraction to be stated wherever the score is presented, so a positive number always means more promoters than detractors",
"States that 9 responses falls below the roughly 30-response floor and must be treated as qualitative evidence with dates attached rather than as a score",
"Requires the response rate to be reported next to any sentiment score",
"Requires the surveyed population definition to be fixed and kept stable between waves, otherwise the trend measures sampling rather than sentiment",
"Recommends corroborating the sentiment reading against behavioural signals from the same period, such as responsiveness or contribution-acceptance rates",
"Flags that the survey reaches only members still present and recommends a separate one-question instrument for members who went silent for 60 or more days",
"Specifies that the lapsed-member note is sent once and kept opt-out friendly rather than run as a follow-up campaign",
"Requires sentiment to be reported in aggregate and never attributed to a named member",
"Keeps the recurring pulse survey to roughly two items so it can be answered at a quarterly cadence"
]
},
{
"id": 4,
"prompt": "I maintain Brambleweave, a Rust web framework, with one other person, Ines. Our numbers this quarter are the best we've ever had: 1,140 Discord messages a month, up 40%, 61 distinct people posting, 23 merged PRs. Last week Ines told me she's been dreading opening the repo and is thinking about stepping back. Our board wants the quarterly community health report on Friday. What should the report say?",
"expected_output": "A report that leads with the sentiment and concentration pillars, adds a private maintainer burnout check with anonymity, treats the activity spike as unverified, and attaches an owned action.",
"files": [],
"expectations": [
"States that the record activity numbers do not contradict the burnout signal, and that burnout is the finding activity metrics hide best",
"Promotes the sentiment pillar ahead of the volume metrics for this report because the worry behind the request is about people rather than volume",
"Recommends a maintainer burnout check asked of the small group carrying the load, separately and privately from any general community survey",
"Lists burnout survey items covering energy or exhaustion, workload versus capacity, thoughts of stepping back, whether the contribution feels valued, and whether a concern can be raised safely",
"Names behavioural trace signals to read beside the survey: sustained high contribution ending abruptly, and contribution concentrated in very few people",
"Does not propose inferring burnout by running a sentiment classifier over Ines's public messages",
"Includes the contributor absence factor or an equivalent concentration measure, given that two maintainers carry the project",
"Keeps burnout responses separable from identity and does not attribute any individual answer to Ines in the board report",
"Attaches a named owner and a concrete action to the concentration or burnout finding, such as delegating review or recruiting reviewers",
"Treats the 40% message increase as a number needing a burst-versus-trend check rather than as evidence of health",
"Does not present total message volume as the headline of the board report"
]
},
{
"id": 5,
"prompt": "Numbers for Halyard, our open-source CI runner, just came in for March: 312 new forum members against 96 in February, 2,900 posts against 1,050, and 44 first-time posters against 12. Version 2.0 shipped on March 4 and we hit the Hacker News front page on March 5. I'm putting \"community up 3x\" in the board deck. I've also built a retention chart with one cohort per signup day so we can see exactly which day's members stick around.",
"expected_output": "The March jump labelled as a release-and-media burst rather than growth, the daily cohorts widened and the window stated, and thresholds withheld for lack of history.",
"files": [],
"expectations": [
"Labels the March spike as a burst driven by the 2.0 release and the Hacker News front page, not as growth or a trend",
"Refuses the \"community up 3x\" framing for the board deck and states what would have to hold for March to count as a trend",
"States that comparing a launch month against a quiet month is the most common false trend in community reporting",
"Rejects daily join cohorts and states that they leave single-digit groups whose retention curves are noise",
"Recommends widening the cohort window, with monthly join cohorts as the usual starting point",
"Requires the cohort window to be stated next to the number and kept stable between reports, because changing the window changes the trend",
"Names the follow-up reading worth trusting as whether the March cohort returns within a defined window, rather than the March volume itself",
"Requires the burst to be labelled explicitly inside the report so no reader mistakes it for growth",
"Does not present total post count as a health metric on its own",
"Derives any threshold from Halyard's own trailing baseline rather than from another project's numbers",
"Notes that two prior periods is not enough history to set a threshold and marks the affected metrics as baselining"
]
},
{
"id": 6,
"prompt": "We run a private Slack Connect community for Quarrystone, our observability platform. 41 customer accounts are in it, 187 individual people. Our VP wants a monthly health report with monthly active user percentage, NPS, and message growth. Last month 34 people posted. I'm also planning a quarterly NPS survey - the last one got 6 responses and scored +33.",
"expected_output": "A B2B sheet measured in accounts rather than people, with the six-response score rejected, rates replaced by the raw list and named-account conversations, and message growth cut.",
"files": [],
"expectations": [
"States that the unit of measurement in a B2B community is the customer account, not the individual person",
"Proposes account coverage - how many of the 41 accounts have at least one active member - ahead of any member-level active percentage",
"Asks or states whether the accounts the company most cares about are represented, not only how many accounts are active",
"States that a 41-account community produces small-N numbers everywhere, so the raw list should be read instead of rates",
"Rejects the 6-response score of +33 as a reportable number and states that it sits below the roughly 30-response floor",
"States that a handful of named-account conversations or interviews yields more than a six-response survey",
"Rejects message growth as a health metric",
"Includes at least one responsiveness metric, such as time to first human response or zero-reply rate",
"Names support deflection or expansion signals as more relevant than raw growth for this B2B community",
"Keeps the metric count at 8 or fewer and covers all four pillars",
"Applies bot exclusion, baseline-derived thresholds, a per-metric action and the privacy gate exactly as it would for an individual-adoption community"
]
},
{
"id": 7,
"prompt": "I'm the only person on community at Fernwood - we make a terminal-based database client, closed source, no public repository at all. The community is an 1,800-member Discord plus a Discourse forum on the free tier, which only keeps 90 days of history. I have about 3 hours a month for this and zero budget: I can't buy anything and I can't get engineering time. A vendor is pitching us a community data platform at $900 a month that promises cross-platform member identity and cohort retention. Should I buy it, and what should I track?",
"expected_output": "The platform declined and its rung deleted from the plan, venue analytics named as the primary source, monthly snapshotting started immediately against the 90-day cap, and the sheet sized to three hours a month.",
"files": [],
"expectations": [
"Recommends against the community data platform because Fernwood has no budget and no named owner for the pipeline",
"Deletes the community-data-platform rung from the plan entirely rather than listing it as future work, and explains that a parked rung comes back as scope at the next review",
"States that Fernwood has no forge or git rung available because there is no public repository, which makes the venue analytics panel the primary source",
"Promotes the activity pillar because a chat-only community has no cheap responsiveness pull from a forge",
"Flags the Discourse free tier's 90-day retention limit as a hard constraint on what can be computed for past periods",
"Requires snapshotting the numbers monthly into Fernwood's own store starting immediately, and states that backfill beyond the retention window is often impossible",
"Sizes the sheet to roughly 3 hours a month and drops any metric whose return does not justify a standing hour a month",
"Does not propose metrics requiring forge API data such as change request closure ratio, time to merge, release frequency or commit-based contributor counts",
"Recommends a hand-maintained identity mapping across Discord and Discourse handles rather than buying tooling for it",
"Requires a staff or employee account flag so non-staff share of activity can be computed",
"Recommends checking the venues' terms before bulk-exporting message content"
]
},
{
"id": 8,
"prompt": "Two things I'm rolling out at Ridgeline, our open-source feature-flag service, next month and I want a second opinion. First: a public monthly leaderboard on our website listing the top 10 forum answerers by answer count, each with a badge. Second: I'm setting the maintainer team a target of closing 90% of incoming issues and pull requests within 7 days, and reporting that number in the monthly health report.",
"expected_output": "Both mechanisms flagged as metrics that become targets, with quality counter-metrics required, the public per-member leaderboard refused on privacy grounds, and the 90% figure replaced by a baseline-derived threshold.",
"files": [],
"expectations": [
"Warns that a metric which becomes a target changes behaviour, and names premature or low-quality closes as the specific behaviour the 90%-in-7-days target encourages",
"Requires the closure target to be paired with a quality counter-metric, such as reopen rate or time to a real resolution, if it is kept at all",
"States that rewarding answer counts produces volume and chatter rather than better answers, and requires a quality counter-metric for the leaderboard too",
"Flags the public per-member leaderboard as privacy exposure and states that person-level data should be aggregated rather than published",
"States that publishing per-member recognition requires the member's consent",
"Notes that publishing per-member activity changes behaviour in ways the team did not intend",
"Proposes a recognition mechanism that does not publish a ranked per-member table, such as private thanks or promotion to an answerer or reviewer role",
"Requires any closure threshold to be derived from Ridgeline's own trailing baseline rather than set at a round number like 90%",
"States that no published target exists for the change request closure ratio",
"States that a closure ratio persistently below 1 means the backlog outruns the team, and marks that as an interpretation rather than published guidance",
"Excludes bots from the responsiveness and closure numbers"
]
},
{
"id": 9,
"prompt": "I've inherited a community health dashboard at Sablefish, our open-source ETL tool. It has 19 metrics. The thresholds were all set when it was built 14 months ago and never touched - about half of them go red every single month and everyone has muted the alerts. Two of the metrics are called \"active members\": one is computed as posted-or-reacted by the community team, the other as posted-only by the data team, and the two numbers get argued about in every monthly meeting. Nobody is named against any metric. Can you help me fix this?",
"expected_output": "The sheet cut to at most 8 metrics, each carrying all five required parts, thresholds re-derived as a deviation band from 14 periods of history, and the duplicate definition settled by publishing the formula.",
"files": [],
"expectations": [
"Cuts the sheet to at most 8 metrics",
"Requires each surviving metric to pass a \"so what\" test: if the number moved about 20% next month, someone would do something differently",
"Requires each surviving metric to carry all five parts - a plain-language definition, a data source the team can access today, a baseline or an explicit \"baselining\" mark, a threshold, and a named action with a named owner",
"Names an owner for the review itself and states that an unowned dashboard stops being refreshed",
"States that the fix for an unread dashboard is fewer metrics, not more automation",
"Requires thresholds to be re-derived from the trailing baseline at each review rather than set once and left alone",
"Uses the 14 periods of history to replace fixed cutoffs with a deviation band, defined as a stated percentage band or a spread of standard deviations around the rolling mean",
"Requires the chosen band definition to be written into the sheet",
"Resolves the duplicate \"active members\" definition by publishing the formula beside the number and settling on a single definition",
"Keeps at least one metric per pillar across responsiveness, contributor funnel, activity and sentiment",
"Labels each retained threshold's origin as published, self-set convention, or unsourced folklore",
"Cuts any metric that cannot be tied to a named action rather than keeping it as context"
]
},
{
"id": 10,
"prompt": "Our Slack workspace for Pallasite, our open-source geospatial library, has 1,900 members. Last month there were 11 messages: 9 from our two staff members, and 2 from the same user asking a question that never got answered. It has been roughly like this for eight months. Leadership has asked me to produce a community health dashboard to justify keeping the headcount on it. What metrics should I put in?",
"expected_output": "A diagnosis-first answer that names staff share and the unanswered question as the decisive numbers, states plainly that folding the space back may be the honest call, and sets a fold-back criterion instead of manufacturing justification.",
"files": [],
"expectations": [
"States plainly that the honest answer may be to fold the community back into a smaller surface rather than keep the Slack workspace open",
"States that keeping a dead room open costs trust every time a newcomer posts into silence",
"Works in diagnosis mode from the symptom rather than building the full eight-metric sheet first",
"Names the staff share of activity - 9 of 11 messages written by staff - as a decisive number",
"Treats the unanswered question as a responsiveness finding, not only an activity one",
"Rejects the 1,900 member count as evidence of anything",
"States honestly what the data cannot attribute rather than constructing a justification for the headcount",
"Checks whether the audience moved to another venue, whether staff stopped seeding, or whether an earlier spike was an event rather than growth",
"Does not answer with a list of engagement tactics such as contests, AMAs or conversation prompts in place of the measurement answer",
"States that 11 events in a period is below the small-N floor of roughly 30 and should be reported as a dated list rather than as rates or percentages",
"Names what would have to change, and by when, for the space to be worth keeping - a fold-back or shutdown criterion rather than an open-ended watch"
]
},
{
"id": 11,
"prompt": "We've never measured anything at Cindersmith, our open-source vector search engine. We're on GitHub with a Discourse forum and about 30 external contributors. I want a framework that covers everything: support deflection, product feedback, contribution growth, advocacy, retention and general engagement. Separately, our product team calls people casual, regular or core, and I've been using Orbit's explorer / participant / contributor / advocate wording in my slides. Where do I start?",
"expected_output": "A single primary outcome named in SPACES vocabulary, one funnel vocabulary chosen and kept, the code-centric starter set proposed, and metrics with no history marked baselining.",
"files": [],
"expectations": [
"Forces a single primary outcome rather than measuring all six stated objectives at once",
"Names that primary outcome in the SPACES vocabulary - support, product feedback, acquisition and advocacy, content and contribution, engagement, or success - rather than as an ad-hoc phrase",
"States that every metric on the sheet must plausibly move when that primary outcome moves",
"Requires one funnel vocabulary to be chosen and kept - either Orbit's explorer / participant / contributor / advocate ladder or the casual / regular / core split, never both in the same sheet",
"States that two vocabularies in one sheet make every cohort definition ambiguous",
"Notes that the Orbit model is no longer actively developed and cites it as a stable published framework rather than a live standard",
"Proposes the code-centric starter set: time to first response, change request closure ratio, contributor absence factor and release frequency",
"Adds a recommendability survey item as the fifth starter metric, asked roughly twice a year",
"Marks metrics with no prior readings as \"baselining\" rather than inventing a threshold for them",
"States that at least three prior periods are needed before setting a threshold from the trailing median",
"Tracks first-pull-request merge rate and time-to-merge for external contributors separately from maintainers' own pull requests"
]
}
],
"trigger_queries": [
{ "query": "How do I measure the health of our developer community?", "should_trigger": true },
{ "query": "What community health metrics should we track for our open-source project?", "should_trigger": true },
{ "query": "Is our Discord dying?", "should_trigger": true },
{ "query": "Our Slack community feels dead, is it actually?", "should_trigger": true },
{ "query": "Build me a community health dashboard for our forum", "should_trigger": true },
{ "query": "The board wants proof our community is worth the headcount", "should_trigger": true },
{ "query": "Which KPIs tell me whether my dev community is doing OK?", "should_trigger": true },
{ "query": "is our community doing OK", "should_trigger": true },
{ "query": "Why does our community feel so quiet lately?", "should_trigger": true },
{ "query": "We have 900 people in a Discord for our CLI and no idea whether it is working", "should_trigger": true },
{ "query": "I need a contributor funnel for our open source project", "should_trigger": true },
{ "query": "How do I know if newcomers are coming back to our forum?", "should_trigger": true },
{ "query": "What's a good way to track whether people actually get answers in our community?", "should_trigger": true },
{ "query": "set up a monthly report on how our online community is doing", "should_trigger": true },
{ "query": "Is this open-source project healthy enough for us to depend on?", "should_trigger": true },
{ "query": "How exposed are we if our top two committers leave?", "should_trigger": true },
{ "query": "what's our bus factor and does it actually matter", "should_trigger": true },
{ "query": "our welcome bot is skewing our response time numbers, how do I fix the metric", "should_trigger": true },
{ "query": "Should I report member count to leadership or is there something better?", "should_trigger": true },
{ "query": "How many community metrics is too many?", "should_trigger": true },
{ "query": "We track 20 things on a dashboard and nobody reads it", "should_trigger": true },
{ "query": "I want to survey our community about how it feels to be here", "should_trigger": true },
{ "query": "how do I measure sentiment in a developer community without a text classifier", "should_trigger": true },
{ "query": "One maintainer does everything and I'm worried, how do I show that in numbers?", "should_trigger": true },
{ "query": "our numbers spiked right after launch, is that real growth?", "should_trigger": true },
{ "query": "how do I set thresholds for community metrics when I have no history", "should_trigger": true },
{ "query": "what counts as an active member and how should I define it", "should_trigger": true },
{ "query": "our two teams compute active users differently and argue about it every month", "should_trigger": true },
{ "query": "measure whether our community actually deflects support tickets", "should_trigger": true },
{ "query": "how do I report on a B2B customer community where members belong to accounts", "should_trigger": true },
{ "query": "we have 40 customer accounts in a Slack Connect channel, what should we measure", "should_trigger": true },
{ "query": "help me write a quarterly community health report", "should_trigger": true },
{ "query": "what should go in a community report for the board", "should_trigger": true },
{ "query": "how often should we review community metrics?", "should_trigger": true },
{ "query": "Are we allowed to publish a leaderboard of our top forum answerers?", "should_trigger": true },
{ "query": "our issue close-rate target is making people close things early", "should_trigger": true },
{ "query": "how do I measure a chat-only community with no public repo", "should_trigger": true },
{ "query": "is it worth buying a community analytics platform if I only have 3 hours a month", "should_trigger": true },
{ "query": "how do I stop losing history when our forum plan only keeps 90 days", "should_trigger": true },
{ "query": "we can't tell staff messages from member messages in our numbers", "should_trigger": true },
{ "query": "do we have a lurker problem or is that normal?", "should_trigger": true },
{ "query": "does the 90-9-1 rule apply to our community", "should_trigger": true },
{ "query": "how do I ask the people who left why they left", "should_trigger": true },
{ "query": "what's a reasonable response time target for our community?", "should_trigger": true },
{ "query": "Someone told me 24 hours is the industry standard for community response time, is that true?", "should_trigger": true },
{ "query": "what's a realistic baseline for our own community numbers", "should_trigger": true },
{ "query": "our community grew 38% last month, how should I present that?", "should_trigger": true },
{ "query": "we want a contributor retention chart by cohort", "should_trigger": true },
{ "query": "our cohort retention curves look like pure noise", "should_trigger": true },
{ "query": "measure how many first-time contributors come back a second time", "should_trigger": true },
{ "query": "what do I do when the numbers say everything's fine but people are burnt out", "should_trigger": true },
{ "query": "how do I prove the community is working", "should_trigger": true },
{ "query": "give me an honest read on whether our user forum is worth keeping open", "should_trigger": true },
{ "query": "diagnose why nobody answers questions in our support channel", "should_trigger": true },
{ "query": "who should own each number in a community report?", "should_trigger": true },
{ "query": "our engagement rate is 12% - is that good?", "should_trigger": true },
{ "query": "what should we stop tracking about our community", "should_trigger": true },
{ "query": "give me a scorecard for our open-source project's contributor base", "should_trigger": true },
{ "query": "Should we start a Discord for our product or not?", "should_trigger": false },
{ "query": "Where should we host a brand new developer community - Discord, Slack or Discourse?", "should_trigger": false },
{ "query": "How do I seed the first 50 members of a community that doesn't exist yet?", "should_trigger": false },
{ "query": "What go/no-go criteria should we set before launching a community?", "should_trigger": false },
{ "query": "Plan the first 90 days of our new developer community", "should_trigger": false },
{ "query": "Write a code of conduct for our open-source project", "should_trigger": false },
{ "query": "We need a moderation playbook and an enforcement ladder", "should_trigger": false },
{ "query": "Someone is harassing people in our Discord, what's the incident runbook?", "should_trigger": false },
{ "query": "How many moderators do we need and how should we roster them?", "should_trigger": false },
{ "query": "Design our issue label taxonomy and stale-issue policy", "should_trigger": false },
{ "query": "How should we route security reports away from the public tracker?", "should_trigger": false },
{ "query": "Set up a triage duty rotation for our maintainers", "should_trigger": false },
{ "query": "Write issue templates that cut down low-quality bug reports", "should_trigger": false },
{ "query": "Write a CONTRIBUTING.md for our repo", "should_trigger": false },
{ "query": "How do I create good first issues that actually get picked up?", "should_trigger": false },
{ "query": "Our clone-to-passing-tests path is broken for new contributors", "should_trigger": false },
{ "query": "How should we review and recognise someone's first pull request?", "should_trigger": false },
{ "query": "Design an ambassador program with perks for our top community members", "should_trigger": false },
{ "query": "What should the published selection criteria be for our developer champions?", "should_trigger": false },
{ "query": "How long should a champion's term run before renewal?", "should_trigger": false },
{ "query": "How do I measure our whole DevRel program across docs, events and content?", "should_trigger": false },
{ "query": "Which DevRel KPIs should I report to the CEO?", "should_trigger": false },
{ "query": "What attribution rule should I write for our developer advocacy work?", "should_trigger": false },
{ "query": "Build an event taxonomy for our docs and website tracking plan", "should_trigger": false },
{ "query": "How do I set up UTM discipline across our developer surfaces?", "should_trigger": false },
{ "query": "We need an identity spine linking docs visitors to product signups", "should_trigger": false },
{ "query": "Map the developer journey from discovery to advocacy for our main segment", "should_trigger": false },
{ "query": "Cut our developer audience into a few named, sized segments", "should_trigger": false },
{ "query": "Benchmark a competitor's DevRel motion from publicly observable signals", "should_trigger": false },
{ "query": "How does our competitor's community compare to ours on public signals?", "should_trigger": false },
{ "query": "Who decides what gets merged - help me write our governance model", "should_trigger": false },
{ "query": "Should we join a foundation or stay independent?", "should_trigger": false },
{ "query": "Plan a recurring monthly meetup with a standing speaker pipeline", "should_trigger": false },
{ "query": "How do I handle no-shows at our user group events?", "should_trigger": false },
{ "query": "Split our DevRel budget across events, content and community", "should_trigger": false },
{ "query": "Plan a quarter of developer content with owners and reviewers", "should_trigger": false },
{ "query": "Design sponsorship tiers for our open-source project", "should_trigger": false },
{ "query": "Which open-source projects should our company fund this year?", "should_trigger": false },
{ "query": "Rewrite our repository README so a newcomer understands the project", "should_trigger": false },
{ "query": "Audit our documentation structure against Diataxis", "should_trigger": false },
{ "query": "Write a quickstart that gets someone to their first success", "should_trigger": false },
{ "query": "Should DevRel report to marketing or to product?", "should_trigger": false },
{ "query": "Write a job description and interview loop for a developer advocate", "should_trigger": false },
{ "query": "How do I break into developer relations from a backend engineering role?", "should_trigger": false },
{ "query": "Plan our open-source launch day", "should_trigger": false },
{ "query": "How do we get onto GitHub Trending?", "should_trigger": false },
{ "query": "Run a health check on our Kubernetes cluster", "should_trigger": false },
{ "query": "Diagnose Azure resource health for our subscription", "should_trigger": false },
{ "query": "Add a /healthz endpoint to our service", "should_trigger": false },
{ "query": "Set up service health monitoring and alerting in our observability stack", "should_trigger": false },
{ "query": "Check the health of our network interfaces on these hosts", "should_trigger": false },
{ "query": "Analyse our codebase health and technical debt", "should_trigger": false },
{ "query": "Build a community marketing strategy to drive word of mouth", "should_trigger": false },
{ "query": "Publish our skill to the community marketplace", "should_trigger": false },
{ "query": "Design an EMR data model for a healthcare app", "should_trigger": false },
{ "query": "Our app submission health check failed in App Store Connect", "should_trigger": false },
{ "query": "How do we pick which conferences to sponsor next year?", "should_trigger": false },
{ "query": "Design a certification program for our developer academy", "should_trigger": false }
]
}
references/health-report-template.md›
# Community health report
## Contents
- [Template](#template)
- [Worked example](#worked-example)
- [Negative example](#negative-example)
## Template
```markdown
# Community health - {community name} - {period}
## Primary outcome
{The one thing this community exists to produce, in one sentence.}
## Metric sheet
| Metric | Pillar | Definition / formula | Source | Baseline | Threshold (origin) | Owner | Action if breached |
| ------ | ------ | -------------------- | ------ | -------- | ------------------ | ----- | ------------------ |
## Reading - {period}
| Metric | This period | Baseline | Direction | Note |
| ------ | ----------- | -------- | --------- | ---- |
## Findings
1. {Finding} - evidence: {numbers, dates, links}. Confidence: {high/medium/low, and why}.
## Actions
| Action | Owner | By when | Metric it should move |
| ------ | ----- | ------- | --------------------- |
## Watch list
{Metrics still baselining, known instrumentation gaps, events that distorted the period.}
## Next review
{Date, and what will be re-checked first.}
```
## Worked example
Illustrative numbers for an open-source project with a Discourse forum, at monthly cadence.
```markdown
# Community health - Acme Toolkit - August 2026
## Primary outcome
Peer support: users answer each other's usage questions, so the two maintainers spend their hours on code.
## Metric sheet
| Metric | Pillar | Definition / formula | Source | Baseline | Threshold (origin) | Owner | Action if breached |
| ------------------------------------------- | -------------- | --------------------------------------------------- | ------------------------- | ------------------- | ---------------------------------------------------------------------- | ----- | -------------------------------------------------- |
| Median time to first human response (forum) | Responsiveness | Question post → first non-bot reply | forum export | 9h (Apr-Jun median) | > 24h for two months (folklore cutoff, unsourced - recalibrate in Nov) | Priya | Recruit 2 answerers in the APAC timezone |
| Zero-reply question rate | Responsiveness | Question threads with no human reply within 7 days | forum export | 12% | > 20% (self-set, ~1.7x baseline) | Priya | Weekly sweep of unanswered threads |
| Non-maintainer share of answers | Activity | Answers not written by the 2 maintainers | forum export + staff flag | 46% | < 35% (self-set, ~0.75x baseline) | Priya | Recognise top answerers; stop answering first |
| Active participants | Activity | Distinct non-bot people posting in the month | forum export | 74 | < 50 (self-set, ~0.7x baseline) | Priya | Diagnose before acting; check for event distortion |
| First-time posters returning within 90 days | Funnel | Second post by a first-time poster, per join cohort | forum export | 22% | < 15% (self-set, ~0.7x baseline) | Sam | Review first-answer time for newcomers |
| Contributor absence factor | Concentration | Smallest number of people producing 50% of commits | git history | 2 | 1 (single point of failure) | Sam | Open the reviewer rung to the top 3 contributors |
| Recommendability | Sentiment | "Would you recommend this project?", NPS-style | quarterly pulse | +21 (n=38) | drop > 15 points (self-set) | Sam | Interview 5 detractors |
## Reading - August 2026
| Metric | This period | Baseline | Direction | Note |
| ------------------------------- | ------------ | -------- | --------- | ------------------------------------------------- |
| Median time to first response | 31h | 9h | worse | Both maintainers were at a conference 8-15 Aug |
| Zero-reply question rate | 24% | 12% | worse | Concentrated in the same two weeks |
| Non-maintainer share of answers | 51% | 46% | better | Two new regulars answered 14 threads between them |
| Active participants | 96 | 74 | up | Release 3.0 landed 6 Aug - burst, not trend |
| First-time posters returning | 19% | 22% | flat | n=31 first-timers; within noise |
| Contributor absence factor | 2 | 2 | flat | - |
| Recommendability | not surveyed | +21 | - | Next pulse: September |
## Findings
1. Responsiveness degraded for exactly one two-week window while both maintainers were away - evidence: 31h median overall, 11h median outside 8-15 Aug. Confidence: high, the split is clean. This is a coverage gap, not a decline.
2. Two new regulars now answer without being asked - evidence: non-maintainer answer share up 5 points, 14 threads. Confidence: medium, one month of data.
3. The August participant spike is release-driven - evidence: 62% of new posts reference 3.0, spike starts the day after release. Confidence: high. Do not carry it into the trend line.
## Actions
| Action | Owner | By when | Metric it should move |
| ---------------------------------------------------------------------------- | ----- | --------- | --------------------------------------- |
| Ask the two new regulars to hold answer duty during maintainer absences | Priya | 12 Sep | Zero-reply rate, time to first response |
| Publish an "away" banner with expected response time before the next absence | Priya | Next trip | Newcomer experience |
| Open the reviewer rung and invite the top 3 contributors | Sam | 30 Sep | Contributor absence factor |
## Watch list
Newcomer-experience survey items still unasked (first wave in September). Chat venue not instrumented - no export on the current plan, so chat activity stays anecdotal.
## Next review
28 September 2026 - re-check time to first response outside travel windows, and whether the two new regulars stayed.
```
What makes this report work:
- Every number is read against the community's own baseline.
- Each threshold carries its origin, so nobody mistakes a working convention for a standard.
- The burst is labeled instead of celebrated.
- The small-N cohort is called out as noise.
- Every finding ends in an owned action.
## Negative example
```markdown
# Community Health Dashboard - Q3
- Total members: 4,812 (+312 this quarter) ↑
- Messages sent: 18,204 (+9%) ↑
- Discord sentiment: 78% positive (auto-analysis)
- Engagement rate: 12%
- Top contributors: @dev_alice (412 msgs), @bob_builds (388 msgs), @carol_hacks (301 msgs)
Overall: community is healthy and growing. Recommend continued investment.
```
Why it fails:
- Total members and message counts only go up. Neither can detect a community going quiet where it matters.
- No baseline, no period comparison, no note that a launch or event happened in the quarter.
- "Engagement rate" is undefined - nobody can recompute it or tell what would move it.
- Automated sentiment on developer chat scores blunt bug reports as negative and jokes as positive. There is no survey behind the 78%.
- A public per-member message leaderboard rewards volume and creates privacy exposure.
- No responsiveness metric at all: 18,204 messages are compatible with every single question going unanswered.
- No owner, no threshold, no action - the report cannot be wrong, so it cannot be useful.
references/metric-catalog.md›
# Metric catalog
Definitions, formulas, data sources and published guidance for the four pillars. Pick at most 8 metrics total, at least one per pillar.
Definitions here are quoted from published metric pages. The _readings_ attached to them are marked as published or as this catalog's own interpretation, because the published model states almost no numeric guidance of its own.
## Contents
- [What is actually published](#what-is-actually-published)
- [Activity](#activity)
- [Responsiveness](#responsiveness)
- [Contributor funnel](#contributor-funnel)
- [Concentration and risk](#concentration-and-risk)
- [Sentiment](#sentiment)
- [The starter set](#the-starter-set)
- [Unsourced practitioner thresholds](#unsourced-practitioner-thresholds)
- [Sources](#sources)
## What is actually published
Quote these verbatim. Everything else in this file is definition plus interpretation.
- "it's important not just to look at the numbers, but also at the trends"
- "we do not compare projects against each other, but instead, we expect teams to use their metrics to make improvements"
- "every project is a little different, so it's important to interpret the metrics in light of a project's needs"
- "automation and bot activity can influence the usefulness of several metrics in this model, in particular Time to First Response and Change Request Closure Ratio. These metrics are only applicable when these activities are primarily driven by humans, not automated responses."
- "Make sure to exclude responses from bots or other automated systems when measuring genuine community engagement." (Time to First Response page)
- "Release frequency is highly variable because software projects belong to different industries and fields, and no two projects have the same needs."
- "A lower Contributor Absence Factor indicates higher dependency on fewer contributors, posing a risk if these individuals leave."
- The only numeric guideline anywhere in the model is a case-study line from VMware: "Our internal guideline for projects is that every PR should receive a response from a human within two business days." One company's internal rule - cite it that way.
Not published, though often repeated as if it were: a target for the change request closure ratio (the page only says projects should "keep up with PRs and resolve them in a timely manner"), and any release-gap threshold.
## Activity
| Metric | Definition | Formula | Source of data |
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | -------------------------------------- |
| Active participants | Distinct humans who posted, commented, committed or reviewed in the period | count distinct non-bot identities after identity merge | venue analytics, forge API |
| Non-staff share of activity | Share of messages/answers not written by employees or maintainers | non-staff activity ÷ total activity | same, plus a staff account flag |
| Channel/venue concentration | Where the conversation actually lives | activity per channel, ranked | venue analytics |
| Collaboration platform activity | "The count of activities across digital collaboration platforms (e.g., GitHub, GitLab, Slack, email) used by a project", carrying timestamp, sender, bot flag, threaded/non-threaded type | count per platform per period | exports or a community-analytics stack |
| Burstiness | "How are short timeframes of intense activity, followed by a corresponding return to a typical pattern of activity, observed in a project?" | deviation from the trailing baseline; volatility bands | any activity series |
Reading notes: raw message volume is not health - its documented use is finding the lowest-barrier channel and mapping where conversation lives. Named burst causes: releases, conferences, hackathons, critical bugs, media coverage, mentorship programs. Label the burst in the report so nobody reads it as growth.
## Responsiveness
| Metric | Definition | Formula | Source of data |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -------------------------------- |
| Time to first response | "The amount of time between when an activity was opened (e.g. Issue or Change Request) and when it received the first response from a human" | median elapsed time, bots excluded | forge API; venue export for chat |
| Unanswered rate | Share of questions/threads that got zero human replies | zero-reply threads ÷ total question threads | forge API, venue export |
| Change request closure ratio | "The ratio between the total number of open change requests during a time period versus the total number of change requests closed in that same period" | closed (merged + rejected) ÷ opened, same window | forge API |
Reading notes: the only attributable response-time figure is VMware's internal two-business-day guideline, quoted in the CHAOSS starter model - a reference point from one company, not a standard. Both metrics are meaningless with bots included.
A closure ratio persistently below 1 means the backlog outruns the team - that reading is this catalog's, not published guidance.
## Contributor funnel
| Metric | Definition | Formula | Source of data |
| ------------------------------------- | ------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------ |
| First-time participants | People whose first activity falls in the period | count of identities with no prior activity | forge API, git history, venue export |
| Second-contribution rate | Share of first-time contributors who come back within 90 days | returning ÷ first-time, per join cohort | same |
| Cohort retention | Share of a join cohort still active at day 30 / 90 | active-at-N ÷ cohort size | same |
| One-and-done rate | Share of all-time contributors with exactly one merged contribution | single-contribution contributors ÷ all contributors | git history |
| First-PR merge rate and time-to-merge | External contributors only, tracked separately from maintainers | median per period | forge API |
Reading notes: a high one-and-done rate is normal in open source. It is a finding only when it moves after a change you made. Maintainers' own PRs are always faster and will mask the number that matters if mixed in.
Cohort width is a decision, not a default. Monthly join cohorts are the usual starting point.
Weekly or daily cohorts in a community that gains a few dozen people a month produce single-digit groups whose retention curves are pure noise. Widen the window until each cohort has enough people to read, state the window next to the number, and keep it stable between reports - changing the window changes the trend.
## Concentration and risk
| Metric | Definition | Formula |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Contributor absence factor | "The smallest number of people that make 50% of contributions" | rank contributors by contribution count, count how many are needed to cross half the total |
| Elephant factor | "The minimum number of companies whose employees contribute a specified percentage of the total commits in a software repository" | same procedure grouped by employer; worked example: commit counts 1000/433/343/332/202/90/42/33 → 50% threshold is 1,237.5, reached by the top two companies → factor 2 |
Reading notes: higher is safer for both. An absence factor of 5 means the project survives several departures, while a factor of 1 is a single point of failure.
A low elephant factor means one or two employers effectively control the project. Neither accounts for knowledge distribution - two people with disjoint expertise are not interchangeable.
## Sentiment
| Metric | Definition | Collection |
| ------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Recommendability | "How likely is it that you would recommend this community/project to other people?" | Likert item, scored NPS-style (promoters minus detractors) |
| Newcomer experience | "How well does an open source community attend to welcoming newcomers?" | Likert items ("I feel welcome in the community", "The community treats new members well") plus open-ended obstacle questions; corroborated by whether first-timers return |
| Maintainer burnout | Chronic-stress state: energy depletion, cynicism, reduced efficacy | Likert items on energy, workload, feeling heard, psychological safety; trace signals: sustained contribution then abrupt stop, contribution concentrated in a few people |
The question bank, cadence, sampling rules and privacy limits live in the survey-instruments reference linked from SKILL.md.
## The starter set
When the user has no framework at all and the community is code-centric, start from the four-metric published minimum and add one sentiment item. Instrument them in this order - decisions bought per hour of collection, matching the pillar ranking in SKILL.md:
- efficiency: time to first response > change request closure ratio > contributor absence factor > release frequency > recommendability
- effort: time to first response == change request closure ratio == release frequency (one forge query each, near-zero) > contributor absence factor (needs an identity mapping first) > recommendability (a survey to field, half a year between readings)
The three-way effort tie is real: all three are single queries against the same API, and no ordering between them survives contact with a different forge. Recommendability sits last on efficiency and stays worth adding, because it is the only item on the list that can contradict the other four.
1. Time to first response (responsiveness)
2. Change request closure ratio (responsiveness/throughput)
3. Contributor absence factor (concentration)
4. Release frequency - count of releases including point releases. No published gap threshold exists ("highly variable... no two projects have the same needs"). What the source does flag is that delayed security releases leave users with no easy upgrade path. Meaningless for docs-only repositories.
5. Recommendability, asked twice a year (sentiment)
For a chat/forum community with no code, swap 2 and 4 for active participants and non-staff share of activity.
## Unsourced practitioner thresholds
These circulate widely in community-management material with no published evidence behind them. Use as starting hypotheses. Replace with the community's own trailing median after three periods, and label them as unsourced whenever you quote them.
| Metric | Commonly quoted "healthy" | Commonly quoted "act now" |
| --------------------------------- | ------------------------- | ------------------------- |
| Support response time | under 24h | over 72h |
| Unanswered questions | under 10% | over 25% |
| New-member 7-day retention | over 40% | under 20% |
| Monthly active ratio | over 20% | under 10% |
| Moderator/staff share of messages | under 30% | over 50% |
Commonly cited warning signs from the same material, which do hold up as qualitative flags:
- Most posts come from the company team.
- Questions sit unanswered past a day.
- The same five people account for most engagement.
- New members stop after their introduction.
## Sources
- CHAOSS, Starter Project Health metrics model - <https://chaoss.community/kb/metrics-model-starter-project-health/>
- CHAOSS, Time to First Response - <https://chaoss.community/kb/metric-time-to-first-response/>
- CHAOSS, Contributor Absence Factor - <https://chaoss.community/kb/metric-contributor-absence-factor/>
- CHAOSS, Release Frequency - <https://chaoss.community/kb/metric-release-frequency/>
- CHAOSS, Elephant Factor - <https://chaoss.community/kb/metric-elephant-factor/>
- CHAOSS, Burstiness - <https://chaoss.community/kb/metric-burstiness/>
- CHAOSS, Collaboration Platform Activity - <https://chaoss.community/kb/metric-collaboration-platform-activity/>
- CHAOSS, Newcomer Experience - <https://chaoss.community/kb/metric-newcomer-experience/>
- Jakob Nielsen, "Participation Inequality: The 90-9-1 Rule" (2006) - <https://www.nngroup.com/articles/participation-inequality/>
references/survey-instruments.md›
# Survey instruments for community sentiment
Sentiment is the pillar most often faked with automated text scoring. The published instruments are surveys, corroborated by behavioural traces - ask people, then check whether their behaviour agrees.
## Contents
- [Pulse survey](#pulse-survey-quarterly)
- [Annual survey](#annual-survey)
- [Maintainer burnout check](#maintainer-burnout-check)
- [Lapsed-member question](#lapsed-member-question)
- [Sampling and response rates](#sampling-and-response-rates)
- [Confidence labels on qualitative findings](#confidence-labels-on-qualitative-findings)
- [Privacy limits](#privacy-limits)
- [Sources](#sources)
## Pulse survey (quarterly)
Two items. Anything longer will not be answered at a quarterly cadence.
1. "How likely is it that you would recommend this community/project to other people?" - Likert scale from "Not at all likely" through "Neutral" to "Extremely likely".
2. "What is the most frustrating thing about participating here?" - free text, optional.
Scoring item 1: split respondents into detractors, passives and promoters, then report **promoters % minus detractors %**. The published CHAOSS page writes the subtraction the other way round, which flips the sign. State the direction on every chart so a positive number always means "more promoters than detractors".
Corroborate the score with implicit signals from the same period - responsiveness and contribution-acceptance rates. A score that disagrees with behaviour usually means the survey reached a different population than the one doing the work.
## Annual survey
Add the newcomer and belonging items once a year, filtered to people who joined in the last 12 months. When they ride inside a larger developer survey, `samber/developer-relations-skills@developer-survey-design` owns recruitment, length, privacy and publication:
- "I feel welcome in the community." (Likert)
- "The community treats new members well." (Likert)
- "What did you need when you arrived that you could not find?" (free text)
- "What stopped you from participating more?" (free text)
Cross-read against behaviour: do first-time participants return within 90 days? Agreement between the survey and the funnel is what makes either credible.
## Maintainer burnout check
Ask the small group carrying the load, separately and privately. Likert items on:
- energy and emotional drainage,
- fatigue and time away,
- thoughts of stepping back or leaving,
- workload versus capacity,
- whether their contribution feels valued,
- whether they can raise a concern safely.
Trace signals to watch beside the survey: sustained high contribution ending abruptly, contribution concentrated in very few people, expressed exhaustion or cynicism in public threads.
Burnout is the finding that activity metrics hide best - a community can post record numbers in the quarter its two maintainers decide to quit.
## Lapsed-member question
Send a one-question note to members who were active and then went silent for 60+ days: "You stopped participating a while back - what changed?" The people who left hold the answer to why, and they are systematically missing from every other instrument.
Keep it opt-out-friendly and send it once. A follow-up campaign to people who already left reads as pressure.
## Sampling and response rates
- Volunteer-community response rates are low. Below roughly 30 responses, treat the result as qualitative evidence with dates attached, not as a percentage.
- Report the response rate next to the score, always. A 68 built from 9 answers is a conversation starter, not a number.
- Keep the population definition stable between waves (e.g. "members active in the last 90 days"), otherwise the trend measures your sampling, not sentiment.
- Never mix a pulse wave sent to everyone with one sent to power users. Segment instead.
## Confidence labels on qualitative findings
Borrowed from customer-research practice, because a sentiment finding travels further than the sample behind it. Label every qualitative finding before presenting it:
| Label | When |
| ------ | --------------------------------------------------------------------------------------- |
| High | five or more independent people said it, across more than one channel or wave |
| Medium | three to four independent people, or one strong wave with agreeing behavioural evidence |
| Low | one or two people, or a single free-text field with no behavioural corroboration |
Below roughly five independent data points on a theme, call it a hypothesis and name the cheapest way to test it. A single loud thread is not a community sentiment, and presenting it as one is how a community team ends up rebuilding something nobody else minded.
An adjacent instrument worth knowing but not adopting blind: the Sean Ellis product-market-fit item ("How would you feel if you could no longer use this?", with a 40% "very disappointed" line popularised by Superhuman). It measures product fit, not community health, and no community-specific validation of the 40% figure exists. If you adapt the wording to a community, say that it is adapted and drop the benchmark.
## Privacy limits
Published metric guidance repeats the same caution: using and disseminating health metrics can create privacy violations, with data-protection law and platform terms named as the exposure. In practice:
- Report sentiment in aggregate only. Never attribute a score to a named member.
- Keep survey responses separable from identity, and say so in the invitation - anonymity is what makes the burnout and lapsed-member instruments work at all.
- Check the venue's terms before bulk-exporting message content for any kind of analysis.
## Sources
- CHAOSS, Project Recommendability - <https://chaoss.community/kb/metric-project-recommendability/>
- CHAOSS, Project Burnout - <https://chaoss.community/kb/metric-project-burnout/>
- CHAOSS, Newcomer Experience - <https://chaoss.community/kb/metric-newcomer-experience/>
scripts/contributor-absence-factor.sh›
#!/usr/bin/env bash
# Compute the contributor absence factor: the smallest number of people
# producing a given share (default 50%) of commits in a git repository.
# Higher is safer; 1 means a single point of failure.
#
# Requires: bash 3.2+ (macOS default) or bash 4/5 (Linux), plus git, grep,
# sort, uniq, awk.
# Portable across macOS (BSD userland) and Linux (GNU coreutils): uses plain
# POSIX ERE and POSIX awk, no GNU-only flags.
set -euo pipefail
usage() {
cat <<'EOF'
Usage: contributor-absence-factor.sh [-r REPO] [-s SINCE] [-p PERCENT] [-x EXCLUDE_REGEX]
-r REPO path to the git repository (default: current directory)
-s SINCE only count commits after this date, e.g. "12 months ago" (default: all history)
-p PERCENT share of commits to cover, 1-100 (default: 50)
-x EXCLUDE_REGEX case-insensitive regex of author names to drop, e.g. "bot|\[bot\]|dependabot"
(default: bot|\[bot\]|dependabot|renovate|github-action)
-h show this help
Output: the absence factor, the covering contributors, and the total commit count.
Bots are excluded by default because automated commits inflate the factor and hide real concentration.
EOF
}
repo="."
since=""
percent=50
exclude='bot|\[bot\]|dependabot|renovate|github-action'
while getopts ":r:s:p:x:h-:" opt; do
case "$opt" in
r) repo="$OPTARG" ;;
s) since="$OPTARG" ;;
p) percent="$OPTARG" ;;
x) exclude="$OPTARG" ;;
h) usage; exit 0 ;;
-) [ "$OPTARG" = "help" ] && { usage; exit 0; }; echo "unknown option --$OPTARG" >&2; exit 2 ;;
:) echo "option -$OPTARG needs a value" >&2; exit 2 ;;
*) usage >&2; exit 2 ;;
esac
done
command -v git >/dev/null || { echo "git not found" >&2; exit 1; }
git -C "$repo" rev-parse --git-dir >/dev/null 2>&1 || { echo "not a git repository: $repo" >&2; exit 1; }
case "$percent" in ''|*[!0-9]*) echo "-p must be an integer" >&2; exit 2 ;; esac
[ "$percent" -ge 1 ] && [ "$percent" -le 100 ] || { echo "-p must be between 1 and 100" >&2; exit 2; }
log_args=(-C "$repo" log --no-merges --pretty=format:%aN)
[ -n "$since" ] && log_args+=(--since="$since")
# Identity note: this counts author names as git records them. Merge aliases of the
# same person before trusting the number, or the factor reads higher than reality.
git "${log_args[@]}" \
| grep -Eiv "$exclude" \
| sort | uniq -c | sort -rn \
| awk -v pct="$percent" '
{ count[NR]=$1; name[NR]=substr($0, index($0,$2)); total+=$1 }
END {
if (total == 0) { print "no commits matched"; exit 1 }
target = total * pct / 100
running = 0
for (i = 1; i <= NR; i++) {
running += count[i]
printf " %2d. %-40s %6d commits\n", i, name[i], count[i]
if (running >= target) { factor = i; break }
}
printf "\nTotal commits: %d (from %d contributors)\n", total, NR
printf "Contributor absence factor at %d%%: %d\n", pct, factor
}'
SKILL.md›
---
name: developer-community-health
description: Designs and runs a developer community health measurement framework: activity, responsiveness, contributor-funnel and sentiment metrics, honest instrumentation, baseline-derived thresholds, and a report that ends in decisions. Use whenever someone asks how to measure their developer community, which community health metrics to track, whether their Discord, Slack or forum is dying, why the community feels quiet, or wants a community health dashboard, contributor funnel, community KPI set or engagement report - even if they only say "is our community doing OK". Do NOT use for launching a community (samber/developer-relations-skills@developer-community-launch) or company-wide devrel KPIs (samber/developer-relations-skills@devrel-metrics).
license: MIT
metadata:
author: Samuel Berthe
version: "1.0.1"
---
# Developer Community Health
You are a community measurement analyst. Turn "how is the community doing?" into a small set of metrics the team can actually collect, read against a baseline, and act on.
Community dashboards fail in one of two ways:
- They count what the platform happens to expose (members, messages) and call it health.
- They list thirty metrics nobody refreshes after month two.
A good framework is short, instrumentable with the access the team has today, and wired to a decision.
## Route before measuring
Check what the user actually needs:
- Deciding whether or where to start a community → `samber/developer-relations-skills@developer-community-launch`.
- Writing a code of conduct, enforcement ladder or incident runbook → `samber/developer-relations-skills@developer-community-moderation`.
- Designing the issue/PR triage workflow itself (labels, templates, stale policy) → `samber/developer-relations-skills@oss-issue-triage`.
- Building the first-contribution path (CONTRIBUTING, good first issues, first-PR review) → `samber/developer-relations-skills@oss-contributor-onboarding`.
- Recruiting and rewarding super-users → `samber/developer-relations-skills@developer-champions`.
- Getting the community's feature requests to a product decision and back → `samber/developer-relations-skills@developer-feedback-loop`.
- Running the yearly survey that carries the sentiment items - sampling, length, privacy, publishing the percentages → `samber/developer-relations-skills@developer-survey-design`. Define the items here. Field them there.
- Measuring the whole DevRel program across docs, content, events, and product → `samber/developer-relations-skills@devrel-metrics`.
- Running the vendor's Stack Overflow tag itself - staff answering, backlog triage, canonical Q&A → `samber/developer-platform-skills@stackoverflow-playbook`. Measure it here as one channel among several.
- Building the tracking plan underneath the numbers - event taxonomy, UTM discipline, identity spine across web and docs → `samber/developer-relations-skills@devrel-analytics`. Define community metrics here. Instrument them there.
This skill measures a community that already exists - a chat/forum space, an open-source contributor base, or both. Say which skill fits and stop if it is not this one.
## Interview
Ask one question at a time, multiple-choice when the options are knowable. Stop once you can fill the metric sheet. Do not run the list mechanically.
**Memory: Store a few info about the current task or project context. When memory lives in a file, use `devrel-context.md`; if a different memory system is in use, rely on that instead. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**
1. What is the community - a chat/forum space, an open-source contributor base, or both? Which venues exactly?
2. Is this B2B (a paid developer tool, members belong to customer accounts) or individual-adoption / open source (members join as themselves)?
3. What is the one outcome this community exists for: support deflection, activation, product feedback, contribution, retention, or word of mouth?
4. How big is it today, and how much activity happens in a typical week? Rough numbers are fine.
5. What data can you get today without asking anyone for budget or access: forge API, git history, the venue's built-in analytics, exports, a community-data platform?
6. Who owns the numbers and who reads the result?
7. What is the effort ceiling: hours per month, headcount, and whether anyone can be asked for new tool budget or new access?
8. By what date does the first reading have to exist - a board date, a planning cycle, or no deadline at all?
9. Do you want a one-off answer to a live question, or a sheet that compounds into a trend nobody has to rebuild?
10. What are you already tracking, and what has that number ever caused you to change?
11. Is there a specific worry behind this - a drop, a quiet channel, a burned-out maintainer, a board asking for justification?
12. Any privacy, legal or venue-terms constraints on exporting member data?
Questions 5, 10 and 12 are not the same question:
- Question 5 establishes what the team _can_ collect.
- Question 10 establishes what it already _does_ collect.
- Question 12 establishes what it _may_ collect.
Ask all three. Skip one and the framework dies at instrumentation or at legal.
Answers to 7, 8 and 9 re-rank the two menus below, and say which moved what:
- A hard date promotes the metrics readable from history on day one (responsiveness, activity) and demotes anything needing three periods before it says anything.
- A compounding mandate promotes snapshotting into your own store from day one and promotes the funnel pillar, whose value only arrives after several cohorts.
- A low effort ceiling deletes the community-data-platform rung outright rather than parking it at the bottom of the sheet.
If the user arrives with a symptom ("nobody posts anymore") rather than a mandate, run the workflow in diagnosis order: measure the pillar the symptom points at first, then fill the rest.
## Workflow
Two rules shape the work before it starts. CMX's own instruction for the SPACES model is to "just focus on one objective." Hence a single primary outcome in step 2.
Steps 5-8 are one pass over the same artifact, the metric sheet: a row is finished when it carries a formula, a baseline, a labeled threshold and an owned action - the same parts the Pass threshold checks.
1. Route, then interview until the outcome and the data access are clear.
2. Name the single primary outcome in the SPACES vocabulary (support, product feedback, acquisition/advocacy, content/contribution, engagement, success), so it is a recognized category rather than an ad-hoc phrase. Every metric on the sheet must plausibly move when that outcome moves.
3. Pick at most 8 metrics (this skill's own ceiling), at least one per pillar. Fewer is better: a sheet of 5 that gets refreshed beats 20 that rot.
4. Reality-check instrumentation before committing to any metric - see Instrumentation reality check. Rank candidates by what each returns per hour of collection, and drop anything whose return does not justify a standing hour a month (a self-set ceiling, not a standard).
5. Write each metric's formula next to its name, in the sheet itself.
6. Capture a baseline: pull at least three prior periods. Without history, mark the metric "baselining" and set its threshold at the next review.
7. Set thresholds from that baseline, and label every number by where it came from - see Sourced, self-set, folklore.
8. Attach a named action and an owner to every threshold breach. A metric with no action attached is decoration - cut it.
9. Set the review cadence - monthly by default, weekly only at high volume (both self-set) - and write the report.
10. Present the report section by section, validating each with the user before moving on.
## The four pillars
Cover all four before publishing a sheet. A community can look busy and be dying, or look quiet and be perfectly healthy for its size - one pillar alone never tells you which. The order below is which pillar earns its first metric first when the hours arrive one at a time, not permission to drop the last one.
Rank the pillars by decisions bought per hour of collection, not by how easy each is to pull:
- efficiency: responsiveness > contributor funnel > activity > sentiment
- effort: responsiveness == activity (near-zero, one scripted query each) > contributor funnel (identity mapping plus three periods of history) > sentiment (a survey to write, field and interpret - a quarter before it reads)
- value: responsiveness == sentiment (each catches a problem before the volume numbers move - the newcomer who got no answer, the maintainer about to quit) > contributor funnel (tells you whether anything you changed worked) > activity (says something is happening, never that it is healthy)
- compliance cost: activity == responsiveness == contributor funnel (aggregate counts over data the venue already exposes, no review) > sentiment (person-level responses, consent, an anonymity threshold below which you cannot publish, and answers you cannot un-collect)
The three-way compliance tie is genuine and not an evasion: none of those three collects anything a member did not already post in the open, so none triggers a review. Sentiment is alone on that axis because it creates data that did not exist before you asked.
| Pillar | Question it answers | Metrics, most decisions per hour first |
| ------------------ | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Responsiveness | Does a person who shows up get an answer? | median time to first human response, % of questions with zero replies, change-request closure ratio |
| Contributor funnel | Do newcomers become regulars? | contributor absence factor, first-time participants per period, second-contribution rate within 90 days (a self-set window), active-member retention by join cohort |
| Activity | Is anything happening, and who makes it happen? | share of activity from non-staff, active contributors/posters per period, burstiness vs steady trend, channel concentration |
| Sentiment | Do people want to be here, and are the load-bearing people still standing? | burnout signals on maintainers, recommendability (NPS-style survey item), newcomer welcome items |
Efficiency starves sentiment: it is the only pillar high on value and high on effort, so it loses every round and a sheet built on the ratio alone never asks anyone anything. Promote it to first anyway when the worry behind the request is about people rather than volume - a maintainer showing burnout signals, a newcomer complaining, or a quiet room whose activity numbers look fine. That case is exactly the one the numbers cannot reach.
This order is a default, not a law: it assumes a code-centric community whose forge is readable and whose team is small. Re-rank it against what the interview already surfaced:
- A chat-only community has no cheap responsiveness pull, so promote activity.
- A team that already owns a community-data platform gets the funnel pillar for near-zero effort, so lead with it.
- A community whose primary outcome is contribution reads the funnel as its headline regardless of ratio.
See [./references/metric-catalog.md](./references/metric-catalog.md) for each metric's definition, formula, data source and the published guidance behind it. See [./references/survey-instruments.md](./references/survey-instruments.md) for the sentiment side: question bank, cadence, sampling and privacy limits.
Keep only metrics that survive the "so what" test: if the number moved 20% next month, would anyone do anything differently? If not, it is not a health metric for this community.
Name the funnel stages with one vocabulary and keep it. Orbit's explorer → participant → contributor → advocate ladder is a published option (the model is no longer actively developed - cite it as a stable framework), and a core/regular/casual split is another. Two vocabularies in one sheet make every cohort definition ambiguous.
## Sourced, self-set, folklore
Every number in a health report arrived by one of three roads, and a table cell hides which. A published figure, your own working convention and an unsourced threshold all look identical on the page - until someone defends the wrong one in the next budget meeting. Label the origin every time.
| Kind | Examples in this skill | How to present it |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Published and attributable | 90-9-1 participation inequality (Nielsen, 2006); VMware's guideline that every PR gets a human response within two business days, quoted inside the CHAOSS Starter Project Health model; the contributor absence factor definition and its worked example | name the source _and its scope_ - VMware's line is one company's internal rule, not an industry standard |
| Self-set baseline | this community's trailing median over three periods; the eight-metric ceiling; the ~30-event small-N floor; the 90-day second-contribution window; monthly review cadence; the hour-a-month instrumentation bar | say it is a working convention and what evidence would change it |
| Folklore | 24-hour response, 20% monthly-active ratio, 40% seven-day new-member retention, 30% staff share of messages | quote only as a hypothesis, marked unsourced, and replace it with the community's own trailing median after three periods |
CHAOSS's Starter Project Health model - the vendor-neutral reference set most of this field quotes - publishes no numeric cutoff of its own. It says the opposite: "it's important not just to look at the numbers, but also at the trends", and "we do not compare projects against each other, but instead, we expect teams to use their metrics to make improvements". So when someone hands you an "industry-standard" community benchmark, it did not come from the reference set - ask where it did come from, and label it accordingly.
## Thresholds and baselines
Thresholds mature with history. Match the mechanism to how many periods the community has:
- **No history yet.** Mark the metric "baselining" and read raw numbers. Below roughly 30 events in a period (this skill's own floor), report the list with dates instead of a rate - small-N percentages swing wildly and invite bad decisions.
- **Three or more periods** (this skill's own minimum). Set the threshold from the trailing median, and compare the community only to its own past - never to another community. Size, ecosystem, product maturity and venue all change what "normal" looks like.
- **Six or more periods.** Replace the fixed cutoff with a deviation band: flag a reading that leaves its own trailing range, defined either as a stated percentage band or as a spread of standard deviations around the rolling mean - pick one and write it in the sheet. A fixed number set once and never revisited fires every month until the team stops reading the report.
Two reading rules apply at every stage:
- Distinguish a burst from a trend. Releases, conferences, hackathons and incidents all spike activity, and a launch month compared against a quiet month is the most common false trend in community reporting.
- Expect participation inequality as the baseline state, not a finding: roughly 90% of members lurk, 9% contribute occasionally, 1% produce most of the activity (Nielsen, 2006). Target growth in the absolute number of active people, not the elimination of lurkers.
## Instrumentation reality check
Rank each candidate metric by what its data source returns per hour spent collecting it, not by what the source costs. The two orderings only coincide when the community lives on a forge:
- efficiency: forge and git history > venue analytics panel > community-data platform
- effort: forge and git history (near-zero once scripted) > venue analytics panel (an hour every month, forever) > community-data platform (a week to stand up, then a standing job)
- value: community-data platform (cross-venue identity, retained history, cohorts that survive a venue change) > forge and git history (the funnel and responsiveness pillars, exact) > venue analytics panel (activity only, coarse, capped by the venue's retention)
- compliance cost: forge and git history (public activity, no review) > venue analytics panel (aggregate reads inside the venue's own terms) > community-data platform (bulk member-data export, venue-terms review, a retention decision, and ingestion you cannot reverse)
1. **Forge API or plain git history** - contributors per period, first-time contributors, issue/PR volumes, time to first response, time to merge, releases. Exact, scriptable, and immune to the venue's retention limits. Leads on efficiency wherever there is code. Returns nothing for a chat-only community, which is the single case that moves it off the top.
2. **The venue's own analytics panel**, read into a sheet monthly. Coarse, and brittle when export formats or retention limits change - but it is the only source of the activity pillar for a chat or forum community the forge cannot see, which is what keeps it above tooling.
3. **A community-data platform or an open-source community-analytics stack.** Highest value on the list and highest effort, so the efficiency order starves it every round. Promote it when three conditions hold together: several venues, a named owner for the pipeline, and a question the rungs above have already failed to answer twice.
Default rung: 1, plus 2 when the community is not code-centric. Move up one only when the metric you actually need is uncomputable below, never because the higher rung looks more rigorous.
Delete a rung the interview ruled out instead of leaving it as future work: a team with no budget and no pipeline owner has no rung 3, and a chat-only community has no rung 1. A ruled-out rung parked at the bottom of the sheet comes back as scope at the next review.
Four traps silently corrupt the numbers:
- **Bots.** Exclude bot accounts from every responsiveness metric. A welcome bot replying in four seconds makes time-to-first-response look excellent while no human ever answered. Keep an explicit bot list and filter at query time.
- **Identity.** The same person appears as a forge handle, a chat handle, a commit email and a forum account. Without a mapping, unique-contributor counts inflate and concentration measures break. A hand-maintained mapping file is enough at low volume.
- **Retention windows.** Rented venues cap history on lower tiers, so a metric defined today may be uncomputable for last quarter. Snapshot monthly into your own store from day one. Backfill is often impossible.
- **Staff attribution.** Flag employee and maintainer accounts. "Share of answers written by non-staff" and "moderator share of messages" are the metrics that separate a community from a support queue, and both need that flag.
Privacy is a hard gate, not a footnote. Health measurement can create real privacy and compliance exposure.
- Aggregate anything person-level.
- Avoid public per-member leaderboards without consent.
- Check the venue's terms before bulk-exporting message content.
Optional helper: `scripts/contributor-absence-factor.sh` computes the contributor absence factor from a local git repository when a shell and git are available. Run `bash scripts/contributor-absence-factor.sh --help` for usage. Compute it by hand from any contributor ranking if you cannot run scripts.
## B2B versus individual-adoption communities
The pillars hold for both. The units and the reading change.
**B2B (members belong to customer accounts).** The unit is the account, not the person. Count how many customer accounts have at least one active member, and whether the accounts you care about are represented.
Low volume is normal: a 40-account community produces small-N numbers everywhere. Lean on the raw list and on named-account conversations rather than rates - a sentiment survey with six responses tells you less than three interviews. Deflection and expansion signals matter more than raw growth.
**Individual adoption and open source.** Volume makes the funnel and the lurker ratios readable, so cohort retention, second-contribution rate and concentration measures do real work here. Sentiment comes from surveys with honest response-rate caveats. A high one-and-done contributor rate is normal in open source - it becomes a finding only when it moves after a change you made.
**Same for both:**
- Bot exclusion.
- Baseline-derived thresholds.
- An action attached to every metric.
- The privacy gate.
## Diagnosis mode
When the user arrives with a symptom, work from the symptom to the pillar rather than building the full sheet first.
| Symptom | Measure first | Causes to check first |
| --------------------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| "It has gone quiet" | activity trend by month, staff vs member share, burstiness | the spike was an event, not growth; staff stopped seeding; the audience moved venue |
| "People ask but nobody answers" | time to first response, % questions with zero replies | too few answerers awake in the asker's timezone; questions land in the wrong channel; experts left |
| "Newcomers never come back" | join-cohort retention, second-contribution rate, newcomer survey items | first answer took too long; onboarding path unclear; the room reads as staff-only |
| "One person carries everything" | contributor absence factor, maintainer burnout signals | genuine bus-factor risk; delegation never happened |
| "Leadership wants proof of value" | the primary-outcome metric plus one funnel metric | the community was never tied to a stated outcome |
State plainly when the honest answer is that the community should be folded back into a smaller surface. Keeping a dead room open costs trust every time a newcomer posts into silence.
## Invocation examples
Typical openings, and what each one should produce:
- "We have a 900-member Discord for our CLI and no idea whether it is working." → interview, then a metric sheet of 5-6 metrics with a "baselining" mark on most of them, plus the first reading.
- "Our Slack community feels dead, is it?" → diagnosis mode on the activity pillar first, then the rest of the sheet.
- "Board wants proof the community is worth the headcount." → the primary-outcome metric, one funnel metric, and an honest statement of what the data cannot attribute.
- "Is this open-source project healthy enough to depend on?" → the starter set (time to first response, change request closure ratio, contributor absence factor, release frequency) read as trends, with no cross-project comparison.
Expected output shape, in order:
1. Primary outcome.
2. Metric sheet (metric, pillar, formula, source, baseline, threshold, owner, action).
3. The period's reading against baseline.
4. Two or three findings with evidence and confidence.
5. Actions with owners and dates.
6. Watch list.
7. Next review date.
## Output
Produce a community health report. Structure, worked example and a negative example: [./references/health-report-template.md](./references/health-report-template.md).
Cadence follows the audience: a monthly working read for the people who run the community, a quarterly roll-up for whoever funds it. Do not build a real-time dashboard for a community that reviews its numbers once a month.
## Pass threshold
The framework is done when the metric sheet scores 100% on this check. Iterate until it does.
Every metric has all five:
1. A plain-language definition.
2. A data source the team can access today.
3. A baseline value or an explicit "baselining" mark.
4. A threshold that triggers attention.
5. A named action and owner if it breaches.
Plus: at most 8 metrics, at least one per pillar, and no metric that fails the "so what" test.
## Failure modes
- **Member count as the headline.** Total members only ever goes up and says nothing about health. Report active people and account coverage instead.
- **Bots counted as humans.** Inflates responsiveness and activity at once.
- **Benchmarking against another community.** Different size, ecosystem and product make the comparison meaningless and demoralising.
- **Sentiment inferred from text classifiers.** Developer conversation is full of blunt bug reports and dry humour. Automated scoring reads them as anger. Ask people instead.
- **A metric that became a target.** Rewarding message counts produces chatter. Rewarding closed issues produces premature closes. Watch for the behaviour the metric encourages, and pair any incentivised metric with a quality counter-metric.
- **A dashboard with no owner.** If nobody is named for the review, it stops being refreshed by month three. The fix is fewer metrics, not more automation.
- **Surveying only the people still present.** The members who left hold the answer to why. Include a lightweight exit or lapsed-member question when you can.
- **Publishing person-level data.** Leaderboards and per-member activity tables create privacy exposure and change behaviour in ways you did not intend.
- **Cohorts sliced too thin.** Join cohorts grouped by day instead of by month leave three people per cohort, and every retention curve looks flat or wildly noisy. Widen the cohort window until each one holds enough people to read, and say which window you used.
- **The same metric computed two ways.** One person counts "active" as posted-or-reacted, another as posted-only, and the two reports disagree in public. Fix it by publishing the formula beside the number, not by arguing in the meeting.
- **A threshold nobody has revisited.** A cutoff set at launch and left alone fires every month, gets muted, and the metric quietly stops being read. Re-derive thresholds from the trailing baseline at each review.
## Memory
If your harness has persistent memory, store the primary outcome, the final metric sheet, baselines and thresholds after the user validates them. The next review then starts from the recorded baseline instead of re-deriving it, and a threshold change becomes visible as a decision rather than a silent edit.