developer-education-strategy

{
  "skill_name": "developer-education-strategy",
  "evals": [
    {
      "id": 1,
      "prompt": "We build Trellisway, a workflow automation API. Northbridge Freight - our biggest prospect, around $400k ARR if it closes - told us last week that their procurement team won't sign unless their integration engineers can get certified on our platform. Our CRO approved $80k for this fiscal year to make it happen, nothing committed beyond that. The team is me (DevRel lead, maybe 10 hours a week on this) and one solutions engineer who can give 5 hours. I was thinking we write about 60 multiple choice questions, set the pass mark at 70%, and launch in Q1. What's the best way to build this out?",
      "expected_output": "A refusal to recommend proctored certification, a badge-rung recommendation instead, the three promotion conditions named with the failing ones identified, an observable certification trigger, and a rejection of the 70% pass mark as indefensible without a standard-setting study.",
      "files": [],
      "expectations": [
        "States a gate verdict in one sentence that names the evidence behind it and the cheaper alternative rejected",
        "Recommends the completion or skill badge rung rather than proctored certification",
        "Names all three conditions that would promote certification: a named buyer or partner contract counting certified heads, a funded second-year recertification cycle, and one person owning the item bank indefinitely",
        "Identifies which of those three conditions Trellisway fails, covering the single-fiscal-year funding and the absence of permanent item-bank ownership",
        "Writes the escalation trigger to certification as an observable condition rather than a date or a stated intention",
        "Rejects the 70% pass mark as indefensible without a standard-setting study",
        "Names at least one standard-setting method among modified Angoff, bookmark, or contrasting groups",
        "States the standard-setting panel size as 5 to 15 subject-matter experts",
        "Does not present a syllabus, course outline, module list or lesson content",
        "Deletes the rungs the answers rule out rather than listing them as later options",
        "Requires one named owner with weekly hours rather than a team label",
        "Frames the badge rung as adding permanent issuance, verification and disputes obligations, not a one-time cost"
      ]
    },
    {
      "id": 2,
      "prompt": "Hey - Quillstone here, we make an observability SDK, about 2,400 active orgs. Over three years we've published 41 tutorials and 12 videos and they sit as a flat list on our docs site. New users email us asking where do I start basically every week. I have a board review in 8 weeks where I need to show something concrete on developer enablement. Realistically I have one technical writer at about 8 hours a week and no budget for a new hire, now or later. What should we build?",
      "expected_output": "A curated learning path recommendation, the efficiency ordering stated explicitly with ad-hoc content above badge, the rungs deleted by the capacity ceiling, and an observable escalation trigger to the curriculum rung.",
      "files": [],
      "expectations": [
        "Recommends the curated learning path rung",
        "States the efficiency ordering explicitly, with the curated learning path first",
        "Places ad-hoc content above the badge rung on the efficiency ordering",
        "States that value and effort both put certification first, and that efficiency is the ordering that disagrees with both",
        "Says the 8-week board review date promoted the curated path",
        "Says the one-part-time-person capacity ceiling deletes every rung above the curated path",
        "Deletes those higher rungs rather than parking them at the bottom of a list as later options",
        "Gives the curated path's effort as an order of magnitude such as a week to curate plus a re-order per release, not a precise cost figure",
        "Notes the curated path is reversible by deleting a page",
        "Writes the escalation trigger to the curriculum rung as an observable condition, such as learners finishing the path and still being unable to do the job",
        "Names the curated path's failure mode: a link list dressed as a curriculum with no evidence anyone finished it",
        "Labels the five-rung ladder as this collection's own synthesis rather than a certified or published maturity model"
      ]
    },
    {
      "id": 3,
      "prompt": "Vantahold here - we do a payments reconciliation API. Support volume is up 3x in six months and our VP wants us to build a developer academy. Looking at the ticket data: the biggest cluster is people asking which endpoint returns settlement batches, which is documented, page 4 of the API reference. Second biggest is people passing an expired token, getting a generic 400 back, and filing a ticket. Separately, 62% of new signups never complete a first successful API call. And one customer's platform team asked us for a training session last month. Can you put together the academy plan?",
      "expected_output": "A gate verdict refusing the academy, with each evidence cluster routed to its cheaper fix: docs information architecture, quickstart and onboarding, product and error-message design, and a one-off workshop plus recording.",
      "files": [],
      "expectations": [
        "Returns a gate verdict that education is not the right fix for the evidence presented",
        "States that verdict in one sentence with the evidence behind it",
        "Names documentation information architecture as the fix for answers that exist but cannot be found",
        "Names quickstart and onboarding as the fix for users failing at first success, rather than a course",
        "Names product and error-message design as the fix for the expired-token case, on the grounds the API could refuse the misuse",
        "Recommends a live workshop plus a recording, not a curriculum, for the single team that asked once",
        "Distinguishes procedural support questions from conceptual misunderstandings, and notes only the conceptual cluster would justify a program",
        "Does not produce a competency map or module list",
        "Does not recommend a badge or a certification",
        "Does not propose buying or evaluating an LMS",
        "Routes the docs and quickstart fixes to the named sibling skills rather than performing them here",
        "Names at least one observable condition that would re-open the education gate later"
      ]
    },
    {
      "id": 4,
      "prompt": "Working on the curriculum for Perigee Academy - Perigee is our data catalog product. Here's the draft module list I put together from our product nav:\n\nModule 1: Dashboard overview\nModule 2: Projects and workspaces\nModule 3: Webhooks\nModule 4: Billing and plans\nModule 5: API keys\nModule 6: Integrations directory\nModule 7: Audit logs\nModule 8: Settings and preferences\n\nCan you flesh each of these out into learning objectives and lesson breakdowns so I can hand it to our writer this week?",
      "expected_output": "A refusal to expand the module list, diagnosed as a feature-tour curriculum, plus a backwards design sequence starting from a job task analysis, the module-to-task trace threshold, and the 20% restart rule labelled as the skill's own baseline.",
      "files": [],
      "expectations": [
        "Refuses to expand the module list as given",
        "Names the draft as a feature-tour curriculum whose modules mirror the product navigation",
        "Requires a job task analysis before any module design, sourced from practitioner interviews, support data and telemetry rather than the feature list",
        "States the design order as job tasks, then competencies, then evidence per competency, then modules",
        "Weights competencies by frequency and business consequence",
        "Defines the evidence per competency as the lab, artifact or exam item that proves it",
        "States the pass threshold that every module must trace to at least one task in the job analysis, and that untraced modules are cut or merged",
        "States the 20% failed-trace threshold above which the curriculum needs a restart",
        "Labels that 20% figure as this skill's own baseline rather than a published or industry standard",
        "Attributes backward design to Wiggins and McTighe, Understanding by Design, 1998",
        "Requires each module to carry a marked source of truth and a refresh trigger",
        "Sequences one new concept at a time and re-uses existing content where it already exists",
        "Hands lesson authoring to a tutorial-writing skill rather than drafting lessons here"
      ]
    },
    {
      "id": 5,
      "prompt": "Six months ago we launched Halcyon Learn, eight modules on our streaming-data platform. 640 people enrolled, 4.6 out of 5 average rating, 82% passed the end-of-module quizzes. Our CFO asked in the QBR what this did for the business and I had nothing. We deliberately didn't ask anyone to log in - everything was open and anonymous so we wouldn't hurt signups. I want to come back next quarter with a real measurement plan. What should I put in it?",
      "expected_output": "A four-level Kirkpatrick plan that classifies the existing numbers as levels 1 and 2, specifies the identity join as a pre-launch decision that cannot be reconstructed, and states attribution limits rather than claiming revenue impact.",
      "files": [],
      "expectations": [
        "Names the four Kirkpatrick levels: reaction, learning, behavior, results",
        "Classifies the 4.6 out of 5 rating as level 1 and the 82% quiz pass rate as level 2",
        "Names stopping at levels 1 and 2 as the framework's own documented failure mode",
        "States the instruction to start with Level 4, Results, attributed to Kirkpatrick Partners rather than presented as generic advice",
        "States that levels 3 and 4 require joining learner identity to product usage",
        "States that the identity join must be decided before launch and cannot be reconstructed after a cohort has learned anonymously",
        "Reproduces level 3's requirement that learners be supported in and accountable for the behavior in their own environment",
        "Says a flat behavior metric can be an employer-support problem rather than a curriculum defect, and to check which before rewriting modules",
        "Specifies exactly one primary metric per level, noting that more than one primary metric per level means none is primary",
        "Notes that Phillips adds a level 5 for ROI and that Kaufman places ROI at level 4, and instructs picking one convention and saying which",
        "Refuses to claim revenue was caused by the course, and frames any link as a labelled correlation using matched comparison groups",
        "Names opt-in selection bias: learners who enrol are already more committed, so raw before-and-after comparison overstates the effect",
        "Names at least one level it will not measure this year and why, in advance rather than at review"
      ]
    },
    {
      "id": 6,
      "prompt": "Drafting the certification policy for Stonecourse, our infrastructure-as-code tool. Our closest competitor uses a 2-year validity with a paid retake, and their docs say courseware is re-aligned within 4 to 8 weeks of each release. That looks like the industry standard so I'm going to mirror both. We ship a minor every 3 weeks. Can you write up the validity, renewal and re-alignment sections of our policy?",
      "expected_output": "A rejection of the industry-standard framing, the four shipped validity and renewal models named with their owners, the 4-8 week window attributed to CKA as a Kubernetes-tracking commitment, and a re-alignment window derived from Stonecourse's own three-week cadence.",
      "files": [],
      "expectations": [
        "Rejects the claim that a two-year validity window with a paid retake is an industry standard",
        "Names CKA's two-year validity with a full-price retake",
        "Names HashiCorp's two-year validity, with a renewal window opening six months before expiry and satisfied by passing a same-level or higher exam",
        "Names AWS's three-year validity with a 50% discount voucher, or its one-year activity-based maintenance path",
        "Names Microsoft's one-year role-based validity, renewed free by a short unproctored open-book assessment inside a six-month eligibility window",
        "Notes that Microsoft Fundamentals credentials never expire",
        "States that the 4 to 8 week window is CKA's own published commitment for aligning to each new Kubernetes minor version, not a norm",
        "Frames that window as a cost promise rather than a badge of quality",
        "States that free renewal maximizes the certified installed base while paid retake maximizes per-credential revenue, and asks which the business needs before copying either",
        "Derives the re-alignment window from Stonecourse's own three-week minor-release cadence rather than from the competitor's calendar",
        "Attributes every vendor figure it quotes to the vendor that chose it",
        "Declines to offer a benchmark pass rate, noting no major vendor certification program publishes one"
      ]
    },
    {
      "id": 7,
      "prompt": "We're hiring 30 backend engineers over the next year at Marrowfield and our internal ramp-up is a mess - new hires take about 10 weeks before they ship anything meaningful. I want an internal engineering academy: structured curriculum, hands-on labs on our own stack, and a certification that each new hire passes at end of week 6 before they get production access. Can you design the program?",
      "expected_output": "A scope refusal: internal employee onboarding and engineering ramp-up are out of scope, routed to people and L&D, with the three reasons given and no charter produced.",
      "files": [],
      "expectations": [
        "Declines the request as out of scope",
        "States that the skill covers education aimed at external developers: users, customers, partners, integrators, students",
        "Routes internal employee onboarding and engineering ramp-up to people or L&D",
        "Gives the reason that an internal audience is captive",
        "Gives the reason that there is no adoption funnel to protect",
        "Gives the reason that there is no external consumer for the credential",
        "Stops rather than producing an education charter anyway",
        "Does not produce a competency map, tier recommendation or operating model for the internal program",
        "Does not design the week-6 internal certification or its pass mark",
        "Offers to re-engage if the audience becomes external customers, partners or integrators"
      ]
    },
    {
      "id": 8,
      "prompt": "Ashgrove Data - we have 6 contracted SI partners and 3 of them shipped implementations last quarter that failed our architecture review. The plan so far: build everything inside a hosted interactive-lab SaaS, the DevRel team owns the program, we review content regularly, film video walkthroughs for all 14 modules, and launch in English, German and Japanese on day one so all our partner regions are covered. We ship a minor release roughly monthly. Poke holes in this and help me firm it up.",
      "expected_output": "An operating model that moves lesson and lab sources into an owned repository, replaces the team label with a named owner and per-role weekly hours, binds the refresh trigger to the monthly release cadence, defers localization and video, and adds partner-specific delivery mechanics.",
      "files": [],
      "expectations": [
        "Requires the source of every lesson and lab to live in a repository the company owns, whatever renders it",
        "Cites the shutdown of Katacoda's free public site on 15 June 2022, after the O'Reilly acquisition, as the precedent for that requirement",
        "Replaces the DevRel team label with one named person accountable for the curriculum's truth",
        "States that owner's commitment in hours per week rather than as headcount",
        "Estimates authoring, technical review, lab maintenance and learner support each in hours per week",
        "Binds the refresh trigger to Ashgrove's monthly release cadence with a stated window, rather than to a periodic content review",
        "Defers localization until one locale proves demand, noting each added language multiplies the refresh bill",
        "Defers video, noting video and screenshots age fastest and cost the most to redo",
        "Recommends a train-the-trainer path and a quality bar authorized trainers must clear",
        "Ties partner tiers to certified or badged staff counts",
        "Frames partner education as delivery-quality control enforced through the partner contract",
        "Budgets year-two recurring cost separately from year-one build, and treats the recurring number as the one that decides the program",
        "Names the labs failure mode: labs rot silently after releases and learners stop trusting the whole program"
      ]
    },
    {
      "id": 9,
      "prompt": "Fernlow is a free CLI tool for database migrations - about 40,000 people hit our docs monthly, all bottom-up individual devs, no sales team, no enterprise contracts. Marketing wants to move all our learning content behind a signup wall so we can capture emails, and they want a certification so people can put Fernlow Certified on LinkedIn. Feels off to me but I can't articulate why. What should we actually do?",
      "expected_output": "A refusal to gate the learning content, the gate-the-credential-not-the-knowledge rule, a bottom-up credential framed as a portfolio signal, a portable Open Badges recommendation, and a named-external-consumer requirement plus badge kill rule.",
      "files": [],
      "expectations": [
        "Refuses to put the learning content behind a login or signup wall",
        "States the rule as gating the credential, not the knowledge",
        "Says a login wall suppresses the exact top-of-funnel reach the bottom-up motion depends on",
        "Characterizes the credential in a bottom-up motion as a portfolio and job-market signal",
        "States that free, public, shareable and low-friction beats rigorous for this audience",
        "Does not recommend proctored certification for this audience",
        "Recommends issuing on Open Badges as a portable open format so the credential sits on profiles Fernlow does not own",
        "Attributes Open Badges to 1EdTech",
        "Declines to use the 27-certified-platforms or 74 million-badges adoption figures in a business case, or explicitly flags them as unverified",
        "Requires a named external consumer who will act on the credential before recommending any credential at all",
        "States the badge kill rule: if no external consumer cites it within two quarters of launch, stop issuing and honor existing badges to their stated expiry",
        "Warns that a badge nobody consumes is a maintenance liability with a marketing veneer"
      ]
    },
    {
      "id": 10,
      "prompt": "Wrenmoor, API gateway product. We're going ahead with a proctored certification, that decision is made. Details: our education team writes and sells the official $1,200 prep course and will also set the pass/fail line. Exam is 60 multiple choice questions, 90 minutes. No budget line for anything anti-cheating, we'll deal with it if it happens. Also we have an old completion badge program from 2024 with about 300 holders that we want to shut down the same week we launch - the plan is to pull the badge pages and revoke the badges so nobody confuses the two. Help me write the program design doc.",
      "expected_output": "A design doc that flags the training/certification impartiality conflict, names the ISO/IEC 17024 scheme contents, pushes toward performance-based items, budgets exam fraud as a permanent cost, and replaces the badge revocation with proper retirement mechanics.",
      "files": [],
      "expectations": [
        "States that the pass/fail decision cannot be outsourced to another body, while delivery can be subcontracted",
        "Flags the conflict in a body that sells the prep course also setting the pass/fail line, and requires showing the training does not compromise the certification's impartiality",
        "Names ISO/IEC 17024 as the international standard for bodies certifying persons",
        "Notes that most vendor devtool certifications are not accredited to it and do not need to be, while it remains the checklist the credential gets measured against",
        "Lists the documented scheme contents, including criteria for recertification and criteria for suspending and withdrawing a certification",
        "Recommends performance-based tasks over the 60 recall items, noting recall exams are the first to be devalued by leaked content",
        "Notes that performance-based items carry the highest per-item build and hosting cost because each needs a disposable, deterministic environment",
        "Names brain dumps as a permanent recurring operating cost rather than an incident to handle later",
        "Lists at least two countermeasures among large rotating item banks, live proctoring with identity verification, item exposure analysis, and legal action against dump sites",
        "Refuses to revoke the roughly 300 already-issued badges retroactively, on the grounds that it destroys trust in every future credential",
        "Requires announcing the badge program's end date before the last enrolment rather than after",
        "Keeps the retired badge content readable at a stable URL with a dated no-longer-maintained banner, or redirects it to the replacement",
        "Names the unfunded items from the certification refusal checklist and either downgrades the recommendation or writes the missing item as the escalation trigger"
      ]
    }
  ],
  "trigger_queries": [
    { "query": "should we build a certification for our API?", "should_trigger": true },
    { "query": "Our enterprise buyers keep asking whether their staff can get trained. Do we need an academy?", "should_trigger": true },
    { "query": "we have forty tutorials and no learning path", "should_trigger": true },
    { "query": "our SI partners keep implementing the product wrong", "should_trigger": true },
    { "query": "what would a badge program cost us to run?", "should_trigger": true },
    { "query": "should we certify our users", "should_trigger": true },
    { "query": "we need more training content for developers", "should_trigger": true },
    { "query": "how do I design a developer curriculum", "should_trigger": true },
    { "query": "planning a developer academy, where do I start", "should_trigger": true },
    { "query": "is a skill badge worth it for a dev tool", "should_trigger": true },
    { "query": "how do we set a defensible pass mark for our exam", "should_trigger": true },
    { "query": "what validity window should our certification have", "should_trigger": true },
    { "query": "we want to launch learning paths on our docs site", "should_trigger": true },
    { "query": "Partner contract says Gold tier needs trained engineers. What do we build?", "should_trigger": true },
    { "query": "how do we measure whether our courses actually work", "should_trigger": true },
    { "query": "our academy has been stale for a year, should we kill it", "should_trigger": true },
    { "query": "do we need hands-on labs or are tutorials enough", "should_trigger": true },
    { "query": "we're thinking about credentials for our community", "should_trigger": true },
    { "query": "who should own our developer training program", "should_trigger": true },
    { "query": "how often do we have to refresh course content", "should_trigger": true },
    { "query": "We want developers to be able to prove they know our product.", "should_trigger": true },
    { "query": "Customers keep telling us their team doesn't really know how to use the platform properly.", "should_trigger": true },
    { "query": "our onboarding for new enterprise accounts takes eleven weeks, can we shorten that", "should_trigger": true },
    { "query": "should our training be free or paid", "should_trigger": true },
    { "query": "Hiring managers say they can't find engineers who already know our stack. What can we do about that?", "should_trigger": true },
    { "query": "what's the cheapest way to get people from zero to productive on our SDK", "should_trigger": true },
    { "query": "Exec wants a Coursera-style thing for our product. Sanity check me.", "should_trigger": true },
    { "query": "how do I write a kill rule for our education program", "should_trigger": true },
    { "query": "do we actually need proctoring for our exam", "should_trigger": true },
    { "query": "we're deciding between a completion badge and a real certification", "should_trigger": true },
    { "query": "should we use an LMS or just put the modules on the docs site", "should_trigger": true },
    { "query": "what goes into a developer education charter", "should_trigger": true },
    { "query": "how many modules should our course have and how do we choose them", "should_trigger": true },
    { "query": "our partners want to resell our training, how should we handle that", "should_trigger": true },
    { "query": "train-the-trainer for our implementation partners", "should_trigger": true },
    { "query": "Marketing wants to gate our tutorials behind a signup to capture leads. Thoughts?", "should_trigger": true },
    { "query": "how do we justify the budget for developer training next year", "should_trigger": true },
    { "query": "is it worth building a certification if only one customer asked", "should_trigger": true },
    { "query": "we built a badge and nobody has claimed one", "should_trigger": true },
    { "query": "how should we sequence our learning content", "should_trigger": true },
    { "query": "our labs keep breaking after every release", "should_trigger": true },
    { "query": "developer enablement program design", "should_trigger": true },
    { "query": "what should we measure for our developer courses beyond completion rate", "should_trigger": true },
    { "query": "we want to be the AWS Certified of our category", "should_trigger": true },
    { "query": "Customer success says deals stall because nobody on the customer side knows the product.", "should_trigger": true },
    { "query": "should students be able to take our certification for free", "should_trigger": true },
    { "query": "how much does it cost to keep a certification running year after year", "should_trigger": true },
    { "query": "our CTO wants a university for our ecosystem", "should_trigger": true },
    { "query": "planning skill credentials for our integrators", "should_trigger": true },
    { "query": "is there something cheaper than building a whole curriculum", "should_trigger": true },
    { "query": "recertification policy for our exam", "should_trigger": true },
    { "query": "we want to tie partner tiers to how many of their engineers are trained", "should_trigger": true },
    { "query": "how do we stop people cheating on our online exam", "should_trigger": true },
    { "query": "should we retire our old training program or keep it going", "should_trigger": true },
    { "query": "structured onboarding for external developers beyond our docs", "should_trigger": true },
    { "query": "we need a way to prove our integrators actually know what they're doing", "should_trigger": true },
    { "query": "write the lesson on webhooks for our tutorial series", "should_trigger": false },
    { "query": "draft a hands-on tutorial teaching OAuth with our Node SDK", "should_trigger": false },
    { "query": "our quickstart takes 40 minutes, get it under five", "should_trigger": false },
    { "query": "write the getting started page so a developer makes their first API call fast", "should_trigger": false },
    { "query": "our docs site is impossible to navigate, restructure it", "should_trigger": false },
    { "query": "where should the API reference sit in our docs information architecture", "should_trigger": false },
    { "query": "plan our whole DevRel program for next year", "should_trigger": false },
    { "query": "how should I split my DevRel budget across content, events and community", "should_trigger": false },
    { "query": "map the developer journey from first visit to production", "should_trigger": false },
    { "query": "which DevRel metrics should I report to the board", "should_trigger": false },
    { "query": "segment our developer audience into personas", "should_trigger": false },
    { "query": "make it easier for new contributors to land their first PR on our open source repo", "should_trigger": false },
    { "query": "build a developer champions program", "should_trigger": false },
    { "query": "launch a Discord for our developer community", "should_trigger": false },
    { "query": "write the script for a five-minute product video", "should_trigger": false },
    { "query": "set standards for the code samples in our docs", "should_trigger": false },
    { "query": "write a troubleshooting page for our five most common errors", "should_trigger": false },
    { "query": "write the v3 migration guide for the breaking changes", "should_trigger": false },
    { "query": "turn these commits into release notes", "should_trigger": false },
    { "query": "start a monthly developer meetup in Lisbon", "should_trigger": false },
    { "query": "outline a conference talk about our storage architecture", "should_trigger": false },
    { "query": "write a CFP abstract for a Kubernetes conference", "should_trigger": false },
    { "query": "write the job description for a developer advocate", "should_trigger": false },
    { "query": "how do I get promoted from developer advocate to head of DevRel", "should_trigger": false },
    { "query": "design a live demo that will not break on stage", "should_trigger": false },
    { "query": "rewrite our README so it converts visitors", "should_trigger": false },
    { "query": "improve organic search traffic to our documentation", "should_trigger": false },
    { "query": "find the keywords developers search for in our category", "should_trigger": false },
    { "query": "set pricing tiers for our API product", "should_trigger": false },
    { "query": "should we build an integrations marketplace or a plugin ecosystem", "should_trigger": false },
    { "query": "how many advocates per region should my DevRel team have", "should_trigger": false },
    { "query": "write a customer case study about how a fintech uses our API", "should_trigger": false },
    { "query": "draft an engineering blog post about our rate limiter rewrite", "should_trigger": false },
    { "query": "what go-to-market motion fits a bottom-up developer tool", "should_trigger": false },
    { "query": "make our docs easier for coding agents to consume", "should_trigger": false },
    { "query": "pick a license for our open source project", "should_trigger": false },
    { "query": "should we take our wire protocol to a standards body", "should_trigger": false },
    { "query": "write a code of conduct and moderation policy for our forum", "should_trigger": false },
    { "query": "improve our employer brand so senior engineers want to join", "should_trigger": false },
    { "query": "should we sponsor this conference at the gold tier", "should_trigger": false },
    { "query": "help me learn this codebase", "should_trigger": false },
    { "query": "teach me Rust from scratch", "should_trigger": false },
    { "query": "create a classroom course plan for my high school computing class", "should_trigger": false },
    { "query": "add educational comments to this file", "should_trigger": false },
    { "query": "improve activation in our product onboarding funnel", "should_trigger": false },
    { "query": "write a lessons-learned retro for last quarter's outage", "should_trigger": false },
    { "query": "build a continuous learning habit for myself as an engineer", "should_trigger": false },
    { "query": "train a model on our support transcripts", "should_trigger": false },
    { "query": "set the training run hyperparameters for our fine-tune", "should_trigger": false },
    { "query": "review this academic paper on educational psychology", "should_trigger": false },
    { "query": "write user onboarding emails for new signups", "should_trigger": false },
    { "query": "how do I get our company SOC 2 certified", "should_trigger": false },
    { "query": "we need an ISO 27001 audit, where do we start", "should_trigger": false },
    { "query": "generate a tutorial from the comments in this file", "should_trigger": false },
    { "query": "write API reference docs for our new endpoints", "should_trigger": false },
    { "query": "help me study for the CKA exam", "should_trigger": false }
  ]
}

# Certification program design

Read before recommending anything at the certification rung. These are the obligations that separate a certification from a quiz with a PDF.

Vendor pricing, validity windows and renewal rules move constantly. Re-check the vendor's own certification page before quoting one to a buyer, and always quote it with its date.

## Scheme requirements

ISO/IEC 17024 is the international standard for bodies certifying persons. Most vendor devtool certifications are **not** accredited to it and do not need to be - it is still the checklist the credential gets measured against once an enterprise buyer, a partner contract or a job posting relies on it.

A documented certification scheme contains:

- Scope of certification, job and task description, required competence, abilities and prerequisites.
- Assessment methods.
- Criteria for initial certification **and recertification**.
- Criteria for suspending and withdrawing a certification.

Two structural rules bite for a vendor program:

- **Impartiality**: the decision to award certification cannot be outsourced to another body. Delivery can be subcontracted; the pass/fail decision cannot.
- **Separation of training and certification**: a body that also sells training must show that the training does not compromise the certification's impartiality. Selling the course that guarantees the exam is exactly the conflict the standard polices.

## Job task analysis is the input, not the syllabus

Derive competencies from what practitioners actually do in the role - sourced from interviews, support data and telemetry. It is also the only defensible source for published domain weights.

- Feature-derived syllabus: certifies familiarity with a menu.
- Job-task-derived syllabus: certifies ability to do the job.

Worked example of published weights (CKA):

- Troubleshooting: 30%
- Cluster Architecture/Installation/Configuration: 25%
- Services & Networking: 20%
- Workloads & Scheduling: 15%
- Storage: 10%

Weights are a claim about the job, and a candidate can hold you to them.

## Cut scores need a standard-setting study

Picking 70% because it looks like a pass mark is indefensible. Fund a study or do not publish pass/fail.

- Panel of **5-15 subject-matter experts** representing key stakeholder groups.
- **Modified Angoff**: each expert estimates, item by item, the proportion of _minimally competent_ candidates who would answer correctly; several rounds, experts revise against peer estimates and real performance data; results averaged or taken at the median.
- **Bookmark**: order items by difficulty, experts mark where a threshold candidate stops succeeding.
- **Contrasting groups**: classify real candidates by known competence, find where the score distributions cross.

Consequence for a small program: a real cut score costs a day of 5-15 experts' time per exam version. A program that cannot fund that should publish a completion badge instead.

## Exam format

Performance-based exams - solving real tasks in a live environment - are the format that survives contact with a developer audience, and the one with the highest per-item build and hosting cost, because every task needs a disposable, deterministic environment. Recall-item exams are cheap to build and the first to be devalued by leaked content.

Reference point: CKA is an "online, proctored, performance-based test that requires solving multiple tasks from a command line running Kubernetes", 2 hours, $445 exam-only, valid two years, and the price includes two exam attempts plus two 36-hour simulator sessions. HashiCorp runs Associate and Professional/Advanced tiers per product, online with a live proctor verifying identity and monitoring the session; each product's own certification page, not the top-level certifications index, publishes exam price and duration. Its Associate-tier exams (Terraform, Vault) are recall-item, multiple-choice, one hour, $70.50 per attempt; its Professional-tier exam (Terraform Authoring and Operations Advanced) is lab-based plus multiple choice, four hours including a 15-minute break, $295 per attempt. Both tiers are valid two years.

## Validity and renewal - four shipped models

| Program              | Validity                                   | Renewal path                                                                                                                                                                                                             | Cost to holder                                                 |
| -------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- |
| CKA                  | 2 years                                    | Retake                                                                                                                                                                                                                   | Full price                                                     |
| HashiCorp            | 2 years                                    | Pass same-level or higher exam; window opens 6 months before expiry                                                                                                                                                      | Full price                                                     |
| AWS                  | 3 years (retake) or 1 year (activity path) | Retake the latest version, pass a higher-level exam, or complete maintenance activities on the vendor's learning platform                                                                                                | 50% discount voucher; game-based recert free at the entry tier |
| Microsoft role-based | 1 year                                     | Short **unproctored, open-book** online assessment inside a six-month eligibility window, taken "as many times as you need as long as you pass before your certification expires"; Fundamentals credentials never expire | Free                                                           |

The spread is a strategy choice, not an industry standard:

- Free renewal: maximizes the installed base of certified people.
- Paid retake: maximizes per-credential revenue and shrinks that base.

Decide which one the business actually needs before copying a competitor's window.

AWS's stated rationale for expiry: recertification "helps strengthen the overall value of your AWS Certification and shows individuals and employers that your credential covers the latest AWS knowledge, skills, and best practices." Use that framing only if you can fund the re-alignment it implies.

## The re-alignment liability

CKA states it will be aligned with the most recent Kubernetes minor version within approximately **4 to 8 weeks** of that release. That is the real cost line: every product release re-dates the item bank, the lab images and the courseware behind them.

Planning rule: a certification's refresh cadence is set by the **product's** release cadence, never by the education team's calendar. A program shipping faster than it can re-align its exam certifies a version nobody runs.

## Exam fraud is a permanent operating cost

"Brain dumps" - reconstructed and resold live exam content - are the standing threat to any high-value IT credential. Once an item bank leaks, the credential measures memorization of the dump.

Documented countermeasures, all recurring costs:

- Performance-based tasks over recall items.
- Large rotating item banks.
- Live proctoring and identity verification.
- Item exposure analysis.
- Legal action against dump sites.

Credential value and operating cost rise together; budget them together.

## Portability

Issue on Open Badges (maintained by 1EdTech, formerly IMS Global) - a free open specification for "information-rich visual tokens of verifiable achievements". Version 3.0 is current: each badge is digitally signed by its issuer and compatible with the Verifiable Credentials Data Model 2.0, and holders can post it to a site, share it socially, or import it into a wallet via the Badge Connect API. A credential locked inside your own dashboard cannot do the job the holder wants it for.

Adoption figures once cited for Open Badges (27 certified platforms, 74M+ badges issued as of 2022) are **not** on 1EdTech's current standard page - treat them as unverified and do not put them in a business case.

## Negative example - the borrowed benchmark

Wrong, and the most common way this reference gets misused:

> "Industry standard is a two-year validity window with a paid retake, and courseware re-aligned within 4-8 weeks of each release, so that is what we will do."

Nothing in that sentence is a standard. Two years with a paid retake is what CKA and HashiCorp chose; Microsoft chose one year renewed free and open-book; AWS chose three years with a discount voucher. The 4-8 week window is CKA's own published commitment for tracking Kubernetes minors, not a norm - and it is a cost promise, not a badge of quality.

Right:

> "We propose two years with a paid retake, matching CKA and HashiCorp, and reject Microsoft's free annual model because we want per-credential revenue more than a large certified installed base. Our re-alignment window is six weeks, set by our own minor-release cadence; CKA's 4-8 weeks is the closest published comparator, not a target we inherit."

The difference is not tone. The second version names who chose what, when it was read, and which trade-off the recommendation is buying.

## Refusal checklist

Refuse to recommend certification when any of these is unfunded:

- The job task analysis.
- The standard-setting study.
- Item security.
- The appeals path.
- The recertification cycle.

Downgrade to a skill badge and name the missing item as the escalation trigger.

# Education charter template

Contents: the charter structure · a worked positive example · the same charter written badly · weak vs. strong per section.

Present the charter section by section and get explicit approval on each before writing the next.

## Table of Contents

- [Structure](#structure)
- [Gate verdict](#gate-verdict)
- [Audience and motion](#audience-and-motion)
- [Chosen tier](#chosen-tier)
- [Competency map](#competency-map)
- [Credential decision](#credential-decision)
- [Operating model](#operating-model)
- [Budget shape](#budget-shape)
- [Measurement plan](#measurement-plan)
- [Kill rule](#kill-rule)
- [First 90 days](#first-90-days)
- [Where each number comes from](#where-each-number-comes-from)
- [Worked example (positive)](#worked-example-positive)
- [Gate verdict](#gate-verdict)
- [Audience and motion](#audience-and-motion)
- [Chosen tier](#chosen-tier)
- [Competency map](#competency-map)
- [Credential decision](#credential-decision)
- [Operating model](#operating-model)
- [Budget shape](#budget-shape)
- [Measurement plan](#measurement-plan)
- [Kill rule](#kill-rule)
- [First 90 days](#first-90-days)
- [The same charter written badly](#the-same-charter-written-badly)
- [Weak vs. strong, per section](#weak-vs-strong-per-section)

## Structure

```
# Developer education charter - <product>, <period>

## Gate verdict
One sentence: education is / is not the right fix, the evidence behind it, and the
cheaper alternative that was rejected and why.

## Audience and motion
Who the learner is, which motion they belong to (enterprise, partner/SI, bottom-up
individual, student), and who consumes the outcome.

## Chosen tier
The rung, the rejected rungs with reasons, and the observable condition that would
trigger the next rung.

## Competency map
Job tasks → competencies → evidence per competency → modules. Every module traces
to a task.

## Credential decision
Rung, the named external consumer who will act on it, validity and renewal, or an
explicit "no credential" with the reason.

## Operating model
Named owner with weekly hours. Authoring, technical review, lab maintenance and
learner support, each in hours per week. Platform class and where the source lives.
Refresh trigger bound to the product's release cadence. Pricing. Localization stance.
Partner delivery.

## Budget shape
Year-one build and year-two recurring, separately. Recurring is the number that
decides the program.

## Measurement plan
One primary metric per Kirkpatrick level, first-cohort target, the identity join
that makes levels 3-4 possible, and the levels deliberately not measured this year.

## Kill rule
Metric, threshold, review date, action.

## First 90 days
Sequenced, with the first learner-visible artefact dated.

## Where each number comes from
Every figure above in one list, each marked sourced (with vendor and read date) or
own baseline (with how it was derived). Anything neither goes back out of the charter.
```

## Worked example (positive)

Illustrative product: a hosted data-pipeline platform with a self-serve tier and an enterprise motion. **Every figure below is invented** - partner counts, hours, pass rates and targets alike - and shows only the shape and level of specificity to aim for. Replace each number with one you measured, and mark it as yours.

```
# Developer education charter - <product>, <year> H2

## Gate verdict
Education is justified for the partner motion only: three of five implementation
partners shipped pipelines that failed our own review, and two enterprise deals
stalled on "who on your side is trained". Rejected cheaper alternative: better docs
IA - partners find the docs, follow them, and still design the wrong topology, which
is a competence gap, not a findability gap.

## Audience and motion
Primary: implementation engineers at the five contracted SI partners (~40 people).
Secondary: platform engineers at enterprise customers (~15 accounts).
Not in scope: self-serve individual users - they succeed on quickstarts today.

## Chosen tier
Structured curriculum with labs (rung 3), plus a skill badge (rung 4) for the two
partner-critical tasks. Rejected: curated path - partners already read everything
and still fail. Rejected: proctored certification - no standard-setting budget and
no third-party consumer beyond our own partner contracts. Escalation trigger to
certification: a partner contract or a customer RFP requires an independently
verifiable credential, twice, in one quarter.

## Competency map
Job analysis from 9 partner-engineer interviews, 6 months of escalations, and the
review notes on the three failed implementations. Four competencies: topology design,
backfill and replay, failure handling and alerting, cost control. Evidence: one graded
lab each. 11 modules, all tracing to a competency; 3 existing tutorials re-used, 8
authored.

## Credential decision
Skill badges for topology design and failure handling. External consumer: our own
partner tier requirements - a Gold partner needs two badged engineers, effective next
contract cycle. Open Badges format so partner engineers keep it on their own profiles.
No expiry in year one; revisit when the topology model changes.

## Operating model
Owner: <named person>, 12h/week, accountable for curriculum truth.
Authoring 16h/week (2 people), technical review 4h/week from the platform team,
lab maintenance 6h/week, learner support 3h/week.
Platform: hands-on lab platform embedded in the docs site; every lesson and lab source
in our own repository, in Markdown plus Terraform.
Refresh trigger: re-align within 4 weeks of any minor release that changes the pipeline
API; full review each quarter.
Pricing: free to contracted partners. Localization: none until a non-English partner
region proves demand. Partner delivery: train-the-trainer for the two largest partners
in year two, not year one.

## Budget shape
Year one: build 240 person-hours plus lab platform subscription.
Year two recurring: ~30h/week steady state, dominated by lab maintenance and
per-release re-alignment. If year two cannot be staffed, do not start year one.

## Measurement plan
Reaction: per-module drop-off, target <25% at any single module.
Learning: lab pass rate, target 70% within two attempts for cohort one.
Behavior: partner implementations passing our architecture review first time,
baseline 2/5, target 4/5 within two quarters.
Results: enterprise deals stalled on trained-staff objections, baseline 2 per quarter,
target 0.
Identity join: partner engineer email → lab platform account → implementation review
record, agreed with the partner ops team before launch.
Not measured this year: revenue attribution - no clean counterfactual with five partners.

## Kill rule
If partner implementations passing first-time review is still below 3/5 at the review
date <date, two quarters out>, stop authoring, keep the two badge labs on maintenance,
and reopen the gate with the new evidence.

## First 90 days
Weeks 1-3 job task analysis write-up and competency sign-off with partner ops.
Weeks 4-7 topology-design module and its lab, piloted with one partner.
Weeks 8-10 failure-handling module and lab.
Weeks 11-13 badge issuance path, first cohort of 8 engineers, baseline metrics captured.
```

## The same charter written badly

```
# Developer Academy

We are launching <product> Academy to educate our developer community and become the
go-to learning destination for our ecosystem.

Audience: all developers.
Program: a full curriculum with courses, videos and a certification, so our users can
prove their expertise.
Platform: we will evaluate LMS vendors.
Team: the DevRel team will own it.
Success: 1,000 course enrolments and high satisfaction scores in year one.
```

Everything wrong with it, in order:

1. No evidence.
2. No rejected alternative.
3. An audience that cannot be excluded from.
4. A top-rung credential chosen by ambition.
5. A platform decision deferred past the point where it changes the cost.
6. A team label instead of a named owner with hours.
7. No refresh trigger.
8. No year-two budget.
9. Level-1-and-2-only metrics.
10. No kill rule.
11. A target of 1,000 enrolments with no stated origin - invented, borrowed from a competitor's press release, nobody can tell.

It will be stale within two releases and nobody will be accountable for that.

## Weak vs. strong, per section

**Gate verdict**

- Weak: "Our users need more education."
- Strong: "Three of five partners shipped failing implementations after reading correct docs - a competence gap, not a findability gap."

**Tier choice**

- Weak: "A full academy with certification."
- Strong: "Rung 3 plus two skill badges; certification rejected for lack of standard-setting budget; escalation trigger written as an observable condition."

**Owner**

- Weak: "Owned by DevRel."
- Strong: "<named person>, 12h/week, accountable for curriculum truth."

**Refresh trigger**

- Weak: "Content will be reviewed regularly."
- Strong: "Re-aligned within 4 weeks of any minor release changing the pipeline API."

**Measurement**

- Weak: "1,000 enrolments and a 4.5/5 rating."
- Strong: "Partner implementations passing architecture review first time: 2/5 today, 4/5 in two quarters, joined via partner engineer email to the review record."

**Kill rule**

- Weak: "We will review the program annually."
- Strong: "Below 3/5 at <date> → stop authoring, keep two labs on maintenance, reopen the gate."

# Education model ladder

The ladder is this collection's own synthesis, not a published maturity model, so treat the rungs as a shared vocabulary for the cost step between options, not as levels anyone certifies against. The vendor facts cited inside it are sourced and dated in the certification reference.

The rungs appear here in inheritance order - each inherits every obligation of the rungs below it - which is not the order they are recommended in. `SKILL.md` orders them by value per unit of effort and that is the ordering to recommend from; this file only carries the per-rung detail that does not fit beside the choice.

## Rung 1 - Ad-hoc content

- **Staffing**: whoever writes content today. No new role.
- **Prerequisites**: none.
- **Escalation trigger**: the same learner asks for "where do I start" more than the team can answer individually, or two artefacts must be consumed in a fixed order to work.
- **Failure mode**: artefacts contradict each other because no one owns the sequence.

## Rung 2 - Curated learning path

- **Staffing**: one part-time curator; hours per week, not a headcount.
- **Prerequisites**: enough existing content to fill the sequence, and gaps named honestly rather than papered over.
- **Escalation trigger**: learners finish the path and still cannot do the job - reading is not producing competence.
- **Failure mode**: a link list dressed as a curriculum, with no evidence anyone finished it.

## Rung 3 - Structured curriculum with labs

- **Staffing**: author, technical reviewer, lab maintainer, learner support. Estimate each in hours per week.
- **Prerequisites**: a job task analysis, a lab platform decision, and a source-of-truth repository you own.
- **Escalation trigger**: someone outside the education team asks to _verify_ that a named person completed it.
- **Failure mode**: labs rot silently; learners hit broken environments and stop trusting the whole program.

## Rung 4 - Completion or skill badge

- **Staffing**: rung 3 plus an owner for issuance and disputes.
- **Prerequisites**: a named external consumer of the badge. Issue on a portable standard - Open Badges (maintained by 1EdTech) is the interoperable format, free and open, so the credential is visible on profiles you do not own.
- **Escalation trigger**: a buyer, a partner contract or a hiring manager starts counting credential holders and asking what the pass mark means.
- **Failure mode**: a badge nobody consumes - a maintenance liability with a marketing veneer.

## Rung 5 - Proctored certification with a partner training ecosystem

- **Staffing**: a program owner, subject-matter expert panel time per exam version, psychometric support, exam operations, partner management.
- **Prerequisites**: job task analysis, standard-setting study, item security plan, funded recertification cycle, published validity window - all detailed in the certification program design reference.
- **Escalation trigger**: none - this is the top rung. The next decision is retirement, not expansion.
- **Failure mode**: certification theater - a pass mark chosen by feel, a leaked item bank, and no recertification budget.

## Reading the ladder

- Every rung's cost is recurring, not one-off. Budget the second year, not the launch.
- A rung above the curated path is only justified when the rung below it has been tried and its ceiling is documented.
- Skipping rungs is the standard failure: an organization with no learning path announces a certification, then discovers it owes an item bank forever.
- Audience changes the top rung worth reaching. Enterprise and partner motions can justify rung 5 because the credential is procurement currency. Bottom-up individual adoption rarely does - free, public and low-friction beats rigorous, and a login wall suppresses the funnel.

# Measurement and kill rules

Measure on four levels - reaction, learning, behavior, results (Kirkpatrick, 1954; popularized 1959 and 1994). The documented failure mode is that trainers stop at levels 1-2: satisfaction scores and quiz results, reported as impact. Design from level 4 backwards, the same order backward design imposes on the curriculum.

Phillips adds a level 5, ROI (benefit vs. cost). Kaufman argues ROI belongs at level 4 and level 5 should measure client/societal impact. Pick one convention, say which, and stop debating it.

Kirkpatrick Partners' own definitions are worth quoting verbatim when someone argues the levels mean something else - level 3 in particular is not "did they like it later":

- Reaction - "the degree to which the target audience finds an experience or initiative favorable, engaging, relevant, and supportive of the work needed to achieve the targeted outcomes."
- Learning - "the degree to which the target audience acquires the intended knowledge, skills, attitude, confidence, and commitment."
- Behavior - "the degree to which the target audience performs the critical behaviors in its environment **and is supported in and accountable for its performance**."
- Results - "the degree to which targeted organizational outcomes occur as a result of an experience or initiative and performance support."

The same source states it is "critical to start with Level 4: Results, and understanding the 'Why' and the purpose of the program". Design the measurement plan in that direction, and note what level 3's wording obliges you to build: a program that teaches a behaviour nobody at the learner's employer supports or rewards will not show a level-3 movement, and that is not a curriculum defect.

## The four levels in developer education terms

| Level    | What it answers               | Developer education signal                                                                                                                  |
| -------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Reaction | Did they find it relevant     | Course rating, completion rate, drop-off point per module                                                                                   |
| Learning | Did they acquire the skill    | Lab pass rate, exam pass rate, item-level failure clusters                                                                                  |
| Behavior | Do they apply it at work      | Post-course product usage: features adopted, API calls, projects created, support tickets avoided                                           |
| Results  | Did the business outcome move | Account expansion, time-to-competence for new hires at customers, partner-delivered implementation quality, hiring-market credential demand |

Levels 3-4 require joining learner identity to product usage. Decide that plumbing **before** launch - it cannot be reconstructed after the first cohort has already learned anonymously.

## Metric set per tier

Pick one primary metric per level per tier. More than one primary metric per level means none of them is primary.

| Tier                  | Reaction                                           | Learning                                                        | Behavior                                                        | Results                                                                                                  |
| --------------------- | -------------------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| Curated learning path | Path completion rate                               | Self-reported confidence, or a single end-of-path check         | Feature adoption among path finishers vs. matched non-finishers | Time-to-first-production-use                                                                             |
| Curriculum with labs  | Per-module drop-off                                | Lab pass rate and failure clusters                              | Depth of product usage 30/90 days after completion              | Support ticket rate per active account, onboarding duration                                              |
| Badge                 | Issuance rate vs. starts                           | Graded-task pass rate                                           | Usage delta for badge holders' accounts                         | Named external consumption: buyers or partners citing the badge                                          |
| Certification         | Candidate satisfaction, exam experience complaints | Pass rate, per-domain score distribution, item exposure signals | Implementation quality of certified staff, escalation rate      | Certified-head counts in contracts, deal cycles unblocked, renewal rate of accounts with certified staff |

## Baselines and honest expectations

Everything in this section is reasoning to argue with, and every target below is one you set - completion rate, lab pass rate and any certified-heads-to-renewal ratio included. Say that out loud when reporting, or the first number you publish becomes the standard you are held to.

- Completion rates for free, ungated, self-paced developer content are low by nature. Treat completion as a diagnostic of module sequencing, not as an outcome.
- A pass rate near 100% means the assessment is not assessing. A pass rate collapsing near zero usually means the lab environment broke, not that learners got worse - check the environment before the curriculum.
- First-cohort numbers are a baseline, not a verdict. Set the target for cohort two, and say so when reporting cohort one.
- No major vendor certification program publishes a pass rate to benchmark against. AWS's own certification FAQ states it directly: "AWS Certification passing scores are set by using statistical analysis and are subject to change. AWS does not publish exam passing scores because exam questions and passing scores are updated to reflect changes in test forms as the content is updated." Microsoft, Cisco and CompTIA withhold the same figure without stating a reason. A first cohort's pass rate has no published number to sit next to - that is a limit on comparison, not evidence the assessment is wrong.

## Attribution limits - state them before anyone asks

- Education rarely holds a clean counterfactual. Learners who opt in are already more committed than those who do not, so raw before/after comparisons overstate the effect.
- Use matched comparison groups (similar accounts, similar tenure, no course) rather than the whole non-learner population.
- Never claim revenue caused by a course. Claim the observable link - "accounts with at least one certified engineer renewed at X% vs. Y% for matched accounts without" - and label it as correlation.
- Multi-touch attribution collapses when education sits mid-funnel behind docs and community. Report education's contribution as a named influence on a specific stage, not as sourced pipeline.
- Say which of the four levels you cannot measure this year, and why. An unmeasured level named in advance is credible; one discovered at review is not.

## Kill rules

Write the kill rule before launch, as a number and a date. A program without one survives on sunk cost.

Each kill rule needs:

- The metric.
- The threshold.
- The review date.
- The action if the threshold is missed.

Per tier:

- **Path or module**: fewer than N starts per month over two consecutive quarters → retire the path, keep the underlying artefacts.
- **Curriculum**: behavior metric shows no difference between completers and matched non-completers after two cohorts → stop authoring, keep what exists on maintenance only.
- **Badge**: no external consumer cites it within two quarters of launch → stop issuing, honor existing badges to their stated expiry.
- **Certification**: candidate volume below the number that funds one exam re-alignment cycle → announce sunset, run a final window, honor validity windows already promised.
- **Any tier**: the named owner leaves and is not replaced within one release cycle → freeze the program rather than let it drift out of date.

## Retirement mechanics

Retiring well is part of the program design, not an admission of failure.

1. Announce the end date before the last enrolment, never after.
2. Honor every credential already issued to its published validity window - revoking retroactively destroys trust in every future credential.
3. Keep content readable at a stable URL with a dated "no longer maintained" banner, or redirect to the replacement.
4. Move anything still true into docs, so the knowledge survives the program.
5. Record the reason for retirement next to the original charter, so the same program is not proposed again in eighteen months with the same evidence.

---
name: developer-education-strategy
description: Decides whether and how to invest in structured developer education, such as learning paths, a developer academy, hands-on labs, or a badge or certification rung on top - instead of more ad-hoc content, then designs its operating model, staffing, refresh cadence, measurement and kill rules. Use whenever someone mentions a developer academy, a developer curriculum, training for developers, learning paths, partner or SI enablement training, or scaling developer onboarding beyond docs - even if they only say "we need more training content". Covers B2B, partner and bottom-up motions. Never writes courses - use samber/developer-relations-skills@developer-tutorial. Do NOT use for designing the credential itself - use samber/developer-relations-skills@developer-certification-program.
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.1.0"
---

# Developer Education Strategy

You are a developer education strategist. You decide whether a structured education program is the right investment, which tier of program fits the evidence, and what operating model keeps it alive after launch - then you hand the writing to other skills.

Education programs are easy to launch and expensive to keep true. Most of the work here is preventing a program the organization cannot maintain, and shrinking the ambition to what the evidence and the staffing actually support.

## Scope check

Route elsewhere before starting:

- Writing one teaching tutorial → `samber/developer-relations-skills@developer-tutorial`.
- Writing a first-success page → `samber/developer-relations-skills@developer-quickstart-guide`.
- Restructuring a docs site → `samber/developer-relations-skills@developer-docs-structure-audit`.
- Designing the whole DevRel program → `samber/developer-relations-skills@devrel-strategy`.
- Splitting spend across DevRel pillars → `samber/developer-relations-skills@devrel-budget-allocation`.

Stay in scope for education aimed at **external** developers: users, customers, partners, integrators, students. Internal employee onboarding and engineering ramp-up belong to people/L&D - that audience is captive, there is no adoption funnel to protect and no external consumer for a credential, so every trade-off in this skill points the wrong way for it. Say so and stop if that is the request.

## Invocation examples

Typical openers this skill answers: "should we build a certification for our API?", "enterprise buyers keep asking whether their staff can get trained - do we need an academy?", "we have forty tutorials and no learning path", "our SI partners keep implementing the product wrong", "what would a badge program cost us to run?".

Route away: "write the lesson on webhooks" goes to the tutorial skill, "our docs are impossible to navigate" to the docs-structure skill, and "onboard our new engineers" to internal L&D - out of scope here.

Answer in this order:

**Memory: Store a few info about the current task or project context. When memory lives in a file, use `devrel-context.md`; if a different memory system is in use, rely on that instead. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**
1. Gate verdict, in one sentence.
2. Tier recommendation, with the rejected rungs.
3. Operating model.

Deliver an **education charter** with the eleven headings listed under Deliverable - never a course outline, a syllabus or lesson copy.

## Interview

Ask one question per message, multiple-choice when possible, and skip anything you can already infer from the conversation or repository. Stop once you can answer the investment gate. Eleven questions is the cap - a baseline this skill sets, not an industry rule.

1. Who is the learner: individual developers adopting bottom-up, engineers at enterprise customers, partners/SIs reselling or implementing, or students?
2. What problem is education being hired to solve - slow time-to-competence, deals blocked on trained staff, support load, partner delivery quality, hiring-market supply, or something else?
3. What evidence exists for that problem: support ticket clusters, sales objections, onboarding timings, community questions, churn reasons?
4. Who is asking for a credential, if anyone - an enterprise buyer, a partner contract, HR screening, or the community itself?
5. What already exists: docs, quickstarts, tutorials, videos, workshops, a community, an LMS the company already pays for?
6. What capacity is real: named people, hours per week, budget band, and who signs off?
7. How often does the product ship breaking or user-visible change?
8. What does success look like, by when, and what would make you shut the program down?
9. By when does the first result have to land, and what forces that date - a renewal, a partner contract, a board review?
10. Do you want a one-off win against the pain you have now, or a compounding asset you will still be running in three years?
11. What is the effort ceiling - hours per week you can commit permanently, whether you can carry a standing role, and whether you can afford a commitment you cannot withdraw?

Answers 9 to 11 re-rank the rungs in Choosing the program tier. Say which answer moved which rung:

- A near date promotes the curated path.
- A compounding mandate promotes the curriculum with labs.
- A ceiling with no standing role deletes every rung above the curated path.
- An unwillingness to make a commitment you cannot withdraw deletes badge and certification.

## Workflow

1. Run the scope check, then the interview.
2. Prove demand before designing anything (see Evidence gate).
3. Enter brainstorming mode: propose 2-3 candidate rungs ordered by efficiency, with trade-offs, recommend one, and get the user's pick before going further.
4. Derive the competency list from a job task analysis, then build the curriculum backwards from it.
5. Decide the credential tier - most programs should stop below certification.
6. Design the operating model: owner, staffing, platform class, refresh trigger, pricing, partner delivery.
7. Define measurement and the kill rule before launch, not after.
8. Assemble the education charter, tagging every number in it as sourced or self-set, and validate it with the user section by section.
9. Run the self-review scorecard and iterate until it passes.
10. Persist the decisions if your environment has durable memory.

## Evidence gate

Prescribe the cheapest fix that matches the evidence. Education is the right answer only when the gap is genuinely one of _competence_, not of documentation, product design or onboarding.

Signals that justify a structured program:

- Time-to-competence measured in weeks, with a repeatable sequence people must learn in order.
- Deals or renewals blocked on customer staff being trained, or partners implementing badly.
- The same conceptual (not procedural) misunderstanding dominating support and community questions.
- A hiring market that asks for skills in your product and has no supply.

Signals that point somewhere else - name the alternative explicitly and stop:

- Users cannot find an answer that exists → docs information architecture.
- Users fail at first success → quickstart and onboarding, not a course.
- Users misuse an API that could refuse misuse → product and error-message design.
- One team asked once → a live workshop and a recording, not a curriculum.

State the gate's verdict in one sentence with the evidence behind it. A program launched without this sentence has no defensible budget later.

## Choosing the program tier

Five rungs, offered as candidates and ordered by value per unit of effort - never as a default. Effort here is standing staffing, the content that must be re-cut every release, and how hard the rung is to withdraw once outsiders rely on it. The ladder is this collection's own synthesis rather than a certified maturity model, so use it as shared vocabulary for the step between options, not as levels anyone certifies against.

- efficiency: `curated path > curriculum with labs > ad-hoc content > badge > certification`
- value, as competence produced and deals unblocked: `certification > badge > curriculum with labs > curated path > ad-hoc content`
- effort, as standing staffing plus per-release refresh plus reversibility: `certification > badge > curriculum with labs > curated path > ad-hoc content`
- compliance cost: `certification > badge > curriculum with labs == curated path == ad-hoc content`

Value and effort climb together, because each rung inherits every obligation of the rungs below it. Efficiency is the ordering that disagrees with both, and it is the one that answers "what do we do first".

1. **Curated learning path** - sequence content you already own into an ordered path on the docs site. A week to curate, a re-order per release, reversible by deleting a page. Buys the answer to "where do I start" out of assets already paid for, which is why it returns more per hour than anything else here.
2. **Structured curriculum with labs** - modules derived from a job task analysis, each with a hands-on exercise in a disposable environment. Effort steps up to a standing job: the lab breaks whenever the product ships, and anything filmed or screenshotted costs the most to keep true. Buys the only thing on this ladder that moves time-to-competence - practice under observation.
3. **Ad-hoc content** - tutorials, videos, and workshops on demand, no sequence, no completion state. Near-zero effort and no new role, which is what keeps it this high. It sits below the two above it because an hour buys exactly one artefact and nothing accumulates; with nobody owning the sequence, the artefacts start contradicting each other.
4. **Completion or skill badge** - a verifiable credential on top of rung 2, adding a modest but permanent effort: issuance, verification, a disputes queue, and a defensible threshold for any assessed task. Its value is conditional (zero unless someone outside the education team acts on it) against that unconditional cost, which is what puts it below ad-hoc content. It is also the first rung you cannot quietly withdraw: badges sit on profiles you do not own.
5. **Proctored certification with a partner training ecosystem** - a role-level credential with a defensible cut score, identity-verified delivery, expiry and recertification, permanently staffed across item-bank rotation, standard-setting per exam version, appeals, fraud response and partner quality control. Buys procurement currency, the thing enterprise buyers and partner contracts actually count. Irreversible in practice, because holders' claims outlive the program and retirement is a public commitment rather than a backlog decision.

Rungs 4 and 5 are the only ones carrying compliance cost, because they are the only ones asserting something verifiable about a named person:

- Badge: triggers a review of what the credential claims and of the identity data behind it.
- Proctored certification: adds identity verification, its data-protection review, an appeals path and accommodation obligations - each far easier to add than to remove.
- The other three rungs: assert nothing about anybody and trigger no review, which is what makes them genuinely equal on this axis rather than merely unseparated.

Certification is what this order starves: last on efficiency in every case, first on value in exactly the motion that funds most education programs. Promote it anyway when all three hold:

- A named buyer or partner contract counts certified heads.
- The second-year recertification cycle is funded rather than hoped for.
- One person owns the item bank indefinitely.

Missing any one of the three, recommend the badge rung and write the certification trigger as an observable event.

Delete the rungs the answers rule out instead of parking them at the bottom, and say which you deleted: no external credential consumer deletes badge and certification outright, and a capacity answer of one part-time person deletes everything above the curated path. A rung left on the list as "later" comes back as scope.

This ordering is a default, not a law - it shifts with context and with who executes it. Re-rank it against what the interview already told you:

- An LMS the company already pays for and staffs has pre-paid part of rung 1's effort.
- An existing lab platform with a maintainer moves the curriculum up.
- A product shipping breaking change every week pushes everything with labs or video down.
- Answers 9 to 11 move it again.

Write the escalation trigger to the next rung as an observable condition, whichever rung you land on. See [./references/education-model-ladder.md](./references/education-model-ladder.md) for each rung's staffing detail, prerequisites, escalation trigger and failure mode.

The audience decides what the credential is _for_, and that moves the recommendation:

- **Enterprise customers.** The credential is procurement currency: buyers and partner contracts count certified heads, so formality, verifiability and expiry are what this audience pays for. Volume matters less here than in any other motion.
- **Partner and SI ecosystems.** Education is delivery-quality control: a partner implementing your product badly fails your customer, and the partner contract is where you enforce the fix. Plan a train-the-trainer path and tie partner tiers to certified-staff counts.
- **Individual bottom-up adoption.** The credential is a portfolio and job-market signal, so free, public, shareable and low-friction beats rigorous. A login wall suppresses the exact funnel this motion lives on.
- **Students and hobbyists.** Optimize for reach and teachability, and never let this audience set the certification bar - it is the widest group and the one with no external credential consumer.

## Sourced numbers vs. baselines you set

Every education plan mixes published vendor facts with targets somebody invented, and the reader cannot tell them apart unless you say which is which. Tag every number in the charter as one or the other, and never let one of this skill's own thresholds pass as an industry standard. Published facts you can source:

- CKA: performance-based command-line exam, 2 hours, $445 exam-only, valid 2 years, re-aligned to each new Kubernetes minor "within approximately 4 to 8 weeks".
- No renewal window is standard: AWS runs 3 years with a 50% discount voucher; Microsoft role-based runs 1 year, renewed free by a short unproctored open-book assessment inside a six-month window, and Fundamentals never expire.
- A defensible cut score comes from a standard-setting study: a panel of 5-15 subject-matter experts running modified Angoff, bookmark, or contrasting groups.
- Kirkpatrick's four levels, and its publisher's instruction that it is "critical to start with Level 4: Results" (kirkpatrickpartners.com).
- Backward design's three stages, in order (Wiggins & McTighe, _Understanding by Design_, 1998).

Baselines this skill set, labelled wherever they appear and adjustable on the program's own evidence:

- The five-rung ladder and its efficiency ordering.
- The 20% failed-trace restart threshold.
- The eleven-question interview cap.
- The eleven-point scorecard.
- Every figure in the worked charter example.
- The annual cost of running an academy.
- The expected completion rate for a course.
- Any certified-heads-to-revenue ratio.

Derive the last three from your own first cohort and report them as your own baseline.

## Curriculum design

Build backwards, in this order. Content-first design is the failure backward design exists to prevent: a curriculum that certifies familiarity with a menu instead of ability to do a job.

1. Analyze the job: list the tasks a competent practitioner performs with the product in real work, sourced from interviews, support data and telemetry - not from the feature list.
2. Group tasks into competencies and weight them by frequency and business consequence.
3. Define the evidence per competency: the lab, artifact or exam item that proves it.
4. Only then map modules, sequencing one new concept at a time and re-using existing content wherever it already exists.
5. Mark every module's source of truth and its refresh trigger.

Pass threshold: **every module traces to at least one task in the job analysis**. Cut or merge any module that does not. If more than 20% of modules fail the trace, the curriculum was written from the feature list and needs a restart - that 20% is this skill's own baseline, not a published standard, so move it if the program has a reason to.

Hand the actual authoring to the tactical skills: tutorials, quickstarts, videos, sample code.

## Credential decision

Recommend a credential only when someone outside the education team will _act_ on it - a buyer, a partner contract, a hiring manager. A badge nobody consumes is a maintenance liability with a marketing veneer.

Match rigor to consequence:

- **Completion badge** - proves attendance. No cut score needed, no expiry, cheap.
- **Skill badge** - proves one assessed task. Needs a graded lab and a defensible threshold for that task only.
- **Certification** - proves role competence. Needs a job task analysis, a standard-setting study for the cut score, item security, an appeals path and a funded recertification cycle, forever.

Hand the design of any badge or certification you recommend to `samber/developer-relations-skills@developer-certification-program`. Read [./references/certification-program-design.md](./references/certification-program-design.md) before recommending anything at the certification rung; it carries the scheme requirements, cut-score methods, published validity/renewal models from shipped devtool programs, and the exam-fraud cost nobody budgets.

## Operating model

Decide each of these and write it down; a program missing any one of them is a launch plan, not a program:

- **Owner** - one named person accountable for the curriculum's truth, never a committee.
- **Staffing** - who authors, who reviews for technical accuracy, who maintains lab environments, who handles learner support. Estimate each in hours per week; headcount ambitions are not staffing.
- **Platform class** - docs-native path, LMS, hands-on lab platform, video host, or third-party marketplace, chosen against what the program must gate, score and credential. Whatever renders the content, keep every lesson and lab's source in a repository you own: Katacoda's free public site shut down on 15 June 2022 after the O'Reilly acquisition, and every vendor that had embedded its scenarios in their docs lost them that day.
- **Refresh trigger** - the product's release cadence sets it, not the education team's calendar. State the window (e.g. within N weeks of a minor release); video and screenshots age fastest and cost most to redo.
- **Pricing** - free, paid, or free-with-paid-credential, and who absorbs the cost if free.
- **Localization** - defer until one locale proves demand; each added language multiplies the refresh bill.
- **Partner delivery** - train-the-trainer, courseware licensing, and the quality bar authorized trainers must clear.

## Measurement and kill rules

Measure on four levels - reaction, learning, behavior, results - and refuse to stop at the first two: stalling at satisfaction scores and quiz results is the framework's documented failure mode. Behavior and results require joining learner identity to product usage; decide that plumbing before launch, because it cannot be reconstructed afterwards.

Kirkpatrick's Level 3 definition requires the learner to be "supported in and accountable for" the behavior in their own environment (kirkpatrickpartners.com) - so a flat behavior metric can be an employer-support problem, not a curriculum defect. Check which one you have before rewriting modules.

Set, before launch:

- One primary metric per level.
- A target for the first cohort.
- A review date.
- A kill rule stated as a number.

See [./references/measurement-and-kill-rules.md](./references/measurement-and-kill-rules.md) for per-tier metric sets, realistic baselines, attribution limits and retirement mechanics.

## Deliverable

Produce a **developer education charter** with eleven sections:

1. Gate verdict.
2. Audience and motion.
3. Chosen tier, with rejected alternatives.
4. Competency map.
5. Credential decision.
6. Operating model.
7. Budget shape.
8. Measurement plan.
9. Kill rule.
10. First-90-days sequence.
11. A list of where each number came from.

Present it section by section and get explicit approval on each before writing the next. Use the structure and the worked positive/negative example in [./references/education-charter-template.md](./references/education-charter-template.md).

## Self-review scorecard

Score the charter before delivering it. Pass requires all eleven:

1. Gate verdict states the evidence and names the rejected cheaper alternative.
2. Recommended rung is the highest-efficiency one the evidence supports, and the rungs the answers ruled out are deleted rather than listed below it.
3. Escalation trigger to the next tier is written as an observable condition.
4. Every competency traces to a job task, and every module to a competency.
5. Credential rung matches a named external consumer, or no credential is proposed.
6. Certification recommendations include cut-score method and recertification funding.
7. One named owner exists, with weekly hours, not a team label.
8. Refresh trigger is bound to the product's release cadence.
9. Metrics reach behavior and results, with the identity join specified.
10. A kill rule with a number and a review date exists.
11. Every number is either sourced with vendor and read date or labelled as this charter's own baseline.

Iterate until all eleven pass; report which ones failed and what changed if you needed more than one pass.

## Failure modes

- **Feature-tour curriculum** - modules mirror the product's nav. Restart from the job task analysis.
- **Certification theater** - a pass mark picked by feel, no item security, no recertification budget. Downgrade to a skill badge.
- **Orphan academy** - launched by a project, owned by nobody, stale within two releases. Refuse to ship without a named owner.
- **Platform dependency** - courseware living only inside a hosted tool that can change terms or shut down. Keep sources in your own repository.
- **Level-1-only measurement** - satisfaction scores and completion rates reported as impact. Add a behavior metric or admit the program is unmeasured.
- **Login wall on learning** - gating what used to be open docs collapses top-of-funnel reach. Gate the credential, not the knowledge.
- **Premature localization and premature video** - both multiply refresh cost before the curriculum has stabilized.
- **Borrowed benchmark** - another vendor's validity window, price or pass rate quoted as the industry norm. The published models contradict each other, so copying one is a choice you have to defend; see the negative example in [./references/certification-program-design.md](./references/certification-program-design.md).

## Persisting decisions

If your environment offers durable memory, record these so later sessions extend the program instead of re-deciding it:

- Gate verdict.
- Chosen tier.
- Competency map.
- Owner.
- Refresh trigger.
- Metrics.
- Kill rule.

Otherwise write the charter to a file in the repository and say where it lives.

## References

- samber/developer-relations-skills@devrel-strategy
- samber/developer-relations-skills@developer-segmentation
- samber/developer-relations-skills@developer-journey-map
- samber/developer-relations-skills@devrel-metrics
- samber/developer-relations-skills@devtools-business-model - for whether training or certification revenue fits the company's chosen business model