api-test-mode-design

{
  "skill_name": "api-test-mode-design",
  "evals": [
    {
      "id": 1,
      "prompt": "I'm the CTO at Payflux, a payments API for marketplaces - we move money between platforms and their vendors, and merchants embed our hosted checkout page. Our biggest prospect, a logistics platform, will only sign if their developers can start integrating within 6 weeks. Today integrators test against our staging environment using real card numbers from our corporate card. We have 4 backend engineers. One is pushing to spend the next two quarters building a completely separate sandbox environment with its own hosts and accounts, because that's what the big platforms do. Another says we should skip building anything and just publish a doc telling partners to create a separate account and be careful with test data. What should we build?",
      "expected_output": "An isolation-model recommendation that picks the soft toggle for the 6-week deadline, deletes the separate-copy build from this pass with a revival condition, rules the documented-convention option out entirely for a money-moving API, and sketches keys, magic values, and the graduation gate.",
      "files": [],
      "expectations": [
        "Recommends the soft toggle (mode flag on the live account with mode-prefixed keys routing to test infrastructure) as the model to build now, not the fully separate sandbox environment.",
        "Explicitly rules the no-test-mode / documented-convention option out entirely for this API rather than listing or ranking it as a viable last option, because money flows through the API.",
        "Ties the choice to the hard 6-week deadline and weeks-level engineering budget as the inputs that promote the soft toggle over the separate-copy environment.",
        "Removes the separate-copy sandbox from this build pass and names a concrete condition that revives it, such as a first cross-mode leakage incident or a compliance regime landing.",
        "Specifies a mode flag (livemode-style boolean) on every API object so test data is invisible to live calls and vice versa.",
        "Specifies mode-prefixed API keys using the recognized type-prefix plus _test_/_live_ mode-substring shape rather than inventing a novel format.",
        "Calls out that testing with real card numbers must stop, treating it as a compliance/terms violation rather than a mere inconvenience.",
        "Proposes magic test card numbers as the first simulation primitive and includes deliberate-failure values (specific declines), not only a success card.",
        "Requires each magic value to map onto a real published error/decline code from the API's error catalog, never a sandbox-only error shape.",
        "States the sandbox's isolation level explicitly in the docs instead of letting the word 'sandbox' imply it.",
        "Notes that graduation to live will be gated by a compliance step (KYC-style activation) and recommends integrators submit it early because review can take days.",
        "Recommends a mirrored test-mode version of the hosted checkout page, so the end-user-facing surface is testable, not just API responses.",
        "Recommends enforcing the mode boundary bidirectionally: live keys reject test card values and test keys reject real cards."
      ]
    },
    {
      "id": 2,
      "prompt": "We run Textwire, an SMS and voice API. Here's the test-mode plan from our platform team - tell us what to change. Magic test numbers will return errors in a dedicated sandbox format, {\"sandbox_error\": \"...\"}, so integrators can tell simulated errors apart from real ones. We'll rotate the magic numbers every quarter so nobody hardcodes them. One master list applies everywhere: +15005550001 means 'invalid' whether it's used as an SMS sender, a voice call destination, or in number purchasing. To let integrators run one true end-to-end check, test credentials will be allowed to send to a verified teammate's real phone. And integrators have asked to test what happens when a queued message gets rejected by a carrier hours after submission - we haven't decided how.",
      "expected_output": "A corrected magic-value design: real error taxonomy instead of a sandbox error shape, stable non-rotating values, per-operation labeling, a hard no-real-message guarantee, and a simulation endpoint for the delayed carrier rejection.",
      "files": [],
      "expectations": [
        "Rejects the dedicated sandbox error format and requires magic numbers to return the API's real published error taxonomy and codes.",
        "Rejects quarterly rotation: magic values are a stable, non-expiring API contract that integrators' CI suites will depend on.",
        "States the asymmetry that adding new magic values is safe while changing or removing existing ones is a breaking change.",
        "Rejects the single flat master list and requires each value labeled per operation and field, because the same number can mean different things in different calls.",
        "Identifies the unlabeled multi-operation catalog as a documented integrator trap (cross-referencing the wrong operation's meaning).",
        "Rejects letting test credentials send to a real teammate's phone: the hard guarantee for a communications API is that no real message ever leaves the sandbox.",
        "Recommends the strong isolation posture where test credentials cannot even reference production resources or real numbers.",
        "Uses phone numbers/addresses as the magic-value carrier, matching the communications domain, rather than card-number or username-style values.",
        "Includes deliberate-failure values (invalid, unroutable, blocked, or not-capable numbers), not just valid/success numbers.",
        "Addresses whether simulated sends fire status callbacks/delivery events, requiring the behavior to be documented explicitly rather than left implicit.",
        "Recommends an on-demand simulation endpoint for the delayed carrier rejection, since that outcome is not a function of the original request input and no magic number can encode it.",
        "Recommends bidirectional mode-boundary enforcement: live credentials reject magic numbers and test credentials reject real ones.",
        "States the sandbox's isolation level explicitly in documentation instead of relying on the word 'sandbox'."
      ]
    },
    {
      "id": 3,
      "prompt": "Renewly is a subscription-billing API. Integrators need to test: a 14-day trial expiring, a monthly renewal charging, dunning after a failed payment (3 retries over 7 days), and an annual plan renewing. Our plan: in the sandbox, trials last 10 minutes instead of 14 days and renewal jobs run every 15 minutes, so developers see everything happen the same day. For anything longer, the docs will say 'contact support to fast-forward your account'. We're also adding a 'fire webhook' button in the dashboard that sends a sample subscription.renewed event with example payload data - and our docs tell integrators to always verify webhook signatures. Developers also complain they have to click through creating a customer, a subscription, and three failed payments every time they want to reach the dunning state. Review this design.",
      "expected_output": "A review replacing compressed durations and support-mediated fast-forwards with a deterministic test-clock primitive, coupling test events to real state, adding seeding/reset endpoints, and holding the design to the event/time coverage gate.",
      "files": [],
      "expectations": [
        "Rejects the compressed-durations approach (10-minute trials, 15-minute renewals) and the contact-support fast-forward as the core time-testing mechanism.",
        "Recommends a deterministic test-clock primitive: a clock object with frozen time that test objects attach to, advanced deliberately, with state transitions and events firing as if the time had passed.",
        "Recommends capping both how far a single clock advance may jump and how much one clock can carry, as a deliberate simulation-complexity cap.",
        "Flags the fire-webhook button's sample payload as decoupled from object state, requiring events either coupled to real state transitions or the decoupling documented so integrators don't wait for events that never come.",
        "Requires every async event type to be fireable on demand by the integrator self-service, never via support tickets.",
        "Confirms the dunning retry schedule (3 retries over 7 days) is exercisable through simulated time without waiting real days.",
        "Applies the event/time coverage gate: every time-dependent flow must be testable via simulated time, and any flow testable only by waiting real calendar time fails the audit.",
        "Mentions auto-aging fixtures (date fields shifting daily) as a cheaper rung for simple cases, distinct from the full clock primitive.",
        "If the dashboard trigger cannot produce a valid webhook signature, requires stating that limitation next to the 'always verify signatures' rule so the rule doesn't break the test tooling.",
        "Recommends object-minting/seeding endpoints so an object in a target state (e.g. mid-dunning) is reachable in one call instead of manual click-through replay.",
        "Recommends reset endpoints that return a developer's test environment to a clean state.",
        "Recommends deriving sandbox configuration from the same infrastructure-as-code as production so any difference is intentional and documented."
      ]
    },
    {
      "id": 4,
      "prompt": "Ledgerlink aggregates bank-account data for fintech apps. Our sandbox v2 plan: export 50,000 real accounts from production, scrub names and emails, and load them as sandbox data so integrators get realistic transaction histories. One shared sandbox for all integrators, and no cleanup jobs since the data is read-mostly. For error testing we'll add /sandbox/force_error?code=X, which makes whatever call comes next fail with that code. Integrators keep asking for accounts shaped like their end users: gig workers with three income streams, small businesses, joint accounts, thin-file students. How should we handle seeding and errors?",
      "expected_output": "A seeding design that bans production data even scrubbed, uses persona-style magic identities with credential-parameterized errors, promotes a config-driven fixture builder with a starter library for the varied shapes, and adds resets to the shared sandbox.",
      "files": [],
      "expectations": [
        "Rejects loading scrubbed/anonymized production accounts into the sandbox: seed data must be synthetic only, never copied from production.",
        "Uses test identities/personas (usernames selecting fixture shapes) as the magic-value carrier, matching the data-aggregation domain.",
        "Replaces the ambient force_error next-call toggle with deterministic per-request error selection, such as parameterizing the error on the credential (e.g. a password suffix selecting the error condition).",
        "Promotes the config-driven fixture builder because integrators' data shapes genuinely vary, naming the data-aggregation domain as the promotion condition.",
        "Ships a starter-fixture library alongside the fixture builder rather than making every integrator hand-write configs.",
        "Maps the requested shapes (gig worker with multiple income streams, small business, joint account, thin-file student) onto named test personas or starter fixtures.",
        "Rejects 'no cleanup needed': a shared sandbox needs automated periodic resets so accumulated test data doesn't drift into an unrepresentative state.",
        "Recommends both object-minting endpoints (seed a known state in one call) and reset endpoints (return to a clean state) for integrators.",
        "States the shared sandbox's isolation level explicitly in docs - shared and not per-integrator isolated - instead of letting 'sandbox' imply isolation.",
        "Recommends auto-shifting fixture date fields so the newest data is always 'today', keeping transaction histories fresh without manual reseeding.",
        "Requires deliberate failure personas/values to map onto the real published error codes, not a sandbox-only error shape.",
        "Applies the failure-path coverage gate: every documented failure outcome deterministically triggerable, with a single untriggerable path failing the audit.",
        "Requires the environment boundary to prevent test credentials from referencing or touching real accounts."
      ]
    },
    {
      "id": 5,
      "prompt": "Chargeline is a payments API and two problems landed the same week. First, fraudsters are card testing us: thousands of small authorizations with stolen cards to see which are still valid. Our security lead wants the sandbox locked down hard - CAPTCHA on sandbox signup and a sandbox rate limit of 10 requests/minute (production allows 100/second) - because 'the sandbox is where attackers practice'. Second, a growth engineer discovered two competitors mass-creating free sandbox accounts and scraping our simulated data to reverse-engineer our pricing model. Design our sandbox quota and abuse posture.",
      "expected_output": "An abuse design that separates card testing (a live-mode control problem) from evaluation-volume abuse (the real sandbox quota case), keeps production rate limits in the sandbox by default, and models the anti-scraping control as an aggregate volume-and-scope cap.",
      "files": [],
      "expectations": [
        "Separates the two abuse classes explicitly and assigns each its own mitigations instead of one blanket sandbox lockdown.",
        "States that card testing targets live checkout endpoints and the sandbox is generally useless for carding because test transactions never touch real card networks.",
        "Redirects card-testing mitigations to live mode: edge bot challenges/CAPTCHA on live checkout and endpoint-scoped rate limits.",
        "Includes live-mode rules capping instruments per customer and consecutive declines per IP as carding controls.",
        "Cites the quantified precedent that managed CAPTCHA on checkout cut card testing by about 80% with under 0.02% impact on authorization rates.",
        "Rejects the 10 requests/minute sandbox limiter as the default and recommends reusing production rate limits unchanged in the sandbox as the default posture.",
        "Gives the reason for limit parity: identical limits keep integrators' throttling and backoff code honest before they hit production.",
        "Treats 'every vendor publishes a distinct sandbox throttle' as false, noting sandbox-specific numeric limits are sparsely documented across vendors and several reuse production limits.",
        "Identifies the competitor scraping as the evaluation-volume abuse case that genuinely justifies a dedicated sandbox control.",
        "Models that control as an aggregate volume-and-scope cap on the pre-production tier, explicitly not as a faster per-minute limiter.",
        "Keeps leaked-key mitigations distinct: server-side-only secrets, scoped/restricted keys, and secret-scanning enrollment.",
        "Notes test keys are not risk-free - on a toggle model they still read and write real account configuration - so they deserve real secret hygiene.",
        "Does not endorse sandbox-side CAPTCHA or a sandbox lockdown as a fix for the card-testing problem."
      ]
    },
    {
      "id": 6,
      "prompt": "Our first integrator, a booking platform, just finished building against Meshpay's sandbox: 400 green test payments, webhook handlers working, and they're launching Friday. Our activation doc currently says: 'Replace sk_test with sk_live in your environment variables and you're live.' Also, when an account that hasn't finished our review calls the live API, we return a plain 401 Unauthorized. Is the key swap enough, or what else should we and the integrator do before Friday?",
      "expected_output": "A graduation design: a compliance gate submitted early, a legible not-yet-approved error instead of the bare 401, a full cutover checklist with fresh webhook secrets and a reversed live canary, and the pitfalls published as integrator docs.",
      "files": [],
      "expectations": [
        "States the key swap alone is not enough: graduation is never purely technical and needs a designed gate plus a cutover checklist.",
        "Identifies a compliance/review gate matching the payments domain (KYC-style activation) and recommends submitting it early because review can take days.",
        "Replaces the plain 401 for unapproved accounts with an explicit, legible 'environment not yet approved' error.",
        "Requires fresh live-mode webhook endpoints and event subscriptions with newly captured live signing secrets, because test-mode signing secrets do not carry over.",
        "Requires test credentials removed from production configuration, not merely live credentials added.",
        "Requires abuse-prevention and fraud rules configured before the first live transaction, not after an incident.",
        "Requires a small live canary transaction per integration path before the integration is declared launched.",
        "Specifies the canary must confirm both the operation's effect/settlement and its event delivery, and ideally be reversed or refunded afterwards.",
        "States that a green sandbox run is necessary but not sufficient: the sandbox does not replicate real card networks, live settlement timing, or full-strength fraud systems.",
        "Recommends publishing the cutover checklist and the recurring cutover pitfalls to integrators as documentation.",
        "Names the cross-environment paste pitfall (live key in a sandbox context or vice versa) and its opaque authentication-failure symptom as a recurring support generator.",
        "Names under-tested real-money failure paths (real declines, disputes, settlement delay) as a pitfall that sandbox happy-path testing never forces.",
        "Includes a reactive go-back trigger: webhooks silently stopping after launch usually means a stale or unregistered live-mode endpoint from a silently skipped cutover step."
      ]
    },
    {
      "id": 7,
      "prompt": "We're Hookline, an order-management API - retailers push orders in and we route fulfillment. A senior dev proposes we generate a mock server from our OpenAPI spec with Prism, host it at sandbox.hookline.dev, and label the docs page 'Sandbox'. Zero backend work, ships this week, and it returns spec-valid responses for every endpoint. Integrators mostly need to test the order lifecycle - create, allocate, ship, deliver - plus cancellations mid-flow, and they listen for shipped and delivered webhook events. Good idea?",
      "expected_output": "A rejection of the mock-as-sandbox proposal on statefulness and drift grounds, with a real stateful test mode designed instead: soft toggle, state-forcing simulation endpoints, on-demand lifecycle events, and an explicit isolation statement.",
      "files": [],
      "expectations": [
        "Rejects shipping the spec-generated mock server as the sandbox.",
        "Gives the statefulness reason: the order lifecycle spans a sequence of calls, and a static request-to-response mapping cannot validate logic that depends on what happened on a prior call.",
        "Names the drift failure mode: when the API changes, the mock keeps simulating the old behavior while production breaks silently, which vendor-exercised real objects cannot do.",
        "States that the bugs a stateless mock misses are the state-dependent ones surfacing weeks into an integration, not day-one request-shape errors.",
        "Recommends building a real stateful test mode instead, with the soft toggle named as the default isolation rung.",
        "Requires the docs to state the isolation level explicitly rather than shipping the bare label 'Sandbox'.",
        "Requires every documented failure outcome in the lifecycle (failed allocation, mid-flow cancellation) to be deterministically triggerable via magic values or simulation endpoints.",
        "Recommends sandbox state-forcing endpoints that move an order into a specific lifecycle state on demand.",
        "Recommends an explicit fire-this-event-now mechanism for the shipped and delivered webhook events.",
        "Applies the failure-path coverage gate at 100%, with one untriggerable failure path failing the audit.",
        "Specifies mode-prefixed test credentials with the recognized _test_/_live_ substring shape for the real test mode, rather than reusing live keys against a different host."
      ]
    },
    {
      "id": 8,
      "prompt": "Vaultic is an HR-data API covering payroll and employee records. We're designing credentials for our new test mode, and the proposal on the table is: key format VLTC-SBX-<uuid> for sandbox and VLTC-PRD-<uuid> for production, deliberately unusual so attackers can't recognize our keys if they leak. Since sandbox keys can't touch real employee data, we'll let developers commit them to public example repos. Our quickstart has developers use their production key locally, with a config flag in one codebase switching between environments. Sample data will be a scrubbed export of a friendly customer's employee list. And today a key sent to the wrong host returns {\"error\": \"invalid_credentials\"}. Review this.",
      "expected_output": "A credential-hardening review: recognized _test_/_live_ prefix shape for free scanner coverage, real hygiene for test keys, no live secrets in local dev, per-environment separation, synthetic-only seed data, and a legible wrong-environment error.",
      "files": [],
      "expectations": [
        "Rejects the deliberately unrecognizable key format and recommends the recognized type-prefix plus _test_/_live_ mode-substring shape.",
        "Gives the reason: recognized prefix shapes are zero-false-positive secret-scanning/push-protection targets, buying free leak detection from tooling the vendor doesn't control, while a novel scheme buys nothing.",
        "Directly counters the obscurity argument: an unrecognizable key loses scanner protection rather than gaining safety.",
        "Rejects committing test keys to public repos: test keys are not risk-free and deserve real secret hygiene.",
        "Explains why test keys carry risk: on a toggle model they still read and write real account configuration, and the prefix communicates mode, not risk level.",
        "Rejects the quickstart's use of the production key in local development: never live secrets in local dev.",
        "Rejects the single-codebase runtime-flag pattern and recommends per-environment separation of credentials, data stores, and redirect allowlists.",
        "Replaces the opaque invalid_credentials response with a clear, legible wrong-environment error.",
        "Frames the wrong-environment paste as a named recurring support-ticket generator with a design target of zero cross-environment incidents, achieved structurally rather than by warning developers.",
        "Requires bidirectional boundary enforcement: live keys reject test data and test keys reject real records.",
        "Rejects seeding the sandbox with the scrubbed customer employee export: seed data must be synthetic only, never copied production data.",
        "Recommends secret-scanning enrollment for the key format, alongside server-side-only and scoped/restricted keys, as leaked-key mitigations."
      ]
    }
  ],
  "trigger_queries": [
    { "query": "Design a test mode for our payments API - integrators currently test on staging with real card numbers", "should_trigger": true },
    { "query": "Should our sandbox be a toggle on the live account or a fully separate environment?", "should_trigger": true },
    { "query": "What test card numbers should our API publish, like Stripe's 4242?", "should_trigger": true },
    { "query": "Integrators want a button to trigger a fake webhook event on demand", "should_trigger": true },
    { "query": "How can developers test a subscription renewing without waiting 30 days?", "should_trigger": true },
    { "query": "We need magic phone numbers for our SMS API so no real texts go out", "should_trigger": true },
    { "query": "Design the graduation path from our sandbox to production", "should_trigger": true },
    { "query": "Partners need somewhere safe to build against our API before launch", "should_trigger": true },
    { "query": "Should test and live API keys look visibly different so devs can't mix them up?", "should_trigger": true },
    { "query": "How do we make every documented decline code reproducible in test mode?", "should_trigger": true },
    { "query": "Add simulated bank accounts and test personas to our data API sandbox", "should_trigger": true },
    { "query": "What needs to happen when an integrator flips from test keys to production keys?", "should_trigger": true },
    { "query": "A big partner wants to integrate but we have no test environment - where do we start?", "should_trigger": true },
    { "query": "developers keep asking for dummy data to build against before go-live", "should_trigger": true },
    { "query": "Should we add a reset endpoint that wipes a developer's test data?", "should_trigger": true },
    { "query": "Do we need different rate limits in the sandbox than in production?", "should_trigger": true },
    { "query": "Let devs simulate time passing in our billing API to test trial expiry", "should_trigger": true },
    { "query": "Design test credentials that structurally can't touch real customer data", "should_trigger": true },
    { "query": "What should a send-test-event feature actually send?", "should_trigger": true },
    { "query": "Our sandbox only does happy paths - integrators can't reproduce failures", "should_trigger": true },
    { "query": "How do Stripe, Plaid and Twilio structure their test environments?", "should_trigger": true },
    { "query": "Test mode for our email API where no real message can ever leave", "should_trigger": true },
    { "query": "We keep getting tickets from devs pasting live keys into the sandbox", "should_trigger": true },
    { "query": "Seeding strategy for a shared developer sandbox", "should_trigger": true },
    { "query": "a prospect wants demo data in the API before they'll sign", "should_trigger": true },
    { "query": "What's the equivalent of a magic test card for a payroll API?", "should_trigger": true },
    { "query": "Simulated webhook events so integrators can test their handlers safely", "should_trigger": true },
    { "query": "Should objects created in test mode expire, or persist forever?", "should_trigger": true },
    { "query": "we told integrators to make small real payments for testing and refund them, is that bad", "should_trigger": true },
    { "query": "How do integrators test our identity-verification flow without real people?", "should_trigger": true },
    { "query": "Fake buyer and seller accounts for testing our marketplace API", "should_trigger": true },
    { "query": "What usage caps should free sandbox accounts get?", "should_trigger": true },
    { "query": "Design deterministic failure triggers for our test environment", "should_trigger": true },
    { "query": "Is a livemode boolean on every API object the right isolation design?", "should_trigger": true },
    { "query": "Test clock design for renewals and dunning simulation", "should_trigger": true },
    { "query": "Integrators say our sandbox behaves differently from production - how do we get parity?", "should_trigger": true },
    { "query": "When does a test environment deserve its own host instead of a flag on the same API?", "should_trigger": true },
    { "query": "What checklist should integrators run before going live from our sandbox?", "should_trigger": true },
    { "query": "dummy input values that trigger specific error codes on purpose", "should_trigger": true },
    { "query": "Our platform moves real money - how isolated does the test environment have to be?", "should_trigger": true },
    { "query": "How do integrators prove their webhook handling works before we approve them for live?", "should_trigger": true },
    { "query": "external devs shouldn't see real customer records while integrating - what do we build?", "should_trigger": true },
    { "query": "Competitors are scraping our sandbox through free accounts - how do we cap that?", "should_trigger": true },
    { "query": "What makes a good developer sandbox for a banking-data API?", "should_trigger": true },
    { "query": "Should we auto-clean abandoned sandbox data periodically?", "should_trigger": true },
    { "query": "Design the error devs see when their account isn't approved for production yet", "should_trigger": true },
    { "query": "Third-party devs need to rehearse the full order lifecycle without creating real orders", "should_trigger": true },
    { "query": "Our hosted checkout page - do integrators need a test version of that too?", "should_trigger": true },
    { "query": "How do we rotate customer API keys without downtime?", "should_trigger": false },
    { "query": "Should we store API keys hashed or keep them retrievable from the dashboard?", "should_trigger": false },
    { "query": "Design the key format for our API keys - prefix and checksum conventions", "should_trigger": false },
    { "query": "What retry and backoff schedule should our webhooks use when a partner endpoint is down?", "should_trigger": false },
    { "query": "Pick an HMAC signing scheme for our outbound webhooks", "should_trigger": false },
    { "query": "Design our API's error code taxonomy and problem-details envelope", "should_trigger": false },
    { "query": "Should rate limits be per-key or per-org, and what tiers should paid plans get?", "should_trigger": false },
    { "query": "Add idempotency keys so clients can safely retry failed POSTs", "should_trigger": false },
    { "query": "Which integration surfaces should we ship first - REST, webhooks, or an SDK?", "should_trigger": false },
    { "query": "Redesign our developer portal so devs get to their first API call faster", "should_trigger": false },
    { "query": "Design the partner onboarding journey from signup to their first submitted app", "should_trigger": false },
    { "query": "How do we provision a sandbox org automatically for each new partner who signs up?", "should_trigger": false },
    { "query": "Sandbox untrusted Python code execution in our plugin system", "should_trigger": false },
    { "query": "What's the best way to mock the Stripe client in our unit tests?", "should_trigger": false },
    { "query": "Set up pytest fixtures and factories for our Django test suite", "should_trigger": false },
    { "query": "Spin up a staging environment for our backend with Terraform", "should_trigger": false },
    { "query": "Configure the iOS app sandbox entitlements for file access", "should_trigger": false },
    { "query": "Run untrusted user code safely in a Docker sandbox", "should_trigger": false },
    { "query": "Build an interactive playground on our marketing site so visitors can try the API before signing up", "should_trigger": false },
    { "query": "Write end-to-end tests for our React checkout with Playwright", "should_trigger": false },
    { "query": "Seed our local dev database with sample rows for the eng team", "should_trigger": false },
    { "query": "Write the QA test plan for next month's mobile release", "should_trigger": false },
    { "query": "Design OAuth scopes and consent screens for third-party apps", "should_trigger": false },
    { "query": "Set up a status page and incident comms for our API", "should_trigger": false },
    { "query": "Plan our API versioning and v1 deprecation timeline", "should_trigger": false },
    { "query": "Which languages should get official SDKs first?", "should_trigger": false },
    { "query": "Review our REST resource naming and pagination choices", "should_trigger": false },
    { "query": "Design the review process for third-party apps on our marketplace", "should_trigger": false },
    { "query": "Should our public API be GraphQL, and how do we design the schema?", "should_trigger": false },
    { "query": "Give partners a log of failed deliveries with a replay button", "should_trigger": false },
    { "query": "Feature flags for rolling out our new billing engine gradually", "should_trigger": false },
    { "query": "Set up a canary deployment pipeline so bad releases roll back automatically", "should_trigger": false },
    { "query": "Load test our API to find the breaking point before launch", "should_trigger": false },
    { "query": "Set up a honeypot to detect attackers probing our network", "should_trigger": false },
    { "query": "Create a demo org our sales team can use in customer calls", "should_trigger": false },
    { "query": "Add synthetic monitoring probes that hit our API every minute", "should_trigger": false },
    { "query": "Raise unit test coverage on the payments service to 90%", "should_trigger": false },
    { "query": "Set up Pact contract tests between our microservices", "should_trigger": false },
    { "query": "Plan a chaos engineering game day for our infra team", "should_trigger": false },
    { "query": "Anonymize production data so analysts can query it under GDPR", "should_trigger": false },
    { "query": "Generate a mock server from our OpenAPI spec for the frontend team to develop against", "should_trigger": false },
    { "query": "Our production API keys leaked in a public repo - walk me through revoking and rotating them", "should_trigger": false },
    { "query": "Tune our chargeback and fraud rules for card-not-present payments", "should_trigger": false },
    { "query": "Design the KYC onboarding flow for new fintech customers", "should_trigger": false },
    { "query": "Add a Faker-based data generator for our integration tests", "should_trigger": false },
    { "query": "Set up an A/B testing framework for pricing experiments", "should_trigger": false },
    { "query": "Dark-launch the new search backend to 1% of production traffic", "should_trigger": false },
    { "query": "Choose between Snowflake shares and S3 Parquet drops for customer data exports", "should_trigger": false }
  ]
}

# Graduation gates and the cutover checklist

Every vendor surveyed gates production access behind some compliance or review step - graduation is never purely technical, even when the credential swap itself is trivial. Use the gate shapes to pick yours, then hand integrators the cutover checklist.

## Five gate shapes (cross-vendor)

| Vendor  | Gate shape                                  | Mechanics                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Stripe  | KYC account activation                      | Testable immediately after signup; accepting real payments requires a completed KYC application (business, product, personal-relationship details) driven by regulator/financial-partner obligations. Often near-instant, but can require ID, proof of address, bank-ownership, or tax-ID verification. Business origin country cannot be changed after activation - a one-way door inside the compliance step. |
| Plaid   | Tiered, partly automated review             | Sandbox free and immediate. US/Canada teams get an auto-approved Trial plan (up to 10 production Items, gated major-bank OAuth) after identity verification, with flagged cases getting manual follow-up in 2-3 business days. EU/UK still runs a slower (~1 week) compliance process. The only surveyed vendor that moved toward self-service, auto-approved graduation.                                       |
| PayPal  | Approved-partner gate                       | Sandbox credentials work before approval; a live call without approval returns a flat 401 Unauthorized. Going live requires approved-partner status, a business account, and settlement reconfiguration - commonly coordinated with a PayPal representative, not self-service.                                                                                                                                  |
| GitHub  | Publisher verification + install thresholds | Apps work immediately with real data, zero gate. The gate appears only when monetizing: a paid Marketplace listing requires publisher verification (org-owned app, 2FA, verified domain) and install thresholds (GitHub Apps ≥100 installs, OAuth apps ≥200 users). Graduation decoupled from "does it work" and tied to commercial listing eligibility.                                                        |
| Shopify | Physical store-object graduation            | A development store cannot process real transactions and cannot be converted; going live means transferring to a merchant/paid-plan store. Shopify Payments can only be tested on a paid plan - the gate is a plan tier, not a review process.                                                                                                                                                                  |

Design implications: match the gate to the actual obligation (regulatory KYC, partner vetting, commercial listing, plan tier) - don't import a heavier gate shape than the domain requires. Tell integrators to submit the compliance step _early_, since review can take days regardless of vendor. And make the pre-approval failure legible: PayPal's flat 401 for an unapproved live call is the anti-pattern - return an explicit "environment not yet approved" error instead.

## Cutover checklist (provider designs it, integrator runs it)

1. Compliance/verification step submitted and approved - before the planned launch date, not on it.
2. Live credentials issued and swapped in every deployed environment; test credentials removed from production configuration.
3. Fresh live-mode webhook endpoints created, event subscriptions re-created, and live signing secrets captured - a test-mode signing secret does not carry over to live.
4. One small real canary transaction (or equivalent live operation) run per integration path, confirming both the operation's settlement/effect and its event delivery - ideally reversed/refunded afterwards.
5. Abuse-prevention and fraud rules configured before the first live transaction, not after an incident.

The canary step exists because a green sandbox run is necessary but not sufficient. Every vendor surveyed cautions that its sandbox doesn't fully replicate production:

- No real card networks or delivery carriers.
- Instant test settlement versus days-long live settlement.
- Fraud systems not running at full strength.

## The five recurring cutover pitfalls

Recurring across the survey regardless of vendor - each maps to a checklist line above:

1. Forgetting to swap test keys for live keys at launch, leaving the core function broken in production.
2. Pasting a live key into a sandbox context (or vice versa) - an opaque authentication failure rather than a clear "wrong environment" error; called out explicitly as a frequent PayPal support issue.
3. Assuming sandbox success predicts live behavior - "sandbox payments do not prove behavior on real card networks or live payment rails."
4. Forgetting fresh live-mode webhook endpoints, signing secrets, and event subscriptions.
5. Under-testing failure paths that only surface with real money or real delivery (real declines, disputes, settlement delay) because sandbox happy-path testing never forces them.

## Reactive triggers - signals to revisit an earlier stage

Not a one-time checklist; these mean "go back":

- A spike in declines or blocked responses at the payment layer signals an abuse attack in progress - tighten fraud rules and edge controls immediately, not as a one-off.
- Environment or institution-access errors from a tiered-access platform mean the integration never actually cleared the review step it assumed it had.
- Webhooks silently stopping (no errors, no deliveries) usually means a stale or unregistered live-mode endpoint - a cutover step that was silently skipped.

# Isolation models and test-key prefixes: cross-vendor comparison

How six platforms structurally separate test data from live data, ranked from hardest to softest isolation, plus the credential conventions that make the mode visible in the key itself.

## Isolation architecture by vendor

| Vendor                  | Model                                | Mechanics                                                                                                                                                                                                                                                                                         |
| ----------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Plaid                   | Hard separate-copy (hardest)         | Two entirely separate environments on different hosts (`sandbox.plaid.com` vs. production), each with its own `client_id`/`secret`. Explicit rule: "Items cannot be moved between environments. The Sandbox environment supports only test Items." No toggle, no shared account.                  |
| Stripe Sandboxes        | Hard separate-copy                   | A separate, isolated copy of the account with its own keys, webhooks, customers, and configuration - teams run parallel tests without polluting shared test data. Stripe made Sandboxes the default for new accounts: "Test mode is no longer the default testing method."                        |
| Shopify                 | Hard separate-copy (non-convertible) | Development stores the integrator owns and controls; they cannot process real transactions, cannot be converted to production, and cannot be transferred to a client directly - a dedicated "client transfer store" mechanism exists for merchant handoff (a three-way split not seen elsewhere). |
| Stripe legacy test mode | Soft toggle                          | A toggle on the same live account: swap to `sk_test_`/`pk_test_` keys and calls route through test infrastructure. A `livemode` boolean on every object marks which side it belongs to; test data is invisible to live calls and vice versa. Kept for existing accounts alongside Sandboxes.      |
| PayPal                  | Soft toggle over disjoint accounts   | A dashboard toggle switches which credentials and transaction data you see; underneath, the sandbox account "is not linked to your production account in any way," and sandbox transactions live on a separate mirrored test site. Soft toggle UX, hard account-level separation.                 |
| GitHub                  | None (integrator convention)         | No test mode at all. Isolation is achieved entirely by developer convention - separate test apps, organizations, and repositories, plus a tunnel tool to forward webhook deliveries locally. The only surveyed vendor pushing isolation responsibility onto the integrator.                       |

Generalizable lesson: a toggle is cheap to build but risks accidental cross-contamination when the toggle state isn't obvious in the UI. A fully separate copy costs more but makes cross-mode leakage structurally impossible rather than merely policy-forbidden. Stripe running both models side by side - and moving its default to the hard one - is the clearest signal of where a payments-grade platform lands as it matures.

Caveat: even a "true sandbox" is often only partially isolated. Stripe test mode, Paddle sandbox, and Plaid sandbox are all stateful but _shared_, rate-limited, and tied to a provider account, not per-session isolated.

One vendor comparison even found a "Mock" product with full per-user isolation while its "Sandbox" counterpart was the shared, cross-contaminating environment. Never let the word "sandbox" imply an isolation level; state the level explicitly in the design.

Two structural surfaces, not one: PayPal splits the Sandbox Test Site (a mirrored end-user site where fictional accounts log in and see transaction history) from the Developer Dashboard (where the integrator manages those accounts and reads credentials). When the product has an end-user-facing side, buyer-experience testing needs the mirrored surface, not just API responses.

PayPal's sandbox also auto-provisions fictional accounts for _both_ sides of its marketplace - a personal (buyer) and a business (seller) account with generated credentials, recognizable by convention (`sb-[random-string]@personal.example.com`). A two-sided product's test mode must model account variety on both sides, not just transaction variety.

## Test-key prefix taxonomy (Stripe)

Each row is a distinct blast-radius level, not just a mode label:

| Prefix                  | Type                   | Mode         | Client-safe?     | Capability                                   |
| ----------------------- | ---------------------- | ------------ | ---------------- | -------------------------------------------- |
| `pk_test_` / `pk_live_` | Publishable            | test / live  | Yes              | Tokenize, identify account in browser/mobile |
| `sk_test_` / `sk_live_` | Secret                 | test / live  | No (server only) | Full account API access                      |
| `rk_test_` / `rk_live_` | Restricted             | test / live  | No (server only) | Scoped least-privilege access                |
| `sk_org_`               | Organization secret    | -            | No               | Org-level access across accounts             |
| `whsec_`                | Webhook signing secret | per-endpoint | No               | Verify webhook authenticity (not an API key) |

The middle substring (`_test_` vs. `_live_`) is the mode switch. Live secret/restricted keys are shown only once at creation - captured and stored immediately, never re-displayed.

**The convention is externally codified**: GitHub secret scanning / push protection treats `sk_live_`, `sk_test_`, `rk_live_`, and `rk_test_` as officially documented, zero-false-positive push-protection targets. Third-party tooling recognizes and blocks these exact prefixes independent of Stripe's own dashboard. Adopt the widely recognized `_test_`/`_live_` shape rather than inventing a novel one - a recognized prefix buys free, automatic leak detection from scanners the vendor doesn't control.

## Sandbox quotas across twelve vendors

Sandbox-_specific_ numeric rate limits are the sparsest-documented axis in this survey - which is itself the finding. Snapshot: vendors revise published limits.

| Vendor                                                         | Sandbox quota posture                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Twilio, PayPal, Shopify, Coinbase (CDP Checkouts/Payments API) | Production limits applied unchanged in test mode - four of twelve, the plurality position. Shopify adds _functional_ rather than numeric test limits (test gateway incompatible with POS and subscription checkouts, a ~$1 minimum test amount). Coinbase states the parity explicitly in its sandbox documentation rather than leaving it implicit.                                                                                                                                                                                       |
| Stripe, Square                                                 | No distinct sandbox rate limit published. Stripe's nearest functional throttle is unrelated to request volume: the test-clock cap on how much simulated time one advance may jump. Square documents only a qualitative warning that high-volume bursts return a `RATE_LIMITED` error, with no numeric threshold disclosed for either environment.                                                                                                                                                                                          |
| Plaid, Adyen (Legal Entity Management API)                     | Numerically stricter sandbox quotas, tiered or split. Plaid: pre-production tiers cap aggregate call volume and gate major-bank access (Trial supports up to 10 production Items) - a volume-and-scope cap, not a per-minute limiter. Adyen's Legal Entity Management API (its KYC/onboarding product, not its core Payments/Checkout API) publishes an explicit numeric split: 700 requests per 5 seconds live versus 200 requests per 5 seconds test, plus a 5-failures-per-10-seconds throttle - roughly 3.5x more restrictive in test. |
| Checkout.com, Braintree                                        | Documented as stricter in sandbox, but without a full published table. Checkout.com states lower rate limits apply to its sandbox environment, with no figures given for either side. Braintree discloses a sandbox-only throttle (50 requests per minute per IP address, then a 5-minute block) with no equivalent production figure published.                                                                                                                                                                                           |
| GitHub, SendGrid                                               | No sandbox concept to compare against. GitHub's published numbers (60 requests/hour unauthenticated, 5,000 requests/hour for user tokens, plus secondary concurrency and per-endpoint point budgets) apply identically everywhere since there is no test mode. SendGrid's "sandbox mode" is a boolean flag on a single endpoint, not a separate environment with its own credentials or rate-limit tier.                                                                                                                                   |

Design takeaway: don't default to building a separate sandbox limiter. Build one when the failure mode is evaluation-volume abuse - free-tier scraping, load tests against a shared sandbox - and model it on Plaid's aggregate volume-and-scope cap, or Adyen's explicit numeric split, rather than on a faster per-minute limiter.

## Vendors that identify mode without a prefix

An alternative design, not a lesser one - but it carries a named cost:

- **Plaid**: environment-specific `client_id`/`secret` pairs with no shared prefix; the environment identity lives in which host you call, not in the credential string.
- **PayPal**: entirely separate sandbox and live Client ID / Secret pairs. Pasting a live key into a sandbox context (or vice versa) produces an authentication failure - flagged in the research as "a very common support issue," i.e. the cost of a non-self-describing credential is a recurring, avoidable support burden.
- **Twilio**: a separate, region-specific Test Account SID and Auth Token used only for magic-value testing.
- **GitHub**: OAuth tokens, App JWTs, and PATs with no test/live prefix at all - consistent with having no test mode.

Design takeaway: a self-describing prefix is optional for correctness, but it materially reduces one specific failure mode, a wrong-environment credential silently accepted or opaquely rejected. This is the shared-ground point with the sibling key-management skill: the prefix communicates _mode_, not _risk level_. Test keys still read and write real account configuration on toggle-model platforms.

# Magic-value catalogs: worked examples across five product domains

A magic value is a fake input that is not random filler - it is a deterministic key that selects a specific test outcome. These catalogs show how five vendors shape the pattern to their domain. Treat every table below as a snapshot: vendors periodically update their lists (Stripe says so explicitly), so verify against the vendor's live testing docs before hard-coding any value into an automated suite.

## Stripe: the card number is the key (payments)

The card number alone decides the outcome. Expiry, CVC, and ZIP can be anything valid. Test values don't expire - the catalog is a stable, long-lived contract integrators memorize, not a rotating fixture set.

| Number                                                                         | Outcome                                                                                            |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
| `4242 4242 4242 4242`                                                          | Always succeeds (the canonical happy path)                                                         |
| `4000 0000 0000 0002`                                                          | Generic decline (`card_declined`)                                                                  |
| `4000 0000 0000 9995`                                                          | Insufficient funds - a distinct `decline_code`, so the integrator's UI can show specific messaging |
| `4000 0000 0000 9987` / `9979`                                                 | Lost card / stolen card                                                                            |
| `4000 0000 0000 0069` / `0127` / `0119`                                        | Expired card / incorrect CVC / processing error                                                    |
| `4000 0000 0000 0259`                                                          | Charge succeeds, then is disputed as fraudulent - tests dispute handling, not just declines        |
| `4100 0000 0000 0019`                                                          | Fraud-review: always blocks (risk "highest")                                                       |
| `4000 0000 0000 4954` / `9235`                                                 | Highest risk but processes / elevated risk                                                         |
| `4000 0027 6000 3184`                                                          | 3DS2 challenge required                                                                            |
| `4000 0025 0000 3155`                                                          | Authentication required only on setup / off-session                                                |
| `4000 0000 0000 3220`                                                          | 3DS required on every transaction (always-challenge variant)                                       |
| `5555 5555 5555 4444` (MC), `3782 822463 10005` (Amex, 15 digits, 4-digit CVC) | Non-Visa happy paths - cover more than one network                                                 |

Two transferable lessons:

1. **Token over raw value**: Stripe recommends a token (`pm_card_visa`) instead of a raw number in test code - raw sensitive values in server-side code become a compliance habit that carries into production paths.
2. **Bidirectional mode boundary**: test values only work with test keys; a live key rejects them outright, and a real card in test mode violates the platform's terms. Enforce both directions, not just "live rejects test."

## Twilio: magic numbers split by operation, restricted to test credentials (communications)

Test credentials can never touch real numbers or carriers - a request made with test credentials cannot even specify a live account number as `From`, so the _only_ usable `From` is a documented magic number. No call is placed, no SMS sent, no status callbacks fire: the request never reaches a real network.

The catalog splits by **operation type**, and the same numeric suffix means different things across operations - a documented trap:

- Number purchasing: `+15005550006` valid/available; `+15005550000` unavailable (21422); `+15005550001` invalid (21421); area code 533 has none available; area code 500 is available.
- SMS `From`: `+15005550006` valid; `+15005550001` invalid (21212); `+15005550007` not SMS-capable/not owned (21606); `+15005550008` queue full (21611).
- SMS `To`: `+15005550009` can't receive SMS (21614); `+15005550002` can't route (21612); `+15005550003` no international permission (21408); `+15005550004` blocked (21610).
- Voice `To`: `+15005550001` invalid (21217); `+15005550002` can't route (21214); `+15005550003` no international permissions (21215); `+15005550004` blocked (21216).

Note "can't route" is `...0002` for voice but SMS uses `...0009` for its closest equivalent - cross-referencing the wrong operation's table is an easy, documented mistake. Design lesson: one flat catalog for a multi-operation API invites exactly this confusion; label every value with the operation and field it applies to. For OTP flows there is no universal always-correct code - a reminder that not every domain admits a magic value.

## Plaid: the username is the magic value (data aggregation)

- Personas select the fake data shape: `user_good`/`pass_good` for basic access, `user_transactions_dynamic` for evolving history, plus role personas (`user_small_business`, `user_yuppie`, …).
- Error simulation rides the password: `user_good` + password `error_ITEM_LOCKED` deterministically triggers that error - failure modes parameterized without one endpoint per error.
- MFA the same way: password `mfa_questions_<n>_<m>` forces a multi-factor flow; two named institutions always launch MFA regardless of credentials.
- `user_custom` accepts a full JSON config shaping accounts, transactions, identity, and forced error codes in one request - the most granular fixture-construction mechanism surveyed. Plaid publishes a starter-fixture repo (`plaid/sandbox-custom-users`) rather than expecting integrators to hand-write configs.
- Identity verification uses reserved sample identities: the name "Leslie Knope" deterministically passes, the surname "Gergich" is deliberately blocklisted - the same pattern applied to an identity product.
- Reset/lifecycle endpoints, distinct from event firing: `/sandbox/item/reset_login` forces re-auth state to test the recovery flow; `/sandbox/item/set_verification_status` skips real micro-deposit timing; `/sandbox/public_token/create` mints test objects directly for seeding.
- Sandbox items auto-age (auto-transition to a login-required state after 30 days) and `user_custom` date fields auto-shift daily so the newest date is always "today" - a low-effort substitute for a true time-manipulation primitive.

## PayPal: the name field drives the outcome (payments, contrast case)

Entering a rejection-trigger string (e.g. `CCREJECT-REFUSED`) in the "Name on Card" field, with any valid test card number, forces that decline - the opposite convention from Stripe, where the number itself varies. Trigger values are case-sensitive, and every rejection returns the same AVS/CVV placeholder values regardless of reason - so rejection _reasons_ cannot be distinguished from those fields. The older Classic API puts magic values directly into transaction fields (amount, authorization ID, CVV2, street) for finer-grained negative testing.

## Shopify and GitHub: the lightweight end of the spectrum

- **Shopify's Bogus Gateway** ("Test payment gateway"): payment method name `Bogus Gateway`, then a card number of `1` (approved), `2` (declined), or `3` (gateway failure) - the simplest scheme surveyed, usable only on a development store. Shopify Payments' own test mode reuses Stripe's card catalog directly - a platform built on a processor inheriting the processor's magic values instead of inventing its own is a reusable pattern.
- Shopify has **no "send test webhook" button**: fire real order webhooks via a Bogus Gateway test order, or use the CLI trigger - whose payloads carry valid JSON but **not a valid HMAC signature** for the app's secret. If guidance says "always verify signatures," it must carve out the vendor's own test trigger, or the rule breaks the test tooling.
- **GitHub** (no sandbox, no fire-webhook endpoint) offers three narrower mechanics:
  - Webhook redelivery: replay a past real delivery.
  - A `ping` event via REST.
  - A REST-triggered test push for repository webhooks, with deliveries inspectable in the UI.

  It also documents an event-suppression limit: pushing more than 3 tags at once fires no `push` event at all, silently, the kind of threshold trap a catalog should name explicitly.

## What the cross-domain comparison teaches

- The magic value's _carrier_ is domain-shaped: card number (payments), phone number per operation (communications), username/password (data aggregation), name field (PayPal), single digit (Shopify). Pick the carrier integrators already type in that domain.
- Every catalog needs deliberate-failure values, not just happy paths - and the failure values should map one-to-one onto the API's real, documented error codes.
- Include the domain's step-up flows (3DS challenges, MFA, identity review), not just success/decline binaries.
- Publish the catalog as documentation with stable values; it is API contract, and integrators' CI suites will depend on every entry.

---
name: api-test-mode-design
description: Design the test/sandbox mode of a public API platform so external integrators build and validate without touching real money, messages, or data - isolation architecture (soft test-mode toggle vs hard separate-copy sandbox), test-key prefixing, magic test values, simulated objects and personas, on-demand test events, deterministic time manipulation, reset and seeding, sandbox quotas, abuse controls, and graduation to live. Use whenever the user mentions a sandbox, test mode, test API keys, magic or dummy test values, simulated webhook events, or demo data - even if they never say "test mode". Not code-execution sandboxing or unit-test mocking. Do NOT use for partner sandbox provisioning - use samber/developer-platform-skills@partner-app-onboarding instead.
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.0.1"
---

# API Test Mode Design

You are a test-mode designer for an API platform. Design the sandbox environment external integrators build against - its isolation boundary, credentials, deterministic test values, simulated events and time, quotas, abuse controls, and the path to live - so an integration is fully validated before it ever touches real money, messages, or data.

A sandbox is table stakes for developer trust on any platform that expects third-party integrators. It is a _stateful_ environment that validates business logic across a sequence of calls, not a mock server's static request→response mapping. The gap shows up around week three of an integration: bugs a stateless mock cannot reproduce because it has no concept of what happened on a prior call.

## Scope

Designs sandbox/test-mode environments (test keys, deterministic fixtures, simulated events) so integrators build without real data. This is product test-mode design, distinct from the existing `developer-sandbox` marketing-playground skill on skills.sh: that skill builds a marketing try-it playground. This one designs the product's own test mode.

Boundaries with siblings:

- **Key lifecycle** (issuance, rotation, hashing, dashboard UX): `samber/developer-platform-skills@api-auth-key-management`. Test-key prefixing is the shared ground; this skill owns only the mode identity inside the prefix.
- **Webhook delivery** mechanics (signing, retries, delivery logs): `samber/developer-platform-skills@webhook-platform-design`. How a test event gets triggered is this skill's.
- **Sandbox controls in the portal**: `samber/developer-platform-skills@developer-portal-design`.

**Memory (advised):** When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

## Clarifying questions

Ask before designing anything; each answer changes a later step. Batch them - this is a tactical design task, not a strategy interview.

1. Product domain: payments-like (money moves), communications-like (messages reach real people), data-aggregation-like (third-party data flows in), or something else? (shapes the magic-value carrier and the graduation gate - see next section)
2. Isolation appetite: is a mode toggle on the live account acceptable, or does the domain demand a structurally separate environment? What is the engineering budget - weeks or quarters? (drives step 1)
3. What compliance regime gates going live - KYC, partner vetting, marketplace listing review, plan tier, or nothing? (drives step 5)
4. What test infrastructure already exists - a staging environment, seeded fixtures, a mock server integrators were told to use? (a retrofit starts from what integrators already depend on)
5. Does the product have time-dependent state (trials, renewals, expirations, scheduled jobs) and/or async events? (decides whether step 3 is a core deliverable or a stub)
6. Is there an end-user-facing surface (checkout page, hosted flow) integrators must also test, or is the API the whole product?
7. By when must integrators be able to build against the sandbox, and is this a one-off unblock (one partner waiting on a deal) or a compounding platform surface you will run for years? (with question 2's budget, these are the re-rank inputs for steps 1 and 2)

Question 7 exists because the isolation menu (step 1) and the simulation menu (step 2) diverge sharply on time-to-effect, durability, and effort. Neither ranking can be picked without it.

- A hard date promotes the soft toggle and magic values.
- A compounding mandate promotes the hard separate-copy and the fixture builder.
- A budget of weeks, rather than quarters, deletes the hard separate-copy from this pass rather than parking it last. Unblocking condition: "the first cross-mode leakage incident, or the compliance regime landing."

## Product domain split

The **product domain** is the axis that changes this design, because it dictates what the magic values look like and what regulators gate graduation on:

- **Payments-like**: magic values ride the transaction instrument (card numbers mapping to specific declines, disputes, fraud-risk levels, step-up auth); graduation is compliance-gated (KYC); the sandbox must simulate settlement it cannot actually perform.
- **Communications-like**: magic values ride addresses (phone numbers, per operation); the hard guarantee is that no real message ever leaves the sandbox - the strongest isolation in the survey refuses to even _reference_ production resources from test credentials.
- **Data-aggregation-like**: magic values ride identities (test usernames/personas selecting fixture shapes, password suffixes selecting error conditions); the design center is fixture variety, not transaction outcomes.

Design for your domain's carrier and gate. The worked catalogs in [references/magic-value-catalogs.md](references/magic-value-catalogs.md) cover all three.

## Workflow

1. Choose the isolation model - the decision every later step builds on.
2. Make every failure path triggerable - deterministic test values and simulated objects.
3. Simulate events and time - test-event triggering, time manipulation, reset and seeding.
4. Harden credentials, quotas, and abuse controls.
5. Design the graduation path - compliance gate, cutover checklist, canary.

Each step has a section below, in order, mirroring how integrators consume a sandbox: failure paths first, async/time behavior, production hardening, then cutover. Ship it in this order, and gate each stage on its benchmark in Measurement.

## 1. Choose the isolation model

Two orthogonal questions get conflated as "have a test mode": _whether_ test data is separated, and _how hard_ the boundary is. Three models, ranked:

- efficiency: `soft toggle > hard separate-copy > no test mode (integrator convention)`
- effort: `hard separate-copy > soft toggle > no test mode`
- value: `hard separate-copy > soft toggle > no test mode`

- **Default rung: the soft toggle.** One account, a mode flag on every object, mode-prefixed keys routing to test infrastructure. Test data stays invisible to live calls and vice versa. Weeks of work on an existing stack, and it serves the common case: one integrator, one integration, moderate stakes.
- **Promote to the hard separate-copy**, a structurally separate environment (own host or account copy, own keys, webhooks, and configuration), when:
  - Teams need parallel testing without polluting shared test data.
  - The compliance regime demands demonstrable isolation.
  - A toggle's failure mode (acting on live data while believing you're in test) has real blast radius.

  This is the starved option: highest value and highest effort, so it loses every efficiency round. Yet the reference payments platform built it anyway and made it the default for new accounts, keeping the toggle only for existing ones. Cross-mode leakage becomes structurally impossible rather than policy-forbidden.

- **No test mode (integrator convention)** - documenting how integrators isolate themselves with separate test apps/orgs/repositories - is viable only when the API is read-mostly and touches no money, no outbound messages, and no sensitive third-party data. If any of those flow through the API, this option is ruled out entirely, not just demoted: don't leave it on the menu.

This ranking is a default, not a law. Re-rank against the answers to questions 1-2 and anything else you know:

- A greenfield platform can build the hard model from day one for far less than a retrofit.
- A regulated domain rules the toggle out.
- A team already running per-tenant infrastructure gets the separate copy nearly free, which flips the effort line and the winner.

Whichever model wins, state the isolation level explicitly in the design and the docs. "Sandbox" guarantees nothing by itself - surveyed vendor sandboxes are frequently shared, rate-limited, and only partially isolated under the same word. And if the product has an end-user-facing surface (question 6), the sandbox needs a mirrored end-user side too, not just API responses.

See [references/isolation-models-and-key-prefixes.md](references/isolation-models-and-key-prefixes.md) for the six-vendor comparison behind this menu.

## 2. Make every failure path triggerable

The sandbox's core deliverable is determinism: an integrator must be able to summon any documented outcome - especially failures - on demand. Three simulation primitives, ranked:

- efficiency: `magic values > on-demand simulation endpoints > config-driven fixture builder`
- effort: `config-driven fixture builder > on-demand simulation endpoints > magic values`
- value: `config-driven fixture builder > on-demand simulation endpoints > magic values`

- **Default rung: magic values.** A documented fake input deterministically selects an outcome. Days of work per outcome family, no new API surface, and integrators memorize them.

  Design rules:
  - Pick the carrier integrators already type in your domain (see Product domain split above).
  - Include deliberate-failure values and step-up flows, not just the happy path.
  - Map each value one-to-one onto a real error code from your published catalog (`samber/developer-platform-skills@api-error-design` territory). Magic values must trigger the _real_ taxonomy, never a sandbox-only error shape.
  - Label values per operation when the same input means different things in different calls. An unlabeled multi-operation catalog is a documented integrator trap.

- **Step up to on-demand simulation endpoints** the moment outcomes stop being a function of request input: sandbox-only endpoints that mint test objects for seeding, force an object into a specific state (login-required, verification-passed), or fire a specific event now. These cover what no input value can encode.
- **The config-driven fixture builder** - one endpoint accepting a full fixture description (accounts, history, forced errors) - is the starved option: the most granular mechanism surveyed and the most work, losing every efficiency round. Promote it when integrators' data shapes genuinely vary (data-aggregation domains) - and ship a starter-fixture library with it rather than making every integrator hand-write configs.

Treat magic values as API contract: stable, documented, non-expiring. Integrators' CI suites will depend on every entry, so additions are free but changes are breaking. Recommend tokens over raw sensitive values in example code - raw card-like values in test code become a compliance habit that leaks into production paths.

Worked catalogs from five vendors: [references/magic-value-catalogs.md](references/magic-value-catalogs.md).

## 3. Simulate events and time

A sandbox that freezes _data_ but leaves _time_ real cannot exercise time-triggered logic - renewals, trial expirations, dunning, scheduled jobs. A deterministic clock primitive is the fix, not a documentation workaround telling integrators to wait.

- **Test-event triggering**: give integrators an explicit "fire this event now" mechanism, a sandbox endpoint or CLI trigger per event type. You own when and how a test event gets triggered; delivery mechanics (signing, retries, logs) belong to `samber/developer-platform-skills@webhook-platform-design`.

  Two traps:
  - If state changes and event delivery are decoupled in your sandbox, document it. Integrators otherwise wait for events that never come.
  - If your CLI trigger cannot produce a valid signature, say so next to your "always verify signatures" rule, or the rule breaks your own test tooling.

- **Time manipulation**: a clock object with a frozen time that test objects attach to, advanced deliberately, with state transitions and events firing as if the time had passed. Cap the advance per call and the complexity of what one clock can carry - the reference implementation limits both, as a deliberate simulation-complexity cap. A cheaper rung for simple cases: auto-aging fixtures whose date fields shift daily so the newest data is always "today".
- **Reset and seeding**:
  - Seed with synthetic data only; never copy in production data.
  - Give integrators both object-minting endpoints (seed a known state in one call) and reset endpoints (return to a clean state).
  - Automate periodic resets of shared sandboxes so accumulated test data doesn't drift into an unrepresentative state.
- **Parity**: derive the sandbox's configuration from the same infrastructure-as-code as production, so any difference is intentional and documented. Environment drift - config, data staleness, version skew - is the "works in sandbox, fails live" root cause.

## 4. Harden credentials, quotas, and abuse controls

- **Mode-prefixed keys**: adopt the externally codified prefix convention, a type prefix plus a `_test_`/`_live_` mode substring.
  - The exact reference prefixes are zero-false-positive push-protection targets in GitHub secret scanning. A recognized shape buys free leak detection from tooling you don't control; a novel scheme buys nothing.
  - Enforce the boundary bidirectionally: live keys reject test values, test keys reject real ones.
  - Key format and lifecycle beyond the mode substring belong to `samber/developer-platform-skills@api-auth-key-management`.
- **Test keys are not risk-free**: on a toggle model, test keys still read and write real account configuration (customers, webhook endpoints), so they deserve real secret hygiene - the prefix communicates mode, not risk level. Tell integrators to never use live secrets in local development, and keep per-environment separation (data stores, redirect allowlists, credentials) rather than one codebase branching on a runtime flag.
- **Two abuse classes, distinct mitigations. Never conflate them.**
  - **Leaked-key risk**: prefixed keys are machine-identifiable and routinely harvested from public repositories. Mitigations: server-side-only secrets, scoped/restricted keys, secret-scanning enrollment.
  - **Card-testing/carding abuse**: attackers validate stolen instrument batches against **live** checkout endpoints. The sandbox is generally useless for carding, since test transactions never touch real networks, so this is a live-mode control, not a sandbox one. Mitigations: edge bot challenges, endpoint-scoped rate limits, and rules capping instruments-per-customer and consecutive declines per IP.

  Stripe's own fraud-team guidance reports that managed CAPTCHA for all Checkout users cut card testing by 80% with under 0.02% impact on authorization rates, a rare fraud control with a quantified, near-negligible legitimate-user cost.

- **Sandbox quotas**: don't assume the sandbox needs its own rate-limit tier. Three of six vendors reviewed here reuse their production limits unchanged in test mode, which also keeps integrators' throttling code honest.
  - Sandbox-specific numeric limits are the sparsest-documented axis across those vendors, so treat "every vendor publishes a distinct sandbox throttle" as false.
  - Build a dedicated sandbox quota only when the failure mode is _evaluation-volume abuse_ (free-tier scraping, load tests against a shared sandbox). Model it as an aggregate volume-and-scope cap on the pre-production tier, not a faster per-minute limiter.
  - Per-vendor quota postures: [references/isolation-models-and-key-prefixes.md](references/isolation-models-and-key-prefixes.md). Limit-tier design mechanics belong to `samber/developer-platform-skills@api-rate-limit-policy`.

## 5. Design the graduation path

Graduation is never purely technical, even when the credential swap is trivial - every surveyed vendor gates production behind a compliance or review step. Design three things:

1. **The gate**: pick the shape matching your actual obligation - KYC activation, tiered/auto-approved review, partner vetting, publisher verification, or plan tier. Don't import a heavier gate than the domain requires. Make pre-approval failures legible: an explicit "environment not yet approved" error, never an opaque 401.
2. **The cutover checklist**:
   - Credentials swapped everywhere.
   - Fresh live-mode event endpoints and signing secrets (test-mode secrets do not carry over).
   - Abuse rules on before the first live transaction.
   - A small live canary per integration path, confirmed effect _and_ confirmed event delivery, ideally reversed afterwards.

   A green sandbox run is necessary but not sufficient: no sandbox replicates real networks, real settlement timing, or full-strength fraud systems.

3. **The publishable version**: hand integrators the checklist and the five recurring cutover pitfalls as documentation - they are the same five support tickets every platform eventually answers.

Gate shapes, the full checklist, pitfalls, and reactive go-back triggers: [references/graduation-gates-and-cutover-checklist.md](references/graduation-gates-and-cutover-checklist.md).

## Failure modes

Anti-pattern checklist - each is a direct design-review finding:

- A mock server shipped as "the sandbox": stateless, it tests the integrator's assumptions about your API, not their correctness - when the API changes, the mock keeps simulating the old behavior while production breaks silently. Vendor-exercised real objects cannot drift this way.
- A documented failure outcome with no way to trigger it deterministically in the sandbox.
- Magic values that return a sandbox-only error shape instead of the real published error taxonomy.
- Time-dependent logic testable only by waiting real calendar time.
- The word "sandbox" in docs with no stated isolation level.
- A wrong-environment credential producing an opaque authentication failure instead of a clear "wrong environment" error - a named, recurring support-ticket generator.
- The five cutover pitfalls (key swap forgotten, cross-environment paste, sandbox-predicts-live assumption, stale live webhook endpoints, untested real-money failure paths) left undocumented for integrators.
- Real production data copied into the sandbox as seed data.
- A shared sandbox with no reset mechanism and no periodic cleanup.

## Measurement

- **Failure-path coverage**: every documented failure outcome (declines, error codes, step-up flows, disputes/reversals) is triggerable deterministically via a magic value or simulation endpoint - a gate at 100%; one untriggerable failure path fails the audit.
- **Event/time coverage**: every async event type is fireable on demand, and every time-dependent flow is testable via simulated time, never by waiting - also a gate.
- **Cross-environment incidents**: track support tickets and auth failures caused by wrong-mode credentials; the design target is zero, achieved structurally (self-describing prefixes plus a legible wrong-environment error), not by warning integrators to be careful.
- **Graduation exercise**: the cutover checklist is walked end-to-end - including one live canary with confirmed event delivery - before any integration is declared launched; treat an unexercised checklist as an unshipped feature.

Iterate the design until both coverage gates pass. They apply from the soft toggle up; the "no test mode" rung of step 1 has no provider-side sandbox to trigger anything in, so its gates are out of scope rather than failed. Holding that rung to the gates would reject an option this skill itself prescribes as viable.

What replaces them there: the published integrator-isolation guidance exists, and step 1's read-mostly/no-money/no-messages/no-sensitive-data conditions are each confirmed in writing. Cross-environment incidents and sandbox-to-live conversion time are trends to baseline from the first month and improve against, since no cross-industry threshold exists to borrow.

## Invocation examples

- "Design a test mode for our payments API - integrators currently test against staging with real card numbers."
- "We're adding webhooks; integrators need a way to trigger test events and simulate a subscription renewing without waiting 30 days."
- "Choose between a test-mode toggle and a fully separate sandbox environment for our new public API, and design the graduation path to live."

## References

- [references/isolation-models-and-key-prefixes.md](references/isolation-models-and-key-prefixes.md) - six-vendor isolation comparison, the test-key prefix taxonomy, per-vendor sandbox quota postures, and the non-prefix alternatives with their support cost.
- [references/magic-value-catalogs.md](references/magic-value-catalogs.md) - worked magic-value catalogs across payments, communications, and data aggregation, with the design lessons each teaches.
- [references/graduation-gates-and-cutover-checklist.md](references/graduation-gates-and-cutover-checklist.md) - five compliance-gate shapes, the cutover checklist, the five pitfalls, and reactive go-back triggers.
- `samber/developer-platform-skills@api-integration-surface-strategy` - the umbrella decision that schedules each surface; every surface it schedules needs a test mode before partners go live, and this skill designs that layer.
- `samber/developer-platform-skills@sample-app-strategy` - the sample portfolio whose CI and playgrounds run against this sandbox; flows the sandbox cannot simulate leave samples compile-only.