api-idempotency-retry

{
  "skill_name": "api-idempotency-retry",
  "evals": [
    {
      "id": 1,
      "prompt": "I'm the backend lead at Lumapay, a B2B payouts platform. When customer apps hit network blips they retry POST /v1/transfers and we end up paying vendors twice - 43 duplicate payouts last month. In our design meeting we agreed the server will compute a SHA-256 of the request body and use that as the dedup key in one global dedup table, so clients don't have to change anything. Worth noting some customers legitimately send two identical transfers in the same day (same vendor, same amount). Sketch the dedup design for us - how keys work, what the API returns on a duplicate, and what we tell integrators about picking keys.",
      "expected_output": "A key-contract design that replaces the server-derived hash with client-minted, per-caller-scoped idempotency keys, ranks derivations, guards payload mismatch, and stores both outcomes with an explicit replay decision.",
      "files": [],
      "expectations": [
        "Rejects the server-computed request-body hash as the dedup key and requires a client-minted idempotency key (e.g. an Idempotency-Key header)",
        "Justifies the rejection with the intentional-duplicate case: two genuinely identical transfers both intended are indistinguishable to a server-side hash, so only a caller-provided key expresses intent",
        "Rejects the single global dedup table and scopes uniqueness per caller as (account_id, idempotency_key)",
        "Recommends event-derived key derivation as the default (a key built from an identifier the caller already holds, e.g. their own transfer or order id)",
        "Presents derivation options as a ranking - event-derived over request-scoped over a random UUID reused across retries - not an unordered list",
        "Names a fresh key value generated per attempt (regenerated UUID or timestamp) as the anti-pattern that defeats the mechanism, not as a fourth ranked option",
        "Specifies that the same key arriving with a different request body is rejected with a conflict (409), never silently replayed",
        "Recommends storing a payload fingerprint or checksum beside the key to enforce the mismatch guard",
        "Stores failed outcomes as well as successes so a retry replays the original result",
        "Makes or demands an explicit decision whether a stored failure replays verbatim or the operation is re-attempted without side effects (the Stripe v1 vs v2 fork)",
        "Does not treat the Idempotency-Key header as a ratified standard - notes the IETF draft expired unratified, so TTL, mismatch and concurrency semantics must be explicitly designed and documented"
      ]
    },
    {
      "id": 2,
      "prompt": "At Orbitshelf (inventory SaaS, Postgres + Rails) we already store idempotency keys: the controller SELECTs the keys table, and if nothing is there it INSERTs the key and creates the invoice. Last Tuesday a customer's integration fired the same request twice within 30ms and both created invoices. A teammate wants to put a Redis SETNX lock with a 10-second TTL in front of the endpoint to serialize the requests. Review the approach and tell us exactly how to fix the race.",
      "expected_output": "Diagnosis of the check-then-insert TOCTOU race, an atomic unique-constraint claim in the same transaction as the mutation, and a reasoned rejection of the SETNX lock.",
      "files": [],
      "expectations": [
        "Names the SELECT-then-INSERT pattern as a check-then-insert (TOCTOU) race in which both concurrent requests observe the key as absent and both proceed",
        "Replaces it with an atomic claim: an INSERT arbitrated by a unique constraint on (account_id, idempotency_key), e.g. INSERT ... ON CONFLICT DO NOTHING",
        "The losing request learns it lost from the insert result itself (constraint violation or no returned row), not from a separate application-level existence check",
        "Requires the key claim and the invoice creation to commit in one database transaction, stated as a hard requirement rather than a nice-to-have",
        "Rejects the Redis SETNX + TTL lock as the claim mechanism",
        "Justifies the SETNX rejection with the fencing problem: a client paused past its lease expiry can still write after the lock has moved on, and no timeout tuning fixes it",
        "Removes the Redis option from the menu entirely (or requires fencing tokens layered on top if Redis were unavoidable) rather than ranking it last as a fallback",
        "Given Postgres is the store, does not offer DynamoDB-style conditional writes as a live alternative",
        "Mentions row lock + SERIALIZABLE as the mechanism to promote only when a request spans external non-rollbackable calls, not for this case",
        "After losing the claim, the request reads the existing key row and branches on its state instead of returning a generic failure",
        "The concurrent duplicate receives a conflict response or the stored replay - under no path does a second invoice get created",
        "Proposes the concurrency verification: fire N identical concurrent requests with one key and require exactly one side effect, as a binary pass/fail gate"
      ]
    },
    {
      "id": 3,
      "prompt": "Our shipping API at Parcelry stores a response per idempotency key and replays it when the key is seen again. Problem: a client's request took 40 seconds (the carrier API was slow), their client timed out at 10s and retried, the retry found no stored response so it ran the full flow again - two shipping labels, two carrier charges. An engineer proposed: if a key exists but has no response yet and it's been more than 10 seconds, assume the first attempt died and let the retry through. Our PM also wants to email the client asking them to stop retrying so aggressively. What should we actually do?",
      "expected_output": "A three-state key design (absent, in-flight, complete) with a 409 in-flight conflict, lock-timeout takeover resuming from a recovery point, and pushback on blaming the client.",
      "files": [],
      "expectations": [
        "Diagnoses the root cause as a binary seen/not-seen key model with no representation of an in-flight request",
        "Specifies the key as a three-state machine: absent, in-flight, complete",
        "Rejects the proposal to let a retry through once the original merely seems stuck past 10 seconds",
        "Defaults the in-flight response to a conflict (409) telling the caller to wait and retry",
        "Offers blocking for the bounded result and 202-plus-status-URL as the alternatives for synchronous callers and long-running effects respectively",
        "Requires the in-flight conflict and the payload-mismatch conflict to carry distinct machine-readable error codes",
        "States that in-flight conflict and completed replay are different responses by design, never fallbacks of each other",
        "Allows takeover of an abandoned request only after a lock timeout, via an atomic conditional update on the lock timestamp",
        "A taken-over or retried request resumes from a persisted recovery point rather than restarting from the beginning",
        "Pushes back on the email to the client: frames the client's retry as correct configured behavior and the server's missing in-flight state as the bug",
        "Recommends an active completer background job that pushes abandoned in-flight requests through to completion, not passive expiry",
        "Recommends a reaper job that deletes expired records and logs what it could not finish for a human"
      ]
    },
    {
      "id": 4,
      "prompt": "Finance at Meterly found 61 duplicate usage-credit purchases in Q3. We traced it: failed API jobs land in a Kafka DLQ and an ops runbook replays them - sometimes 5 or 6 days later - and our idempotency records expire after 24 hours, so replayed jobs look brand new. Storage on the keys table grows about 9 GB a month and the infra team's proposal on the table is to cut retention to 6 hours to control costs. What retention should we set, and is anything else wrong with our retention model?",
      "expected_output": "Retention set from the longest retry chain (covering the DLQ horizon), the two-horizon replay-window/reaper split, post-expiry semantics, and verbatim response storage.",
      "files": [],
      "expectations": [
        "Attributes the duplicates to the replay window (24h) being shorter than the longest retry chain (the 5-6 day DLQ replay runbook)",
        "Rejects the 6-hour retention proposal",
        "States the replay window is set from the longest retry chain, explicitly not from storage cost",
        "Recommends a replay window covering the DLQ replay horizon (about 7 days or more), or shortening the replay runbook's horizon to fit - either way the two must be reconciled",
        "Splits retention into two horizons: a replay window governing correctness and a longer reaper threshold governing operational recovery",
        "Explains the reaper threshold keeps failed records alive for human recovery (e.g. a Friday incident's failed requests surviving the weekend)",
        "States a key reused after the replay window is treated as a brand-new request, not answered with an error",
        "Requires the stored record to hold the full response verbatim (status and body) so the replay is indistinguishable from the original",
        "Requires the replay window published to integrators as a concrete number, with the post-expiry behavior spelled out",
        "Advises integrators to complete their retry logic well within the documented window",
        "Treats the in-window duplicate side-effect rate as a zero-by-definition alarm on regression, not a threshold to tune"
      ]
    },
    {
      "id": 5,
      "prompt": "I need to rewrite the 'Retrying requests' page of Ledgerhawk's public API docs. Our integrators mostly call raw HTTP. Today the page just says 'on any error, retry up to 5 times with 100ms between attempts'. Several integrators hammered us during last month's outage and made it worse. Rewrite the guidance - which errors to retry, how to wait between attempts, and a code example we can paste in.",
      "expected_output": "Integrator retry guidance with an explicit retry allowlist, full-jitter backoff as the ranked default, dual attempt/duration bounds, retry logging, a library-first pointer, and a concrete worked schedule.",
      "files": [],
      "expectations": [
        "Replaces 'retry on any error' with an explicit allowlist: 429, 502, 503, 504, and connection timeouts/resets",
        "Retries on 429 honor the Retry-After header",
        "Treats 500 separately from the allowlist - retried cautiously, since some 500s are real bugs that fail identically on retry",
        "States non-429 4xx responses and auth failures are never retried",
        "Warns against retrying a timed-out non-idempotent request unless idempotency keys are in place",
        "Recommends full jitter as the default backoff, with the formula sleep = random(0, min(cap, base * 2^attempt)) or an exact equivalent",
        "Names decorrelated jitter as the variant to promote when a tight user-facing deadline makes time-to-success outweigh fleet load",
        "Excludes equal jitter and no-jitter from the recommended menu (equal jitter identified as the loser among jittered options; no jitter re-synchronizes the fleet into the original spike)",
        "Attributes the outage amplification to synchronized retries, crediting jitter rather than backoff alone with reducing total client work",
        "Bounds both the attempt count and the total retry duration, not attempts alone",
        "Requires logging every retry with attempt number, error, and next wait",
        "Points integrators at an established retry library (tenacity, Resilience4j, or equivalent) ahead of hand-rolled formulas",
        "Provides a worked schedule with concrete numbers (e.g. 1s, 2s, 4s, 8s capped at 30s, plus jitter)"
      ]
    },
    {
      "id": 6,
      "prompt": "We're finalizing v2 of the Quotanet Python and Node SDKs. Current proposal from the team: retries off by default (integrators can opt in), up to 10 attempts when enabled, and each attempt gets its own fresh 30-second timeout. Several of our biggest customers run 200-instance fleets against us. Review the proposal and give us the defaults you'd ship instead.",
      "expected_output": "SDK defaults reversed to retries-on with a low bounded cap, one top-level deadline with per-attempt budgets, a fleet retry budget with token-bucket figures, p99-based timeouts, and auto-generated keys.",
      "files": [],
      "expectations": [
        "Reverses the opt-in proposal: SDK retries default ON with backoff and jitter",
        "Rejects 10 attempts and bounds the cap low (3 total attempts is the cited anchor)",
        "Rejects the fresh 30-second timeout per attempt: one overall deadline is set at the top and per-attempt timeouts are budgeted inside it",
        "Explains that a fresh overall timeout per attempt silently extends the caller's real wait beyond what it committed to",
        "Adds a fleet-wide retry budget as a distinct control from per-caller attempt caps",
        "Quantifies the amplification risk: a 200-instance fleet each retrying a few times multiplies load on a struggling downstream several-hundred-fold while every instance looks reasonable in isolation",
        "Names the token-bucket retry-quota implementation with its figures: 14 tokens per transient-error retry, 5 per throttle retry",
        "Sets single-attempt timeouts at 2-3x measured p99 latency, never an OS default or a round number",
        "Has the SDK auto-generate idempotency keys for retry-unsafe operations where it controls the request",
        "Treats SDK defaults as the de-facto contract for the majority of callers who never read the docs, reviewed with API-contract rigor",
        "When pairing with a circuit breaker, states the stacking order Retry then CircuitBreaker then RateLimiter then Bulkhead",
        "For call chains, propagates the remaining deadline downward with the request (relative timeouts on the wire) so no hop starts work exceeding what the top-level caller has left"
      ]
    },
    {
      "id": 7,
      "prompt": "Vaultberry's POST /v1/subscriptions does three things: creates the subscription row, charges the card through our PSP, then emails the receipt through our transactional mail provider. We wrapped the handler in one database transaction and added idempotency keys, but when the process crashed right after the PSP charge last week, the transaction rolled back, the retry ran the whole thing again, and the customer was charged twice. Separately, marketing's new API landing page says 'guaranteed exactly-once delivery, powered by idempotency keys' and wants engineering sign-off on the wording. Fix the design and review the copy.",
      "expected_output": "A phase-decomposed design with recovery points around each external call, intent recorded before calling out, resume-not-restart retries, a crash-recovery test, and the exactly-once claim replaced with accurate wording.",
      "files": [],
      "expectations": [
        "States the handler cannot be one database transaction because the PSP charge is an external, non-rollbackable mutation",
        "Decomposes the request into atomic phases separated by each foreign state mutation (the charge, the email), each phase transactional",
        "Persists a recovery point between phases",
        "Records intent before each external call, so a crash between the call and its response leaves evidence to reconcile instead of a silent duplicate",
        "A retry resumes from the recovery point - once the charge is recorded it is never re-executed, and the flow continues to the email",
        "Promotes the richer claim lifecycle for this multi-step case: in-flight as a first-class state (e.g. a lock column) that anchors resumable recovery",
        "The key claim commits inside the first atomic phase's transaction",
        "Specifies the crash-recovery test: kill the process between phases, retry, and require resumption with zero repeated external calls",
        "Rejects the 'guaranteed exactly-once delivery' copy as provably false over an open network, not just risky wording",
        "Explains what the mechanism actually delivers: at-least-once delivery plus duplicate collapse (effectively-once), bounded to the scope where idempotency is implemented",
        "Supplies replacement wording along the lines of: retries are safe within the documented key scope and replay window"
      ]
    },
    {
      "id": 8,
      "prompt": "Doing an idempotency pass on Cratebase's API before our enterprise launch. Endpoints: GET /v2/orders, POST /v2/orders, PUT /v2/orders/{id}/notes (each call appends a note entry to the order's activity log), DELETE /v2/credits/{id} (each call decrements the account's credit balance by the credit's value), POST /v2/refunds. The plan is to add Idempotency-Key to the two POSTs only - GET, PUT and DELETE are idempotent per the HTTP spec so they're covered. Also draft the checklist of what our integrator docs must cover for the keys.",
      "expected_output": "An audit that flags the appending PUT and decrementing DELETE as retry-unsafe despite their methods, plus a complete integrator documentation checklist for the key contract.",
      "files": [],
      "expectations": [
        "Flags PUT /v2/orders/{id}/notes as retry-unsafe despite the method, because each call appends to the activity log",
        "Flags DELETE /v2/credits/{id} as retry-unsafe despite the method, because each call decrements the balance",
        "Applies the rule: verify claimed idempotency against actual behavior, never trust the HTTP method",
        "Notes RFC 9110 defines method-level idempotency but says nothing about the Idempotency-Key retrofit, so 'per the HTTP spec' does not cover these endpoints",
        "Requires every state-changing endpoint to either accept the key or be explicitly documented as unsafe to retry - the appending PUT and decrementing DELETE get keys or an explicit warning",
        "Docs checklist covers which endpoints accept the key, the header name, and the maximum key length",
        "Docs checklist covers a recommended key derivation for clients",
        "Docs checklist covers per-caller scoping and the replay window as a concrete number",
        "Docs checklist covers post-expiry behavior (a reused key is treated as a new request)",
        "Docs checklist covers both conflict behaviors - payload mismatch and in-flight duplicate - each with its own distinct error code",
        "Docs checklist covers whether a stored failure replays verbatim or is re-attempted",
        "Grounds the checklist in the fact that no ratified standard exists, so this documentation is the integrators' only normative source"
      ]
    }
  ],
  "trigger_queries": [
    { "query": "Add Idempotency-Key header support to our POST /payments endpoint", "should_trigger": true },
    { "query": "Customers get double-charged when their app retries a timed-out request - how do we fix this server-side?", "should_trigger": true },
    { "query": "What TTL should we set on idempotency key records before purging them?", "should_trigger": true },
    { "query": "Write the 'retrying requests' section of our public API docs", "should_trigger": true },
    { "query": "Pick the backoff defaults for the Ruby SDK we ship to integrators", "should_trigger": true },
    { "query": "Two concurrent requests with the same idempotency key both created an order - find the race", "should_trigger": true },
    { "query": "full jitter vs decorrelated jitter for our API client retries?", "should_trigger": true },
    { "query": "how do I make a POST endpoint safe to retry", "should_trigger": true },
    { "query": "Our mobile app resubmits checkout on flaky connections and we end up with duplicate orders", "should_trigger": true },
    { "query": "duplicate rows whenever clients retry POST /transfers", "should_trigger": true },
    { "query": "is a Redis SETNX lock good enough to dedupe retried API requests?", "should_trigger": true },
    { "query": "A client retried while the first request was still processing and both went through - what should the server have returned?", "should_trigger": true },
    { "query": "exponential backoff settings for calling a third-party API - attempts, caps, jitter?", "should_trigger": true },
    { "query": "Marketing wrote 'exactly-once delivery' on our API landing page - can engineering sign off on that?", "should_trigger": true },
    { "query": "What should happen when the same idempotency key arrives with a different request body?", "should_trigger": true },
    { "query": "we need a retry budget so our fleet stops hammering a struggling downstream", "should_trigger": true },
    { "query": "Two instances of our webhook handler both POSTed the same invoice to our billing API - audit the idempotency implementation", "should_trigger": true },
    { "query": "Should our SDK retry 500s? Which status codes are safe to retry?", "should_trigger": true },
    { "query": "how to propagate the caller's deadline through retries so we don't answer someone who already gave up", "should_trigger": true },
    { "query": "users tap the pay button twice and we charge them twice", "should_trigger": true },
    { "query": "our DLQ replays jobs days later and the API applies them again - how do we dedupe?", "should_trigger": true },
    { "query": "what does Stripe actually do with idempotency keys and should we copy it?", "should_trigger": true },
    { "query": "safe POST retries", "should_trigger": true },
    { "query": "support queue is full of 'I was billed twice' tickets after every network blip", "should_trigger": true },
    { "query": "Postgres or DynamoDB for storing idempotency keys, and how do we claim a key atomically?", "should_trigger": true },
    { "query": "our client generates a new UUID on every retry attempt - is that right?", "should_trigger": true },
    { "query": "jitter formula for API retries", "should_trigger": true },
    { "query": "should a stored failed response be replayed to a retry, or should we re-run the request?", "should_trigger": true },
    { "query": "add request deduplication to our public API", "should_trigger": true },
    { "query": "How many times should our Node client retry a timed-out call and how long between tries?", "should_trigger": true },
    { "query": "what should the second caller get while the first request with the same key is still running?", "should_trigger": true },
    { "query": "recommend how API clients should derive their idempotency keys", "should_trigger": true },
    { "query": "double-submitted form creates two records - what's the server-side fix?", "should_trigger": true },
    { "query": "review our retry loop: it retries every exception 10 times with a fixed 100ms sleep", "should_trigger": true },
    { "query": "which 409 should we return when a key is reused with a different payload?", "should_trigger": true },
    { "query": "circuit breaker plus retry - which wraps which?", "should_trigger": true },
    { "query": "help me write the idempotency section of our developer docs", "should_trigger": true },
    { "query": "per-attempt timeout vs overall timeout for retried calls?", "should_trigger": true },
    { "query": "our idempotency records expire after a day but batch replays arrive three days later - should keys live longer?", "should_trigger": true },
    { "query": "prevent duplicate side effects when integrators retry our POST endpoints", "should_trigger": true },
    { "query": "an integrator asks if they can retry on ETIMEDOUT - what do we tell them?", "should_trigger": true },
    { "query": "we accept an Idempotency-Key header but never store anything - is that a problem?", "should_trigger": true },
    { "query": "What should our 429 response body look like and which rate-limit headers should we send?", "should_trigger": false },
    { "query": "Design the retry and backoff schedule for the webhooks we deliver to customers", "should_trigger": false },
    { "query": "Pick an error-code taxonomy and RFC 9457 problem-details envelope for our public API", "should_trigger": false },
    { "query": "How should we signal retryability in our error responses - a retryable flag and Retry-After?", "should_trigger": false },
    { "query": "Set quota tiers and burst limits for our API pricing plans", "should_trigger": false },
    { "query": "Design HMAC signing for our outbound webhook payloads", "should_trigger": false },
    { "query": "Our Kafka consumer duplicates messages - debug the transactional exactly-once producer config", "should_trigger": false },
    { "query": "Zero-downtime API key rotation with a dual-key overlap window", "should_trigger": false },
    { "query": "Should we version our API by URI path or by header?", "should_trigger": false },
    { "query": "Write the deprecation notice policy and sunset headers for our v1 endpoints", "should_trigger": false },
    { "query": "Review our REST resource naming and status-code choices", "should_trigger": false },
    { "query": "How granular should our status page components be during a partial outage?", "should_trigger": false },
    { "query": "Design magic test card numbers for our sandbox mode", "should_trigger": false },
    { "query": "Which languages should get official SDKs first, and generated or handwritten?", "should_trigger": false },
    { "query": "My database migration failed halfway - how do I make the migration script idempotent so re-running is safe?", "should_trigger": false },
    { "query": "Make our Terraform apply idempotent so re-runs don't recreate resources", "should_trigger": false },
    { "query": "My Ansible task keeps re-adding the same line to the config file - how do I make it idempotent?", "should_trigger": false },
    { "query": "How do I dedupe rows in Postgres with a unique index and ON CONFLICT?", "should_trigger": false },
    { "query": "Design the refresh-token rotation policy for our OAuth provider", "should_trigger": false },
    { "query": "How many cost points should we allow per GraphQL query?", "should_trigger": false },
    { "query": "Our React form double-submits - add a debounce to the submit button", "should_trigger": false },
    { "query": "Retry and failure thresholds for our Kubernetes liveness probes", "should_trigger": false },
    { "query": "Configure Sidekiq retry counts and the dead-letter queue for our background jobs", "should_trigger": false },
    { "query": "How should webhook consumers verify signatures and reject replay attacks?", "should_trigger": false },
    { "query": "What Retry-After value should we send with 429s while load-shedding?", "should_trigger": false },
    { "query": "My endpoint receives Stripe webhooks twice sometimes - dedupe the inbound events on my side", "should_trigger": false },
    { "query": "Choose between REST, GraphQL and gRPC for our new integration surface", "should_trigger": false },
    { "query": "Design per-integration delivery logs with a replay button for partners", "should_trigger": false },
    { "query": "What take-rate should our app marketplace charge developers?", "should_trigger": false },
    { "query": "Tune TCP retransmission timeouts on our load balancer", "should_trigger": false },
    { "query": "Add optimistic locking with a version column so two admins can't overwrite each other's edits", "should_trigger": false },
    { "query": "Audit whether every endpoint and error code is documented in our API reference", "should_trigger": false },
    { "query": "Reconcile mismatches between our ledger and the PSP settlement report", "should_trigger": false },
    { "query": "Choose a dead-letter policy and redelivery schedule for our RabbitMQ workers", "should_trigger": false },
    { "query": "Should repeating a DELETE return 404 or 204 the second time?", "should_trigger": false },
    { "query": "Draft the SLA and uptime commitments for our enterprise API tier", "should_trigger": false },
    { "query": "Implement a distributed lock so only one node runs the nightly cron job", "should_trigger": false },
    { "query": "Model our GraphQL mutation errors as union result types", "should_trigger": false },
    { "query": "Which events belong in our webhook catalog and how do we version the payloads?", "should_trigger": false },
    { "query": "Add request signing so clients can't tamper with query parameters", "should_trigger": false },
    { "query": "Our analytics batch hits serialization_failure under SERIALIZABLE - tune Postgres isolation", "should_trigger": false },
    { "query": "Set up automatic re-runs for flaky tests in GitHub Actions", "should_trigger": false }
  ]
}

# Backoff, jitter, retry budgets, and deadlines - the math

Canonical source for the jitter comparison: Marc Brooker, AWS Architecture Blog, "Exponential Backoff and Jitter" (2015, updated 2023 - the update states the guidance "continues to serve as a pillar for how Amazon builds remote client libraries"). His core finding: capped exponential backoff _alone_ "helps only a small amount" under contention - jitter, not backoff, is what reduces total client work in a retry storm.

## The four strategies

| Strategy                          | Formula                                          | Trade-off                                                                                          |
| --------------------------------- | ------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
| Full jitter (recommended default) | `sleep = random(0, min(cap, base * 2^attempt))`  | Lowest total client work and lowest server load; slightly more total time to completion            |
| Decorrelated jitter               | `sleep = min(cap, random(base, sleep_prev * 3))` | More total work than full jitter, completes slightly faster                                        |
| Equal jitter                      | half the backoff fixed, other half randomized    | Brooker's "clear loser" among the jittered approaches - more work than full jitter and much slower |
| No jitter (plain exponential)     | `sleep = base * 2^attempt`                       | Categorically the worst - so slow in total that Brooker's own comparison graph leaves it off       |

Why no-jitter fails at fleet scale: 1,000 clients that failed together all retry at exactly t+100ms, t+200ms, t+400ms - re-creating the overload spike that caused the failures. Full jitter spreads the same population across the whole window.

Brooker's 2019 follow-up ("Timeouts, retries and backoff with jitter", AWS Builders' Library) extends jitter to any periodic job, with one caution: apply jitter _consistently per host_, so an overload pattern stays diagnosable instead of looking like random noise.

## A concrete documented schedule

Resend's documented client loop - a citable worked example for integrator docs:

```js
const delay =
  Math.min(1000 * Math.pow(2, attempt), 30000) + // 1s → 2s → 4s → 8s ... cap 30s
  Math.random() * 1000; // jitter
```

Prefer the retry library over hand-rolled math wherever one exists - the formula is one line there, pre-debugged:

- Python (tenacity): `wait_exponential_jitter(initial=1, max=10)`, bounded with `stop_after_attempt(5) | stop_after_delay(60)`
- JVM (Resilience4j): `IntervalFunction.ofExponentialBackoff(...)` with `exponentialBackoffMultiplier: 2`

Bound **both** attempts and total duration (the tenacity combinator above): each attempt can look reasonable while the loop blows past the caller's own budget. Log every retry - attempt number, error, next wait; both libraries treat retry logging as a default, not an opt-in.

## Retry budget: the fleet-wide cap

Per-caller attempt caps cannot prevent amplification: a fleet of 100 instances each retrying 3 times amplifies load on a struggling downstream by 300x while every instance's cap looks reasonable in isolation.

- Mechanism: track the ratio of retried to original requests over a sliding window (e.g. 1 minute); once it exceeds the budget (e.g. 20%), stop retrying and fail fast.
- Named implementation - AWS SDK standard mode's token bucket: each retry deducts tokens, each success replenishes; when the bucket empties, the SDK returns the error without retrying. 2026 rebalance: a transient-error retry now costs **14 tokens** (up from 5); a throttling retry still costs **5** - the higher transient cost makes the quota engage sooner during a sustained outage, so the downstream recovers faster.

## Deadline propagation

- Set **one overall deadline at the top** of the call; budget per-attempt timeouts inside it. Never create a fresh overall timeout inside each retry attempt - that silently extends the caller's real wait past what they committed to, and the "successful" retry may answer a caller who already gave up.
- Down a call chain, pass the remaining budget with the request so each hop subtracts its own time before calling deeper - a mid-chain service must never start work whose timeout exceeds what the top-level caller has left. gRPC is the citable mechanics: deadlines are absolute in the API but sent as _relative_ timeouts on the wire, converted at each hop, precisely to dodge clock skew between machines.
- Setting a single timeout value: measure p99 latency under normal load and set the timeout at 2-3x that p99 - not a round number, and never the OS default (120-300s, which pins a thread to a hung connection for minutes).

## Pattern stacking

When retry is combined with other resilience patterns, the execution order matters (Resilience4j's documented order):

```
Retry → CircuitBreaker → RateLimiter → Bulkhead → call
```

- Retry wraps the outermost layer, so one logical call can still retry across circuit-breaker rejections during the half-open probe.
- The circuit breaker is retry's complement, not its substitute: once a downstream is confirmed down, the OPEN state short-circuits further retries instead of making every caller pay the full backoff schedule.
- Half-open recovery must gate to one or two trial requests, not the whole backlog, or it re-creates the thundering herd.

# Idempotency key lifecycle - worked walkthrough

A complete pass through one key's life, from first claim to reaping, using the default-rung mechanism (DB unique constraint) and the patterns from the Postgres reference implementation (Brandur Leach, brandur.org/idempotency-keys). No vendor publishes their production internals, so treat this as an independent, Stripe-inspired reference design, not Stripe's disclosed code.

## Storage schema

Relational-flavored for concreteness; every column maps to a concept any store needs.

```sql
CREATE TABLE idempotency_keys (
  id              BIGSERIAL PRIMARY KEY,
  account_id      BIGINT NOT NULL,          -- per-caller scope, never global
  idempotency_key TEXT NOT NULL,            -- client-minted, e.g. up to 255 chars (Stripe's cap)
  request_fingerprint TEXT NOT NULL,        -- payload checksum for the mismatch guard
  locked_at       TIMESTAMPTZ,              -- non-NULL = in-flight; drives lock recovery
  recovery_point  TEXT NOT NULL DEFAULT 'started',  -- resumable position for multi-step requests
  response_status INT,                      -- stored outcome: both success AND failure
  response_body   JSONB,                    -- replayed verbatim on completed retry
  created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (account_id, idempotency_key)      -- the atomic claim: the DB picks one winner
);
```

## The atomic claim

Never `SELECT` then `INSERT` - under concurrent retries both callers see "absent" and both proceed (TOCTOU). Insert first and let the constraint arbitrate:

```sql
INSERT INTO idempotency_keys (account_id, idempotency_key, request_fingerprint, locked_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (account_id, idempotency_key) DO NOTHING
RETURNING id;
```

- A returned row: this caller won the claim; proceed as a first attempt.
- No row: the key exists. Read it and branch on its state (next section).
- The claim insert and the business mutation commit in one transaction (or the first atomic phase's transaction) - the AWS hard requirement: token recorded and effect applied are all-or-nothing.

## Branching on an existing key - the three states

**State: complete** (`response_status` set) - first compare `request_fingerprint`.

- Fingerprint matches → replay `response_status` + `response_body` verbatim. No re-execution, even if the stored outcome is a 500 (Stripe v1 semantics: the retry gets exactly what the original got).
- Fingerprint differs → payload mismatch, a client bug. Reject:

```json
HTTP/1.1 409 Conflict
{ "code": "invalid_idempotent_request",
  "message": "This idempotency key was already used with a different request payload. Use a new key, or resend the original payload." }
```

(Resend's documented code name; fix advice belongs in the message - see `samber/developer-platform-skills@api-error-design` for message rules.)

**State: in-flight** (`locked_at` recent, no stored response) - the original is still running. Default handling, per the field convention:

```json
HTTP/1.1 409 Conflict
{ "code": "concurrent_idempotent_requests",
  "message": "A request with this idempotency key is currently in progress. Wait and retry." }
```

Never let the second caller through because the first "seems stuck" - the alternatives (block for the bounded result; 202 + status URL for long-running effects) also keep exactly one executor. The expired IETF draft made this deliberate: in-flight conflict and completed replay are different responses, not fallbacks.

**State: in-flight but lock expired** (`locked_at` older than the lock timeout) - the original crashed or was abandoned. This caller may take over the lock (an atomic conditional `UPDATE ... SET locked_at = now() WHERE locked_at < $threshold`) and resume from `recovery_point` - not from the beginning.

## Multi-step requests: recovery points around foreign state mutations

A request that calls an external, non-rollbackable system mid-flight (a card charge, an outbound email) cannot be one transaction. Decompose into atomic phases separated by each foreign call:

```
Phase 1 (txn): claim key, validate, create local order row     → recovery_point = 'order_created'
  -- foreign state mutation: charge the card --
Phase 2 (txn): record charge id                                 → recovery_point = 'charge_recorded'
  -- foreign state mutation: send confirmation email --
Phase 3 (txn): store final response                             → recovery_point = 'finished'
```

- Within each phase, everything commits or nothing does (ACID).
- Record intent _before_ each foreign call - a crash between the call and its response then leaves evidence to reconcile, instead of a silent duplicate on retry.
- A retry (or the completer) reads `recovery_point` and resumes at the right phase: `charge_recorded` means the card is charged - never charge again, continue to the email.

## Retention: two horizons, then reaping

- **Replay window (~24h)** - governs correctness: retries inside it replay; a reused key after it is treated as a brand-new request (every surveyed provider does this; none errors on post-TTL reuse). Set it from the longest retry chain - DLQ replays, dispute windows - not from storage cost. Tell integrators the operational consequence, as Resend does: "complete your retry logic well within 24 hours."
- **Reaper threshold (~72h)** - governs operations: records outlive the replay window so a bad Friday deploy's failed requests survive the weekend for a developer to push through on Monday.

## The two background jobs

- **Completer**: finds requests that look abandoned mid-flight (claim held, lock expired, client gave up) and pushes them to completion using only idempotency keys and recovery points - no per-endpoint knowledge.
- **Reaper**: deletes records past the reaper threshold, attempts cleanup on anything unfinishable, and logs what it couldn't finish for a human to chase.

Recovery is these active processes, not passive expiry - expiry alone leaves half-finished requests half-finished forever.

## Client-side pairing

The key only works if the client reuses it across retries of one intent. For critical calls, write the intent to a durable queue with a pending status _before_ the first attempt, so the retry state - key included - survives an application restart.

Derivation, in the recommended order:

- Event-derived (`order-confirm-${orderId}`) - deterministic by construction.
- Request-scoped (`reset-${userId}-${resetRequestId}`).
- A random UUID generated once and reused.

The anti-pattern that defeats everything: `Date.now()` or a fresh `crypto.randomUUID()` per attempt.

# Vendor SDK retry and idempotency defaults - cross-vendor comparison

How five platforms actually default and document this behavior, to calibrate your own SDK defaults and docs against. What converges is safe to copy; what diverges is exactly the part no standard settles - decide it explicitly for your own API.

## The comparison table

| Vendor                        | Default retry mode                                             | Max attempts      | Backoff / idempotency behavior                                                                                                                               |
| ----------------------------- | -------------------------------------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| AWS SDK v2 / CLI v2           | `standard`                                                     | 3 (incl. initial) | Exponential + jitter, gated by a token-bucket retry quota (14 tokens per transient-error retry, 5 per throttling retry, since the 2026 rebalance)            |
| AWS SDK `adaptive` mode       | opt-in                                                         | 3, same jitter    | Adds a client-side rate limiter reacting to throttling; not recommended for multi-tenant apps unless tenants get isolated clients                            |
| Stripe SDKs                   | client sets or omits the idempotency key; no fixed attempt cap | n/a               | v1: retry with the same key replays the previously saved response verbatim; v2: attempts to complete without side effects and may return an updated response |
| Google Cloud client libraries | per-service, varies                                            | varies            | Distinguishes _unconditionally_ idempotent operations (GET, list) from _conditionally_ idempotent ones needing preconditions/ETags before retry is safe      |
| Twilio                        | 24-hour idempotency window on config/orchestrator operations   | n/a               | Outbound webhook deliveries retried with exponential backoff up to 48 hours                                                                                  |
| Resend                        | idempotency opt-in per request                                 | n/a               | 24-hour key retention; two named `409` codes split payload-conflict from concurrent-in-flight                                                                |

## What to copy - the convergent core

- Retries on by default, bounded (AWS's 3 total attempts is the concrete anchor), with exponential backoff + jitter - since the 2023 update, most AWS SDKs ship this in standard/adaptive modes.
- A ~24-hour key retention window (Stripe, Resend, Twilio independently) - provided your retry chains actually fit inside it.
- Client-minted keys, V4 UUID recommended, generous length cap (Stripe: up to 255 characters).
- Payload-mismatch on a reused key rejected, never silently replayed.

## What to flag, not generalize

- **Stripe v1 vs v2 is a real behavioral fork - name the generation.**
  - v1: pure replay - same key returns the identical stored response, _including a stored 500_.
  - v2: supervised retry - may complete the operation without duplicating side effects, returning a response that differs from the original attempt.
  - "Stripe's idempotency model" without a version is an ambiguous claim.
- **Google Cloud has no single cross-product idempotency-key spec analogous to Stripe's.**
  - Cloud Storage uses object preconditions.
  - Individual APIs document their own retry guidance.
  - AIP-155 defines request IDs for the broader surface.
  - Scope any "Google does X" claim to the specific API.
- **Verified vs inferred:** none of Stripe, AWS, or Resend publish the isolation level or lock implementation behind their production key stores. Only the independent Postgres reference implementation (Brandur Leach's, Stripe-inspired) is concrete there - treat claims about "how Stripe's database enforces this" as inference from that reference design, never as disclosed internals.

## What this means for the SDK you ship

- The defaults decide behavior for the majority of callers who never read past installation - treat them as API contract, reviewed like one.
- Auto-generate an idempotency key for retry-unsafe operations when the SDK controls the request (Stripe's SDKs do) - the caller gets safe retries without knowing the mechanism exists.
- Expose the knobs (attempts, max delay, per-request key override) but make the zero-config path the safe one.
- Document the defaults in the SDK README with the same precision as the raw-HTTP contract: mode, attempt count, backoff shape, which operations get automatic keys.

---
name: api-idempotency-retry
description: Design idempotency-key support and client retry guidance for a public API so integrators retry safely - key derivation and per-caller scoping, storage TTL and replay windows, atomic claim mechanisms (DB unique constraint, SERIALIZABLE row lock, conditional writes), payload-mismatch rejection, in-flight duplicate handling, exponential backoff with jitter, retry budgets, timeout propagation, and SDK retry defaults. Use whenever the user mentions an Idempotency-Key header, duplicate or double-submitted requests, safe POST retries, backoff, or jitter - even if they never say "idempotency". Do NOT use for what the error response signals (samber/developer-platform-skills@api-error-design) or webhook delivery retries (samber/developer-platform-skills@webhook-platform-design).
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.0.1"
---

# API Idempotency and Retry

You design the two halves of one guarantee: server-side idempotency keys that make a retried write safe, and client-side retry behavior that uses them responsibly. Shipping either half alone is the trap - Resilience4j's own docs state it as a hard constraint, not a suggestion: retried operations must be idempotent, so retry guidance without key support (or the reverse) leaves the guarantee open exactly where a dropped connection tests it.

Two facts frame everything below:

- Exactly-once delivery is impossible over an unreliable network (Tyler Treat's canonical argument, grounded in the FLP result). What a key actually buys is _effectively-once_: at-least-once delivery plus duplicate collapse, bounded to the scope where idempotency is actually implemented.
- There is no ratified standard behind the `Idempotency-Key` header. The IETF draft (`draft-ietf-httpapi-idempotency-key-header`) expired in April 2026 without ratification. The header name is consistent across the industry because Stripe popularized it, but the semantics (TTL, mismatch behavior, concurrency handling) are whatever each provider decided, so yours must be designed and documented, never assumed.

**Memory (advised): When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.**

## Clarifying questions

Ask these before designing anything; each answer changes a later step. Batch them - this is a tactical design task, not a strategy interview.

1. Paradigm: REST, gRPC, or both? (gRPC carries deadline propagation natively; REST needs the header convention and explicit deadline plumbing)
2. Which endpoints are non-idempotent? Request the endpoint list or OpenAPI spec and enumerate every state-changing operation.
3. What storage backs the key store - a relational DB with unique constraints and transactions, a conditional-write KV store, or only a cache? (rules atomic-claim options in or out - see step 4)
4. Do any requests call an external, non-rollbackable system mid-flight (a charge, an outbound email)? (drives recovery points - see step 5)
5. Which SDK languages do you ship, and do integrators also call raw HTTP? (drives how much of the retry mechanics you can default for them - see step 6)
6. What is the longest possible retry chain - dead-letter-queue replay windows, dispute windows, scheduled re-runs? (drives TTL - see step 3)
7. By when must the guarantee hold: a one-off fix (stop the duplicate charges one client is filing tickets about), or a compounding contract every future endpoint inherits? What is the effort ceiling on the claim mechanism - a ship date that rules out a schema or isolation-level migration, or existing fluency with SERIALIZABLE workloads? (re-ranks the claim menu - see step 4: a hard date promotes whatever the current store already supports, a compounding mandate promotes the richer claim lifecycle, and a ceiling that forbids a schema migration deletes the rungs it cannot reach rather than ranking them last)

## Who controls the retry loop

Every retry decision is made by a developer or a program, whoever bought the product. The design splits by who controls the retry loop:

- **First-party developers** - you control both ends; internal convention plus code review covers most of it. The cheapest audience to serve.
- **Third-party integrators calling raw HTTP** - they write their own retry loop from your docs alone. They need the full contract published: key semantics, TTL, mismatch and in-flight behavior, and a worked backoff example, because no standard fills the gaps for them.
- **SDK-mediated callers** - your SDK _is_ their retry loop. Its defaults (backoff, jitter, attempt caps, key auto-generation) are the contract most callers will never read past, so wrong defaults ship duplicates at fleet scale.

Design for raw-HTTP integrators first - a contract complete enough for them is what your own SDKs then automate - and treat SDK defaults as the highest-leverage artifact, since they decide behavior for the majority who never read the docs.

## Workflow

1. Identify retry-unsafe endpoints.
2. Design the key contract.
3. Set storage and TTL.
4. Claim the key atomically.
5. Design the response semantics.
6. Write client retry guidance and SDK defaults.
7. Document the contract for integrators.

Each step has a section below, in order.

## 1. Identify retry-unsafe endpoints

- RFC 9110 - the one ratified document relevant here - defines GET, PUT, and DELETE as idempotent methods and POST as not. It says nothing about the `Idempotency-Key` retrofit pattern.
- Enumerate every state-changing endpoint. Each must either accept an idempotency key or be explicitly documented as unsafe to retry - the review-level check `samber/developer-platform-skills@public-api-design-review` performs; this skill builds what makes the first option true.
- Verify claimed idempotency, don't trust the method: a PUT that appends to a log or a DELETE that decrements a counter is not idempotent whatever the method promises.

## 2. Design the key contract

- The client mints the key, the server never derives it. AWS's Builders' Library names the rejected alternative: a server-side hash of request parameters breaks when two genuinely identical requests are both intended (launching two identical instances on purpose) - a caller-provided key expresses intent an inferred one cannot. Stripe, AWS, Twilio, and Resend all put the client in charge.
- Scope keys per caller - `(account_id, idempotency_key)` - never globally. Stripe, AWS, and the standard Postgres reference implementation all key on the pair, so the same key string reused by different accounts never collides.
- Recommend a derivation to clients. Three work; rank them for the client rather than listing them (Resend's own recommendation order):
  - value: `event-derived > request-scoped > random UUID reused across retries`
  - effort: `event-derived > request-scoped > random UUID reused across retries`
  - efficiency: `event-derived > request-scoped > random UUID reused across retries` - value and effort rank identically here, so their spread decides the ratio: minutes wide on effort (name the key after an identifier the caller already holds) against the whole guarantee on value.
  - **Default rung: event-derived** (`order-confirm-${orderId}`) - deterministic by construction, so a retry cannot fail to reuse the key. Move down to request-scoped when no stable business identifier exists at call time, and to a reused random UUID when the caller is a stateless script. Nothing is starved here: every rung costs one line of client code.
  - Never rank a fourth rung below these: a fresh value per attempt (`Date.now()`, `randomUUID()` regenerated on retry) defeats the mechanism entirely, so document it as the anti-pattern instead.
- Guard the payload: the same key with a different request body is a client bug - reject it (409 is the field convention; Resend names it `409 invalid_idempotent_request`), never silently replay. The expired IETF draft generalizes this into an optional payload _fingerprint_ stored beside the key; a checksum of the payload is the simplest form.
- Store both outcomes. Stripe saves failures as well as successes - including 500s - so a retry replays the exact original result. Decide explicitly whether a stored failure replays verbatim (Stripe v1) or the operation may be re-attempted without side effects (Stripe v2); leaving it undecided is a design gap.

## 3. Set storage and TTL

- Retention is two separate concerns, not one TTL (Brandur Leach's reference implementation):
  - Replay window (~24h) - governs idempotent-replay correctness; also the documented convention at Stripe, Resend, and Twilio.
  - Reaper threshold (~72h) - a longer operational window that keeps records alive for human recovery, so a bad Friday deploy's failed requests survive the weekend for Monday's fix.
- Set the replay window from the longest retry chain (question 6), never from storage cost. A 24-hour TTL behind a 7-day DLQ replay is a duplicate waiting to happen. AWS retains EC2 run tokens for the resource's lifetime plus a grace period - which is what lets a late retry get a consistent answer even after the resource was deleted.
- After expiry, treat a reused key as a brand-new request - every provider surveyed does; none returns an error for post-TTL reuse.
- Store the full response verbatim (status, headers that matter, body), not a summary - the replay must be indistinguishable from the original.

## 4. Claim the key atomically

A check-then-insert is a TOCTOU race under concurrent retries: both callers see "absent" and both proceed. The claim must be a single atomic operation the storage layer arbitrates. Three mechanisms appear in production, ranked:

- efficiency: `DB unique constraint > conditional-write OCC > row lock + SERIALIZABLE`
- effort: `row lock + SERIALIZABLE > conditional-write OCC > DB unique constraint`
- value: `row lock + SERIALIZABLE > DB unique constraint == conditional-write OCC`

- **Default rung: a unique constraint on `(account_id, idempotency_key)`.** The database picks the single winner; the loser gets its 409 from the `INSERT` failure itself, not an application-level check. Nearly free - one index - and transaction isolation gives the fencing guarantee below at no extra cost.
- **Conditional-write OCC (DynamoDB-style)** delivers the same single-winner claim on stores without unique-constraint transactions - the tie on the value line is real: same guarantee, different storage family. Your answer to question 3 rules one of these two out rather than ranking it; a ruled-out option is deleted, not demoted.
- **Row lock + SERIALIZABLE transaction is the starved option.**
  - Value: a `locked_at` column inside a SERIALIZABLE transaction represents _in-flight_ as a first-class state, survives lock expiry, and anchors resumable recovery points.
  - Effort: isolation-level tuning, serialization-abort retry handling - so it loses every efficiency round.
  - Promote it when requests span foreign state mutations (question 4): multi-step requests need the richer claim lifecycle, not just a winner-picker.
- Whatever the mechanism: the claim and the business mutation must commit as one ACID unit - AWS states this as a hard requirement, not a best practice. A claim outside the transaction it guards can record a token for a resource that was never created, or the reverse.
- A bare Redis `SETNX` + TTL lock is not a fourth option. Kleppmann's analysis: without a fencing token, a client paused past its lease expiry can still write after the lock moved on - no timing assumption saves you. No surveyed vendor uses a distributed lock for the claim; if Redis is all you have, layer fencing tokens on top, and know the DB constraint sidesteps the problem entirely.
- This ranking is a default, not a law. Re-rank against the storage answer and the effort ceiling (questions 3 and 7): a team already running SERIALIZABLE workloads gets the starved option far cheaper than the effort line assumes. A ship date that forbids a schema migration promotes whichever mechanism their current store already supports.

## 5. Design the response semantics

The bug in most homegrown implementations is treating a key as binary - seen or not-seen. A key is a three-state machine: **absent**, **in-flight**, **complete**. The retry that hits an in-flight key is not the bug ("your client library did exactly what it was configured to do when the connection dropped"); the bug is a server with no way to say "seen, not finished."

| Outcome                  | What happened                            | Correct behavior                                                                                    |
| ------------------------ | ---------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Never received           | Network drop before processing           | Retry proceeds as a first attempt; the claim wins normally                                          |
| Received, not processed  | Claim held, no terminal result yet       | Return an in-flight conflict, or resume from the last recovery point once the original lock expired |
| Processed, response lost | Result committed; response never arrived | Replay the stored response verbatim - no re-execution                                               |

- For the in-flight case, pick one:
  - Reject with 409 (Resend's `409 concurrent_idempotent_requests`, documented advice "wait and retry") - simplest and safest default.
  - Block until the bounded result, when the caller needs it synchronously.
  - Return 202 plus a status URL, for long-running effects.

  Never let the second caller through because the first "seems stuck." The expired IETF draft deliberately made in-flight conflict and completed replay _different_ responses, not fallbacks of each other.

- For multi-step requests (question 4), decompose into atomic phases separated by each foreign state mutation, persisting a recovery point between phases - a retried request resumes exactly where it left off instead of re-running an external call already made. Record intent _before_ calling out, so a crash between call and response leaves evidence.
- Run recovery as active background jobs, not passive expiry (Brandur's model):
  - A _completer_ pushes abandoned in-flight requests through to completion.
  - A _reaper_ deletes expired keys and logs what it couldn't finish for a human.
- What the error responses look like - codes, envelope, `retryable` flag, `Retry-After` - is `samber/developer-platform-skills@api-error-design`'s territory; consume its signals, don't redesign them here. Distinguishing your two 409 meanings (payload mismatch vs in-flight) with distinct machine-readable codes is that skill's taxonomy applied to this skill's states.

See [references/key-lifecycle-walkthrough.md](references/key-lifecycle-walkthrough.md) for the full worked lifecycle: schema, claim query, all three states' responses, and a multi-step recovery-point example.

## 6. Write client retry guidance and SDK defaults

- Retry an explicit allowlist, never a blanket catch: 429 (honoring `Retry-After`), 502, 503, 504, connection timeouts and resets - 500 cautiously (some 500s are a real bug that retries identically). Never retry non-429 4xx, auth failures, or exceptions that are bugs. Four independent sources (tenacity, Resilience4j, Resend, and the response-signaling side) converge on this exact split - cite it as convergent practice, not one vendor's opinion.
- Never retry a read timeout on a non-idempotent operation without an idempotency key already in place. This is the exact boundary condition the server half of this skill exists to close.
- Backoff must carry jitter. Ranked (Marc Brooker's AWS analysis, the canonical source):
  - value: `full jitter > decorrelated jitter > equal jitter > no jitter`
  - effort: `full jitter == decorrelated jitter == equal jitter == no jitter` - a real tie, not a dodge: each is one formula or one library flag, identical to write, test, and reverse, so no rung buys its value with more of the reader's time.
  - efficiency: the value order unchanged, since equal effort cancels out of the ratio.
  - **Default rung: full jitter** (`sleep = random(0, min(cap, base * 2^attempt))`) - lowest total client work and lowest server load, at slightly more total time to completion.
  - Starved by that order: decorrelated jitter, which completes faster for more total work. Promote it when a user-facing call sits inside a tight deadline and time-to-success outweighs fleet load. Equal jitter is Brooker's "clear loser" among jittered options, and no jitter re-synchronizes the fleet into the exact spike that caused the failures - keep neither in the menu you hand an integrator.
  - Re-rank against what you know about the caller: a single-instance batch job pays none of the fleet synchronization cost this order assumes.
- Bound both attempt count and total duration, not attempts alone - a per-attempt-reasonable loop can blow past the caller's own budget. Log every retry (attempt, error, next wait); a silent retry loop hides an outage in the making.
- Add a fleet-wide retry budget, distinct from per-caller caps: stop retrying once retries exceed a set share of traffic over a sliding window. A hundred instances each retrying three times amplify load 300x while every instance looks reasonable in isolation. AWS's token bucket is the named implementation, tuned to engage the quota sooner in a real outage: 14 tokens per transient-error retry, 5 per throttle retry.
- Propagate the deadline: one overall deadline set at the top, per-attempt timeouts budgeted inside it - never a fresh overall timeout per attempt, which silently extends the caller's real wait. Set single timeouts at 2-3x measured p99, not an OS default. gRPC's relative-timeout-on-the-wire model is the citable mechanics for chains.
- Point integrators at their language's retry library (tenacity, Resilience4j, or equivalent) before describing formulas - the formula is a one-liner there, pre-debugged. When pairing with a circuit breaker, state the stacking order: `Retry → CircuitBreaker → RateLimiter → Bulkhead`.
- For SDKs you ship: default retries ON with backoff + jitter and a bounded attempt count (AWS defaults to 3 total), and auto-generate idempotency keys for unsafe operations where the SDK can. Defaults are the contract for callers who never read docs. How your retry defaults interact with the platform's rate-limit policy - whether SDK retries count against quota, adaptive throttling - is `samber/developer-platform-skills@api-rate-limit-policy` territory.

Formulas, budget mechanics, and deadline math: [references/backoff-jitter-formulas.md](references/backoff-jitter-formulas.md). Cross-vendor defaults to calibrate yours against: [references/vendor-sdk-retry-defaults.md](references/vendor-sdk-retry-defaults.md).

## 7. Document the contract for integrators

Because no standard defines the semantics, your documentation is the only normative source your integrators have. Document, per the audience split above:

- Which endpoints accept the key, the header name, max key length, and the recommended derivation.
- Scoping, the replay window as a concrete number, and what happens after it expires ("treated as a new request").
- Both conflict behaviors, each with its distinct error code: payload mismatch and in-flight duplicate.
- Whether a stored failure replays or re-attempts (the Stripe v1/v2 distinction, applied to your API).
- A worked retry loop with concrete numbers - schedule, cap, jitter - like Resend's documented `1s → 2s → 4s → 8s` capped at 30s.
- Never write "exactly-once delivery" - the claim is provably false across an open network. Write what is true instead: retries are safe within the documented key scope and window. Correct the claim wherever marketing already made it (Treat's steelman: Kafka's exactly-once holds only inside a closed transactional system; a public API is by definition not one).

## Failure modes

Anti-pattern checklist - each is a direct audit finding:

- A key accepted but never stored or honored - tells the client retrying is safe when it isn't; worse than no key at all.
- A fresh key value generated per attempt.
- Check-then-insert claiming (TOCTOU race).
- Binary key state with no in-flight representation.
- Same key, different payload, silently replayed.
- Replay window shorter than the longest retry chain (the 24h-TTL-behind-7-day-DLQ trap).
- A bare `SETNX` lock with no fencing token as the claim.
- No decision on whether stored failures replay or re-execute.
- Retry loop that resets its overall timeout each attempt.
- Backoff without jitter.
- "Exactly-once" anywhere in the docs.

## Measurement

- Concurrency gate: fire N identical concurrent requests with one key against a state-changing endpoint - exactly one side effect must execute; the other N-1 get the in-flight conflict or the stored replay. Binary pass; iterate the claim design until it holds. This is the definitional guarantee, testable directly.
- Crash-recovery gate: kill the server between atomic phases of a multi-step request, then retry - the request must resume from its recovery point with zero repeated external calls. Binary pass where question 4 answered yes.
- Duplicate side-effect rate in production: within the documented scope and window the design target is zero by definition - track it as the alarm that the gates regressed, not as a tunable threshold.
- Orphaned-key rate (claimed, never completed - the completer's queue): no published industry threshold exists, so set the baseline from your own first month and watch the trend, as with any unsourced operational metric.

## Invocation examples

- "Add Idempotency-Key support to our POST /payments and POST /orders endpoints - we're seeing duplicate charges when mobile clients retry on timeout."
- "Write the retry guidance section of our API docs, and pick the backoff defaults for the Python and Node SDKs we ship."
- "Two concurrent webhook handlers both created the same invoice - audit our idempotency implementation and find the race."

## References

- [references/key-lifecycle-walkthrough.md](references/key-lifecycle-walkthrough.md) - worked key lifecycle: schema, atomic claim, three-state responses with real 409 codes, recovery points, completer/reaper.
- [references/backoff-jitter-formulas.md](references/backoff-jitter-formulas.md) - the three jitter formulas with trade-offs, retry-budget mechanics, deadline propagation, timeout setting.
- [references/vendor-sdk-retry-defaults.md](references/vendor-sdk-retry-defaults.md) - how AWS, Stripe, Google Cloud, Twilio, and Resend default retry and idempotency behavior; what to copy and what to flag.