integration-error-observability

{
  "skill_name": "integration-error-observability",
  "evals": [
    {
      "id": 1,
      "prompt": "I run the platform team at Freightwise - we do shipment tracking APIs for 3PLs and freight brokers. We push about 4M webhooks a month, and our delivery worker already writes an attempts table with the HTTP response code and the timeout reason for every attempt, but it's internal-only, nobody outside engineering can see it. Support is burning roughly 15 hours a week on integration tickets. Leadership just approved 6 engineering weeks starting next month to build a developer-facing dashboard. Where do we start?",
      "expected_output": "A build order that starts from a classification of the recent support tickets, separates the three log surfaces, and recommends exposing the delivery-attempt data Freightwise already records as the first rung rather than building everything at once.",
      "files": [],
      "expectations": [
        "Recommends classifying the most recent integration support tickets by failure class (request not found, webhook never arrived, auth or credential failure, rate-limit confusion, silent data mismatch) before choosing what to build",
        "Names the request log, the event log, and the webhook delivery-attempt log as three distinct surfaces rather than one combined dashboard",
        "States that the delivery-attempt log answers 'did you tell me about it', the request log answers 'did my call work', and the event log answers 'what did it change'",
        "Recommends exposing the existing delivery-attempt data as the first surface, on the grounds that it is already recorded and the remaining work is mostly access control and interface",
        "Does not recommend building all three surfaces simultaneously inside the six-week window",
        "States that a delivery-attempt row must carry the actual failure reason (timeout, connection refused, TLS failure, or a specific non-2xx status) rather than a generic 'delivery failed'",
        "Lists the fields a delivery-attempt row carries, including event ID, destination endpoint, attempt number, attempt timestamp, and next scheduled retry time",
        "Warns that a surface shipping only failure counts with no detail behind it converts a silent failure into a support ticket rather than into a fix",
        "Asks or establishes what an integrator can see today without contacting Freightwise, as the audit baseline",
        "Asks whether request and response bodies carry PII, secrets, or another tenant's data, and who signs off on exposing them to a third party",
        "Asks whether this is a one-off fix after a bad support quarter or a compounding surface Freightwise will run for years, and ties that answer to which rung it recommends",
        "Requires the dashboard to display the same identifiers Freightwise returns to integrators rather than internal primary keys"
      ]
    },
    {
      "id": 2,
      "prompt": "Palmate is a B2B billing API. We return an X-Palmate-Request-Id header on 4xx and 5xx responses. Partners quote it in tickets and our support team tells me that roughly half the time the answer is 'that ID isn't in anything I can search'. Our edge assigns the ID, and then each internal service generates its own one for its own logs. We also call an external tax engine and a card processor on most requests. Fix this properly.",
      "expected_output": "A correlation-identifier design that echoes one unchanged ID on every response, makes it resolvable in a surface the integrator can reach, and propagates it downstream and across retries.",
      "files": [],
      "expectations": [
        "Identifies that the ID is echoed only on failures, and requires echoing it on every response including 2xx, because a call that returned 200 and did the wrong thing is a real ticket class with no error body to carry the ID",
        "Identifies per-service regeneration as a defect and requires one identifier to survive unchanged from the edge through every internal hop",
        "States that echoing the header without an external surface that resolves it leaves integrators quoting a string nobody outside Palmate can look up",
        "Recommends making the identifier searchable in a developer-facing log the integrator can reach themselves, and pairs that with the header rather than treating either rung as sufficient alone",
        "Distinguishes a request ID, which identifies one HTTP call and its response, from a correlation ID, which identifies a multi-call transaction spanning several request IDs",
        "Recommends forwarding the same identifier to the tax engine and the card processor instead of generating new ones at those hops",
        "Justifies promoting downstream propagation by pointing at Palmate's current inability to see into failures that came from a provider it calls",
        "Requires Palmate's own internal retries to carry the original identifier rather than generating a fresh one per attempt",
        "Recommends logging the identifier on both sides of every boundary: inbound at the edge and outbound at each downstream call",
        "Recommends accepting and echoing a client-supplied identifier when one arrives rather than overwriting it",
        "Specifies an opaque random value such as a UUIDv4 and rejects sequential or timestamp-derived identifiers because they leak volume or timing and collide under concurrency",
        "Does not redesign the error response payload shape, or explicitly defers the question of whether the ID rides in the error body to a separate error-design scope",
        "Recommends documenting the lookup path next to the error documentation rather than on a separate observability page"
      ]
    },
    {
      "id": 3,
      "prompt": "Nooklet sends an email to the partner's technical contact every time one of their webhook deliveries fails. We have 340 partner apps. Six of our top ten partners have admitted they filter us into a folder they never open. Our VP wants to add severity levels and route the important ones to Slack instead. Most of our partners reconcile bookings weekly, but two of them are payment-adjacent and lose real money inside an hour when their integration breaks. What should the notification design actually be?",
      "expected_output": "A notification design that removes per-failure alerting entirely, defaults to a per-integration digest, and promotes threshold alerting with a cooldown only for the two money-losing partners.",
      "files": [],
      "expectations": [
        "Rejects per-failure alerting on error volume as a design option outright rather than ranking it last, naming it as the alert-fatigue failure mode itself",
        "Recommends a periodic per-integration digest as the default notification rung for the majority of Nooklet's partners",
        "Recommends threshold alerting with a cooldown specifically for the two payment-adjacent partners, tying that promotion to their losing money within hours",
        "Specifies a cooldown period during which the same integration cannot alert again, and explains that without it one ongoing incident produces a continuous stream until the recipient mutes the channel permanently",
        "Advises deriving the cooldown length from Nooklet's own typical incident duration rather than copying a borrowed number",
        "Treats integration state changes such as revoked credentials, a disabled integration, or a lapsed subscription as a separate immediate rung with no threshold and no cooldown",
        "Requires a recovery notification when the rate returns to baseline, on the grounds that silence is not an all-clear",
        "Requires every alert to name the top machine-readable error codes and to link directly into the log filtered to those failures",
        "Requires a threshold alert to state whether a Nooklet release falls in the window, so the partner knows what the platform has already ruled out",
        "Requires a threshold alert to state the earliest time the next alert for that integration can arrive",
        "States that the digest and the threshold alert must not be routed to the same destination at the same cadence",
        "Recommends tracking the share of alerts partners act on, and treats a sustained decline as the design failing rather than the partners failing",
        "Specifies an escalation path naming where a repeated unacted-on alert goes and after what interval",
        "Does not present adding severity levels to the existing per-failure emails as the fix"
      ]
    },
    {
      "id": 4,
      "prompt": "Gridwell exposes an HR data API to payroll and benefits vendors. Our partner dashboard shows one number: 'error rate, last 24h'. We page ourselves when any partner crosses 5%. Last week a partner sat at 4.2% all week while 100% of one specific app's calls were failing, and we never noticed. We also don't tag requests by app at the edge - we can join back to the API key afterwards from the auth service logs. And our biggest partner scaled 8x last month and went red purely from throttling. Rebuild the aggregation.",
      "expected_output": "A per-identity aggregation schema tagged at ingestion, split into separate 4xx/5xx/429 buckets, compared against each integration's own baseline across two detection windows, with deploy markers overlaid.",
      "files": [],
      "expectations": [
        "Requires tagging each request with the integration identity at ingestion, and rejects reconstructing the tag afterwards by joining auth-service logs",
        "States that retrofitting the identity tag leaves the preceding history unqueryable per integration, which is exactly the period any investigation needs",
        "Splits the error rate into at least three separately counted buckets: 4xx, 5xx, and 429",
        "Explains that 429 is isolated because rate-limit rejection is neither party's defect, and that blending it makes a partner scaling successfully look identical to a partner whose code is broken",
        "Attributes 4xx to the integrator's side and 5xx to Gridwell's side, and states that a blended number cannot say whose problem the errors are",
        "Replaces the single 5% platform-wide threshold with a comparison of each integration against its own recent baseline",
        "Explains that one global percentage is either noise for low-traffic integrations or blindness for high-traffic ones",
        "Requires the aggregate row to carry a success count as the denominator alongside the error counts",
        "Requires explicit window start and end on each aggregate rather than a floating rate",
        "Requires the top machine-readable error codes per bucket in the aggregate",
        "Overlays Gridwell's own deploy or release markers on the error-rate view, and requires checking a 5xx spike against that timeline before telling a partner anything",
        "Specifies both a short detection window measured in minutes for sudden breaks and a long window of hours to a day for slow degradation, and states that a single window misses half the failure types",
        "Recommends reading how errors are distributed rather than volume alone, distinguishing a clustered pattern (logic bug or bulk operation) from an evenly spread one (infrastructure)"
      ]
    },
    {
      "id": 5,
      "prompt": "Kettleback partners keep asking for two things. First, a button to re-send webhook events their endpoint missed during their own maintenance windows. Second, a 'retry this call' button on failed POST /orders requests in our log viewer. We already retry webhooks automatically 6 times over 12 hours. One partner wants to bulk re-send everything that failed during a 4-hour outage on their side, which is about 90,000 events. Design both features.",
      "expected_output": "Event replay designed with a stable event ID, a bounded published window, rate-controlled bulk scope, and independence from the automatic retry schedule - plus a refusal to build synchronous request replay.",
      "files": [],
      "expectations": [
        "Refuses to build a replay button that re-issues the failed POST /orders request on the integrator's behalf, and explains that re-issuing it creates side effects nobody authorised",
        "Recommends instead surfacing the full request detail so the integrator's own client re-issues the call",
        "Approves event replay for the webhook case and ties the promotion to partners losing events during their own downtime, which is Kettleback's stated situation",
        "Requires the replayed delivery to carry the same event ID as the original so the consumer can deduplicate",
        "States that without a stable event ID replay converts a missed event into a duplicated side effect, a worse failure than the one it fixed",
        "Requires manual replay to run independently of Kettleback's automatic six-attempt retry schedule, without cancelling or resetting it",
        "Warns that a manual resend issued while automatic retries are still scheduled can produce a duplicate delivery, and addresses that specific overlap",
        "Defines a bounded window measured from event creation beyond which replay is not offered, and requires publishing it",
        "Requires rate control on the 90,000-event bulk replay, and states that an unthrottled bulk replay turns the partner's recovery into Kettleback's next incident",
        "Decides the replay scope explicitly among per-event, per-endpoint, and per-time-range rather than leaving it implicit",
        "Identifies replay as the highest-effort item on the visibility menu rather than a quick win",
        "Does not recommend having a manual resend cancel or supersede the in-flight automatic retries"
      ]
    },
    {
      "id": 6,
      "prompt": "Tuliphase has a per-customer dashboard where any admin of that customer's account can see every API call their integrations made - full request log, 30 days, searchable, with bodies. We're genuinely proud of it. We have 22 ISV partners whose apps are installed across roughly 6,000 of our customers. Every single time one of those apps breaks, the ISV finds out from their own customers, not from us. Our head of partnerships thinks we need to promote the dashboard harder. Diagnose it.",
      "expected_output": "A diagnosis that the per-account dashboard is structurally wrong for fleet partners, requiring cross-install aggregation keyed on the app ID plus push notification, and a separate access-model fix.",
      "files": [],
      "expectations": [
        "States the problem is structural rather than a matter of awareness, and rejects promoting the existing dashboard as the fix",
        "Explains that a per-account dashboard cannot serve an ISV whose app is installed across thousands of accounts, because nobody watches thousands of dashboards",
        "Requires cross-install aggregation keyed on the app or client ID, spanning every install of one ISV's app, as the missing surface",
        "Requires push notification to the ISV rather than relying on them to come and look",
        "Identifies the access model as a distinct gap: the ISV engineer who maintains the integration is typically not a user of the customer's account and therefore cannot open the per-account dashboard at all",
        "Distinguishes the single-account integrator segment from the ISV-fleet segment, and states that a surface designed for the first does not finish the job for the second",
        "Recommends an installs-affected against installs-total measure per ISV app, and reads that ratio as the diagnosis",
        "Interprets a high installs-affected ratio as a partner-side release and a low one as configuration trouble at a few customers",
        "Does not conclude that Tuliphase's existing per-account request log was the wrong thing to have built",
        "Recommends deciding explicitly who can read each surface: the account owner, any user of the account, and the partner engineer outside the account",
        "Scopes the notification audience to the 22 ISVs rather than to the individual customer accounts"
      ]
    },
    {
      "id": 7,
      "prompt": "Verandaq's integration health board was all green for eleven days. Then a partner emailed asking why none of their stock syncs had landed since the 3rd. Their scheduler had died on the 3rd and they simply stopped calling us - zero requests, zero errors, board stayed green the whole time. Our SRE lead wants to add a 'no traffic' alert that fires when any integration drops below 10 calls per hour. What's the right design here?",
      "expected_output": "A volume-drop signal designed as its own item, baselined per integration rather than against a global floor, surfaced in the fleet view and pushed to the partner.",
      "files": [],
      "expectations": [
        "Rejects a single platform-wide floor of ten calls per hour, and explains that integrations differ in traffic shape by orders of magnitude",
        "Requires comparing each integration's call volume against that integration's own recent baseline instead of a fixed number",
        "States explicitly that an absence of calls produces no errors and therefore appears nowhere in an error log or error-rate view",
        "Treats the volume-drop signal as its own design item rather than a side effect of the error logging already in place",
        "Requires silent integrations to appear in the provider-side fleet view alongside the erroring ones",
        "Notes that silent failures are frequently the most expensive class precisely because neither side has noticed",
        "Recommends notifying the partner about the drop rather than waiting for them to ask",
        "Places the volume-drop signal outside the 4xx, 5xx and 429 status-code buckets rather than inside them",
        "Uses a long enough comparison window that a single quiet interval does not fire the signal",
        "Records this failure class as its own row in the visibility audit alongside the error classes",
        "Does not claim that better error alerting or a lower error threshold would have caught this"
      ]
    },
    {
      "id": 8,
      "prompt": "Sablecrest wants two things built. An internal partner error leaderboard: every partner sorted by total errors in the last 24 hours, descending, top 20 on a TV in the partner team's area. And a 0-100 Integration Health Score shown to each partner in their portal, which we'll use to decide who keeps Premier tier next year. Write the spec.",
      "expected_output": "A fleet view ranked by blast radius with contact ownership and a separate regression queue, plus hard constraints on publishing any partner-facing score that drives commercial tiering.",
      "files": [],
      "expectations": [
        "Rejects sorting the fleet view by raw error count and requires ranking by blast radius instead",
        "Explains that a partner app failing across many installs outranks a single account throwing more absolute errors, because one conversation fixes many problems",
        "Requires installs affected against installs total per partner app as a dimension of the view",
        "Requires a dominant status-code bucket per row so the queue separates the partner's payloads from Sablecrest's own failures from quota rejection",
        "Requires a named contact owner on each row, on the grounds that a ranked list with no contact becomes a report nobody acts on",
        "Requires first-seen timestamps aligned against both Sablecrest's release timeline and, where known, the partner's",
        "Separates integrations broken since Sablecrest's own deploy into a distinct view or tab from the partner-outreach queue, and identifies the first as Sablecrest's own regression queue",
        "Requires silent integrations whose call volume dropped toward zero to appear in the view",
        "Warns against publishing a per-partner reliability number Sablecrest cannot defend line by line, because it will be quoted back in commercial conversations",
        "Requires the inputs behind any partner-facing score to be inspectable by that partner: error counts, install coverage, and the windows used",
        "Requires a documented recomputation and dispute process before the score can affect Premier tier, rather than after the first complaint",
        "Rules out showing a partner where they rank against named competitors",
        "States that volume alone is not health, and that a large partner at a stable low error rate is healthier than a small partner at a high one",
        "Requires Sablecrest to set and defend its own weights rather than borrowing another product's weighting"
      ]
    },
    {
      "id": 9,
      "prompt": "Our new platform director came from an SRE background. She wants each of our 40 integration partners at Marrowpoint to get a 99.9% availability SLO with a monthly error budget, published in the partner portal, and she wants us to explain to partners that because they depend on us plus two other vendors, their achievable availability is capped below 99.7%. She's asked me to write the spec this week. Sanity-check it before I do.",
      "expected_output": "A separation of which parts of SRE error-budget practice transfer to a provider-side integration view and which do not, with a refusal to publish a per-partner SLO derived from integration health.",
      "files": [],
      "expectations": [
        "States that the published error-budget and SLO literature is written consumer-side, for a team measuring its own dependence on someone else's API, and does not transfer wholesale to a provider publishing numbers for partners",
        "Rejects presenting the multi-vendor multiplicative availability argument to Marrowpoint's partners, because that arithmetic describes the consumer's position rather than Marrowpoint's",
        "Warns that an integration measurement includes the partner's own code and their customers' configuration, so committing to a number derived from it is a promise Marrowpoint only partly controls",
        "Routes formal SLA and uptime publication to a separate status-and-incident-communication scope rather than deriving it from the integration health view",
        "Keeps per-integration and per-endpoint granularity as a part that does transfer, against a single blended platform number",
        "Keeps multi-window burn-rate detection as a part that does transfer: a long window confirming the problem is sustained plus a short window confirming it is still active",
        "States that nothing in the published record establishes what a provider-side integration SLO should contain",
        "Advises against publishing any per-partner number before Marrowpoint can defend how it is computed",
        "Frames the goal of being the trustworthy real-time source of integration health as a design intent rather than an industry standard to cite to partners",
        "Does not simply approve writing the 99.9% per-partner SLO spec as requested",
        "Recommends an internal prioritisation view first, and distinguishes it from a partner-facing published grade as two different products with different obligations",
        "Notes that a partner-facing number becomes a contract the moment partners start optimising against it"
      ]
    },
    {
      "id": 10,
      "prompt": "Clovermast processes medical claims. Legal has given us a written answer: request and response bodies contain PHI and must never be visible outside our tenancy boundary, no exceptions, not negotiable. Meanwhile about 70% of our support pile is 'my call failed and I don't know why' on synchronous POSTs. Our current log viewer shows timestamp, method, path, and status code, and nothing else. Design what we can actually ship.",
      "expected_output": "A metadata-only request log treated as the ceiling rather than a compromise, with error codes, identifiers and identity added, and structural tenant isolation plus unconditional credential stripping.",
      "files": [],
      "expectations": [
        "Removes payload-level logging from the option set entirely rather than ranking it lower or proposing a masked version of it",
        "States that metadata-only logging is the ceiling in Clovermast's case rather than a compromise position",
        "Adds a machine-readable error code to every row, and states that a bare 400 tells the integrator nothing their own client did not already know",
        "Adds a request ID to every row, copy-pasteable and searchable by the integrator",
        "Adds the integration identity and the account it acted on to every row",
        "Adds latency to the row",
        "Requires an error detail or reason field naming which field or condition failed, constructed without echoing the payload itself",
        "Requires credentials, tokens and signing secrets to be stripped unconditionally rather than partially masked, because a prefix plus a length is often enough to attack",
        "Requires any withheld field to be visibly marked as withheld so the integrator can tell it was suppressed rather than absent",
        "Requires tenant isolation to be enforced structurally rather than by an application-code filter on account ID",
        "Treats the redaction rule set as a published contract once integrators build against it, noting that loosening it later is a privacy incident and tightening it later breaks their tooling",
        "Prioritises the synchronous request log over the other surfaces on the strength of the 70% synchronous share of the support pile",
        "Does not propose negotiating with legal or seeking an exception as the path forward"
      ]
    },
    {
      "id": 11,
      "prompt": "Plumridge is a payroll API. Two problems. First, we keep integration logs for 24 hours to keep storage costs down, but our customers reconcile payroll monthly. Second, about 60% of the people building against us aren't engineers - they're ops staff wiring us into no-code automation tools, and their tickets literally say 'it's not working'. Our log viewer today shows raw JSON request and response pairs plus HTTP status codes. What do we change?",
      "expected_output": "A retention window sized to the monthly reconciliation cycle and published, plus a separate plain-language failure surface for the non-engineer segment that does not replace the detailed one.",
      "files": [],
      "expectations": [
        "States that a 24-hour retention window against monthly reconciliation guarantees the evidence expires before the investigation starts",
        "Requires the retention window to exceed the time it realistically takes a Plumridge customer to notice a failure, and sizes it against the monthly cycle",
        "Requires the retention window to be published rather than discovered during an incident",
        "Identifies the no-code and non-engineer builders as a distinct audience segment with different needs from engineers integrating directly",
        "Recommends a plain-language failure list scoped to that builder's own flow rather than raw JSON request and response pairs",
        "Requires a 'what to do about it' line next to each failure entry for that segment",
        "States that the detail useful to an engineering integrator reads as noise to the no-code segment",
        "Keeps the detailed surface for the engineering segment rather than replacing it wholesale with the simplified one",
        "Recommends classifying Plumridge's 'it's not working' tickets by failure class to decide which surface to build first",
        "Treats the retention change and the audience change as two separate items rather than one",
        "Names log retention shorter than time-to-notice as a failure mode that generates tickets regardless of how good the log row is"
      ]
    }
  ],
  "trigger_queries": [
    {
      "query": "How do we show partners what's failing in their integration without them opening a ticket?",
      "should_trigger": true
    },
    {
      "query": "Design a developer-facing request log for our public API",
      "should_trigger": true
    },
    {
      "query": "our ISV partners always hear about outages from their own customers before they hear from us",
      "should_trigger": true
    },
    {
      "query": "We want to expose webhook delivery attempts to the developers who subscribed to them",
      "should_trigger": true
    },
    {
      "query": "what should a per-integration error dashboard actually show",
      "should_trigger": true
    },
    {
      "query": "Should we email partners when their error rate spikes?",
      "should_trigger": true
    },
    {
      "query": "partners quote a request id in tickets and support can't find it anywhere",
      "should_trigger": true
    },
    {
      "query": "integration support tickets are eating 20 hours a week, how do we cut that down",
      "should_trigger": true
    },
    {
      "query": "Design proactive partner notification for broken integrations",
      "should_trigger": true
    },
    {
      "query": "we need a view of which partner apps are breaking across their installs",
      "should_trigger": true
    },
    {
      "query": "how do I let customers replay failed webhook events",
      "should_trigger": true
    },
    {
      "query": "our error rate looks fine platform-wide but one app is failing every single call",
      "should_trigger": true
    },
    {
      "query": "Should 429s be in the same error rate as 400s when we report to partners?",
      "should_trigger": true
    },
    {
      "query": "what does a good API log row need to contain for an external developer",
      "should_trigger": true
    },
    {
      "query": "we alert on every failed delivery and everyone has muted us",
      "should_trigger": true
    },
    {
      "query": "how long should we keep integration logs that partners can see",
      "should_trigger": true
    },
    {
      "query": "design the cooldown for our partner alerting",
      "should_trigger": true
    },
    {
      "query": "one of our partners stopped calling us entirely and nothing flagged it",
      "should_trigger": true
    },
    {
      "query": "can we show request and response bodies to third-party developers",
      "should_trigger": true
    },
    {
      "query": "Our support team can't resolve the correlation ID integrators send us",
      "should_trigger": true
    },
    {
      "query": "we want to rank our partners by how broken their integrations are",
      "should_trigger": true
    },
    {
      "query": "should we give each partner an integration health score",
      "should_trigger": true
    },
    {
      "query": "how do we tell a partner the errors are on their side and not ours",
      "should_trigger": true
    },
    {
      "query": "build the spec for a partner-facing failure digest email",
      "should_trigger": true
    },
    {
      "query": "external devs can't see why their calls are failing",
      "should_trigger": true
    },
    {
      "query": "what's the cheapest thing we can ship so integrators stop asking us what went wrong",
      "should_trigger": true
    },
    {
      "query": "we're adding a debug console to our developer portal, what goes in it",
      "should_trigger": true
    },
    {
      "query": "how should we tag incoming requests so we can aggregate errors per API key",
      "should_trigger": true
    },
    {
      "query": "Design the notification we send when a partner's credentials get revoked",
      "should_trigger": true
    },
    {
      "query": "the same incident sends 400 emails to one partner",
      "should_trigger": true
    },
    {
      "query": "our partner engineers can't log into the customer accounts where their app is installed",
      "should_trigger": true
    },
    {
      "query": "how do we detect that a partner's integration silently stopped working",
      "should_trigger": true
    },
    {
      "query": "should a manual resend cancel our automatic retries",
      "should_trigger": true
    },
    {
      "query": "we return X-Request-Id but nobody outside the company can look it up",
      "should_trigger": true
    },
    {
      "query": "what do we put on the screen in the partner ops room",
      "should_trigger": true
    },
    {
      "query": "how do we know whether an error spike was our deploy or their release",
      "should_trigger": true
    },
    {
      "query": "Design a self-service debugging surface for third-party integrators",
      "should_trigger": true
    },
    {
      "query": "give integrators a way to see their own failed API calls",
      "should_trigger": true
    },
    {
      "query": "our webhook consumers keep asking whether we even tried to deliver",
      "should_trigger": true
    },
    {
      "query": "should we forward our request id to the payment processor we call downstream",
      "should_trigger": true
    },
    {
      "query": "how do we decide which integration visibility feature to build first",
      "should_trigger": true
    },
    {
      "query": "we're getting 'the webhook never arrived' tickets constantly",
      "should_trigger": true
    },
    {
      "query": "what should we alert partners on and what should we leave in a dashboard",
      "should_trigger": true
    },
    {
      "query": "our 4xx and 5xx sit in one bucket and partners argue about whose fault it is",
      "should_trigger": true
    },
    {
      "query": "design the fleet view for our partner team",
      "should_trigger": true
    },
    {
      "query": "how do we stop being the last to know when an integration breaks",
      "should_trigger": true
    },
    {
      "query": "we want to publish a reliability number per partner",
      "should_trigger": true
    },
    {
      "query": "what belongs in an event log versus a request log",
      "should_trigger": true
    },
    {
      "query": "should the partner be able to re-run a failed POST from our dashboard",
      "should_trigger": true
    },
    {
      "query": "how do we redact PII before showing logs to a third party",
      "should_trigger": true
    },
    {
      "query": "our log dashboard can only be opened by the account owner",
      "should_trigger": true
    },
    {
      "query": "what metric tells us whether our developer debugging tools are working",
      "should_trigger": true
    },
    {
      "query": "we're rebuilding our integrations page, what visibility do partners actually need",
      "should_trigger": true
    },
    {
      "query": "should we use traceparent or our own request id header",
      "should_trigger": true
    },
    {
      "query": "how do we avoid alert fatigue with partner notifications",
      "should_trigger": true
    },
    {
      "query": "design the escalation path when a partner ignores our error alerts",
      "should_trigger": true
    },
    {
      "query": "we have no idea which of our 200 integrations are healthy right now",
      "should_trigger": true
    },
    {
      "query": "the no-code people building on our API just say 'it's broken'",
      "should_trigger": true
    },
    {
      "query": "how do we bound bulk event replay so it doesn't take us down",
      "should_trigger": true
    },
    {
      "query": "what should our partner error emails actually say",
      "should_trigger": true
    },
    {
      "query": "design our status page and the incident update cadence",
      "should_trigger": false
    },
    {
      "query": "write the public postmortem for last week's API outage",
      "should_trigger": false
    },
    {
      "query": "what should our published uptime SLA say",
      "should_trigger": false
    },
    {
      "query": "we need to announce a scheduled maintenance window to our API consumers",
      "should_trigger": false
    },
    {
      "query": "design our error code taxonomy and problem+json envelope",
      "should_trigger": false
    },
    {
      "query": "should our 400 responses follow RFC 9457",
      "should_trigger": false
    },
    {
      "query": "how do we word API error messages so they're actionable",
      "should_trigger": false
    },
    {
      "query": "add a retryable flag and Retry-After to our error responses",
      "should_trigger": false
    },
    {
      "query": "design our idempotency key storage and TTL",
      "should_trigger": false
    },
    {
      "query": "what backoff and jitter should our SDK use on retries",
      "should_trigger": false
    },
    {
      "query": "how do we reject a payload mismatch on a reused idempotency key",
      "should_trigger": false
    },
    {
      "query": "design the retry schedule and dead-letter policy for our outbound webhooks",
      "should_trigger": false
    },
    {
      "query": "what HMAC signing scheme should our webhooks use",
      "should_trigger": false
    },
    {
      "query": "design our webhook event taxonomy and payload envelope",
      "should_trigger": false
    },
    {
      "query": "how do we version webhook payload schemas without breaking consumers",
      "should_trigger": false
    },
    {
      "query": "set our rate limit tiers and burst quotas",
      "should_trigger": false
    },
    {
      "query": "which rate limit headers should we return, X-RateLimit or the IETF RateLimit fields",
      "should_trigger": false
    },
    {
      "query": "how much notice do we owe a partner before lowering their quota",
      "should_trigger": false
    },
    {
      "query": "design our API key prefix and checksum format",
      "should_trigger": false
    },
    {
      "query": "how do we rotate API keys with zero downtime",
      "should_trigger": false
    },
    {
      "query": "should we store API keys hashed or encrypted",
      "should_trigger": false
    },
    {
      "query": "design our OAuth consent screen and scope taxonomy",
      "should_trigger": false
    },
    {
      "query": "should we require PKCE for confidential clients too",
      "should_trigger": false
    },
    {
      "query": "what should our access token TTL be",
      "should_trigger": false
    },
    {
      "query": "design our sandbox isolation model and magic test values",
      "should_trigger": false
    },
    {
      "query": "how do developers reset and reseed their test account data",
      "should_trigger": false
    },
    {
      "query": "design the information architecture of our developer portal",
      "should_trigger": false
    },
    {
      "query": "how do we measure time to first call for new signups",
      "should_trigger": false
    },
    {
      "query": "audit our OpenAPI reference for undocumented endpoints",
      "should_trigger": false
    },
    {
      "query": "our docs have drifted from the spec, what CI gates do we add",
      "should_trigger": false
    },
    {
      "query": "when do we cut /v2 and how long should the deprecation window be",
      "should_trigger": false
    },
    {
      "query": "should we send Sunset and Deprecation headers",
      "should_trigger": false
    },
    {
      "query": "which languages get official SDKs first",
      "should_trigger": false
    },
    {
      "query": "should we generate our SDKs or hand-write them",
      "should_trigger": false
    },
    {
      "query": "should we add a GraphQL layer or stay REST-only",
      "should_trigger": false
    },
    {
      "query": "design Relay cursor connections for our public schema",
      "should_trigger": false
    },
    {
      "query": "what query depth and complexity limits should our public GraphQL API have",
      "should_trigger": false
    },
    {
      "query": "review our REST endpoint naming and status code usage",
      "should_trigger": false
    },
    {
      "query": "should this operation be a POST or a PATCH",
      "should_trigger": false
    },
    {
      "query": "design our gRPC proto packages and buf breaking-change gates",
      "should_trigger": false
    },
    {
      "query": "should we build our own app marketplace or just list on other people's",
      "should_trigger": false
    },
    {
      "query": "what take rate should our app store charge developers",
      "should_trigger": false
    },
    {
      "query": "write the review and approval criteria for third-party apps on our marketplace",
      "should_trigger": false
    },
    {
      "query": "our AppExchange listing isn't converting, fix the screenshots and tagline",
      "should_trigger": false
    },
    {
      "query": "set the screenshot and demo video specs every listing must meet",
      "should_trigger": false
    },
    {
      "query": "pick the launch cohort for our marketplace reveal at the keynote",
      "should_trigger": false
    },
    {
      "query": "design the partner developer signup and sandbox provisioning flow",
      "should_trigger": false
    },
    {
      "query": "which technology partners should we integrate with next quarter",
      "should_trigger": false
    },
    {
      "query": "should we build a managed Fivetran connector or ship a Singer tap",
      "should_trigger": false
    },
    {
      "query": "design the Parquet file drop and partitioning scheme for our bulk export",
      "should_trigger": false
    },
    {
      "query": "how do we give customers a JDBC endpoint onto their own data",
      "should_trigger": false
    },
    {
      "query": "which twelve tools should our MCP server expose to agents",
      "should_trigger": false
    },
    {
      "query": "how do we hire an API product manager",
      "should_trigger": false
    },
    {
      "query": "am I ready to interview for a partner engineer role",
      "should_trigger": false
    },
    {
      "query": "where should I start with our new public API program",
      "should_trigger": false
    },
    {
      "query": "instrument our Go services with OpenTelemetry traces and metrics",
      "should_trigger": false
    },
    {
      "query": "set up Datadog dashboards and SLO monitors for our Kubernetes cluster",
      "should_trigger": false
    },
    {
      "query": "my Sentry error rate is spiking in production, help me debug it",
      "should_trigger": false
    },
    {
      "query": "add structured logging and error wrapping to our Python codebase",
      "should_trigger": false
    },
    {
      "query": "design query observability for our Postgres cluster",
      "should_trigger": false
    }
  ]
}

# Alerting and Notification

Per-integration error aggregation, detection windows, and how to notify a partner without training them to ignore you.

## Contents

- Aggregation schema
- Status-code buckets
- Detection windows
- Threshold and cooldown mechanics
- Notification templates
- Anti-fatigue rules

## Aggregation schema

The minimum a per-integration aggregate has to carry, one row per identity per window:

| Field                        | Why it exists                                                                                   |
| ---------------------------- | ----------------------------------------------------------------------------------------------- |
| Integration identity         | API key or app/client ID, tagged at ingestion - the aggregation is only as granular as this tag |
| Account                      | Which of your customers the integration was acting on; a fleet app needs both dimensions        |
| Window start and end         | Explicit, so a rate is interpretable rather than a floating number                              |
| Success count                | The denominator; an error count without it cannot produce a rate                                |
| Count per status-code bucket | 4xx, 5xx, 429 separately - never one blended error total                                        |
| Top error codes              | The machine-readable codes, ranked; this is what makes a notification actionable                |
| Deploy markers               | Your release timeline, overlaid, so a platform-caused spike is distinguishable                  |

Tag at ingestion rather than reconstructing later. A tag added after the fact leaves the preceding history unqueryable by integration, which is exactly the period any investigation wants.

## Status-code buckets

Three buckets, kept apart because each has a different owner and a different remediation:

- **4xx - the integrator's side.** Malformed payloads, missing fields, invalid auth, calls against deleted objects. Actionable by them, and the bucket where naming the top error codes does most of the work.
- **5xx - your side.** Never notify a partner about a 5xx spike without first checking it against your deploy timeline, and never phrase it as though it were theirs to fix.
- **429 - neither party's bug.** Rate-limit rejection is a capacity and quota conversation, not a defect. Isolating it prevents the most common misread of a blended rate: a partner scaling successfully looks identical to a partner whose code is broken. Quota design itself belongs to `samber/developer-platform-skills@api-rate-limit-policy`.

A fourth signal sits outside the buckets entirely: **volume drop to zero**. An integration that stopped calling produces no errors and appears nowhere in an error-rate view, so detect it separately by comparing call volume against the integration's own recent baseline.

## Detection windows

Two speeds catch two different failures, and a design with only one will miss half of them:

- **A short window** (minutes) catches a sudden break - a deploy on either side, a revoked credential, an endpoint gone.
- **A long window** (hours to a day) catches slow degradation - an error rate creeping from 1% to 6% over a week, which never trips a short-window threshold but is the more expensive failure because it runs unnoticed for longer.

This is the multiwindow, multi-burn-rate pattern from SRE alerting practice: pair a long window that confirms a problem is sustained with a shorter window that confirms budget is still actively being consumed before firing (sourced: Google SRE Workbook, "Alerting on SLOs"). The two-speed detection instinct transfers cleanly to per-integration alerting. The surrounding error-budget framing does not, because it was written for a team measuring its own dependence on somebody else's API.

Compare each integration against its own baseline rather than a platform-wide threshold. Traffic shapes differ by orders of magnitude between integrations, so any single global percentage is either noise for the small ones or blindness for the large ones.

## Threshold and cooldown mechanics

The concrete pattern worth copying comes from Salesforce's Agentforce health monitoring (sourced: Salesforce): notify immediately when an error-rate or latency threshold is crossed, then enforce a **cooldown period - 30 minutes by default there - before the same subject can alert again.**

Why the cooldown is the load-bearing part: without it, a single ongoing incident generates a continuous alert stream, and the recipient's only defence is to mute the channel - permanently, and for every future incident too. The cooldown converts "we alert on incidents" from a claim into something that survives contact with a real incident.

Design decisions to make explicitly:

- **Cooldown length.** Long enough that one incident produces one alert. Short enough that a second, distinct incident is not swallowed. Start from your typical incident duration, not from a borrowed number.
- **Recovery notification.** Send one when the rate returns to baseline. Without it, the partner has no signal to stop investigating, and the missing all-clear is itself a source of tickets.
- **Threshold ownership.** Decide whether partners can tune their own thresholds. Self-service tuning is the honest answer for a diverse partner base, since no single threshold fits every traffic shape - but it is a build, not a config flag.
- **Escalation path.** Where a repeated alert goes if nobody acts: your partner team, and after what interval. An alert with no escalation is a notification, not a control.

## Notification templates

**Digest (default rung).** A scheduled summary as the deliberate alternative to an instant per-failure alert is a real, named pattern in adjacent observability tooling - Elementary Data ships an "Incidents Digest" explicitly as a scheduled complement to real-time alerts (sourced: Elementary Data docs) - though no platform surveyed documents it scoped specifically to a per-integration or per-partner audience the way this rung is; that scoping is this skill's own application of the pattern. One scheduled message per integration:

```
Integration health - acme-connector - 30 Aug

  Calls          12,480   (-3% vs prior day)
  Failed           1,204   (9.6%)

  Integrator-side (4xx)  1,190
    invalid_shipping_address   842
    missing_customer_email     201
    object_not_found           147
  Platform-side (5xx)        14
  Rate-limited (429)           0

  Full request log: <link>
```

**Threshold alert (promoted rung).** One message, sent on crossing, then held for the cooldown:

```
Alert: acme-connector error rate at 41% (baseline 2%)
Started 14:05 UTC, ongoing. 3,912 failed calls across 84 installs.
Top error: invalid_shipping_address (94% of failures).
No platform release in this window - this pattern indicates a change on your side.
Request log filtered to these failures: <link>
Next alert for this integration no earlier than 14:35 UTC.
```

Two things make the second template work: it states what the platform already ruled out (its own deploys), and it names when the next alert can arrive, which is what stops the recipient from bracing for a flood.

**State-change alert (narrow rung).** Credentials revoked, integration disabled, subscription lapsed - one discrete event, sent immediately, no threshold and no cooldown, because there is nothing to repeat.

## Anti-fatigue rules

- One alert per incident, not per failure. If the design cannot express that, it is not ready to send.
- Never alert on a 5xx spike before checking your own deploy timeline.
- Every alert names the top error codes and links directly into the filtered log. An alert that only says "errors are up" costs the recipient a search and buys them nothing.
- Send a recovery message. Silence is not an all-clear.
- Track the share of alerts partners act on and treat a sustained decline as the design failing, not the partners. Widen the threshold or lengthen the cooldown before adding another channel.
- Never route the digest and the threshold alert to the same place at the same cadence. The point of two rungs is that they interrupt differently.

# Correlation ID Conventions

How to identify a failed call so an external developer can hand your support team one string that locates it exactly.

## Contents

- Request ID versus correlation ID
- Header conventions and platform variants
- Echo and propagation rules
- The support-lookup workflow
- A negative example

## Request ID versus correlation ID

Two identifiers, routinely conflated, doing different jobs:

- **Request ID** - identifies exactly one HTTP call and its response. Generated at your edge if the client did not supply one.
- **Correlation ID** - identifies an entire multi-call transaction. One user action can produce a dozen request IDs under a single correlation ID.

Most platforms ship the request ID and stop. The correlation ID is what a partner needs when their failure is "the checkout flow broke" rather than "this call returned 500" - a flow spanning several of your endpoints, where the interesting question is which step failed, not what one call returned.

If you support only one, support the request ID: it is the one that appears in an error the integrator is already holding. Add the correlation ID when the audit shows multi-call flows in the ticket pile.

## Header conventions and platform variants

An opaque per-request correlation header is the de facto convention, generated or forwarded by essentially every gateway, load balancer and cloud edge - but `X-Request-Id` is not the one literal name they converge on. Every major platform brands its own header instead (sourced: AWS, GitHub, Vercel, Twilio docs, see variants below), so treat "an opaque request identifier exists" as the convention and the header's name as a per-platform choice. The value should be an opaque random string - a UUIDv4 is the default choice, because it is collision-safe under load and carries no information, unlike a sequential counter (leaks volume) or a timestamp-derived value (leaks timing and collides under concurrency).

Named variants exist and are citations rather than requirements:

- GitHub returns `X-GitHub-Request-Id` with colon-separated hex segments, e.g. `1C73:26D4:E2E500:1EF78F4:62EC2479` (sourced: GitHub docs).
- Vercel returns `x-vercel-id`, listing the regions a request hit and the region a function executed in, which also lets Vercel detect infinite loops (sourced: Vercel docs). Vercel's own support terms ask users filing a case to include this header's value - a first-party instance of the support-lookup workflow below (sourced: Vercel support terms).
- Cloudflare returns `Cf-Ray`, a hashed value encoding the data center that handled the request, meant to be logged at the origin so a request can be matched between Cloudflare's edge and the origin's own server logs (sourced: Cloudflare docs).
- AWS load balancers return `X-Amzn-Trace-Id` in the form `Root=1-<hex-timestamp>-<hex-id>`, generated by the load balancer when the client didn't supply one (sourced: AWS docs).
- Twilio returns `Twilio-Request-Id` and its own REST API best practices tell developers to check for it before contacting support (sourced: Twilio docs) - the same first-party support-lookup pattern as Vercel's.

Pick one header name, document it, and never change it - integrators write parsers against it.

The standards-track alternative is the W3C `traceparent` header: `version-trace_id-parent_id-trace_flags`, where the trace-flags byte's low bit carries the sampling decision (sourced: W3C Trace Context spec). Choose it when you already emit distributed tracing internally and want external calls to join those traces. Choose the plain request ID when you do not, since `traceparent` without a tracing backend behind it is a more complicated string that resolves to nothing.

Accept a client-supplied ID when one arrives, and echo it rather than overwriting it - an integrator who generates their own correlation ID is trying to join your logs to theirs, which is the behaviour you want.

## Echo and propagation rules

- **Echo in response headers on every response**, not only failures. A call that returned 200 and did the wrong thing is a real ticket class, and it has no error body to carry an ID.
- **Echo in the error body too**, coordinating with `samber/developer-platform-skills@api-error-design`, which owns the payload shape. Header-only means the ID is lost the moment an integrator's client logs the parsed body and discards the response headers - which most do.
- **Never regenerate mid-flight.** If your edge assigns an ID and an internal service assigns another, the string the integrator holds resolves to nothing.
- **Propagate on downstream calls.** When handling the request causes calls to sub-processors, carriers or other providers, forward the same identifier so one lookup surfaces the whole chain.
- **Propagate on retries.** Your own internal retries must carry the original ID. A retry that generates a fresh one splits a single failure into unlinked fragments.
- **Log it on both sides of every boundary** - inbound at your edge, outbound at each downstream call - so the chain reconstructs even when a hop fails.

## The support-lookup workflow

The design target: an integrator resolves their own failure before writing to you, and when they do write, one string is enough. Three things have to be true:

1. The ID is in something the integrator already has (response header, error body, or their own client logs).
2. A surface they can reach resolves it - the developer-facing request log. This is the rung platforms skip, and skipping it converts the ID from a self-service tool into ticket decoration.
3. Your support team resolves the same string against internal logs, including the downstream hops the integrator cannot see.

Document the lookup path in the developer docs, next to the error documentation rather than in a separate observability page: the moment an integrator needs it, they are reading about an error.

Two platforms document this exact workflow, by name, in their own first-party docs rather than leaving it implicit: Plaid's API reference and error docs both instruct developers to "include the `request_id` when contacting support regarding a specific API call," and PayPal's integration checklist and troubleshooting docs name `PayPal-Debug-Id` as the identifier to retain and hand to support (sourced: Plaid docs, PayPal docs). Both name the support-resolution use case explicitly, which is what separates this from generic request tracing.

## A negative example

What a broken implementation looks like from the outside:

```
Integrator: "Call failed at 14:22 UTC, request ID req_8f2c1ab9d3e04c7f."
Support:    "That ID isn't in a system I can search. Can you send the full
             request body, the endpoint, and your account ID?"
```

Everything the ID exists to prevent happens anyway. The usual causes, in order of frequency:

- The ID is generated at the edge for tracing but never persisted alongside the application logs support actually searches.
- Each internal service generates its own, so the returned ID belongs to a hop that has no record of the failure.
- The ID is returned but dropped on the downstream call that actually failed, so the chain ends one hop short of the answer.
- The ID resolves internally, but no external surface exists, so every lookup still costs a support round-trip - the header shipped, the log never did.

# Fleet View and Health Scoring

The provider-side view of which integrations are breaking, how to score partner health, and why consumer-side SLO framing does not transfer.

## Contents

- What the fleet view is for
- Layout and ranking
- Per-partner health scoring
- The consumer-side SLO adaptation gap
- What not to publish

## What the fleet view is for

Steps 2 through 5 of the skill build outward-facing surfaces. This one faces inward: your partner, support and platform teams looking across the whole integration base and asking which partner to contact first. It runs on the same aggregation as the outward surfaces, read along a different axis.

It exists because proactive outreach needs a queue. Without it, "we notice partner problems before they do" is aspirational - someone has to be looking at a ranked list on a Monday morning, and the ranking is what makes that a five-minute job rather than an afternoon.

## Layout and ranking

Rank by blast radius rather than raw error volume. A partner app failing across 200 installs outranks a single high-traffic account throwing more absolute errors, because the first is one conversation that fixes 200 problems. A flat leaderboard sorted by error count inverts exactly that priority.

Dimensions worth having on the view:

- **Installs affected** versus **installs total**, per partner app. The ratio is the diagnosis: 200/200 is a partner-side release, 3/200 is a configuration problem at three customers.
- **Error rate against that integration's own baseline**, not a global threshold.
- **First-seen timestamp**, aligned against both your release timeline and - where you know it - the partner's. Broken since your deploy is yours outright. Broken since theirs is the outreach case.
- **Dominant status-code bucket**, so the queue separates "their payloads changed" from "we broke them" from "they hit quota" before anyone opens a ticket.
- **Silent integrations** - call volume dropped to or near zero against baseline. These appear in no error ranking and are frequently the worst failures, because nobody on either side has noticed.
- **Contact ownership**, so the row is actionable. A ranked list with no named partner contact becomes a report nobody acts on.

Give the view a second tab for the platform's own view of itself: integrations whose failures correlate with your last release, aggregated. That is your regression queue, and it should never be mixed into the partner-outreach queue.

## Per-partner health scoring

A weighted health score per partner is a real shipped pattern, not an invention: third-party tooling built specifically for ISVs on Salesforce computes a per-managed-package weighted health score and a churn-risk signal from it (sourced: ISVapp product documentation). That the pattern exists as a third-party product is itself informative - it fills a gap the underlying platform did not expose to its ISVs natively.

If you build one, three rules keep it honest:

1. **Set and defend your own weights.** Borrowing another product's weighting produces a number nobody in your organisation can explain when a partner disputes it - and they will dispute it, because a low score is a commercial signal.
2. **Make the inputs inspectable.** A partner shown a score must be able to see the error counts, install coverage and windows behind it. An opaque score is a trust liability, and the first dispute converts it into a support burden larger than the one it saved.
3. **Decide internal versus external before building.** An internal prioritisation score and a partner-facing health grade are different products with different obligations. The internal one can be rough and iterated. The external one becomes a contract the moment partners optimise against it.

Score composition that holds up in practice:

- Error rate relative to baseline
- Install coverage of the failures
- Trend direction over the long window
- Time-since-last-clean-period

Volume alone is not health - a large partner with a stable 0.5% error rate is healthier than a small one at 30%.

## The consumer-side SLO adaptation gap

The published literature on error-budget and SLO framing is consistently **consumer-side**: guidance for a company that depends on third-party APIs, not for a platform providing one to external integrators (sourced: Nobl9 "A Complete Guide to Error Budgets", Stytch "Understanding SLAs, SLOs, SLIs and Error Budgets"). Borrowing the framing wholesale is a documented failure mode of this design. The consumer-side literature on reliability states these principles, some of which transfer to the provider side and others do not:

- **Give each third-party dependency its own SLO** rather than folding vendor downtime into one blended number (sourced: Nobl9, Stytch; also Google Cloud's CRE blog "Defining SLOs for services with dependencies", and formalized in Treynor, Dahlin, Rau and Beyer's "The Calculus of Service Availability", ACM Queue / SRE Workbook). _Transfers as a shape_: per-integration and per-endpoint granularity beats a blended platform number, which is the same argument step 4 of the skill makes.
- **Multi-vendor reliability compounds multiplicatively** - several 99.9% dependencies cap the consumer's achievable availability below any one of them (sourced: Nobl9, Stytch). _Does not transfer_: this is arithmetic about the consumer's position, not yours.
- **Do not trust a vendor's status page as your monitoring source.** The gap between a real outage and a status update is error budget draining unwatched (sourced: Nobl9, Stytch; also ThousandEyes, "Why You Shouldn't Trust (Only) the Status Page"). _Transfers inverted_, and this is the interesting one - you are the vendor being distrusted.
- **Endpoint-level granularity matters** because a blended uptime number hides partial outages of specific functionality (sourced: Nobl9, Stytch; the same point recurs across API-observability vendor documentation, e.g. Zuplo, Datadog, without one canonical origin). _Transfers directly._
- **Multi-window burn-rate alerting** catches both sudden spikes and slow degradation (sourced: Google SRE Workbook, "Alerting on SLOs"). _Transfers directly_: a short window for sudden breaks, a long one for creeping degradation.

The mirror-image framing - the platform working to be the trustworthy, real-time source about integration health so its partners do not have to build independent monitoring of it - is **this skill's own construction, not a sourced practice**. It is a useful design intent. It is not something to present to partners as an industry standard, and nothing in the published record establishes what a provider-side integration SLO should contain.

## What not to publish

- **A per-partner reliability number you cannot defend line by line.** Once published it is quoted back at you in commercial conversations.
- **An availability commitment derived from the fleet view.** The fleet view measures integrations, which include the partner's own code and their customers' configurations. Committing to a number you only partly control is a promise you cannot keep. Formal SLA and uptime publication belongs to `samber/developer-platform-skills@api-status-communication`.
- **Cross-partner comparisons.** A partner learning where they rank against named competitors turns an operational tool into a commercial incident.
- **A health grade with no appeal path.** If a score affects a partner's tier, placement or commercial terms, it needs a documented recomputation and dispute process before it goes live, not after the first complaint.

# Visibility Surfaces

The three developer-facing log surfaces, what each row must carry, replay semantics, and the access and redaction rules that gate all of it.

## Contents

- The three surfaces
- Required fields per surface
- Good and bad log rows
- Replay semantics
- Retention, access scoping, redaction

## The three surfaces

They answer three different questions and are not substitutes. Stripe's developer dashboard keeps them separate and is the clearest public reference for the split (sourced: Stripe docs).

| Surface              | Question it answers         | Granularity                               | Primary reader                                              |
| -------------------- | --------------------------- | ----------------------------------------- | ----------------------------------------------------------- |
| Request log          | "Did my call work?"         | One HTTP request/response pair            | Any integrator debugging a synchronous failure              |
| Event log            | "What did it change?"       | One business-level event, with payload    | An integrator reconciling state, or auditing after the fact |
| Delivery-attempt log | "Did you tell me about it?" | One delivery attempt against one endpoint | A webhook consumer whose handler went quiet                 |

A platform that ships only the request log leaves every "the webhook never arrived" ticket unanswerable, and one that ships only the delivery log cannot explain a synchronous 400. Build the one matching your ticket pile first. Plan for all three.

## Required fields per surface

**Request log row** carries:

- Timestamp with timezone
- HTTP method and endpoint path
- Status code
- Machine-readable error code
- Request ID
- Integration identity (API key or app ID, and the account it acted on)
- Latency
- Request and response bodies, subject to redaction

Omitting the error code and keeping only the status code is the common shortfall: `400` alone tells an integrator nothing their own client did not already know.

**Event log row** carries:

- Event ID
- Event type
- Creation timestamp
- The object it concerns
- The full payload as delivered
- A pointer to the request that caused it

The link back to the causing request is what turns two logs into one investigation.

**Delivery-attempt log row** carries:

- Event ID
- Destination endpoint
- Attempt number
- Attempt timestamp
- Response status or transport-level failure reason (timeout, connection refused, TLS failure, non-2xx)
- The response body your server received back, if any
- The next scheduled retry time

The failure reason must be the actual one, not a generic "delivery failed" - the whole value of the surface is giving the integrator the detail their own logs would have shown if their handler had ever run. This is a real, shipped external surface rather than an internal-only convenience: Twilio's Debugger, HubSpot's webhook monitoring log, Zapier's Zap history, and Workato's job history each expose attempt-level status and the actual failure reason (including a distinction between a timeout and other causes) to their own external developers (sourced: Twilio, HubSpot, Zapier, Workato docs).

Across all three: make every identifier copy-pasteable and searchable, and show the same identifiers the integrator sees in their own responses. A dashboard that displays an internal primary key instead of the request ID you returned in the header breaks the lookup path entirely.

## Good and bad log rows

Bad - a row that produces a support ticket:

```
2026-08-31 14:22  POST /v1/orders   400   Error
```

Nothing here is actionable. No error code, no request ID, no indication of which field failed, no identity, no link to detail.

Good - a row the integrator resolves alone:

```
2026-08-31 14:22:07 UTC  POST /v1/orders  400  invalid_shipping_address
  request_id  req_8f2c1ab9d3e04c7f
  app         acme-connector v2.3   account acct_19KdQ2
  detail      shipping.postal_code: "SW1A" is not a valid postal code for country "US"
  latency     84 ms
```

The difference is not volume of text - it is that each field answers a question the integrator would otherwise have asked you.

## Replay semantics

Replay is the highest-effort rung of the visibility ladder. Design it around four decisions:

1. **Window.** Bound how far back a replay can reach. Stripe allows manual resend of a failed event for up to 15 days after creation (sourced: Stripe docs). Pick your own bound from your retention and state model, and publish it.
2. **Independence from automatic retry.** A manual replay must not cancel, reset or interfere with the platform's own retry schedule, and the two must not double-deliver in a way the integrator cannot detect. Stripe's resend explicitly does not cancel its automatic schedule (sourced: Stripe docs) - the two mechanisms run independently. PayPal ships the same pattern: a "Resend" action on its Webhooks Events dashboard, separate from its automatic retry schedule (sourced: PayPal docs). Checkout.com documents the same feature and is explicit about the risk this decision creates: manual resend can produce a duplicate delivery "if any automatic retries are scheduled" (sourced: Checkout.com docs) - exactly the failure this rule exists to prevent.
3. **Idempotency.** Every replayed delivery carries the same event ID as the original so the consumer can deduplicate. Without a stable ID, replay converts a missed event into a duplicated side effect, which is a worse failure than the one it fixed.
4. **Scope.** Decide whether replay is per-event, per-endpoint (everything that failed in a window), or per-time-range. Bulk replay needs rate control, or an integrator recovering from an outage becomes your next incident.

Do not offer replay of synchronous API requests the way you offer replay of events. Re-issuing a POST on the integrator's behalf creates side effects nobody authorised. Give them the full request detail and let their own client re-issue it.

## Retention, access scoping, redaction

**Retention.** The window has to exceed the time it realistically takes a partner to notice a failure. A 24-hour window on a failure class noticed at weekly reconciliation guarantees the evidence expires before the investigation starts. Publish the window, since an undocumented one gets discovered during an incident.

**Access scoping.** Decide explicitly who can read each surface: the account owner, any user of the account, and - separately - the partner engineer who maintains the integration but is not a user of the customer's account. That third case is the one platforms usually miss, and it is the reason a partner with a fleet of installs cannot debug through a per-account dashboard even when one exists.

**Redaction.** Before any payload becomes visible to an external party, run a data-classification pass:

- Strip credentials, tokens and signing secrets unconditionally - never partially masked, since a prefix plus a length is often enough to attack.
- Mask PII to the level your legal answer (question 7 of the interview) permits, and make the masking visible so the integrator knows a field was withheld rather than absent.
- Guarantee tenant isolation structurally, not by query convention. A log surface that filters by account ID in application code is one missing clause away from showing one customer another's data.
- Treat the redaction rule set as a published contract once integrators build against it - loosening it later is a privacy incident, tightening it later breaks their tooling.

---
name: integration-error-observability
description: Design how a platform surfaces integration failures to the external developers and partners who built against it - per-integration request/event/delivery logs with replay, correlation IDs that survive from response header to support ticket, error-rate aggregation per API key or app split by 4xx/5xx/429, proactive partner notification with cooldowns against alert fatigue, and a provider-side fleet view of which integrations are breaking. Use whenever the user mentions a developer-facing error dashboard, exposing request logs to integrators, alerting partners about broken integrations, or cutting integration support tickets - even if they never say "observability". Do NOT use for platform-wide incident comms - use samber/developer-platform-skills@api-status-communication instead.
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.0.1"
---

# Integration Error Observability

You are designing what an external developer can find out about their own broken integration, without opening a support ticket. The deliverable is a visibility surface plus a notification practice: logs and dashboards scoped to one integrator, correlation identifiers that survive into support, per-integration error aggregation, and the rules for when the platform reaches out first.

The asymmetry is the whole problem. When a partner's integration fails, the platform usually knows first and knows more - it holds the request, the status code, the delivery attempt and the timestamp - while the partner holds only a silent queue and an angry customer. Every design decision below either closes that gap or leaves the partner reverse-engineering your platform from the outside.

**Memory (advised):** When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

## Clarifying questions

Ask these before designing anything. Each answer moves a later ranking. Batch them - this is a tactical design task, not a strategy interview.

1. What can an integrator see today without asking you? Send the URL if any surface exists. "Nothing" is a valid and common answer.
2. Who breaks: engineers at a single customer integrating for their own use, commercial partners whose app runs across a fleet of installs, no-code and agent builders, or a mix? (see next section)
3. Classify the last 20 integration support tickets: request not found, webhook never arrived, silent data mismatch, auth or credential failure, rate-limit confusion. The shape of that pile decides which surface to build first.
4. Is every inbound request already tagged with an integration identity - API key, app or client ID - at ingestion? Aggregation is only ever as granular as the tag it is keyed on.
5. Do webhooks exist, and does the delivery machinery already record per-attempt outcomes? (decides the default rung in step 2)
6. When an integration breaks, does the partner lose money within hours, or is it noticed at the next reconciliation? (decides the notification posture in step 5)
7. What can you legally show? Do request and response bodies carry PII, secrets, or data belonging to another tenant, and who signs off on exposing them to a third party?
8. Deadline, payoff shape and effort ceiling: by what date must this be live, is it a one-off fix after a bad support quarter or a compounding platform surface you will run for years, and what can you spend - engineering weeks, log storage, an on-call rotation to answer the alerts you start sending?

Question 8 exists because every menu below diverges sharply on effort and on how long the payoff lasts. A hard deadline promotes whatever you already record and merely have to expose, while a compounding-surface mandate promotes the rungs that need new storage, new state, or a new outbound channel.

Question 7 can delete rungs outright rather than reorder them - a legal answer of "those bodies can never leave our tenancy boundary" removes payload-level logging from the menu instead of demoting it.

If your harness has persistent memory, store the surface's core decisions - which logs are exposed, the retention window, the identity the aggregation is keyed on, the alert thresholds and cooldown, the redaction rules - so a later run adding an event type or tuning an alert starts from the design instead of re-deriving it.

## Integration ownership

How many end users sit behind the failing integration decides whether a dashboard is enough:

- **Single-account integrators** - a customer's own engineers wiring your API into their systems. One account, one dashboard, one set of credentials. A good self-serve log surface genuinely finishes the job for them: they will go look when something breaks, because their own users are already complaining.
- **Partners and ISVs with a fleet of installs** - their app runs across tens or thousands of customer accounts, and each install can fail independently. A per-account dashboard is structurally useless to them: nobody watches a thousand dashboards, so they need cross-install aggregation and push notification instead, or they learn about breakage from their own customers. This is the segment that makes steps 5 and 6 load-bearing rather than optional.
- **No-code and agent builders** - assemble a flow, cannot read a stack trace, and will not correlate anything. They need one plain-language failure list scoped to their flow, and a "what do I do about it" line next to each entry. Detail that helps the other two segments reads as noise here.

Serve every segment your answer to question 2 names. A surface designed only for the first segment is the documented default failure: the platform ships logs, calls integration visibility done, and its partner base still finds out about outages from end users.

## Workflow

1. Audit what an integrator can see today.
2. Expose per-integration failure detail.
3. Make the correlation ID reach support.
4. Aggregate error rates per integration identity.
5. Decide the notification posture.
6. Build the provider-side fleet view.

Each step has a section below, in order.

## 1. Audit what an integrator can see today

Start from the support pile (question 3), not from an architecture diagram. For each of the last 20 integration tickets, answer one question: could the integrator have answered this themselves with a surface you could reasonably ship? The share that says yes is your baseline, and it is the number every later step is trying to move.

- Walk each failure class separately: synchronous API errors, webhooks that never arrived, auth or credential expiry, rate-limit rejections, and silent data mismatches. Platforms routinely ship a surface for one and assume it covers the others.
- Check the failure that produces no error at all: an integration that stopped calling you entirely. Nothing in an error log shows an absence, so a volume-drop signal is a separate design item, not a side effect of error logging.
- Record retention honestly. A log an integrator cannot reach back to the moment of the incident is a log that generates a support ticket anyway.
- Note who can see it. A surface that only the account owner can open fails the partner whose engineer is not a user of that account - an access-model gap, not a logging gap.

Write the audit as a table of failure class against current visibility, and carry it into every step below. The gaps are the build order.

## 2. Expose per-integration failure detail

Three distinct surfaces exist here, and conflating them is the common design error. Stripe's developer dashboard is the reference implementation to copy the shape of, and it keeps them separate (sourced: Stripe docs):

- **Request log** - every API call with its outcome. Answers "did my call work".
- **Event log** - the business-level events those calls produced. Answers "what did it change".
- **Webhook delivery log** - every delivery attempt per endpoint with the failure reason. Answers "did you tell me about it".

An integrator debugging in the dark needs whichever one matches their failure class.

Rungs, from what you can expose cheapest to what costs most to build:

- value (support tickets deflected): `request log with per-request detail > delivery-attempt log > replay > failure counts only`
- effort: `replay > request log with per-request detail > delivery-attempt log > failure counts only`
- compliance cost: `request log with per-request detail > delivery-attempt log > replay == failure counts only` - exposing request and response bodies to a third party triggers a data-classification and redaction review, and it is close to irreversible: once integrators build tooling against a field set you cannot quietly withdraw it.
- efficiency: `delivery-attempt log > request log with per-request detail > failure counts only > replay`

**Default rung: expose the delivery-attempt log you already write**, when the platform sends webhooks. The delivery machinery records every attempt and its failure reason regardless. Exposing it is mostly an access-control and interface job, and it retires the single most support-expensive question in webhook platforms ("your webhook never arrived").

Where there are no webhooks, the request log is the default instead.

**Move up to the full request log** - per call: timestamp, endpoint, status code, error code, request ID, and as much of the payload as question 7 permits - as soon as the audit shows synchronous API failures in the ticket pile. This is the rung that covers every integrator segment rather than only the webhook consumers.

**Failure counts alone** are the floor rung: a number per integration with no detail behind it. It is cheap, and worth shipping the same week if nothing exists today. But understand what it buys: it tells a partner something is wrong and gives them no way to act, which converts a silent failure into a support ticket rather than into a fix.

**The starved option: replay.** Letting an integrator re-send a failed event or re-run a request recovers value no other rung measures - the deflection axis above does not count it. But it carries the highest effort on the menu: idempotency guarantees, a bounded window, and a clear relationship to your own retry schedule. It loses every efficiency round as a result.

Promote it when the audit shows integrators regularly losing events during their own downtime, which is exactly when a dashboard that only explains the loss is cold comfort.

Design it as independent of automatic retry. On Stripe, a failed event can be manually resent for up to 15 days after creation, and doing so does not cancel the platform's own retry schedule (sourced: Stripe docs). So a partner's manual replay never fights the platform's machinery.

This ranking is a default, not a law. Re-rank it against what you already know:

- A platform with no webhooks has no delivery log to expose, and starts at the request log.
- A team already storing full request/response bodies for its own debugging gets the request-log rung for the cost of an access-scoped view.
- A "no bodies leave our boundary" answer to question 7 deletes payload detail from the menu, leaving metadata-only logging as the ceiling rather than a compromise.

See [references/visibility-surfaces.md](references/visibility-surfaces.md) for the three-surface taxonomy expanded, the fields a log row has to carry, a good/bad log-row pair, replay semantics, and the tenant-isolation and redaction rules.

## 3. Make the correlation ID reach support

Two identifiers, kept separate in your vocabulary:

- **Request ID** - identifies one HTTP call and its response.
- **Correlation ID** - identifies a whole multi-call transaction that may span many request IDs.

Most platforms need both and ship only the first.

The convention is `X-Request-Id` carrying an opaque random value - a UUIDv4 by default, since it is collision-safe under load and leaks nothing, unlike a sequential or timestamp-derived identifier. Named platform variants exist (GitHub's `request-id`, Vercel's `X-Vercel-Id`, Cloudflare's `Cf-Ray`) and are citations, not requirements. The W3C `traceparent` header is the standards-track alternative that carries a span hierarchy and sampling flag instead of a bare string.

- value (a support ticket resolved in one lookup): `searchable in the developer-facing log > echoed in the response header > present in the error body > propagated to sub-processors`
- effort: `propagated to sub-processors > searchable in the developer-facing log > echoed in the response header == present in the error body` - that tie is genuine rather than an evasion: both write an identifier you already hold into the response you are already serialising, in the same code path, with no new storage behind either.
- efficiency: `echoed in the response header > present in the error body > searchable in the developer-facing log > propagated to sub-processors`

**Default rung: echo the ID in the response headers of every response**, success and failure alike. An integrator cannot paste an identifier into a support ticket if it was never returned to them, and this rung is usually a middleware change measured in hours. Success responses matter too - plenty of tickets are about a call that returned 200 and did the wrong thing.

**Promote to searchable-in-the-log the moment step 2 ships.** These two rungs are the pair that actually closes the loop: the header hands the integrator a string, the log lets them resolve it themselves before they ever write to you. Shipping the header alone produces a distinctive failure - integrators dutifully quoting an ID nobody outside your platform can look up.

The error body rung exists but belongs to a sibling: `samber/developer-platform-skills@api-error-design` owns the shape of the error payload, including whether the ID rides in it. Coordinate rather than redesign it here.

**The starved option: propagation to sub-processors.** Forwarding the same identifier onward when your platform calls a payment processor, carrier or other downstream provider, and keeping it on retries, is high effort and invisible until it matters. Promote it when your support answer today is "that failure came from our provider, we cannot see it" - at that point one lookup spanning the whole chain is the difference between a resolved ticket and a handoff.

This ranking is a default, not a law. Re-rank it against your stack:

- A platform already emitting W3C trace context internally gets propagation nearly free, which flips the effort line.
- A platform whose failures are overwhelmingly its own code gains little from propagation regardless of cost.

See [references/correlation-id-conventions.md](references/correlation-id-conventions.md) for header conventions, request-versus-correlation-ID modelling, propagation rules, the support-lookup workflow, and a negative example.

## 4. Aggregate error rates per integration identity

A platform-wide error rate can look healthy while one specific integration is failing 100% of its calls. Per-identity aggregation is what makes that visible, and the architecture is consistent across API-monitoring practitioner sources (sourced: Zuplo, Last9, dotcom-monitor, Odown):

1. **Tag at ingestion, not after the fact** - API key, app or client ID, attached to the request as it arrives. Retrofitting the tag later means your history is unqueryable for exactly the period you want to investigate.
2. **Split by status-code family rather than one blended rate** - 4xx (the integrator's payloads or auth), 5xx (yours), and 429 isolated on its own because rate-limit rejection is neither party's bug and has a different remediation. Blending them produces a number nobody can act on: you cannot tell a partner "your error rate is 8%" without saying whose fault the 8% is.
3. **Compute a rolling rate per identity**, and compare each integration against its own recent baseline rather than a global threshold - integrations differ so widely in traffic shape that one platform-wide percentage is either noisy for the small ones or blind to the large ones. Moesif's API monitoring documents both halves of this: status-code distribution recorded per endpoint and per customer in real time, and a "Dynamic Alert" that builds a historical baseline from a few days of data before alerting on deviation from it (sourced: Moesif docs).
4. **Correlate against your own deploy timeline** before concluding anything. A spike that lines up with your last release is your regression, and telling a partner otherwise is a trust cost you pay once. This is a named, shipped practice rather than a hypothetical - Datadog's Deployment Tracking overlays deploy and version markers on error-rate graphs specifically to answer whether a new release introduced the errors (sourced: Datadog docs).
5. **Read distribution, not just volume** - errors clustered around one time or one action pattern point to a logic bug or a bulk operation. Errors spread evenly point at infrastructure instead. This changes both who you notify and what you say.

429 specifically hands off to `samber/developer-platform-skills@api-rate-limit-policy`: the surfacing is yours, the quota design and headers are that skill's.

See [references/alerting-and-notification.md](references/alerting-and-notification.md) for the aggregation schema, the status-code bucket definitions, and burn-rate detection windows.

## 5. Decide the notification posture

Aggregation without notification only helps the integrators who think to go look - which excludes the entire fleet segment from the audience split. But push notification is where this design most easily destroys itself: an alert stream that fires on every failure gets muted, and a muted channel delivers nothing at all.

- value (partner learns before their own customers do): `threshold alert with cooldown > digest > immediate alert on state changes > pull-only dashboard`
- effort: `threshold alert with cooldown > digest > immediate alert on state changes > pull-only dashboard`
- efficiency: `digest > threshold alert with cooldown > immediate alert on state changes > pull-only dashboard`

**Default rung: a periodic digest per integration** - a scheduled summary of failures by class, with links into the log from step 2. It needs an aggregation window and a scheduler, nothing more, and it is the rung that respects the partner's attention by default rather than by tuning.

**The starved rung is threshold alerting with a cooldown** - top of the value line, top of the effort line, and beaten on efficiency by the digest every round. Promote it when question 6 says the partner loses money within hours. The concrete mechanic to copy comes from Salesforce's Agentforce health monitoring (sourced: Salesforce): notify immediately on an error-rate spike or latency creep, then hold a cooldown - 30 minutes by default there - before the same integration can alert again.

The cooldown is what makes immediacy survivable. Without it, the same incident alerts continuously until someone mutes the channel permanently.

**Immediate alerts on integration state changes** - credentials revoked, integration disabled, subscription expired - are a separate, narrow rung that coexists with the others. These are single discrete events, not a stream, and they are the class where an hour of digest delay is genuinely too slow.

Per-failure immediate alerting on error _volume_ is not on this menu. It is the alert-fatigue failure mode itself rather than a rung, and parking it at the bottom of a list is how it quietly reappears as scope later.

Ship the logging first, but do not mistake that for finishing. This is a documented real-world gap rather than a hypothetical one: Shopify's partner dashboard surfaces function error counts and billable-event errors while threshold-based email alerting on those errors remains a developer request rather than a shipped capability (sourced: Shopify dev docs and developer community). Logging existing without push alerting is the normal resting state of platforms that considered the job done at step 2.

This ranking is a default, not a law. Re-rank it against the user:

- A platform that already runs an outbound notification channel to partners gets every push rung cheaper, which compresses the effort line.
- A partner base of single-account integrators who watch their own systems closely can legitimately stop at the digest.
- Question 8's effort ceiling decides how much threshold tuning you can commit to, because an alert you cannot afford to tune becomes an alert nobody reads.

Notification templates and the anti-fatigue rules are in [references/alerting-and-notification.md](references/alerting-and-notification.md).

## 6. Build the provider-side fleet view

The same aggregation from step 4, read from your own side, answers a different question: which of our integrations are failing right now, and which partner should we contact? This is the surface your partner and support teams use, and it is what makes proactive outreach possible instead of reactive.

- Rank integrations by blast radius, not by raw error count - a partner app failing across 200 installs matters more than a single account throwing more errors, and a flat leaderboard by volume inverts that.
- Separate "broken since our last deploy" from "broken since their last release". You own the first outright. The second is the outreach case.
- Include the silent-failure signal from step 1: an integration whose call volume dropped to zero shows up nowhere in an error ranking.
- A weighted per-partner health score is a real shipped pattern rather than an invention. Third-party tooling built specifically for ISVs on Salesforce computes exactly this per managed package (sourced: ISVapp product documentation). Treat the specific weights as yours to set and to defend, not as something to borrow.

On borrowing SLO framing here, be careful. The published error-budget and SLO literature is written consumer-side, for a team measuring its own dependence on somebody else's API, and it does not transfer wholesale to a provider publishing reliability numbers _for_ partners. The mirror-image framing - the platform being the trustworthy real-time source about integration health, so the partner does not have to build independent monitoring of you - is this skill's own construction, not a sourced practice.

Use it as a design intent. Do not present it to partners as an industry standard, and do not publish a per-partner reliability number until you can defend how it is computed.

See [references/fleet-view-and-health-scoring.md](references/fleet-view-and-health-scoring.md) for the fleet-view layout, health-score construction, and the SLO adaptation gap in full.

## Failure modes

Each of these is a direct audit finding:

- Logging shipped, alerting called out of scope, and the platform still learns about partner outages from end users.
- Alert fatigue: an alert per failure, no cooldown, and a partner base that has muted the channel - after which the platform is functionally back to pull-only while believing it notifies.
- A request ID echoed in headers that no external surface can resolve, so integrators quote a string nobody but your engineers can look up.
- A correlation ID dropped on downstream or retried calls, breaking the chain at exactly the hop the partner cannot see.
- One blended error rate per integration, with 4xx, 5xx and 429 mixed together, so no notification can say whose problem it is.
- A healthy global error rate hiding one integration failing every call.
- Consumer-side SRE error-budget framing lifted wholesale into a partner-facing reliability promise the platform never designed to meet.
- A dashboard scoped to the account owner, invisible to the partner engineer who actually maintains the integration.
- Log retention shorter than the time it takes a partner to notice, so the evidence expires before the investigation starts.
- Payloads exposed without a redaction pass, leaking PII, secrets, or another tenant's data into a third party's console.
- Replay that fights the platform's own retry schedule, producing duplicate side effects the partner then has to reconcile.
- An integration that stopped calling entirely and shows up in nothing, because absence produces no errors.

## Measurement

- **Failure-class coverage** - the share of failure classes from the step 1 audit that an integrator can now diagnose unaided. This is the gate: iterate until every class in your ticket pile has a surface, or a written reason it does not.
- **Request-ID resolvability** - every error response carries an identifier that resolves in a surface the integrator can reach. Binary, and cheap to test by taking one ID from a real response and looking it up as an outside user would.
- **Self-serve deflection** - re-run the step 1 classification on a fresh batch of tickets after shipping and compare. The direction is the signal. The level is not, so baseline from your own before-and-after rather than any published figure (self-set: no industry benchmark exists for this).
- **Time from failure to partner awareness** - measured from first failure to the partner acting, not to the alert being sent. Track it per notification rung, since it is the number that justifies promoting a rung.
- **Alert action rate** - the share of alerts a partner acts on. A falling rate is the alert-fatigue failure mode arriving. Treat a sustained decline as a signal to widen thresholds or lengthen the cooldown (self-set: threshold is yours to pick from your own baseline).

The first two are gates. The last three are trends to watch, never pass thresholds - none of them has a defensible industry number behind it.

## Invocation examples

- "Our partners find out their integration is broken when their customers complain - design what we should be showing them."
- "Build the developer-facing request log for our API; we already store every request internally."
- "We alert partners on every failed webhook and they've all muted us. Fix the notification design."
- "Audit what an external integrator can actually see about their own failures on our platform today."
- "Design a fleet view so our partner team knows which ISV apps are breaking across installs."

## References

- [references/visibility-surfaces.md](references/visibility-surfaces.md) - the request/event/delivery three-surface taxonomy, required log-row fields, a good/bad log-row pair, replay semantics, retention and access scoping, redaction rules.
- [references/correlation-id-conventions.md](references/correlation-id-conventions.md) - request versus correlation ID, header conventions and platform variants, echo and propagation rules, the support-lookup workflow, a negative example.
- [references/alerting-and-notification.md](references/alerting-and-notification.md) - per-identity aggregation schema, status-code buckets, threshold and cooldown mechanics, multi-window burn-rate detection, notification templates, anti-fatigue rules.
- [references/fleet-view-and-health-scoring.md](references/fleet-view-and-health-scoring.md) - provider-side fleet-view layout, blast-radius ranking, per-partner health-score construction, the consumer-side SLO adaptation gap.

See also, same collection:

- `samber/developer-platform-skills@webhook-platform-design` - the delivery semantics, retry schedule and dead-lettering underneath the delivery log. This skill is the visibility layer over that machinery.
- `samber/developer-platform-skills@developer-portal-design` - where these logs and dashboards live in the portal's information architecture and access model.
- `samber/developer-platform-skills@partner-app-onboarding` - the partner lifecycle that decides who is watching these surfaces in the first place.