app-marketplace-review

{
  "skill_name": "app-marketplace-review",
  "evals": [
    {
      "id": 1,
      "prompt": "I run the platform team at Larkfield, we make field-service software and we opened an app marketplace 14 months ago. 62 listed apps, about 4,100 customer tenants with at least one app installed. Last Thursday one of our listed reporting apps started shipping customer job records to an endpoint nobody recognized. Forensics came back this morning: the publisher's account was taken over after someone phished their credentials, and the attacker pushed version 3.2.1, which reached every installed tenant inside an hour. Our CEO's answer is 'make review stricter' - she wants a mandatory third-party penetration test on every new app submission before anything gets listed. Updates go out to installed tenants automatically today, no admin action needed. I have to write up what we actually change by Friday.",
      "expected_output": "A remediation plan that separates the incident from the admission gate, refuses to attribute it to a review failure, and puts publish-channel hardening (FIDO publisher 2FA, short-lived scoped publish tokens, a 24-48h propagation hold with kill switch and security-fix expedite path, OIDC trusted publishing as the promotion) ahead of deepening initial review, while declining the blanket per-submission pen test.",
      "files": [],
      "expectations": [
        "States that the initial admission review could not have prevented this incident, because the compromise was of publisher credentials and rode the update channel",
        "Declines to treat the incident as a review-gate failure and does not assert a causal review-process change from it",
        "Prioritizes publish-channel hardening ahead of deepening the admission review",
        "Recommends phishing-resistant (FIDO) two-factor authentication on publisher accounts as a default control",
        "Recommends short-lived scoped publish tokens, citing a 7-day default and 90-day maximum lifetime",
        "Specifies a publish-to-propagation hold of 24-48 hours between a cleared update and auto-propagation to installed tenants",
        "Includes an operator kill switch usable inside the propagation-hold window",
        "Includes an expedite path through the hold for genuine security fixes",
        "Treats the propagation hold as mandatory rather than optional, because updates auto-propagate to installed tenants",
        "Names OIDC trusted publishing from CI/CD as the promotion once publisher count outgrows token hygiene",
        "States the controls must be layered rather than relying on any single one, noting that published marketplace controls have demonstrated real-world bypasses",
        "Does not adopt a mandatory third-party penetration test on every new submission as the primary remediation"
      ]
    },
    {
      "id": 2,
      "prompt": "I'm drafting the update policy for Meridian Stack's integration marketplace - 180 listed apps, roughly 140 updates a month, two reviewers. The line I have written right now is: 'Any substantial change to an app's features, purpose or functionality requires resubmission for full review; routine updates do not.' Legal signed off on it. My reviewers hate it because they spend half their week arguing with developers about what counts as substantial, and I suspect a bunch of partners have just decided nothing they ship is substantial. Tighten this into something I can publish in the partner docs.",
      "expected_output": "A rewritten re-review policy that deletes the subjective trigger and replaces it with the objective five-item trigger list, pairs the minor-update fast path with per-release automated scanning, and positions unconditional re-certification and periodic expiry as conditional promotions rather than the default.",
      "files": [],
      "expectations": [
        "Rejects 'substantial change' as a re-review trigger rather than refining its wording",
        "Gives the reason that a subjective trigger cannot be self-assessed by the developer and decays into apps that are never resubmitted",
        "Replaces it with an objective trigger list",
        "Trigger list includes a request for new OAuth scopes",
        "Trigger list includes new external endpoints",
        "Trigger list includes a change to data handling",
        "Trigger list includes a payment-model change",
        "Trigger list includes a new app type or edition",
        "States the test of an objective trigger is that a developer can self-assess it before submitting",
        "Pairs the minor-update fast path with automated scanning on every release",
        "Warns that without that scanning the fast path becomes the gap it was meant to close",
        "Recommends the objective minor-update fast path as the default rung",
        "States unconditional re-certification of every post-approval change is viable only where a deterministic automated test suite returns a report fast enough to keep it cheap, benchmarked around 24 working hours",
        "States periodic approval expiry is the only rung that catches drift in apps that never ship an update, and promotes it as an annual floor for the restricted-scope tier rather than as the main policy"
      ]
    },
    {
      "id": 3,
      "prompt": "Tessellate runs an app marketplace for HR platforms; about 40% of our listed partners are EU-based companies. Today when we reject a submission the developer gets an email saying 'your app did not meet our marketplace quality standards, please review the guidelines and resubmit.' We also run a partner office hours call every second Tuesday where developers can raise problems with review decisions, and internally we've been calling that our appeals process. Two partners escalated to my VP last month saying they have no way to contest a decision at all. What should this actually look like?",
      "expected_output": "An appeal design that strips the 'appeal' label off office hours, requires decision-independent human review with a stated turnaround and policy-citing rejections, and promotes the dashboard-integrated appeal on the strength of EU platform-to-business mediation exposure.",
      "files": [],
      "expectations": [
        "States the office hours call is an escalation channel, not an appeal, and must not be labelled one",
        "Gives the reason that an escalation channel is guidance rather than an independent re-review of the decision",
        "Requires the appeal to be reviewed by a human who did not make the original decision",
        "Requires a stated turnaround commitment on appeal decisions",
        "Requires every rejection to cite the specific policy it violated, as a prerequisite for any appeal to function",
        "Notes the current generic rejection wording leaves an appellant nothing to argue against",
        "Identifies EU platform-to-business rules as giving EU business users mediation rights over listing decisions",
        "Uses that EU exposure to promote the appeal rung above the default rather than treating it as a footnote",
        "Recommends the dashboard-integrated appeal, pre-populated with the violation context, as the rung EU exposure promotes",
        "States that a defined human-reviewed appeal is cheap to stand up: a documented channel, a decision-independence rule, and a turnaround commitment",
        "States fix-and-resubmit alone is acceptable only pre-launch at trivial volume",
        "Names appeal overturn rate as a metric to track once the process is live"
      ]
    },
    {
      "id": 4,
      "prompt": "Quick sanity check. Corvid Systems sells warehouse-management software to mid-market and enterprise logistics companies. We're standing up an app directory: partner apps can read our public product catalog and our published carrier-rate metadata through a read-only API. No customer orders, no shipment records, no personal data, no write access of any kind. Our CTO's position is that because we sell to enterprise logistics, we should run the same sort of multi-week manual security review Salesforce does before anything gets listed, otherwise procurement teams won't take the directory seriously. Realistically I have one engineer who can spend about 4 hours a week on this. How deep does this review need to go?",
      "expected_output": "A tiering answer that sets depth from the maximum data grant rather than the enterprise buyer profile, recommends a light functional and scope-focused gate for metadata-only apps with post-publication scanning retained, and writes an objective escalation trigger for the day the grant widens.",
      "files": [],
      "expectations": [
        "Ties review depth to the maximum data grant an app can hold, not to the marketplace's market, buyer size, or enterprise positioning",
        "Rejects the enterprise-B2B framing as the input that sets review depth",
        "States the consequence explicitly: a consumer-adjacent marketplace whose apps hold restricted scopes should out-review a B2B marketplace whose apps only read metadata",
        "Recommends a lighter depth - functional and scope-focused rather than a mandated multi-week security review - for the metadata-only grant",
        "States that functional-only review is the right depth for a metadata-only tier inside a larger process, not a whole-marketplace shape once apps reach business data",
        "Keeps post-publication automated scanning in place despite the lighter admission gate",
        "Defines an objective escalation trigger that moves an app up a review tier if it later requests write access or restricted scopes",
        "Requires that escalation trigger to be written so a developer can self-assess it before submitting",
        "Refuses a no-review registry as an option rather than offering it as the cheapest choice",
        "Fits the recommendation inside the stated 4-hour-per-week reviewer ceiling rather than proposing a process that exceeds it",
        "Warns that the sensitivity tiering must be settled early, because changing it late forces a rewrite of the re-review, hardening and enforcement sections built on top of it"
      ]
    },
    {
      "id": 5,
      "prompt": "Halyard Cloud. Our marketplace has 2 full-time reviewers. We take in about 40 new apps a month and 600 app updates a month, and every single one of them goes through the same full manual review. The queue is at 6 weeks and three of our largest partners have threatened to pull their apps. We bought a SAST product last year and wired it so that every finding opens a Jira ticket for the review team - it generated 1,900 tickets last quarter and the reviewers have basically stopped opening them. About half of first submissions fail on something. I have budget for one more headcount, that's it. What do I do?",
      "expected_output": "A staged-gate redesign with an objective fast path for minor updates, a triage rule that auto-tickets only critical and high findings and routes only secret-scan hits to humans, cost-tiered scan cadence, a developer-facing pre-submission checker, and the matching operating metrics.",
      "files": [],
      "expectations": [
        "Promotes a staged gate with an objective fast path, on the grounds that update volume has made reviewer cost the binding constraint",
        "Keeps full review for new apps and major versions",
        "Grants near-instant clearance only to objectively defined minor updates",
        "Keeps continuous automated scanning covering what the fast path skips",
        "Changes the ticketing rule so that only critical and high findings auto-raise a ticket",
        "Routes to human reviewers only judgment calls plus a single genuinely ambiguous signal class, naming secret-scan hits as that class",
        "Gives the reason: scanning everything and ticketing everything drowns reviewers",
        "Tiers scan cadence by cost - cheap checks daily, expensive per-version checks on release, slow checks monthly",
        "Hands developers the reviewers' own checking tool to run before submission",
        "Connects that pre-submission self-check to reducing the first-pass failure rate",
        "Treats the roughly-half-of-first-submissions-fail figure as a benchmarked floor to beat rather than as an acceptable norm",
        "Names automated-scan false-positive rate as a tracked metric, because rising false positives retrain reviewers to ignore the scanner",
        "Names share of updates clearing the fast path as a tracked metric"
      ]
    },
    {
      "id": 6,
      "prompt": "Two things for the marketplace policy doc at Northcut. First: a researcher reported a stored XSS in a listed app that has about 900 installs. The app's publisher answered once, asked for a repro, and has gone quiet for nine days. My head of security wants the policy to say 'apps with confirmed vulnerabilities are delisted immediately' and be done with it. Second, unrelated: we're about to require every listed app to ship an SBOM. We have 300 apps already listed. I want to announce it in two weeks and enforce it from the first of next month. Write me the enforcement side of the policy.",
      "expected_output": "A severity-tiered enforcement ladder with per-rung owner, notice period and firing evidence, an explicit note that this ladder is not effort-ranked, publisher-level enforcement for chronic non-responders, standing discovery channels, and a rejection of the one-month SBOM runway in favour of a six-month minimum.",
      "files": [],
      "expectations": [
        "Replaces blanket immediate delisting with a severity-tiered enforcement ladder",
        "The ladder runs: finding raised, remediation SLA by severity, listing hidden, new installs suspended, tokens revoked or kill switch, then delisting with notification to affected customers",
        "Every rung names an owner",
        "Every rung names a developer-notice period",
        "Every rung names the evidence that fires it",
        "States explicitly that this ladder carries no efficiency or effort ranking, because severity and the developer's response pick the rung rather than the operator's effort budget",
        "Adds graduated publisher-level enforcement for chronic non-responsive or low-quality submitters, distinct from rejecting or removing a single app",
        "Rejects the proposed announce-in-two-weeks, enforce-next-month runway for the SBOM requirement",
        "Sets at least six months of notice before a new requirement takes effect on already-listed apps",
        "Gives the reason: retroactive rule changes without runway burn the partner trust the review program exists to build",
        "Requires a named security contact per partner as a standing discovery channel",
        "Requires an incident-reporting channel feeding the ladder",
        "Lists vulnerability discovery through internal scanning, a disclosure program, and a bug bounty as the program matures"
      ]
    },
    {
      "id": 7,
      "prompt": "I'm reviewing a submission on Pallas, our marketplace for accounting software. The app is a spend-analytics dashboard. It's requesting ledger.read, vendors.read and ledger.write. In the submission form, under justification for ledger.write, the developer wrote: 'Required for planned auto-reconciliation features on our Q3 roadmap.' Their screenshots only show read-only charts and exports. Their backend posts to two endpoints we've never seen in a submission before, one of which resolves to a third-party data-enrichment vendor. How should I handle this one, and what should the written rule be so the next reviewer handles it the same way?",
      "expected_output": "A decision that refuses the roadmap justification on demonstrated-necessity grounds, escalates the app a tier for the write scope, extends review to post-consent data flow and subprocessors, and writes the per-scope rule plus the five reviewer verification techniques into the process.",
      "files": [],
      "expectations": [
        "Rejects the roadmap justification, on the rule that high-risk scopes require demonstrated necessity rather than declared intent",
        "States the review's job is to check the app does not ask for more than it actually uses",
        "Writes scope minimization as a named per-scope rule rather than a general least-privilege principle",
        "Treats the ledger.write request as escalating the app up one review tier",
        "Requires the tier-escalation trigger to be objective enough for a developer to self-assess it before submitting",
        "Extends the data-access review past the grant itself to where data goes after consent - external endpoints, subprocessors, and retention",
        "Treats the unrecognized third-party enrichment vendor as a subprocessor question inside that review",
        "States secrets-handling rules as prohibitions covering source repositories, URL strings, and logs",
        "Applies codebase scanning for prohibited patterns as a reviewer verification technique",
        "Applies configuration and manifest inspection, plus UI/UX verification, as reviewer verification techniques",
        "Applies OAuth, billing and install flow tracing as a reviewer verification technique",
        "Applies scope-necessity assessment against the app's declared functionality as a reviewer verification technique"
      ]
    },
    {
      "id": 8,
      "prompt": "Pre-launch question. Arbolet is a CRM for construction companies; our app marketplace opens in 11 weeks. Listed apps will get read and write access to customer CRM records - contacts, deals, job files, the lot. My head of ecosystem wants to launch open: anyone can list after a basic completeness check, and then we run an 'Arbolet Certified' badge program that the good apps can earn later. Her argument is that we need 50 apps at launch for the marketplace to look alive, and a heavy gate gets us maybe 12. We also want an enterprise trust tier at some point, because our top 20 accounts ask about app security in every single renewal conversation. Give me your recommendation and the reasoning behind it.",
      "expected_output": "A recommendation of the single hard gate for a read-write business-data grant, with the two-rung ladder named and argued down, the three separate pipeline orderings emitted with the trust/efficiency inversion explained, and the enterprise tier positioned as a later stackable layer.",
      "files": [],
      "expectations": [
        "Recommends a single hard gate as the default rung, given apps reach read-write access on business records",
        "States the two-rung ladder starves the quality floor because 'listed' clears almost no bar",
        "States the ladder is promoted only when the binding risk is developer-adoption friction, which is rare once the data grant passes metadata-only",
        "Warns that an uncertified-but-listed tier is an incident waiting to be attributed to the operator",
        "Emits reviewer effort per submission as its own ordering across the pipeline shapes",
        "Emits buyer trust signal as its own separate ordering",
        "Emits efficiency as its own separate ordering",
        "Explains that the trust and efficiency orderings invert, because a uniform hard bar gives every buyer the same guarantee at full reviewer cost while a fast path or ladder means 'listed' certifies less than 'certified'",
        "Treats the enterprise trust tier as stackable on top of whichever base gate exists, never as a replacement for one",
        "Sequences the enterprise trust tier after the base gate is proven rather than at launch",
        "Names the staged gate with an objective fast path as the later promotion once update volume makes reviewer cost the binding constraint, not as the launch shape",
        "States which input moved which rung, specifically that the read-write grant on business records is what holds the hard gate"
      ]
    },
    {
      "id": 9,
      "prompt": "Here's the summary of the review policy doc we're about to ship at Ferrule. B2B procurement SaaS, marketplace with 210 listed apps, and updates auto-install into customer tenants overnight. Section 1, Admission: every app gets a completeness check, a scope review and a security questionnaire; apps touching purchase-order data go to 'enhanced review' at the reviewer's discretion. Section 2, Updates: minor updates publish without re-review, major updates require re-review. Section 3, Publish: developers push builds from the partner portal using their username and password plus an SMS code, and builds go live to tenants that night. Section 4, Enforcement: the trust and safety team may hide, suspend or remove any app that violates our policies. Section 5, Curation: we also need to settle in this doc whether Ferrule should be an open marketplace or a curated one. Is this ready to ship?",
      "expected_output": "A gate-by-gate verdict against the four design-document gates, failing sections 1-4 with the specific missing objective criteria, owners, notice periods and publish-channel controls, plus a ruling that the curation question belongs upstream in marketplace strategy.",
      "files": [],
      "expectations": [
        "Checks the document against a fixed set of four design-document gates rather than giving general impressionistic feedback",
        "Fails Section 1 because 'enhanced review at the reviewer's discretion' is not an objective escalation trigger a developer can self-assess",
        "Fails Section 2 because 'minor update' is left undefined",
        "Supplies the objective minor-update definition: no new scopes, no new external endpoints, no change to data handling",
        "Fails Section 4 because no enforcement rung names an owner, a notice period, or the evidence that fires it",
        "Fails Section 3 because the publish channel carries none of the default controls",
        "Requires a publish-to-propagation hold specifically because updates auto-install into tenants overnight",
        "Flags the SMS second factor as not phishing-resistant and replaces it with FIDO-based 2FA",
        "Replaces the long-lived portal password publish path with short-lived scoped publish tokens",
        "States the open-versus-curated decision in Section 5 belongs to marketplace strategy, and that this review process enforces that decision rather than re-making it",
        "Instructs iterating on the document until all four gates pass rather than shipping with known gaps",
        "Lists operating metrics for once it is live, including time-to-first-decision against a published SLA and post-approval security findings per quarter with time-to-remediation"
      ]
    }
  ],
  "trigger_queries": [
    {
      "query": "Design the security review process for third-party apps on the marketplace we're launching next quarter",
      "should_trigger": true
    },
    {
      "query": "what should our approval criteria be for apps submitted to our marketplace",
      "should_trigger": true
    },
    {
      "query": "how do we decide which partner apps actually get into our app store",
      "should_trigger": true
    },
    {
      "query": "our app review takes six weeks and partners are furious, where can we add a fast path",
      "should_trigger": true
    },
    {
      "query": "when should an app update have to go through review again",
      "should_trigger": true
    },
    {
      "query": "write the policy for what counts as a minor update that skips re-approval",
      "should_trigger": true
    },
    {
      "query": "a listed integration just leaked customer data through an auto-update, what do we change",
      "should_trigger": true
    },
    {
      "query": "how do we audit the OAuth scopes a submitted app is asking for before we list it",
      "should_trigger": true
    },
    {
      "query": "a partner wants read-write on all CRM records and says it's for a feature on their roadmap",
      "should_trigger": true
    },
    {
      "query": "do we need an appeals process when we reject a partner's app",
      "should_trigger": true
    },
    {
      "query": "developers keep complaining that our rejections are opaque and they can't contest them",
      "should_trigger": true
    },
    {
      "query": "how do we delist an app that turned out to be malicious",
      "should_trigger": true
    },
    {
      "query": "we need a written policy for pulling an app out of the marketplace",
      "should_trigger": true
    },
    {
      "query": "what stops a compromised publisher from pushing a bad version to all our customers",
      "should_trigger": true
    },
    {
      "query": "should we hold cleared updates for a day before they reach installed tenants",
      "should_trigger": true
    },
    {
      "query": "our scanner opens a ticket for every finding and the reviewers have stopped reading them",
      "should_trigger": true
    },
    {
      "query": "how many reviewers do we need for 600 app updates a month",
      "should_trigger": true
    },
    {
      "query": "design the vetting process for integrations before they go into our directory",
      "should_trigger": true
    },
    {
      "query": "what security checks should we run on a third-party app before approving it",
      "should_trigger": true
    },
    {
      "query": "we want a certified tier badge sitting on top of a basic listing bar, thoughts",
      "should_trigger": true
    },
    {
      "query": "should we require every submitted app to pass a penetration test",
      "should_trigger": true
    },
    {
      "query": "our marketplace goes live in 11 weeks and we have no process for deciding what gets in",
      "should_trigger": true
    },
    {
      "query": "someone impersonated one of our partners and uploaded a fake build of their app",
      "should_trigger": true
    },
    {
      "query": "how do we handle a publisher who keeps submitting sloppy apps and ignoring our feedback",
      "should_trigger": true
    },
    {
      "query": "write the enforcement ladder for apps that fail a post-approval security check",
      "should_trigger": true
    },
    {
      "query": "what KPIs tell us whether our marketplace vetting is actually working",
      "should_trigger": true
    },
    {
      "query": "how do I tier how strictly we check apps based on what data they can touch",
      "should_trigger": true
    },
    {
      "query": "we're adding an SBOM requirement for all listed apps, how much notice do partners get",
      "should_trigger": true
    },
    {
      "query": "our EU partners are claiming they have mediation rights over our rejection decisions",
      "should_trigger": true
    },
    {
      "query": "can we just let apps publish updates instantly the way npm does",
      "should_trigger": true
    },
    {
      "query": "what keeps a stolen publish token from turning into a supply chain attack on our customers",
      "should_trigger": true
    },
    {
      "query": "how should we split automated scanning versus human reviewers on submissions",
      "should_trigger": true
    },
    {
      "query": "how do we know an app isn't asking for more permissions than it actually uses",
      "should_trigger": true
    },
    {
      "query": "we approved an app two years ago and nobody has looked at it since",
      "should_trigger": true
    },
    {
      "query": "should approvals expire?",
      "should_trigger": true
    },
    {
      "query": "we need to decide what re-triggers a full re-check once an app is already live",
      "should_trigger": true
    },
    {
      "query": "the exec team wants stricter gates after the breach but I don't think the gate was the problem",
      "should_trigger": true
    },
    {
      "query": "design the intake and approval workflow for our integrations directory",
      "should_trigger": true
    },
    {
      "query": "what should a rejection email to a partner developer actually say",
      "should_trigger": true
    },
    {
      "query": "our reviewers are the bottleneck, how do we scale without lowering the bar",
      "should_trigger": true
    },
    {
      "query": "how should we handle apps that request restricted mailbox and file scopes",
      "should_trigger": true
    },
    {
      "query": "give partners a way to check their own app before they submit it to us",
      "should_trigger": true
    },
    {
      "query": "what happens to a listed app when its publisher's account gets taken over",
      "should_trigger": true
    },
    {
      "query": "we want to be able to kill an installed app remotely if it goes bad",
      "should_trigger": true
    },
    {
      "query": "how do we decide when to revoke an app's tokens versus just hiding the listing",
      "should_trigger": true
    },
    {
      "query": "our app store has basically no quality bar and it's starting to hurt us",
      "should_trigger": true
    },
    {
      "query": "set up the gate that decides what gets into our integration catalog",
      "should_trigger": true
    },
    {
      "query": "should a change to an app's pricing model force a new review",
      "should_trigger": true
    },
    {
      "query": "how strict does our vetting need to be if apps only ever see public metadata",
      "should_trigger": true
    },
    {
      "query": "write the onboarding flow for new partner developers signing up to build on our platform",
      "should_trigger": false
    },
    {
      "query": "how do we provision sandbox tenants for partner developers before their first build",
      "should_trigger": false
    },
    {
      "query": "what screenshot dimensions and video specs should we require on marketplace listings",
      "should_trigger": false
    },
    {
      "query": "design the category taxonomy for our app directory",
      "should_trigger": false
    },
    {
      "query": "write the quality bar for listing descriptions and marketing copy in our app store",
      "should_trigger": false
    },
    {
      "query": "what revenue share should we take on paid apps in our marketplace",
      "should_trigger": false
    },
    {
      "query": "how and when do we pay out developers for marketplace sales",
      "should_trigger": false
    },
    {
      "query": "should we be merchant of record for marketplace transactions or let developers bill directly",
      "should_trigger": false
    },
    {
      "query": "pick the ten launch partners for our marketplace announcement and set the embargo",
      "should_trigger": false
    },
    {
      "query": "how should we run an app-of-the-month spotlight program",
      "should_trigger": false
    },
    {
      "query": "should we build our own marketplace or just list our integrations on HubSpot's",
      "should_trigger": false
    },
    {
      "query": "is our ecosystem mature enough to justify having a marketplace at all",
      "should_trigger": false
    },
    {
      "query": "how do we get more installs on our AppExchange listing",
      "should_trigger": false
    },
    {
      "query": "our app ranks badly in Atlassian Marketplace search results, fix our listing",
      "should_trigger": false
    },
    {
      "query": "design the consent screen third-party apps see when requesting access to a customer account",
      "should_trigger": false
    },
    {
      "query": "what scope granularity should our OAuth authorization server expose",
      "should_trigger": false
    },
    {
      "query": "should our refresh tokens rotate on every use",
      "should_trigger": false
    },
    {
      "query": "design our API key prefix and checksum format",
      "should_trigger": false
    },
    {
      "query": "how do we rotate customer API keys without downtime",
      "should_trigger": false
    },
    {
      "query": "build a test mode with magic values so integrators can exercise our API safely",
      "should_trigger": false
    },
    {
      "query": "how should integrators reset and reseed their sandbox data",
      "should_trigger": false
    },
    {
      "query": "which languages should get official SDKs first",
      "should_trigger": false
    },
    {
      "query": "write our deprecation notice window for breaking API changes",
      "should_trigger": false
    },
    {
      "query": "how should we structure our 429 responses and Retry-After headers",
      "should_trigger": false
    },
    {
      "query": "design the component model for our public status page during incidents",
      "should_trigger": false
    },
    {
      "query": "give partners a dashboard showing their failed webhook deliveries",
      "should_trigger": false
    },
    {
      "query": "design the retry, backoff and dead-letter policy for our outbound webhooks",
      "should_trigger": false
    },
    {
      "query": "review our REST endpoint naming and pagination style for consistency",
      "should_trigger": false
    },
    {
      "query": "audit our OpenAPI reference for undocumented endpoints and error codes",
      "should_trigger": false
    },
    {
      "query": "what tools should our MCP server expose to AI agents",
      "should_trigger": false
    },
    {
      "query": "should we publish a GraphQL API alongside our REST one",
      "should_trigger": false
    },
    {
      "query": "set query depth and complexity limits on our public GraphQL schema",
      "should_trigger": false
    },
    {
      "query": "should our public gRPC surface be transcoded to JSON at the edge",
      "should_trigger": false
    },
    {
      "query": "design Parquet file drops to S3 for our enterprise data export product",
      "should_trigger": false
    },
    {
      "query": "price a JDBC access add-on for customers connecting Tableau to their data",
      "should_trigger": false
    },
    {
      "query": "plan our listing as a source connector on Fivetran and Airbyte",
      "should_trigger": false
    },
    {
      "query": "which technology partners should we integrate with next quarter",
      "should_trigger": false
    },
    {
      "query": "how do we attribute partner-sourced versus partner-influenced pipeline",
      "should_trigger": false
    },
    {
      "query": "redesign the information architecture of our developer portal",
      "should_trigger": false
    },
    {
      "query": "how do we cut time to first API call for newly signed-up developers",
      "should_trigger": false
    },
    {
      "query": "how do I hire an API product manager for our platform team",
      "should_trigger": false
    },
    {
      "query": "prep me for a partner engineer interview next week",
      "should_trigger": false
    },
    {
      "query": "which platform skill should I start with for our new public API program",
      "should_trigger": false
    },
    {
      "query": "review this pull request for security vulnerabilities before I merge it",
      "should_trigger": false
    },
    {
      "query": "do a code review of our authentication middleware",
      "should_trigger": false
    },
    {
      "query": "our iOS app got rejected by Apple's App Store review team, help me write the response",
      "should_trigger": false
    },
    {
      "query": "improve our Google Play store listing conversion rate",
      "should_trigger": false
    },
    {
      "query": "write the vendor security questionnaire we send to SaaS tools before we buy them",
      "should_trigger": false
    },
    {
      "query": "audit our npm dependencies for known vulnerabilities",
      "should_trigger": false
    }
  ]
}

# Platform review benchmarks

Eight marketplace operators' documented review processes from their published docs and security guides. Every figure carries its source. Where a platform discloses nothing, that non-disclosure is stated rather than filled in.

## Per-platform pipelines

**Salesforce AppExchange / AgentExchange**, the single-hard-gate archetype (Salesforce developer docs, review policy effective 2023-08-09):

- **Stages** (five sequential, published duration bands):
  1. Submission.
  2. Verification (1-2 weeks) - the Security Review Operations team checks materials.
  3. Initial testing (3-4 weeks) - the Product Security team's full manual pass.
  4. Results notification.
  5. Resubmission testing on denial (2-3 weeks).
- **Fee:** $999 per review (Salesforce partner docs).
- **SAST:** mandatory and named - Checkmarx (branded "Source Code Scanner"), 3 free scans per review, alongside Salesforce's own in-house Code Analyzer the developer must run pre-submission. Its report is "useful but not sufficient" per the docs.
- **DAST:** Salesforce retired its own Chimera scanner on June 16, 2025. Partners now bring OWASP ZAP, Burp Suite, or Qualys for external endpoints.
- **Test surface:** the OWASP Top 10 plus Salesforce-specific surfaces (CRUD/FLS enforcement, SOQL injection, sharing violations).
- **Status tracking:** a color-coded Security Review Wizard in the Partner Console.
- **Re-review:** approvals expire, forcing periodic re-review with zero code change.
- **First-pass failure rate:** "Salesforce estimates 50 percent of applications fail the first time" (ISV consultancy Aquiva Labs, relaying Salesforce's own estimate) - re-verified as still current and now converged on independently by multiple other ISV/consulting sources (Magicfuse, Digital Flask, Concret.io, Appnigma), though none point to an official Salesforce-published statistic either.

**Atlassian Marketplace**, the staged-gate-with-fast-path archetype (Atlassian app-approval guidelines and security-workflow docs):

- **Four mandatory security parts:** a security questionnaire (manually reviewed - specific answers can fail the app), one-time partner KYC/KYB via a third-party vendor (2-3 business days, informational only), automated vulnerability scanning that hard-blocks on any critical or high finding, and Privacy & Security tab disclosures.
- **Timelines by submission type:**
  - New app: 5-10 business days.
  - Major version: 5-10 days.
  - Minor Cloud update: instant, security checks skipped.
  - Minor Data Center update: minutes-to-hours (automated scan only).
- **Base functional/quality review:** 10-15 business days, run separately.
- **Annual review:** every listed app, in the anniversary month of first approval, regardless of updates.
- **Ecoscanner** (run by the named Ecosystem Security Team, the most transparent operator scanner documented anywhere):
  - Named per-framework scanners: CSRT for Connect, FSRT for Forge, CVS.
  - TruffleHog-based secret scanning, Endor Labs dependency scanning.
  - Cadence tiered by cost: Connect checks daily, Forge checks per version release, the long-running XSS check monthly.
  - Critical/high findings auto-raise a ticket; fixes auto-close within 24 hours.
  - Only secret-scan hits route to human validation, to control false positives (Atlassian Ecoscanner docs).

**Shopify App Store**, mandatory human review with non-gating automation (shopify.dev app-store-review requirements):

- **Automated pre-submission checks** (expanded April 30, 2025: auth-after-install, redirect, uninstall/reinstall), plus a developer-run AI self-review tool (launched April 20, 2026) that Shopify states "doesn't replace human review". A human reviewer installs and tests every app regardless.
- **SLA:** none fixed. Multi-week, multi-round reviews are common per 2025-2026 developer-forum reports.
- **Scope minimization:** numbered requirements operationalize it per named scope (e.g. `read_all_orders` must justify needing order history beyond the default 60-day window).
- **Billing:** off-platform billing is an outright disqualifier.
- **Five reviewer verification techniques:** codebase scanning for prohibited patterns, configuration/manifest inspection, UI/UX verification, OAuth-billing-install flow tracing, and scope-necessity assessment against declared functionality.
- **Partner-level suspension** (distinct from app rejection): triggers on 2+ unaddressed review exchanges, repeat submissions with growing issue counts, or non-response.

**Slack App Directory**, the functional-only archetype (Slack's review guide):

- **Two phases:** automated feedback cleared during submission, then manual review where the App Directory team installs and tests the app. The checklist has historically run ~36 items.
- **Security-adjacent check:** OAuth scope review - "your app only uses scopes that it needs to work" - with legacy umbrella scopes ineligible for listing at all.
- **SLA:** none fixed ("we will not be able to skip or accelerate the review"). No mandated SAST/DAST/pen test.

**Microsoft Teams / AppSource**, unconditional re-certification made viable by speed (Microsoft Learn):

- **Testing:** "More than 400 tests" run against every app by a named App Validation team ("concierge validators") across desktop, web, and mobile. A self-service validation tool replays the same test cases pre-submission.
- **Timeline:** detailed test report within 24 working hours; clean apps publish in 1-2 business days.
- **Gating:** Publisher Verification plus Publisher Attestation (a self-assessment, not an audit).
- **Optional tier:** Microsoft 365 Certification adds an elevated compliance tier.

**Google Workspace Marketplace**, dual independent gates (Google Workspace developer docs):

- **Base gate:** a manual Marketplace listing review sits on top of Google's OAuth app verification.
- **Restricted scopes** (Gmail, Drive): additionally pass a CASA security audit built on OWASP ASVS and run by App Defense Alliance labs (named: TAC Security, Leviathan, DEKRA), not by Google.
- **CASA tiers:**
  - Tier 1: self-assessment.
  - Tier 2: independent lab DAST plus questionnaire (required for restricted scopes - Google removed the self-scan option at this tier).
  - Tier 3: full manual pen test (required only for the "independent security verification" badge).
- **Cost:** roughly $500-$6,000 per assessment depending on lab and turnaround speed, with annual revalidation regardless of change (App Defense Alliance / Google docs). Re-verified: TAC Security (Google's named preferred partner) prices Tier 2 at $540-$1,800 and Tier 3 at $4,500; Leviathan Security (an ADA founding member) prices Tier 2 at $3,000-$6,000 depending on how fast the assessment must start, with Tier 3 quoted case-by-case. The App Defense Alliance is transitioning its Tier 1-3 naming to Assurance Levels (AL1/AL2) under Linux Foundation governance; Google may still map apps to the legacy tier names.

**Chrome Web Store**, per-update review, ungated propagation:

- Every update must pass review with all code included, but there is no publish-to-install hold - a cleared update auto-propagates immediately (see the incidents file).
- Chrome is also the strongest benchmarked appeal path (see the re-review and appeals file).

**npm**: no pre-publication review of any kind - a registry, not a curated store. Included as the boundary case: its 2025 hardening was authentication-layer, because there was no review gate to strengthen (see the incidents file).

## Reviewer team design, cross-cutting

Every operator that discloses structure names at least two functions - intake/triage and technical testing (Salesforce: Security Review Operations + Product Security; Atlassian: Ecosystem Security Team; Microsoft: App Validation team). No marketplace operator discloses review-team headcount, and none states whether its reviewers are in-house or outsourced - treat headcount or outsourcing claims found elsewhere as unverified. One rare disclosed operational metric: Microsoft's VS Marketplace reported 136 extensions reviewed, 110 removed in one reporting period (Microsoft marketplace security blog).

The automation-pre-screen → human-triage pattern is confirmed at three operators: Atlassian (Ecoscanner → tickets), Microsoft VS Marketplace (multi-engine AV + sandbox → security engineers), and Microsoft Teams (validation tool → concierge validators).

- Shopify keeps automation and AI self-review explicitly non-gating, with mandatory human review.
- Salesforce is the outlier, pairing automated scanning with full manual re-testing of every submission.

## Named scanning vendors documentably inside review pipelines

Confirmed vendor-to-marketplace mapping:

- Checkmarx → Salesforce (mandatory SAST).
- TruffleHog + Endor Labs → Atlassian Ecoscanner.
- Bugcrowd → Atlassian's marketplace bug bounty.
- CASA labs (TAC Security, Leviathan, DEKRA) → Google Workspace.
- OWASP ZAP / Burp Suite / Qualys → accepted DAST for Salesforce.
- Multi-engine AV → Microsoft VS Marketplace.

Slack and Shopify disclose no third-party scanning vendor.

Boundary to keep: SSPM/OAuth-governance vendors (AppOmni, Obsidian Security, Nightfall AI) score and police apps on the customer-tenant side - an org's own security team watching what is connected to its instance. They are not operator-side admission gates. Do not conflate the two when designing who scans what.

# Re-review triggers and appeal tiers across platforms

## What re-triggers review after approval

Re-review-on-update is the dimension where the eight benchmarked marketplaces diverge most sharply - and the divergence maps exactly onto which platforms the 2024-2026 supply-chain incidents hit (see the update-channel incidents file).

- **Atlassian**, the most explicit objective fast-path (Atlassian approval guidelines):
  - Routine version updates skip re-approval entirely.
  - Full re-review re-triggers only on a payment-model change (free→paid, or vendor-collected→Atlassian-collected billing) or a new app type/edition, regardless of whether code changed.
  - Forge apps get automated checks on every version release anyway.
  - Connect apps are scanned daily regardless of releases.
  - Every listed app gets an annual anniversary-month review even with zero updates.
- **Salesforce** - new package versions generally require re-review, and approvals themselves **expire**, forcing periodic re-review with no code change. Salesforce documents no lightweight patch fast-path. Treat third-party claims of auto-approved updates after a first pass as unverified.
- **Slack** - re-review only for "substantial changes or updates to the features, purpose or functionality" (Slack's own wording). "Substantial" is subjective and undefined: a developer cannot self-assess it, which is why this skill deletes subjective triggers as a design option.
- **Microsoft Teams / AppSource** - unconditional: "If you make changes after your submission is certified, it must go through the certification process again" (Microsoft Learn), with no minor-update exemption. The 24-working-hour report SLA is what keeps this viable.
- **Google Workspace** - scope-based, not version-based: any change to restricted-scope usage re-triggers OAuth/CASA review, and annual CASA revalidation applies regardless of change - a restricted-scope app never reaches a steady state where re-review stops.
- **Chrome Web Store** - every update "must pass review with all code included", but with **no publish-to-install hold**: a cleared update auto-propagates immediately.
- **npm**: no review at any point. Instant self-service publish.

The pattern: platforms with an objective narrow trigger (Atlassian) or a fast unconditional re-check (Microsoft Teams) close the update-channel gap without slowing legitimate patches. The platforms with instant, ungated, auto-propagating updates (Chrome, npm) are where every documented 2024-2026 major incident actually landed.

**Objective "minor update" definition to reuse:**

- No new OAuth scopes.
- No new external endpoints.
- No change to data handling.
- No payment-model change.
- No new app type.

Anything else clears the fast path.

## Appeal tiers

Genuine appeal mechanisms - distinct from addressing reviewer feedback and resubmitting - are rare. The eight platforms split three ways:

**Real appeal:**

1. **Chrome Web Store** (strongest):
   - A dedicated Appeal button on the item detail page and in the developer dashboard, pre-populated with extension ID and violation data.
   - Rejection emails cite the violated policy.
   - Appeals are human-reviewed, with developers reporting 1-2 week turnaround.

   EU/UK business users additionally get voluntary mediation under P2B platform-to-business regulations.

2. **Shopify**: "Each appeal is reviewed by a Shopify team member, which ensures that a human reviews your case" (Shopify's own wording).
   - A defined email-based process, for account/store restrictions and terms violations.
   - A separate Partner Governance escalation path, for delistings (which developers report as opaque).

**Escalation, not an appeal** (a channel to raise concerns, not an independent re-review of the decision):

3. **Atlassian**: an escalation process for compliance concerns, and security tickets allowing direct dialogue with security engineers - framed around compliance disputes, not rejected-submission appeals.
4. **Salesforce**: a written findings report plus Partner Security Portal office hours and support cases. No independent appeal board - resolution stays iterative remediation.

**No documented formal appeal:** Slack, Microsoft Teams/AppSource (validation report + resubmission cycle only), Google Workspace Marketplace, npm (nothing to appeal - no pre-publication review exists).

**Design implication:** six of eight platforms still default to fix-and-resubmit. A marketplace that wants lower developer churn and lower regulatory exposure - the P2B angle applies to any EU-facing platform - builds a human-reviewed appeal with a stated turnaround, and graduates to the dashboard-integrated Chrome model when volume or EU exposure justifies the build.

# Update-channel incidents and hardening controls

The 2024-2026 incident record behind this skill's central claim: the update/publish channel, not initial review, is where documented marketplace damage actually happened. Read the working rule at the bottom before citing any of it.

## The one fully attributable incident-to-change chain: npm, 2025

Two chained attacks:

1. The **Josh Junon ("Qix") maintainer phish** (September 8, 2025): phishing impersonating npm support led to a 2FA reset and malicious releases of ~18 high-profile packages including chalk, debug, ansi-styles, strip-ansi, supports-color. Blast radius: a billion-plus weekly downloads; payloads targeted browser crypto wallets.
2. The **Shai-Hulud worm** (notified September 14, 2025): the first self-replicating npm worm, compromising maintainer accounts via malicious post-install scripts, stealing secrets, and auto-republishing trojanized packages. GitHub removed 500+ malicious package versions across ~195 packages, confirmed by a CISA alert September 23, 2025.

Follow-on waves: Sha1-Hulud/"2.0" (November-December 2025) and a forced token reset ("Mini Shai-Hulud", May 19, 2026).

GitHub's response is **fully attributable and concrete**, per its own blog ("Our plan for a more secure npm supply chain", September 2025):

- Mandatory 2FA for local publishing.
- Granular tokens with a **7-day default and 90-day maximum expiration** (classic-token creation disabled, migration deadline November 19, 2025).
- Expanded **Trusted Publishing** (OIDC-based, no long-lived tokens, provenance attestations).
- Deprecation of classic tokens and TOTP in favor of FIDO-based 2FA.

GitHub reported completing this overhaul in December 2025. Notably, the worm itself used secret scanning offensively before the defensive hardening landed.

## Highest incident volume: VS Code / Visual Studio Marketplace

ReversingLabs reported malicious detections on VS Code "almost quadrupled from 27 in 2024 to 105" in the first 10 months of 2025 - still the most current confirmed full-year comparison as of this check; no superseding 2026 full-year count has been published yet, though ReversingLabs' 2026 Software Supply Chain Security Report frames the marketplace as now spanning seven distinct campaign types. Named cases:

- **`prettier-vscode-plus`** (November 21, 2025): publisher "publishingsofficial" impersonated Prettier, delivering the Anivia loader → OctoRAT, discovered by Checkmarx Zero/Hunt.io. Removed within ~4 hours after ~6 downloads - the one benchmarked case that is a fresh malicious upload rather than an update-channel compromise, and its blast radius stayed in single digits.
- **GlassWorm** (2025): a self-propagating worm documented by Koi Security, spreading through Open VSX and Microsoft's marketplace by hiding payloads in invisible Unicode characters and harvesting npm/GitHub/Git/Open VSX credentials.
- **Wiz's research** (February 2025 onward): 100+ cases of secret leakage in `.vsix` packages, including tokens that could push malicious updates to a ~150,000 cumulative install base.
- **Fake-PNG trojan campaign** (active since February 2025, discovered December 2, 2025, by ReversingLabs): 19 extensions shipped a pre-packaged `node_modules` folder with a modified `path-is-absolute` or `@actions/io` dependency that auto-executes on VS Code startup and decodes an obfuscated dropper disguised as a PNG file. Reported to Microsoft and removed.
- **Name-reuse loophole** (discovered August 2025 by ReversingLabs): an unpublished extension's name stays reserved, but a _removed_ one frees its name for reuse - attackers republished malicious code under a deleted extension's former name (the "shiba" ransomware campaign). A distinct mechanism from typosquatting or update-channel compromise: it targets the marketplace's own name-lifecycle rules.

Microsoft's response is **attributable**, per its own blog ("Security and Trust in Visual Studio Marketplace", June 11, 2025), which documents the layered pipeline:

- Multi-engine AV static scan on submit.
- A rescan shortly after publish.
- Periodic marketplace-wide bulk rescans.
- Sandbox dynamic detection.
- Community "Report a concern".
- Mandatory signature verification enforced at install.
- Impersonation prevention.
- Secret detection, shipped later in blocking mode.

Caveat that belongs next to any citation of these controls: independent researchers (Wiz; Mazin Ahmed) **demonstrated bypasses of them in 2025** - the documented controls have real-world gaps, not just theoretical ones.

## The over-attribution cautionary cases: Chrome Web Store and Shopify

**Cyberhaven** (December 24-26, 2024): a phishing email tricked an employee into granting OAuth consent to a malicious app. "The same code had been injected into at least 35 extensions collectively used by roughly 2,600,000 people" (BleepingComputer). This was a **consent-phishing / publisher-compromise attack, not a review failure**, and there is **no confirmed public evidence Google changed the Chrome Web Store's review process specifically in response to it**.

Manifest V3's ban on remotely hosted code was a separate, pre-existing multi-year rollout completed ~2024-2025 that does not address stolen-publisher-credential update pushes. Attributing a CWS review change to Cyberhaven would be speculation.

The one documented, dated CWS process change, integrating appeals into the developer dashboard, is not tied to any specific malware incident either.

**Shopify's three reported incidents:**

1. The "888" third-party data leak (2024, disclosed July 2024, ~179,873 rows, app unnamed).
2. Consentik/Omegatheme's consent-banner plugin (2025-2026, carried a "Made for Shopify" badge, quietly closed after Cybernews notification).
3. Disputifier (disclosed January 2026, 200,000+ merchant records reportedly exposed).

All **third-party-app breaches, not Shopify-review failures**. Shopify has issued no official statement attributing any 2025-2026 review-process change (expanded automated checks, AI self-review, protected-customer-data access flow) to any single one of them.

## Working rule for using this record

State incidents and process changes as **two separate claims with two separate confidence levels**:

- **"Confirmed as reported"**: the breach happened, sourced to the vendor or the primary security researcher.
- **"Causal review-process change"**: claim it only when the operator's own blog or docs state the causal link, as GitHub/npm and Microsoft/VS Marketplace do. Never infer it from timing alone, as the Cyberhaven and Shopify cases show.

## The control set, in detail

1. **Phishing-resistant (FIDO) publisher 2FA** - the Qix phish defeated TOTP-style 2FA via a reset flow. FIDO is the control the npm hardening converged on.
2. **Short-lived, scoped publish tokens** - 7-day default / 90-day maximum lifetime (the shipped npm model). A stolen token expires before most attackers use it.
3. **Publish-to-propagation hold** - 24-48 hours between a cleared update and auto-propagation to installed clients, with an operator kill switch inside the window and an expedite path for genuine security fixes. Chrome and npm's lack of exactly this hold is the gap third-party supply-chain analyses (Aikido/Nx) flag. Some third-party tools impose their own default 48-hour hold as a mitigation the stores don't provide.
4. **OIDC trusted publishing from CI/CD** - no long-lived credential exists to steal. Provenance attestations tie the artifact to the build that produced it.
5. **Signed artifacts, signature enforced at install** - survives a fully compromised publish pipeline. This is the VS Marketplace model. Highest effort: requires client-side enforcement, key management, and a revocation story.

---
name: app-marketplace-review
description: Design the operator-side review and approval process for third-party apps on a B2B SaaS marketplace - review-pipeline shape scaled to data sensitivity, permission and data-access audit checkpoints, automated pre-screening with human triage, objective re-review-on-update triggers, publish-channel hardening against supply-chain attacks, appeal paths, and a severity-tiered revocation policy. Use whenever the user mentions app review, marketplace approval criteria, third-party app permission audits, re-review after an app update, rejection appeals, or delisting and revocation - even if they never say "app review". Operator side only. Do NOT use for submitter onboarding - use samber/developer-platform-skills@partner-app-onboarding instead.
license: MIT
metadata:
  author: Samuel Berthe
  version: "1.0.1"
---

# App Marketplace Review

You are a marketplace-trust advisor to the platform team that operates a B2B SaaS app marketplace. Design the operator-side process that decides which third-party apps get in, what re-opens that decision later, and what gets an app removed. The output is a review-process design document the team can staff and a later reader can falsify - not a one-time checklist, because every major documented 2024-2026 marketplace incident rode the update channel or a compromised publisher account, not a malicious first submission.

**Memory (advised):** When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

## Interview

Ask these before proposing anything. One question per message, multiple-choice where offered - each answer redirects a menu below.

1. Where is the review process today? (a) designing from scratch pre-launch (b) marketplace live, review is ad-hoc (c) formal process exists, revisiting after an incident (d) formal process exists, revisiting for scale or friction.
2. What can a listed app touch at its maximum grant? (a) public data or metadata only (b) read access to business records - CRM, files, tickets, messages (c) read-write on business records (d) regulated data - health, financial, personal data at legal-exposure level. This answer sets review depth more than any other.
3. Submission volume, now and in 12 months: new apps per month, and updates per month. Updates dominate reviewer cost at scale.
4. After approval, how does an update reach customers? (a) auto-propagates to installed tenants (b) each customer admin upgrades explicitly. Auto-propagation promotes publish-channel hardening above everything else.
5. Which curation rung did the marketplace strategy choose - open, hybrid with a certified tier, single hard gate, or partnership-gated? This process enforces that decision. It does not re-make it (see `samber/developer-platform-skills@connector-marketplace-strategy`).
6. Who reviews today, and with what tooling? Dedicated security engineers, borrowed product engineers, or nobody yet. Existing SAST/DAST/secret scanning, or none.
7. Does the marketplace serve EU-based business users? EU platform-to-business (P2B) rules give them mediation rights over listing decisions - this moves the appeal-path rung.
8. By when must the process be live - a hard date, or open-ended?
9. Is this a one-off win (unblock a launch, satisfy a named enterprise deal) or a compounding asset (a trust program that appreciates)?
10. What is the effort ceiling - reviewer hours per week, engineering time for automation, and appetite for developer friction?

Questions 8-10 exist because the menus below diverge sharply on time-to-effect, durability, and effort:

- A hard deadline (question 8) promotes rungs that ship as policy text.
- A compounding mandate (question 9) promotes automation and certification tiers.
- A low reviewer-hour ceiling (question 10) rules whole shapes out rather than demoting them.

## Data sensitivity

Review rigor tracks the data sensitivity of what an app can touch, not the marketplace's market, size, or age.

- **Reaches business or regulated data**: heaviest gates - Salesforce's mandatory multi-week security review, Google Workspace's third-party CASA audit for restricted scopes.
- **Consumer-adjacent, narrow-scope, or sandboxed** (Slack, Chrome, Shopify): lighter functional review plus post-publication scanning.

Tier review depth by each app's maximum data grant (question 2), and accept the consequence: a consumer-adjacent marketplace whose apps hold restricted scopes should out-review a B2B one whose apps only read metadata. Cross-platform comparison: [references/platform-review-benchmarks.md](references/platform-review-benchmarks.md).

## Workflow

1. Fix the design inputs: data-sensitivity tiers, volume forecast, propagation model, and the curation rung inherited from strategy.
2. Choose the review-pipeline shape (menu in step 3) and the review depth per sensitivity tier.
3. Specify the admission review: permission audit, data-access review, security testing, functional review - each marked automated or human.
4. Set the re-review-on-update policy with objective triggers.
5. Harden the publish channel.
6. Design the appeal path.
7. Write the revocation and post-approval enforcement ladder.
8. Set measurement, then check the document against the gates in Measurement.

Draft the document section by section in that order, validating each with the user before the next.

- A wrong sensitivity tiering caught in step 2: cheap to fix.
- A wrong sensitivity tiering discovered at step 7: rewrites everything between.

## 1. Review-pipeline shape

Four shapes, ranked. Efficiency here means risk caught per reviewer hour:

- reviewer effort per submission (descending): `single hard gate > staged gate with fast-path > two-rung ladder > functional-only review`
- buyer trust signal: `single hard gate > two-rung ladder > staged gate with fast-path > functional-only review`
- efficiency: `staged gate with fast-path > two-rung ladder > functional-only review > single hard gate`

The trust and efficiency orders invert because a uniform hard bar gives every buyer the same guarantee at full reviewer cost per submission, while a fast path or a ladder means "listed" certifies less than "certified" does.

- **Default rung: single hard gate** - every submission clears one full bar: completeness, compliance, security testing including scope minimization, functional review (the Salesforce and Zoom shape, per their published pipelines). Right while submission volume is small enough that reviewer capacity is not the constraint, or while question 2 lands on (c)/(d) - the common B2B SaaS case, where an uncertified-but-listed tier is an incident waiting to be attributed to you.
- **Promote to a staged gate with an objective fast-path** once update volume makes reviewer cost the binding constraint:
  - Full review for new apps and major versions.
  - Instant or near-instant clearance for objectively minor updates.
  - Continuous automated scanning covering what the fast path skips.

  Atlassian's shape (the only benchmarked marketplace that scales review cost to change risk): new apps and major versions 5-10 business days, minor cloud updates instant with security checks skipped, per Atlassian's approval guidelines.

- **Starved option: the two-rung ladder** - a minimal bar to list at all, plus an optional certified tier behind quantitative gates (HubSpot's shape). Lowest entry friction and a real trust badge, but it starves the quality floor: "listed" clears almost no bar. Promote it only when the binding risk is developer-adoption friction rather than app quality - rare when question 2 is past (a).
- **Functional-only review** (automated feedback plus a manual functional and scope check, no mandated security scan - Slack's published shape) is the right depth for a metadata-only tier inside a larger process, not a whole-marketplace shape once apps reach business data.
- **Deleted, not demoted: the no-review registry.** An operator asking for a review process has ruled it out, and the ungated publish channel that shape implies is exactly where the 2025 npm worms landed ([references/update-channel-hardening.md](references/update-channel-hardening.md)).
- **Stackable, not a rung: an elevated enterprise-trust tier** - security, reliability, and support commitments re-verified annually (the Atlassian Cloud Fortified / Microsoft 365 Certification shape) layered on top of whichever base gate exists, to unlock enterprise buyers. Add it once the base gate is proven, never instead of one.

Whatever the rung, run it as **automation-pre-screen then human-triage** - the pattern four benchmarked operators converged on independently.

- **Ticket routing:** auto-ticket only critical and high findings. Route to humans only judgment calls and the one genuinely ambiguous signal class (Atlassian routes only secret-scan hits to human validation, to control false positives, per its Ecoscanner docs).
- **Cadence:** cheap checks daily, expensive per-version checks on release, slow checks monthly.
- **Developer pre-check tool:** hand developers the reviewer's own tool before submission - Microsoft replays its validation test cases pre-submission, Salesforce mandates a self-run analyzer scan. Self-screening attacks the first-pass failure rate directly (Salesforce's ecosystem reports roughly half of first submissions fail - Salesforce's own estimate as relayed by ISV consultancy Aquiva Labs).

This ranking is a default, not a law, and so is every menu below it. Re-rank it against questions 2, 3, and 5, and against assets the team already owns:

- An existing SAST pipeline: moves automation up a rung.
- A compliance-exposed vertical: keeps the hard gate regardless of volume.

Re-rank each later menu against the answers its own section names. Say which interview answer moved which rung.

## 2. Permission and data-access audit

The substance the admission review checks, whatever the pipeline shape:

- **Scope minimization as a named, per-scope rule:** an app requests only the scopes its functionality requires, nothing more (Atlassian's stated security requirement), operationalized per named scope.
  - High-risk scopes require _demonstrated necessity_, not declared intent (Shopify's numbered requirements make `read_all_orders` justify needing history beyond the default window).
  - Scope taxonomy and the granting side: `samber/developer-platform-skills@oauth2-provider-design`.
  - This review's job: check the app doesn't ask for more than it uses.
- **Objective tier-escalation trigger:** any request for restricted scopes (mailbox, file, or CRM read-write) or write access to customer data moves the app up one review tier. Write the trigger so a developer can self-assess it before submitting.
- **Data-access review beyond the grant:** where data goes after consent - external endpoints, subprocessors, retention. Secrets-handling prohibitions stated as prohibitions: never in source repositories, URL strings, or logs (Atlassian names the anti-patterns explicitly because they were observed).
- **Five reviewer verification techniques** (Shopify's published methodology, reusable as-is):
  1. Codebase scanning for prohibited patterns.
  2. Configuration and manifest inspection.
  3. UI/UX verification.
  4. OAuth-billing-install flow tracing.
  5. Scope-necessity assessment against declared functionality.

Per-platform checklists, SLAs, and the named scanning vendors that run inside these pipelines: [references/platform-review-benchmarks.md](references/platform-review-benchmarks.md).

## 3. Re-review-on-update policy

The sharpest cross-platform divergence sits here, and it is the exploited one - decide it deliberately. Three rungs, ranked:

- effort, reviewer plus developer friction (descending): `periodic expiry re-review > unconditional re-certification > objective minor-update fast-path`
- update-channel exposure closed: `unconditional re-certification == objective fast-path with continuous scanning > periodic expiry alone`
- efficiency: `objective fast-path > unconditional re-certification > periodic expiry`

The `==` holds because both re-inspect every risky change: one by re-checking everything cheaply, the other by scoping full review to objective risk triggers while scanning covers the rest. Periodic expiry alone leaves months of unreviewed updates between cycles.

- **Default rung: objective minor-update fast-path.** Routine version updates skip re-approval. Full re-review re-triggers only on objective criteria (Atlassian's published trigger list):
  - New OAuth scopes.
  - New external endpoints.
  - Changed data handling.
  - Payment-model change.
  - New app type.

  Objective means the developer can self-assess before submitting. Pair it with automated scanning on every release, or the fast path becomes the gap.

- **Promote to unconditional re-certification** when a deterministic automated test suite already exists to keep it cheap - Microsoft Teams re-certifies every post-certification change and stays viable only because its test report lands within 24 working hours (Microsoft Learn).
- **Starved option: periodic expiry** - approvals expire and re-review fires with zero code change (Salesforce's expiring approvals; Google's annual CASA revalidation). Highest friction per unit of change reviewed, so it loses every efficiency round - but it is the only rung that catches drift in apps that never ship an update. Promote it for the restricted-scope tier only, as an annual floor under the default rung.
- **Deleted, not demoted: two shapes the constraints rule out.**
  - A _subjective_ trigger ("substantial changes", undefined) gives developers no way to self-assess and quietly decays into never-resubmitted.
  - _Instant ungated auto-propagating_ updates are the documented channel every major 2024-2026 incident used ([references/update-channel-hardening.md](references/update-channel-hardening.md)).

  Cross-platform trigger detail: [references/rereview-and-appeal-policies.md](references/rereview-and-appeal-policies.md).

## 4. Publish-channel hardening

The documented 2024-2026 incidents all rode compromised publisher credentials or an unguarded update channel: compromised Chrome extensions pushed via a phished publisher's OAuth grant, a self-replicating npm worm, an editor-extension worm spreading on harvested publisher credentials. Malicious first submissions still happen - an impersonation upload to an editor marketplace, pulled within hours at single-digit installs - but none of the damaging incidents started there.

Sourced incident-by-incident in [references/update-channel-hardening.md](references/update-channel-hardening.md), including which incidents provably changed a review process and which attributions would be speculation. Harden the channel that ships code to customers before hardening initial review further. Five controls, ranked:

- effort (descending): `signed artifacts enforced at install > OIDC trusted publishing > propagation hold > short-lived publish tokens > phishing-resistant publisher 2FA`
- efficiency: `phishing-resistant publisher 2FA > short-lived publish tokens > propagation hold > OIDC trusted publishing > signed artifacts enforced at install`

- **Default rung: phishing-resistant (FIDO) 2FA on publisher accounts, plus short-lived scoped publish tokens** (the model npm's operator shipped after its 2025 worm: 7-day default, 90-day maximum token lifetime - GitHub's own blog). These close the credential-theft vector that actually fired.
- **Propagation hold:** 24-48 hours between a cleared update and auto-propagation, with a kill switch inside the window and an expedite path for security fixes. Promoted to mandatory whenever question 4 answered auto-propagate - the absence of exactly this hold is the documented gap third-party supply-chain analyses flag in the ungated stores.
- **OIDC trusted publishing** from CI/CD replaces long-lived credentials entirely. Promote it as publisher count grows past what token hygiene can police.
- **Starved option: signed artifacts enforced at install.** Highest engineering effort across operator and clients, so it loses every efficiency round - but it is the only control that survives a fully compromised publish pipeline. Promote it when clients execute auto-updated code (extension-style runtimes), where the benchmarked operator that ships it enforces signature verification at install (Microsoft's marketplace security blog).

Layer these. Do not rely on one. Independent researchers demonstrated real bypasses of a benchmarked marketplace's published controls in 2025 (Wiz; Mazin Ahmed) - documented controls have real-world gaps.

## 5. Appeal path

An appeal path is a deliberate design choice: six of the eight benchmarked marketplaces offer nothing beyond fix-and-resubmit ([references/rereview-and-appeal-policies.md](references/rereview-and-appeal-policies.md)). Four rungs, ranked:

- effort (descending): `dashboard-integrated appeal > defined human-reviewed appeal > escalation channel > fix-and-resubmit only`
- developer churn and regulatory exposure reduced: `dashboard-integrated appeal > defined human-reviewed appeal > escalation channel > fix-and-resubmit only`
- efficiency: `defined human-reviewed appeal > escalation channel > dashboard-integrated appeal > fix-and-resubmit only`

- **Default rung: a defined, human-reviewed appeal with a stated turnaround**, explicitly distinct from resubmission - a named process where a human who did not make the original decision reviews the case (Shopify states this guarantee in writing). Cheap to stand up: a documented channel, a decision-independence rule, a turnaround commitment. Prerequisite whatever the rung: every rejection cites the specific violated policy, or the appeal has nothing to argue against.
- **Fix-and-resubmit only** is acceptable solely pre-launch at trivial volume. If chosen, write the promotion date into the document.
- **Escalation channels and office hours** (a security portal where partners discuss findings, a compliance-concern channel) are guidance, not an independent re-review of the decision - never label one an appeal.
- **Starved option: the dashboard-integrated appeal** - an appeal button on the rejection itself, pre-populated with the violation context, human-reviewed with a stated turnaround (the Chrome Web Store model; developers report 1-2 week turnarounds). Highest build effort, so it loses the efficiency round. An EU-facing marketplace (P2B mediation exposure), or appeal volume that itself needs tooling, promotes it.

## 6. Revocation and post-approval enforcement

Post-approval security is a standing program, not a one-time gate. Write four things:

1. **A severity-tiered enforcement ladder** with an owner and a developer-notice period per rung:
   1. Finding raised.
   2. Remediation SLA by severity.
   3. Listing hidden.
   4. New installs suspended.
   5. Tokens revoked / kill switch.
   6. Delisting, with notification to affected customers.

   These rungs carry no efficiency ranking on purpose: severity and the developer's response pick the rung, not the operator's effort budget. A rule with no rung behind it is a preference, not a policy.

2. **Standing discovery channels feeding the ladder:** a required security contact per partner, an incident-reporting channel, and vulnerability discovery via internal scanning, a disclosure program, and (as the program matures) a bug bounty - the most transparent benchmarked operator runs all three concurrently.
3. **Graduated publisher-level enforcement, distinct from single-app rejection:** chronic low-quality or non-responsive submitters get escalating partner-level consequences (Shopify suspends partners after repeated unaddressed review exchanges) - one bad app and a bad-faith publisher are different problems.
4. **A notice runway for new requirements** applied to already-listed apps - the benchmarked commitment is at least six months before a new requirement takes effect (Atlassian's Cloud Fortified program docs). Retroactive rule changes without runway burn the partner trust the review program exists to build.

## Failure modes

- **Guarding the front door, leaving the back door open.** A heavy initial gate with ungated updates is the exact 2024-2026 incident pattern. Fix: steps 3-4 before deepening step 1.
- **Subjective re-review trigger.** "Substantial changes" decays into never-resubmitted because nobody can self-assess it. Fix: the objective trigger list.
- **Over-attributing incidents to review.** A third-party app breach is not automatically a review failure. Claim a causal process change only when the operator itself states one ([references/update-channel-hardening.md](references/update-channel-hardening.md) carries the working rule). Redesigning the gate after an incident the gate could never have caught spends effort on the wrong control.
- **Uniform depth held past its volume.** A single hard gate, kept after reviewer cost becomes the constraint, produces multi-week queues and developer churn - the benchmarked no-SLA marketplace draws exactly those complaints in its own forums.
- **Automation without triage design.** Scanning everything and ticketing everything drowns reviewers. Auto-ticket critical/high only, and route only genuinely ambiguous signal classes to humans.
- **An appeal that is secretly resubmission.** Labeling office hours an appeal erodes trust and, for EU-facing marketplaces, leaves the P2B exposure unaddressed.

## Measurement

Design-document gates first, self-set since no industry pass-standard exists for a review-process design. Iterate until all four pass:

1. Every review tier has an objective escalation trigger a developer can self-assess.
2. "Minor update" is defined by objective criteria - no new scopes, no new external endpoints, no data-handling change.
3. Every enforcement rung names an owner, a notice period, and the evidence that fires it.
4. The publish channel carries at least the default control rung, and the propagation hold whenever updates auto-propagate.

Operating KPIs once live:

- First-pass approval rate (the benchmarked floor to beat: roughly half of first submissions fail at the strictest gate - Salesforce's estimate via Aquiva Labs).
- Time-to-first-decision against the published SLA.
- Share of updates clearing the fast path.
- Appeal overturn rate.
- Post-approval security findings per quarter, with time-to-remediation against SLA.
- Automated-scan false-positive rate (rising false positives silently retrain reviewers to ignore the scanner).

If your harness has persistent memory, record the chosen pipeline shape, the sensitivity-tier triggers, the "minor update" definition, and each menu's rung with its promotion condition - onboarding, listing, and monetization runs should inherit these decisions instead of re-asking.

## Invocation examples

- "We're launching an app marketplace for our CRM next quarter - design the security review process for third-party apps."
- "Our app review takes six weeks and partners are furious. Where can we add a fast path without reopening the update channel?"
- "An installed app just leaked customer data through an auto-update. Rewrite our re-review and revocation policy so that can't happen again."

## References

- [references/platform-review-benchmarks.md](references/platform-review-benchmarks.md) - eight-platform comparison: pipeline shapes, SLAs, review checklists, reviewer team design, named scanning vendors.
- [references/update-channel-hardening.md](references/update-channel-hardening.md) - the 2024-2026 incidents with sourced attribution (confirmed process changes versus unattributed), and the publish-channel control set in detail.
- [references/rereview-and-appeal-policies.md](references/rereview-and-appeal-policies.md) - re-review triggers and appeal tiers across platforms, with the objective-versus-subjective trigger contrast.

See also, same collection:

- `samber/developer-platform-skills@connector-marketplace-strategy` - chooses the curation rung this process enforces. Run it first when marketplace strictness is still an open question.
- `samber/developer-platform-skills@oauth2-provider-design` - owns the scope taxonomy and the granting side of consent. This skill's permission audit checks apps against that taxonomy.
- `samber/developer-platform-skills@partner-app-onboarding` - the submitter's journey through the gate designed here: docs, sandbox tenancy, certification steps.
- `samber/developer-platform-skills@app-marketplace-listing-standards` - listing-content rules and content-quality rejections. Badge eligibility is decided here, badge display there.
- `samber/developer-platform-skills@app-marketplace-monetization-model` - billing and revenue-share mechanics. A payment-model change is a re-review trigger here, its design lives there.
- `samber/developer-platform-skills@app-marketplace-launch-marketing` - the launch and promotion program that markets a cohort as vetted. The gate designed here is what makes that claim true, so a marketplace with no live review gate cannot honestly use it.