[stefanjudis]

Gets the Checkly plugin to the point where it can be submitted to the OpenAI plugin directory, and adds the review material the submission form asks for.

Why the previous submission was stuck

The earlier listing showed its MCP server as "Not connected" with Connect, Reconnect and Rescan all disabled, and no error anywhere in the UI. The dashboard API explained it:

"mcp_servers": [{ "key": "checkly", "can_configure": false, ... }]
"app_id": null, "associated_apps": [], "app_ids": []
"submission": { "issues": [{
  "field": "mcpServers.checkly",
  "description": "No app is attached to this MCP server. Apps are created for the first supported remote MCP in a new plugin upload."
}]}

No app was attached, so there was nothing for those buttons to act on — and apps are only attached on the first upload of a new plugin, which is why no amount of re-uploading fixed it. The package it was created from used the Codex compatibility layout with "type": "http", which isn't a transport the Agent Plugins schema knows (stdio, streamable-http, sse).

A fresh plugin created from the portable package in this PR attached an app immediately, connected over OAuth, and scanned 46 tools.

What changed

Packaging (scripts/package.ts) — the ZIP is now a portable Agent Plugins bundle: root plugin.json and mcp.json generated from the repo's compatibility manifests, with skills/, assets/, LICENSE and README.md. The compatibility manifests and .app.json are no longer shipped. The repo's own files are untouched, so Claude Code, Cursor and Codex marketplace installs are unaffected.

• mcp.json declares streamable-http, and extensions["com.openai"].auth with client.mode: "cimd", baseScopes and defaultScopes: [].
• plugin.json keeps portable identity at the root and moves interface under extensions["com.openai"].
• The package name ships as checkly again; the app-6a7f1be5… override is gone. That value was a ChatGPT app ID reused as a package name, and the dashboard reports can_edit_package_name: true with its own plugins_… identifier.

CIMD instead of a registered OAuth client — auth.checklyhq.com advertises no registration_endpoint, so DCR isn't available. A Client Identifier Metadata Document works and needs no client registration: the authorize request goes out with client_id=https://chatgpt.com/oauth/client.json and PKCE. No Auth0 work needed.

defaultScopes: [] — this is the one with teeth. ChatGPT's connection form defaults to requesting every scope the server advertises, including checkly:environment-variables:write. Without the override, the scope narrowing from c3042bd was undone at connect time and the two environment-variable tools OpenAI flagged came back. With it, the authorize request carries exactly our 14 scopes, and the tool scan shows only list-account-environment-variables and get-account-environment-variable.

Review material (.codex-plugin/plugin.json) — the submission form shows test cases read-only ("managed by your plugin ZIP"), so five positive and three negative cases live in the manifest under extensions["com.openai"].review, along with commerce and release notes. The positive prompts name checks that exist in the review demo account.

On the negative test cases

A negative case here can't rest on "the MCP server has no tool for it". Diffing the server's advertised tools against the scan shows only two are withheld from the plugin:

create-account-environment-variable
update-account-environment-variable

Everything else is exposed, and the checkly skill's generic checkly api pass-through covers most account operations besides. So the three negatives are:

1. Secrets - setting CHECKLY_API_KEY as an account environment variable. The only case grounded in a withheld tool, and it answers the October rejection about tools accepting raw secret values.
2. Actions outside monitoring - rolling back a deploy after checks start failing. This replaced "deploy the Playwright Check Suite", which was wrong: the checkly skill lists Playwright Check Suites among the things it handles, so the plugin should engage with that prompt rather than stay out.
3. Billing - upgrading the subscription and changing the payment method.

Incidentally, get-error-group-root-cause-analyses is exposed even though the package never requests checkly:error-groups:read; the server gates it on rca:read. Positive case 5 depends on that tool, so it's worth knowing the scope list doesn't need widening for it.

Not in this PR

Reviewer credentials, the demo walkthrough URL and country availability are portal-side and already filled in the dashboard. Still outstanding before submission: readOnlyHint / openWorldHint / destructiveHint with a justification on every tool, which has to come from the MCP server itself.