v1.1.0: sync security and correctness stabilization, payload-markdown 1.6 peer, upgrade migration helper

[cooperlarson]

Summary

1.1.0 is the docs plugin's stabilization release, the counterpart of payload-markdown 1.6.0. It fixes the sync, security and CLI findings from the joint audit, and makes @valkyrianlabs/payload-markdown (now ^1.6.0) and @payloadcms/plugin-seo peer dependencies. CI now tests a real consumer install.

This release needs a database migration. The new schema is additive, but one unique index fails on duplicate nonce rows that older versions allowed. The release ships a helper for that, plus an upgrade guide: docs/reference/upgrade-1-1.md.

Upgrading (for users)

1. Install the peers. Run pnpm add @valkyrianlabs/payload-markdown@^1.6.0 @payloadcms/plugin-seo, then payload generate:importmap.
2. Migrate. Run payload migrate:create, then add one line at the top of up, then run payload migrate:

   import { prepareDocsSyncMigration } from '@valkyrianlabs/payload-markdown-docs/migrations'
   
   export async function up({ db, payload, req }: MigrateUpArgs) {
     await prepareDocsSyncMigration({ payload, req }) // removes expired + duplicate nonces
     await db.execute(sql`...generated...`)
   }

   • Dev push mode: if startup fails on keyId_nonce_idx, run the two DELETE statements in the guide, then start again.
   • MongoDB: a payload run script with the same helper.
3. Review the changed defaults:
   • plugin collections are admin-only;
   • asset changes from draft syncs apply on the next --publish sync;
   • forwarded headers are opt-in.

Schema delta:

• new tables docs_sets_repositories and docs_access_rels;
• new columns docs_sets.allow_tag_refs (existing rows get true) and docs.sync_fields_hash_at_last_sync, plus their version-table copies;
• unique index keyId_nonce_idx on docs_sync_nonces (key_id, nonce).

Upgrade validated on Postgres. I built a 1.0.5 schema from origin/main via migrations and seeded it with expired and duplicate nonces. Then I upgraded to this branch:

Upgrade path	Result
migrate:create without the helper	migrate fails on the index and rolls back cleanly
migrate:create with the helper	Migrates; removes 2 expired and 1 duplicate nonce, keeping the oldest; the existing docs set gets allow_tag_refs = true; a follow-up migrate:create --skip-empty finds no drift
Dev push mode with duplicates	Startup fails; after the documented DELETEs, it starts and finishes the schema

Security

• Access: plugin collections are admin-only by default (config.admin.user) and can be overridden. Sync runs and nonces are read-only for humans.

• Replay protection:

  • nonces are inserted first, under a unique (keyId, nonce) index;
  • concurrent duplicates are rejected;
  • nonce and jti TTLs are bounded by the skew window;
  • expired nonces are pruned.

  If the index is missing (failed Mongo index build, migration not run), a post-insert check still rejects replays and fails closed.

• Credential scope:

  • Ed25519 keys and OIDC owner records can be limited to specific docs sets (unscoped keys still work, with a warning);
  • OIDC tokens can be bound to repositories;
  • allowTagRefs is on by default, so tag-triggered publishing keeps working;
  • pull request tokens are checked against base_ref.

• Pre-auth surface:

  • the request body read is bounded;
  • source.id must be slug-shaped;
  • authentication runs before docs-set lookup;
  • unauthenticated callers get a uniform 401;
  • JWKS fetches are cached with a timeout, and an unknown kid triggers a rate-limited refetch.

• Assets:

  • a shared content-type allowlist;
  • nosniff and a CSP sandbox;
  • attachments for non-text types;
  • asset routes are confined to their prefix.

• Origins: the configured origin wins; X-Forwarded-* headers are trusted only with endpoint.trustForwardedHeaders.

Sync correctness

• Visibility: one module owns public visibility (drafts, archived records, unpublished sets) for pages, nav, sitemap, llms.txt, and CTA/hero blocks. Unpublished docs no longer leak through any of them.
• Atomic apply:
  • route claims are resolved before any write;
  • archives tombstone their routes;
  • apply runs in one transaction where the adapter supports it, and a failed sync leaves no partial state.
• Removals:
  • removals reach the published record;
  • assets: [] archives existing assets;
  • the shared TS and C++ planners skip records that are already archived.
• Admin drafts: docs-set bookkeeping no longer publishes admin drafts.
• Manual-edit protection: the field hash covers title, description, navTitle and order.
• Errors: an error.issues contract, no raw DB messages, and correct sync-run codes.

CLI (pmdocs) and TS↔C++ contract

• Shared rules: shared contract vectors (contracts/vectors) pin frontmatter, order, titles, routes, body limits, content types and UTF-8 handling to identical behavior in TS and C++. pmdocs matches the server on all 112 valid corpus files.
• Nothing dropped silently:
  • hidden files are published with a warning (--skip-hidden);
  • build, dist and .next are skipped only at the docs root, with a warning.
• New output and flags:
  • prints issues, routeCollisions and conflicts;
  • validate detects route collisions;
  • new --route-base, --asset-route-base, --publish and --existing-assets.
• Security:
  • refuses plain http except on loopback (--allow-insecure-http);
  • writes keys and output atomically with 0600/0700 permissions.
• Packaging:
  • the Homebrew formula bundles skill data;
  • doctor reports degraded when skill data is missing;
  • requires Meson ≥ 1.3.

Architecture

• Sync endpoint: split into stages (src/endpoints/sync/*: parse → authenticate → policy → plan → apply).
• Shared modules: record helpers (src/shared/records.ts), one route derivation (src/routing/docsSetRoutes.ts, 7 copies removed), one docs-set resolver.
• Performance: root llms.txt runs 10 queries instead of 2+2N.
• Packaging: core, plugin-seo, next, react and payload ^3.0.0 are peers; the new ./migrations export holds the upgrade helper.
• Fixes:
  • apps without a media collection no longer crash;
  • the llms-full link rewriter ignores code spans and fences and handles linked images.

CI and release

• Build and Test:
  • actionlint, lint, build, dev-app typecheck and unit tests;
  • the Postgres-backed suite and Playwright e2e against postgres:16 service containers;
  • native CLI build, tests and install smoke;
  • release tooling pytest and tools.release check.
• Consumer smoke: packs this plugin and payload-markdown main into a fresh Next + Payload app with pnpm, runs next build, does a signed pmdocs sync and checks the rendered output (13 checks).
• Release:
  • DeepSeek changelog provider support (org-level VL_DEEPSEEK_API_KEY / VL_AI_RELEASE_PROFILE);
  • least-privilege permissions and SHA-pinned actions;
  • pull requests run on GitHub-hosted runners only.
• Self-hosted runner: e2e installs Playwright's system packages with plain apt-get, which the runner's sudoers allows.
• Version: 1.0.5 → 1.1.0 via tools.release bump minor (VERSION, package.json, meson, debian, Homebrew).

Validation

All gates were run locally on this branch's final tree, on fresh scratch Postgres databases:

• Static checks: pnpm install --frozen-lockfile, pnpm build, pnpm lint, dev-app typecheck: pass.
• vitest unit: 716 passed, 82 DB-gated skipped.
• vitest against Postgres: 798/798, 27 files. Includes the helper and concurrency regressions.
• Playwright e2e: 2/2.
• Native CLI: meson test 13/13; pmdocs --version prints 1.1.0.
• Release tooling: pytest 346 passed; tools.release check OK.
• Workflows: actionlint clean.
• Consumer smoke with the published @valkyrianlabs/payload-markdown@1.6.0 tarball, this package's tarball and the 1.1.0 pmdocs: 13/13. That covers no peer warnings, a single core copy, the import map resolving, next build, a signed sync and rendered pages.
• Upgrade from 1.0.5: validated on Postgres as described above.

Known follow-ups (not in this PR)

• X-4: docs pages render two <h1> elements and a nested <article>. Removing one changes appearance and needs an owner decision.
• DOCS-11, doc-level: a --publish sync still publishes CMS draft edits to fields the sync does not manage (e.g. hero image).
• X-5: the llms-full link rewriter does not handle indented code blocks; this needs an AST URL hook.
• Identifier length: Payload 3.90 warns that the _docs_sets_v_version_advanced_security_allowed_workflow_refs_* index names exceed Postgres's 63-character limit. This predates the PR, and no schema drift was observed. Shortening the names with dbName would itself need a migration.
• Debian changelog: debian/changelog still has the skeleton entry text (the version header is bumped).
• ESLint 10: deferred; it is a major upgrade together with @payloadcms/eslint-config.