Skip to content

feat: track E2E coverage by required scenarios, not only routes - #2367

Open
aasimsyed wants to merge 19 commits into
activist-org:mainfrom
aasimsyed:feat/e2e-scenario-coverage
Open

feat: track E2E coverage by required scenarios, not only routes#2367
aasimsyed wants to merge 19 commits into
activist-org:mainfrom
aasimsyed:feat/e2e-scenario-coverage

Conversation

@aasimsyed

@aasimsyed aasimsyed commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Contributor checklist

Description

Route coverage still counts a page as done if a spec opens the URL. This adds a required-scenario catalog and scores it against current specs.

  • frontend/test-e2e/e2e-coverage/catalog/flows/: catalog of flows and cases, one file per flow (e.g. events.mjs), with ID_PREFIXES and CATEGORIES legends (EF-PERM-01 = event FAQ, permissions, first case)
  • frontend/test-e2e/e2e-coverage/{utils,parsers,catalog,scoring,report}/: coverage engine. Old import paths (scripts/e2e-coverage.mjs, scenario-matrix.mjs) stay as thin shims
  • Default report is summary, coverage by category, and missing rows (with why: no spec vs title miss). --verbose prints every row and the test() titles that counted. --routes is the old URL table. --out writes frontend/test-results/e2e-coverage-latest.md (gitignored, and now on by default)
  • Optional @coverage Playwright gate (yarn test:e2e:coverage:gate): fails a local run when required scenario coverage drops below the 90% baseline. Static analysis only, no browser needed, not wired into default e2e CI
  • Stub pages are detected from :underDevelopment or an empty template (not a hardcoded list). Does not close Remove stub route detection from e2e-coverage.mjs #1972
  • yarn test:e2e:coverage and docs in FRONTEND_TESTING.md, including the gate's known limitations (no CI enforcement, fragile title matching, manual staleness cleanup, a calibrated rather than derived 90% threshold)

From frontend/: yarn test:e2e:coverage (same as node test-e2e/e2e-coverage/scripts/e2e-coverage.mjs).

Related issue

@aasimsyed
feat: add E2E required-scenario catalog
7e2ba47 
Give coverage scoring a stable list of product behaviors, with prefix and category legends, instead of treating a route hit as full coverage.
@aasimsyed
feat: score E2E scenarios in the coverage script
8a6927e 
Match catalog cases to current specs (--full, --out, --json) and detect stub pages from page source so the hardcoded route list is not required.
@aasimsyed
chore: add yarn test:e2e:coverage
0b4cf6c 
Expose the coverage script as a frontend yarn command.
@aasimsyed
docs: document E2E scenario coverage
eaef9a9 
Describe how to run the regenerable report and how to add a catalog row.
@aasimsyed
docs: explain scenario coverage ids with examples
5610139 
Make the catalog easier to read by walking through what an id means and when a row is covered, partial, or missing.
@aasimsyed
feat: make the E2E scenario report actionable
0b6bdb1 
Show why a case is missing, coverage by category, and which test title counted so loose matches are visible.
@aasimsyed
feat: default E2E coverage to a compact scenario report
4bc5298 
Print summary, categories, and missing rows with no flags. Use --verbose for every row and --routes for the old URL table.
@netlify

netlify Bot commented Aug 26, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ Deploy Preview for activist-org ready!

Name Link
🔨 Latest commit a424d34
🔍 Latest deploy log https://app.netlify.com/projects/activist-org/deploys/6a9e3c87ef4044000865bbec
😎 Deploy Preview https://deploy-preview-2367--activist-org.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Thank you for the pull request! ❤️

The activist team will do our best to address your contribution as soon as we can. The following are some important points:

  • Those interested in developing their skills and expanding their role in the community should read the mentorship and growth section of the contribution guide
  • If you're not already a member of our public Matrix community, please consider joining!
    • We'd suggest that you use the Element client as well as Element X for a mobile app
    • Join the General and Development rooms once you're in
  • Also consider attending our bi-weekly Saturday developer syncs!
    • Details are shared in the Development room on Matrix each Wednesday before the sync
    • It would be great to meet you 😊

Note

activist uses Conventional Comments in reviews to make sure that communication is as clear as possible.

@github-actions

github-actions Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Maintainer Checklist

The following is a checklist for maintainers to make sure this process goes as well as possible. Feel free to address the points below yourself in further commits if you realize that actions are needed :)

  • Tests for changes have been written and the TypeScript, pytest and formatting workflows within the PR checks do not indicate new errors in the files changed

  • The Playwright end to end and Zap penetration tests have been ran and are passing (if necessary)

  • The changelog has been updated with a description of the changes for the upcoming release and the corresponding issue (if necessary)

@andrewtavis andrewtavis added the frontend Relates to the project frontend label Aug 26, 2026
@aasimsyed
aasimsyed marked this pull request as ready for review August 31, 2026 18:43
@nicki182

nicki182 commented Sep 3, 2026

Copy link
Copy Markdown
Member

@aasimsyed I kind of understand what you did here but maybe break it up and clean the code here so that if someone hops on can understand what is happening. Maybe create a parser folder since you are doing so with each file within a folder called coverage and maybe once cleaning and refactoring is done we can bring it in and maybe we could run it as a separate test for e2e. Right now, as it is. Having files of 1000 lines is not great and I think this could be separated made cleaner.

@aasimsyed
refactor(e2e): extract coverage engine into e2e-coverage/
fe33efc 
Split the two ~1,000-line files (scripts/e2e-coverage.mjs,
scenario-matrix.mjs) into utils/, parsers/, catalog/, scoring/, and
report/ modules. CLI flags and output are unchanged. Old import paths
stay as thin shims.
@aasimsyed
chore(e2e): remove the scenario-matrix.mjs shim
d790786 
Nothing imports it once the catalog lives in e2e-coverage/catalog/.
@aasimsyed
test(e2e): add optional @coverage Playwright gate
20f4996 
Fails a local run when required scenario coverage drops below the
baseline in scoring/constants.mjs. Not wired into default e2e CI.
Skips global-setup auth since the gate only does static analysis.
@aasimsyed
docs(e2e): document the coverage module and gate
ad80e6e 
Point catalog edits at e2e-coverage/catalog/flows/, add the folder
README, and document yarn test:e2e:coverage:gate.
@aasimsyed
fix(e2e): raise the coverage gate to 90%
846f11c 
The next two commits close the gap this uncovers.
@aasimsyed
fix(e2e): make the scenario catalog honest
9b0f1a4 
- Defer org/group PERM rows: that role-based UI is still in development
- Tighten S-INT-01 and C-VAL-01 matchers so a page-load spec cannot
  also claim behavior it does not test
- Drop H-INT-04/H-INT-05: duplicates of S-INT-01 and T-INT-03
- Defer T-INT-03: /home has no query handler wired to its topic filter
@aasimsyed
test(e2e): manage resources with one CREATE/UPDATE/DELETE path
fc0d368 
Adopt the FAQ suite's pattern for event, organization, and group
resources: one test walks create, update, and delete against the real
API, replacing separate create/edit/delete tests that a validation or
server-error spec could partially satisfy. Consolidate each surface's
three CRUD catalog rows into one that matches the new test name.

Closes the gap the 90% gate raised, from 89% to 91%.
@aasimsyed
feat(e2e): write the coverage report to test-results/ by default
2cbbec1 
yarn test:e2e:coverage now passes --out, so the markdown report is
always saved alongside stdout. Output stays under the gitignored
test-results/ directory.
@aasimsyed
docs(e2e): simplify the coverage section for first-time editors
e1ffdf6 
Lead with a concrete c(...) example, reorder subsections so adding a
scenario comes before reference tables, and note the gate's known
limitations (no CI enforcement, fragile title matching, manual
staleness cleanup, a calibrated rather than derived 90% threshold).
@aasimsyed

aasimsyed commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

Fair callout, those files had grown past the point of being readable in one sitting.

done: split scripts/e2e-coverage.mjs and scenario-matrix.mjs into e2e-coverage/{utils,parsers,catalog,scoring,report}/. Catalog rows now live under catalog/flows/, one file per page area (e.g. events.mjs for all events pages), and scoring/parsing/reporting are each their own module. The documented command path, scripts/e2e-coverage.mjs, stays as an 11-line shim so nothing else needed to change.

done: also added the optional @coverage Playwright gate you mentioned, it fails a local run if required-scenario coverage drops below the agreed baseline. It's not wired into default CI.

@nicki182

nicki182 commented Sep 4, 2026

Copy link
Copy Markdown
Member

@aasimsyed how much time it takes to run it if we do so?

@aasimsyed

aasimsyed commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

@nicki182 Ran it locally a few times, consistently ~1s wall clock, 300ms of that is the actual assertions. It's a static scan of source files, no browser or server, so that cost wouldn't grow with the size of the E2E suite the way the real Playwright job does.

@nicki182

nicki182 commented Sep 4, 2026

Copy link
Copy Markdown
Member

@andrewtavis what do you think of adding it in the ci tests?

@andrewtavis
andrewtavis self-requested a review September 4, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewtavis andrewtavis Awaiting requested review from andrewtavis

Assignees

No one assigned

Labels

frontend Relates to the project frontend

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat: track E2E coverage by required scenarios, not only routes Remove stub route detection from e2e-coverage.mjs

3 participants

@aasimsyed @nicki182 @andrewtavis