diff --git a/.agent/skills/webstatus-backend/SKILL.md b/.agent/skills/webstatus-backend/SKILL.md index bc7c5496c..400ca1a39 100644 --- a/.agent/skills/webstatus-backend/SKILL.md +++ b/.agent/skills/webstatus-backend/SKILL.md @@ -18,6 +18,21 @@ This skill provides guidance for developing the Go-based backend API for `websta For a technical deep-dive into the backend implementation patterns, request flows, and auth middleware, see [references/architecture.md](references/architecture.md). +## Development Environments + +The project supports two primary development environments: + +- **VS Code DevContainer**: A Docker-based environment with all tools pre-installed. +- **Nix (Alternative)**: A lightweight environment with pinned tool versions (Go 1.26.1). Enter via `nix develop`. + +### Tool Management + +- **Core Tools**: Go and Node are provided by the environment (Nix or DevContainer). +- **CLI Tools**: Tools like `wrench` and `oapi-codegen` are managed via `tools/go.mod` and executed via `make` targets (which use `go tool`). + +> [!IMPORTANT] +> A **container runtime** (Docker or Podman) is required on the host machine in both environments to run Spanner integration tests (via `testcontainers-go`) and the local emulator. + ## Guides - **[Add a New API Endpoint](references/add-api-endpoint.md)**: Mandatory spec-first process. @@ -52,7 +67,7 @@ We use a Hexagonal-style **Adapter Pattern** to decouple application logic from ## Testing & Linting -- **Precommit Suite**: Run `make precommit` to execute the full suite of Go tests, formatting, and linting. +- **Precommit Suite**: Run `make precommit` to execute the full suite of Go tests, formatting, and linting. This uses the pinned tool versions provided by your environment. - **Linting**: Run `make go-lint` to lint all Go code using `golangci-lint`. - **Quick Test Iteration**: Because this project uses a multi-module workspace (`go.work`), to run tests quickly for a single package without running the whole suite, execute `go test` from _within_ the specific module directory, or provide the full module path: ```bash @@ -60,7 +75,7 @@ We use a Hexagonal-style **Adapter Pattern** to decouple application logic from # Or go test -v github.com/GoogleChrome/webstatus.dev/lib/gcpspanner/... ``` -- **Integration Tests**: Any changes to `lib/gcpspanner` **MUST** include integration tests using `testcontainers-go` against the Spanner emulator. +- **Integration Tests**: Any changes to `lib/gcpspanner` **MUST** include integration tests using `testcontainers-go` against the Spanner emulator. Remember that this requires a working container runtime on your host! ## Documentation Updates diff --git a/.agent/skills/webstatus-e2e/SKILL.md b/.agent/skills/webstatus-e2e/SKILL.md index ac6f68e33..1225dca57 100644 --- a/.agent/skills/webstatus-e2e/SKILL.md +++ b/.agent/skills/webstatus-e2e/SKILL.md @@ -35,6 +35,17 @@ For a detailed technical guide on the local development environment (Skaffold/Mi - **Retries**: Playwright tests are configured to retry twice on failure only when running in a CI environment. If you want to simulate this locally and test flakiness, you can prefix your command with `CI=true` (e.g., `CI=true make playwright-test`). - **Browsers**: If you ever need to test against new browsers (e.g., mobile viewports, branded Edge/Chrome), modify the `projects` array within `playwright.config.ts`. +## Nix Environment & Docker Browser + +When working in the Nix development environment (via `nix develop` or `direnv`), special handling is required for Playwright: + +- **Isolated Browsers**: Browsers are isolated to `.nix/browsers` to avoid affecting host installations. +- **Docker Browser Server**: To guarantee 100% font parity for screenshots, the environment defaults to `USE_DOCKER_BROWSER=true`. This automatically spins up a Docker container and connects to it via WebSocket. +- **Known Limitation**: The "Show browsers" option in the VS Code Playwright extension is **broken** when using the Docker server. +- **Workarounds**: + - Use UI Mode (`make playwright-ui` or `npx playwright test --ui`). + - Temporarily disable Docker to use local Nix-patched browsers: `USE_DOCKER_BROWSER=false npx playwright test --headed`. + ## Execution & Debugging - For detailed instructions on rapid iteration, debugging CI failures, and using traces, see [references/execution-and-debugging.md](references/execution-and-debugging.md). diff --git a/.agent/skills/webstatus-frontend/SKILL.md b/.agent/skills/webstatus-frontend/SKILL.md index 34c3fd379..14227d1c9 100644 --- a/.agent/skills/webstatus-frontend/SKILL.md +++ b/.agent/skills/webstatus-frontend/SKILL.md @@ -14,6 +14,13 @@ This skill provides architectural guidance and conventions for the `frontend/` d - **State Management**: Uses **Lit Context** for dependency injection and state management via a service container pattern (``). - **API Interaction**: Communicates with the Go backend using TypeScript types generated from the OpenAPI specification (`make node-openapi`). +## Development Environments + +The project supports two primary development environments: + +- **VS Code DevContainer**: A Docker-based environment with all tools pre-installed. +- **Nix (Alternative)**: A lightweight environment with pinned tool versions (Node.js 24.14.0). Enter via `nix develop`. + ## Architecture For a technical breakdown of the Lit component hierarchy, frontend identity flows, and theming patterns, see [references/architecture.md](references/architecture.md). @@ -36,7 +43,11 @@ For a technical breakdown of the Lit component hierarchy, frontend identity flow ## Testing & Linting -- **Test Execution**: `npm run test -w frontend`. +- **Test Execution**: `npm run test -w frontend` or `make node-test`. +- **Nix Environment**: + - Browsers are isolated to `.nix/browsers`. + - If browsers are missing, run `npx playwright install` within the `nix develop` shell. + - Note: Frontend unit tests always use local browsers, even in Nix (unlike E2E tests which may default to a Docker-based browser). - **Linting**: Run `make node-lint` to run ESLint and Prettier for the frontend code, or `make lint-fix` to attempt auto-fixing. `make style-lint` is also available for CSS. - **ES Module Testing**: When testing components that use ES module exports directly (e.g. Firebase Auth), use a helper property (e.g. `credentialGetter`) that can be overridden with a Sinon stub. - **Typing**: Use generic arguments for `querySelector` in tests (e.g. `querySelector(...)`) for type safety. @@ -54,6 +65,7 @@ If a frontend unit test is timing out or failing mysteriously, Web Test Runner's - **Watch Mode**: Instruct the user to run `npm run test:watch -w frontend` in their own terminal. - **Visual Debugging**: Ask the user to open the provided localhost URL (e.g., `http://localhost:8000/`) in their web browser and inspect the developer console/DOM to see where the test is getting stuck. +- **Wayland Support**: In the Nix environment, Firefox is automatically configured to run natively if Wayland is detected (`MOZ_ENABLE_WAYLAND=1`). - **DON'T** arbitrarily increase the timeout in `web-test-runner.config.mjs` to fix timeout issues. Address the root cause of the hang instead. ## Documentation Updates diff --git a/.agent/skills/webstatus-ingestion/SKILL.md b/.agent/skills/webstatus-ingestion/SKILL.md index 39b1ab64d..89ba89534 100644 --- a/.agent/skills/webstatus-ingestion/SKILL.md +++ b/.agent/skills/webstatus-ingestion/SKILL.md @@ -18,6 +18,16 @@ This skill provides guidance for developing and deploying the scheduled data ing For a detailed map of data sources, Spanner target tables, and job orchestration patterns, see [references/architecture.md](references/architecture.md). +## Development Environments + +The project supports two primary development environments: + +- **VS Code DevContainer**: A Docker-based environment with all tools pre-installed. +- **Nix (Alternative)**: A lightweight environment with pinned tool versions. Enter via `nix develop`. + +> [!IMPORTANT] +> Even when using the Nix environment, a **container runtime** (Docker or Podman) is still required on your host machine to run the local Spanner emulator and test local job execution via Skaffold. + ## Infrastructure Abstraction (The Adapter Pattern) Ingestion jobs must be decoupled from the core DB logic and the "Backend" API. @@ -44,7 +54,7 @@ Ingestion jobs must be decoupled from the core DB logic and the "Backend" API. ## Testing & Linting -- **Precommit Suite**: Run `make precommit` to execute the full suite of Go tests, formatting, and linting. +- **Precommit Suite**: Run `make precommit` to execute the full suite of Go tests, formatting, and linting. Ensure you are inside your chosen environment (Nix or DevContainer) to use the correct tool versions. - **Linting**: Run `make go-lint` to lint all Go code using `golangci-lint`. - **Quick Test Iteration**: Because this project uses a multi-module workspace (`go.work`), to run tests quickly for a single package without running the whole suite, execute `go test` from _within_ the specific module directory: ```bash diff --git a/.agent/skills/webstatus-maintenance/SKILL.md b/.agent/skills/webstatus-maintenance/SKILL.md index e68fdf03d..3be3b4a1e 100644 --- a/.agent/skills/webstatus-maintenance/SKILL.md +++ b/.agent/skills/webstatus-maintenance/SKILL.md @@ -1,6 +1,6 @@ --- name: webstatus-maintenance -description: Use when upgrading toolchain versions (Go, Node.js, Terraform, Playwright) or updating the DevContainer and Github CI configurations. +description: Use when upgrading toolchain versions (Go, Node.js, Terraform, Playwright), updating the DevContainer, Nix environment, or Github CI configurations. --- # webstatus-maintenance @@ -27,6 +27,7 @@ The Go version must be kept in sync across: 2. `.github/workflows/ci.yml` (`GO_VERSION` environment variable) 3. `.github/workflows/devcontainer.yml` (`GO_VERSION` environment variable) 4. `images/go_service.Dockerfile` (`FROM golang:X.Y.Z-alpine...`) +5. `flake.nix` (If pinning a specific `nixpkgs` commit for Go) After updating the files, you should run `make go-update && make go-tidy` to ensure the `go.mod` dependencies are compatible with the new version. @@ -39,6 +40,7 @@ The Node.js version must be kept in sync across: 3. `.github/workflows/ci.yml` (`NODE_VERSION` environment variable) 4. `.github/workflows/devcontainer.yml` (`NODE_VERSION` environment variable) 5. `images/nodejs_service.Dockerfile` (`FROM node:X.Y.Z-alpine...`) +6. `flake.nix` (If pinning a specific `nixpkgs` commit for Node.js) After updating, run `make node-update` and test the frontend build. @@ -48,17 +50,33 @@ Playwright requires its NPM package and OS-level dependencies to stay in sync: 1. Update `playwright` and `@web/test-runner-playwright` in `frontend/package.json`. 2. Update the system dependencies in `.github/workflows/ci.yml` (the `npx playwright install --with-deps` step). +3. On Nix: Playwright browsers are cached in `.nix/browsers` and patched automatically by the shell hook in `flake.nix`. + +### Upgrading via Nix + +The Nix environment provides an alternative toolchain. + +- **Bumping All Tools**: Run `nix flake update` to update all tools to their latest versions in `nixpkgs-unstable`. +- **Pinning Versions**: To pin a specific tool version, update `flake.nix` to use a specific `nixpkgs` commit hash (see `docs/nix-setup.md`). + +### Upgrading Go-based Tools + +Tools like `wrench`, `oapi-codegen`, and `golines` are managed via `tools/go.mod`. + +- To upgrade a tool, navigate to the `tools/` directory and run `go get @`. +- Run `go mod tidy` in the `tools/` directory. ### Upgrading DevContainer Features -Other DevContainer tools (Terraform, Skaffold, Shellcheck, GitHub CLI) are managed within `.devcontainer/devcontainer.json`. +Other DevContainer tools (Skaffold, Shellcheck, GitHub CLI) are managed within `.devcontainer/devcontainer.json`. - Find the relevant feature under the `features` object and update its `"version"`. -- If modifying Terraform, also ensure the `.terraform-version` file (if one exists) or CI checks match the new version. +- If modifying Terraform, also ensure `infra/providers.tf` (`required_version`) and CI checks match the new version. ## Documentation Updates If you change how versions are managed or introduce a new critical dependency: - Update `docs/maintenance.md` to reflect the new update path. +- Update `docs/nix-setup.md` if the Nix environment logic changes. - Trigger the "Updating the Knowledge Base" prompt in `GEMINI.md` to ensure I am aware of the changes. diff --git a/.agent/skills/webstatus-pr-creation/SKILL.md b/.agent/skills/webstatus-pr-creation/SKILL.md index e2811706b..54eca96b2 100644 --- a/.agent/skills/webstatus-pr-creation/SKILL.md +++ b/.agent/skills/webstatus-pr-creation/SKILL.md @@ -9,10 +9,14 @@ When you have finished implementing a feature or fixing a bug and the user asks 1. **Verify the State**: Run `git status` to see what files have been modified. 2. **Review the Diffs**: Before committing, use `git diff` to quickly review the changes. Ensure you haven't left any stray `console.log` statements, commented-out debugger code, or unresolved merge conflict markers. -3. **Format, Lint, and Style**: Run standard project linters (`make precommit`, `make go-lint`, or `make node-lint`). Additionally, cross-reference your changes against the project's specific skills (e.g., `webstatus-backend`, `webstatus-frontend`) and standard Google Style Guides (e.g., Google Go Style Guide, Google TypeScript Style Guide). If you are unsure about a style rule, you may search for it or ask the user. +3. **Format, Lint, and Style**: Run standard project linters (`make precommit`, `make go-lint`, or `make node-lint`). + - **Environment**: Ensure you are in the `nix develop` shell or the project's DevContainer so you use the correct tool versions. + - **Regeneration**: If you modified OpenAPI specs, JSON schemas, or ANTLR grammars, you MUST run `make gen` before committing to ensure generated code is in sync. + - **Style Guides**: Cross-reference your changes against the project's specific skills (e.g., `webstatus-backend`, `webstatus-frontend`) and standard Google Style Guides. If you are unsure about a style rule, you may search for it or ask the user. 4. **Create a New Branch**: NEVER commit directly to `main`. - Run `git checkout -b feature/` or `git checkout -b fix/` 5. **Stage Files**: Add the specific files you modified using `git add `. Try to avoid `git add .` unless you are absolutely certain no unrelated files (like local IDE configs) are present. + - **Nix**: If you modified `flake.nix`, ensure you also run `nix flake update` and stage the updated `flake.lock`. 6. **Write a Descriptive Commit Message**: You MUST use the Conventional Commits format (`type(scope): subject`). Make sure to prefix the commit with `feat:`, `fix:`, `chore:`, `docs:`, `test:`, or `refactor:`. - _Example_: `feat: implement dark mode component` - _Example_: `fix(api): handle missing search name in payload` diff --git a/.agent/skills/webstatus-search-grammar/SKILL.md b/.agent/skills/webstatus-search-grammar/SKILL.md index 3e9ecda14..dc1dc1c04 100644 --- a/.agent/skills/webstatus-search-grammar/SKILL.md +++ b/.agent/skills/webstatus-search-grammar/SKILL.md @@ -16,6 +16,15 @@ For a technical breakdown of the ANTLR grammar, search node transformation, and - The canonical source of truth for the search syntax is `antlr/FeatureSearch.g4`. - **DON'T** edit the generated parser files in `lib/gen/featuresearch/parser/` directly. +## Tooling & Environment + +The search grammar relies on **ANTLR v4**. The project is configured to handle tool execution seamlessly across different development environments: + +- **Nix**: The environment provides the `antlr4` package and sets the `ANTLR=antlr4` environment variable in the shell. +- **DevContainer**: The environment vendors the ANTLR JAR at `/usr/local/lib/antlr-4.13.2-complete.jar` and the `Makefile` defaults to this path if `ANTLR` is not set. + +**Always use `make antlr-gen`** to regenerate the parser. This target abstracts the environment differences and ensures the correct Go-specific flags (`-Dlanguage=Go -visitor -no-listener`) and output directories are used. + ## General Guidelines - **DO** cross-reference all Go and test code against the official Google Go Style Guide. If you are unsure about a specific style rule, DO NOT assume; you MUST ask the user for clarification. diff --git a/.agent/skills/webstatus-workers/SKILL.md b/.agent/skills/webstatus-workers/SKILL.md index f5d3a289d..5f0a2add8 100644 --- a/.agent/skills/webstatus-workers/SKILL.md +++ b/.agent/skills/webstatus-workers/SKILL.md @@ -28,6 +28,16 @@ Workers must use the shared structs in [lib/workertypes/types.go](../../lib/work - The workers run locally via Skaffold and connect to local emulators for Spanner (`spanner:9010`) and Pub/Sub (`pubsub:8060`). - The `FRONTEND_BASE_URL` locally is usually `http://localhost:5555`. +## Development Environments + +The project supports two primary development environments: + +- **VS Code DevContainer**: A Docker-based environment with all tools pre-installed. +- **Nix (Alternative)**: A lightweight environment with pinned tool versions. Enter via `nix develop`. + +> [!IMPORTANT] +> Even when using the Nix environment, a **container runtime** (Docker or Podman) is still required on your host machine to run the local emulators and test local execution via Skaffold. + ## Infrastructure Abstraction (The Adapter Pattern) All workers must be decoupled from GCP-specific SDKs. diff --git a/.github/workflows/ci-nix.yml b/.github/workflows/ci-nix.yml new file mode 100644 index 000000000..4c697acea --- /dev/null +++ b/.github/workflows/ci-nix.yml @@ -0,0 +1,116 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: 'ci-nix' +on: + pull_request: + merge_group: + workflow_dispatch: + +permissions: + contents: read + id-token: write + actions: write + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Maximize build space + uses: easimon/maximize-build-space@fc881a613ad2a34aca9c9624518214ebc21dfc0c + with: + remove-dotnet: 'true' + remove-android: 'true' + remove-haskell: 'true' + remove-codeql: 'true' + root-reserve-mb: 2048 + overprovision-lvm: 'true' + build-mount-path: '/home/runner/work/webstatus.dev' + continue-on-error: true + + - name: Create Workspace + run: mkdir -p ${{ github.workspace }} + + - name: Checkout (GitHub) + uses: actions/checkout@v6 + + - name: Move Docker + run: | + sudo service docker stop + mkdir -p ../docker-storage + sudo rm -rf /var/lib/docker + sudo ln -s $(realpath ../docker-storage) /var/lib/docker + sudo service docker start + + - name: Install Nix + uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 + + - name: Magic Nix Cache + uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 + + - name: Run Nix environment setup + run: nix develop --command make nix-setup + + - name: Run precommit target for CI via Nix + run: nix develop --command make precommit + + playwright: + runs-on: ubuntu-latest + steps: + - name: Maximize build space + uses: easimon/maximize-build-space@fc881a613ad2a34aca9c9624518214ebc21dfc0c + with: + remove-dotnet: 'true' + remove-android: 'true' + remove-haskell: 'true' + remove-codeql: 'true' + root-reserve-mb: 2048 + overprovision-lvm: 'true' + build-mount-path: '/home/runner/work/webstatus.dev' + continue-on-error: true + + - name: Create Workspace + run: mkdir -p ${{ github.workspace }} + + - name: Checkout (GitHub) + uses: actions/checkout@v6 + + - name: Move Docker + run: | + sudo service docker stop + mkdir -p ../docker-storage + sudo rm -rf /var/lib/docker + sudo ln -s $(realpath ../docker-storage) /var/lib/docker + sudo service docker start + + - name: Install Nix + uses: DeterminateSystems/nix-installer-action@ef8a148080ab6020fd15196c2084a2eea5ff2d25 + + - name: Magic Nix Cache + uses: DeterminateSystems/magic-nix-cache-action@565684385bcd71bad329742eefe8d12f2e765b39 + + - name: Run Nix environment setup + run: nix develop --command make nix-setup + + - name: Run playwright-test target for CI via Nix + run: nix develop --command make playwright-test + env: + CI: 'true' + + - uses: actions/upload-artifact@v7 + if: ${{ !cancelled() }} + with: + name: playwright-report-nix + path: playwright-report/ + retention-days: 30 diff --git a/.gitignore b/.gitignore index d186421c6..9dc98d9f0 100644 --- a/.gitignore +++ b/.gitignore @@ -74,3 +74,10 @@ tsconfig.tsbuildinfo # Docker build log files infra/*.log + +# Direnv +.envrc +.direnv/ + +# Nix +.nix/ diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 6b040a21a..9aece920e 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -13,11 +13,11 @@ ## Requirements -The recommended way to do development is through the provided devcontainer. To -run a devcontainer, check out this -[web page](https://code.visualstudio.com/docs/devcontainers/containers#_system-requirements) -for the latest requirements to run devcontainer. The devcontainer will have -everything pre-installed. +The recommended way to do development is through the provided **VS Code DevContainer**. The devcontainer will have everything pre-installed. + +Alternatively, you can use **Nix** to set up your development environment. See the [Nix Setup Guide](./docs/nix-setup.md) for detailed instructions. + +For either path, a container runtime (like Docker Desktop) is still required on your host machine to run the local services. ## Project Structure diff --git a/GEMINI.md b/GEMINI.md index a8a9754df..058ddbde5 100644 --- a/GEMINI.md +++ b/GEMINI.md @@ -13,6 +13,7 @@ This document provides context to Gemini Code Assist to help it generate more ac This section describes the tools and commands for local development. - **Devcontainer**: The project uses a devcontainer for a consistent development environment. +- **Nix**: An alternative development environment that avoids heavy Docker containers for the toolchain. See `docs/nix-setup.md` for details. - **Skaffold & Minikube**: Local development is managed by `skaffold`, which deploys services to a local `minikube` Kubernetes cluster. - **Makefile**: Common development tasks are scripted in the `Makefile`. See below for key commands. diff --git a/Makefile b/Makefile index 17db0c479..84312d29d 100644 --- a/Makefile +++ b/Makefile @@ -39,8 +39,17 @@ DOCKERFILES := \ build: gen go-build node-install +# Note: We only install deps for webkit because full --with-deps fails on newer systems +# (like Ubuntu 24.04) due to virtual package resolution issues (e.g. libasound2). +nix-setup: node-install go-install-tools gen + npx playwright install + npx playwright install-deps webkit + clean: clean-gen clean-node port-forward-terminate minikube-delete +clean-nix: + rm -rf .nix/ + precommit: license-check go-fix go-tidy lint test unstaged-changes ################################ @@ -312,8 +321,10 @@ node-test: playwright-install ################################ # ANTLR ################################ +ANTLR ?= java -jar /usr/local/lib/antlr-$${ANTLR4_VERSION}-complete.jar + antlr-gen: clean-antlr - java -jar /usr/local/lib/antlr-$${ANTLR4_VERSION}-complete.jar -Dlanguage=Go -o lib/gen/featuresearch/parser -visitor -no-listener antlr/FeatureSearch.g4 + $(ANTLR) -Dlanguage=Go -o lib/gen/featuresearch/parser -visitor -no-listener antlr/FeatureSearch.g4 clean-antlr: rm -rf lib/gen/featuresearch/parser @@ -349,6 +360,8 @@ ADDLICENSE_ARGS := -c "${COPYRIGHT_NAME}" \ -ignore 'infra/storage/spanner/schema.sql' \ -ignore 'antlr/.antlr/**' \ -ignore '.devcontainer/cache/**' \ + -ignore '.nix/**' \ + -ignore 'direnv/**' \ -ignore 'workers/email/pkg/digest/testdata/digest.golden.html' license-check: go-install-tools @@ -374,10 +387,19 @@ unstaged-changes: # fresh-env-for-playwright prerequisite. If unset, the fresh environment will be created. SKIP_FRESH_ENV ?= -fresh-env-for-playwright: $(if $(SKIP_FRESH_ENV),,playwright-install delete-local build deploy-local port-forward-manual dev_fake_users dev_fake_data) +playwright-docker-stop: + @echo "Stopping Playwright Docker container if running..." + docker stop playwright-server || true + docker rm playwright-server || true + +fresh-env-for-playwright: $(if $(SKIP_FRESH_ENV),,playwright-docker-stop playwright-install delete-local build deploy-local port-forward-manual dev_fake_users dev_fake_data) playwright-install: - npx playwright install --with-deps + @if [ -z "$$PLAYWRIGHT_BROWSERS_PATH" ]; then \ + npx playwright install --with-deps; \ + else \ + echo "Skipping playwright install because PLAYWRIGHT_BROWSERS_PATH is set to $$PLAYWRIGHT_BROWSERS_PATH. This should have been handled by nix-develop already to prevent sudo prompt."; \ + fi playwright-update-snapshots: fresh-env-for-playwright npx playwright test --update-snapshots diff --git a/README.md b/README.md index 1f5367530..31c6144ff 100644 --- a/README.md +++ b/README.md @@ -33,9 +33,11 @@ capabilities, please refer to our [Search Syntax Guide](./antlr/FeatureSearch.md ## Get the code -This repository relies heavily on [devcontainers](https://code.visualstudio.com/docs/remote/create-dev-container) to get started. +To get started with the codebase, you can use either a **VS Code DevContainer** or **Nix**. -To continue setting up locally: +### Option 1: VS Code DevContainer (Recommended for VS Code users) + +This repository supports [devcontainers](https://code.visualstudio.com/docs/remote/create-dev-container) to provide a fully configured environment. ```sh git clone https://github.com/GoogleChrome/webstatus.dev @@ -47,9 +49,23 @@ code webstatus.dev # Opens Visual Studio Code with the webstatus.dev folder. # 2. Select the option: Dev containers: Rebuild and Reopen in Container ``` +### Option 2: Nix (Alternative) + +If you prefer not to use Docker for your toolchain or don't use VS Code, you can use Nix. See the [Nix Setup Guide](./docs/nix-setup.md) for detailed instructions. + +```sh +git clone https://github.com/GoogleChrome/webstatus.dev +cd webstatus.dev +nix develop +make nix-setup +``` + +> [!NOTE] +> To clean up the downloaded browsers and start fresh, you can run `make clean-nix`. + ### Running the services locally -After getting the code with or without devcontainer, check out the [DEVELOPMENT.md](./DEVELOPMENT.md) for more information to get started and running locally. +After setting up your environment, check out the [DEVELOPMENT.md](./DEVELOPMENT.md) for more information to get started and running locally. ### Using Gemini CLI diff --git a/docs/maintenance.md b/docs/maintenance.md index 7414c948d..9caa496cb 100644 --- a/docs/maintenance.md +++ b/docs/maintenance.md @@ -5,20 +5,22 @@ any other dependencies that may not be clear. Files with `(*)` mean they are use ## Dependencies -| Binary | File to update | -| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| gcloud | devcontainer [Dockerfile](../.devcontainer/Dockerfile) | -| go | [devcontainer.json](../.devcontainer/devcontainer.json)
GO_VERSION in [ci](../.github/workflows/ci.yml) and [devcontainer](../.github/workflows/devcontainer.yml) workflows
`(*)`[Generic Go Dockerfile](../images/go_service.Dockerfile) | -| node | [devcontainer.json](../.devcontainer/devcontainer.json)
NODE_VERSION in [ci](../.github/workflows/ci.yml) and [devcontainer](../.github/workflows/devcontainer.yml) workflows
`(*)`[Generic Node + Nginx Dockerfile](../images/nodejs_service.Dockerfile)
[DevContainer Node Version](../.devcontainer/Dockerfile) | -| Github CLI | [devcontainer.json](../.devcontainer/devcontainer.json) | -| terraform | [devcontainer.json](../.devcontainer/devcontainer.json) | -| shellcheck | [devcontainer.json](../.devcontainer/devcontainer.json) | -| kubectl, helm, minikube | [devcontainer.json](../.devcontainer/devcontainer.json) | -| skaffold | [devcontainer.json](../.devcontainer/devcontainer.json) | -| oapi-codegen | [post_create.sh](../.devcontainer/post_create.sh) | -| wrench | [post_create.sh](../.devcontainer/post_create.sh) and [.dev/spanner/Dockerfile](../.dev/spanner/Dockerfile) | -| datastore emulator | [Dockerfile](../.dev/datastore/Dockerfile) | -| spanner emulator | [Dockerfile](../.dev/spanner/Dockerfile) | -| pub/sub emulator | [Dockerfile](../.dev/pubsub/Dockerfile) | -| wiremock | [Dockerfile](../.dev/wiremock/Dockerfile) | -| gcs emulator | [Dockerfile](../.dev/gcs/Dockerfile) | +| Binary | File to update | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| gcloud | devcontainer [Dockerfile](../.devcontainer/Dockerfile) | +| go | [devcontainer.json](../.devcontainer/devcontainer.json)
GO_VERSION in [ci](../.github/workflows/ci.yml) and [devcontainer](../.github/workflows/devcontainer.yml) workflows
`(*)`[Generic Go Dockerfile](../images/go_service.Dockerfile)
[flake.nix](../flake.nix) | +| node | [devcontainer.json](../.devcontainer/devcontainer.json)
NODE_VERSION in [ci](../.github/workflows/ci.yml) and [devcontainer](../.github/workflows/devcontainer.yml) workflows
`(*)`[Generic Node + Nginx Dockerfile](../images/nodejs_service.Dockerfile)
[DevContainer Node Version](../.devcontainer/Dockerfile)
[flake.nix](../flake.nix) | +| Github CLI | [devcontainer.json](../.devcontainer/devcontainer.json) | +| terraform | [devcontainer.json](../.devcontainer/devcontainer.json)
[providers.tf](../infra/providers.tf) | +| shellcheck | [devcontainer.json](../.devcontainer/devcontainer.json) | +| kubectl, helm, minikube | [devcontainer.json](../.devcontainer/devcontainer.json) | +| skaffold | [devcontainer.json](../.devcontainer/devcontainer.json) | +| oapi-codegen | [tools/go.mod](../tools/go.mod) and [post_create.sh](../.devcontainer/post_create.sh) | +| wrench | [tools/go.mod](../tools/go.mod), [post_create.sh](../.devcontainer/post_create.sh) and [.dev/spanner/Dockerfile](../.dev/spanner/Dockerfile) | +| golines | [tools/go.mod](../tools/go.mod) | +| addlicense | [tools/go.mod](../tools/go.mod) | +| datastore emulator | [Dockerfile](../.dev/datastore/Dockerfile) | +| spanner emulator | [Dockerfile](../.dev/spanner/Dockerfile) | +| pub/sub emulator | [Dockerfile](../.dev/pubsub/Dockerfile) | +| wiremock | [Dockerfile](../.dev/wiremock/Dockerfile) | +| gcs emulator | [Dockerfile](../.dev/gcs/Dockerfile) | diff --git a/docs/nix-setup.md b/docs/nix-setup.md new file mode 100644 index 000000000..0105d4c1f --- /dev/null +++ b/docs/nix-setup.md @@ -0,0 +1,134 @@ +# Nix Development Environment Setup + +This guide explains how to use the Nix development environment for `webstatus.dev`. + +## Overview + +We are introducing a Nix-based development environment as an alternative to the VS Code DevContainer. This allows developers to have a consistent environment with pinned tool versions without requiring a heavy Docker container for the entire toolchain. + +We are currently supporting both workflows side-by-side. + +## Prerequisites + +1. **Nix**: You must have Nix installed on your host machine. Follow instructions at [nixos.org](https://nixos.org/download.html). +2. **Container Runtime**: A container runtime (like Docker Desktop, Podman, or OrbStack) is still required on your host machine to run Minikube and the Spanner emulator via `make start-local`. + +## Getting Started + +To enter the development environment, navigate to the project root and run: + +```bash +nix develop +make nix-setup +``` + +This will drop you into a new shell with all the required tools in your `PATH`: + +- Go +- Node.js +- OpenJDK +- Terraform, Cloud SDK, Minikube, Skaffold, etc. + +You can verify the versions by looking at the output message when you enter the shell. + +## Tools Managed by Go + +Some tools used in the `Makefile` (like `wrench`, `oapi-codegen`, `golines`, `addlicense`) are not packaged in the Nix flake. Instead, they are managed by Go via `tools/go.mod` using the Go 1.24+ `tool` directive. + +Running `make` commands will automatically use `go tool` to resolve and run the correct versions of these tools. + +## Wayland vs X11 + +The Nix environment attempts to detect if your host machine is using Wayland by checking the `$WAYLAND_DISPLAY` environment variable. + +- If detected, it sets `MOZ_ENABLE_WAYLAND=1` to ensure Firefox (used in tests) runs natively on Wayland. +- If not detected, it falls back to standard X11 settings. + +## Playwright and Browsers + +If you need to run E2E tests headfully (where the browser pops up), ensure you have a display server running on your host. For headless tests (default in CI and most local runs), no display is required. + +## Verifying Package Versions + +You can check what version of a package is provided by the current locked flake without entering the shell by running: + +```bash +nix eval nixpkgs#go.version +nix eval nixpkgs#nodejs.version +nix eval nixpkgs#golangci-lint.version +``` + +## Pinning Specific Tool Versions + +If you need to pin a tool like `golangci-lint` to a specific older version (as recommended by its maintainers to avoid using Go tools), you can pull in a specific commit of `nixpkgs` in `flake.nix`. + +1. Find the commit hash for the desired version on [nixhub.io](https://nixhub.io). +2. Add a new input in `flake.nix` pointing to that commit. For example, for `golangci-lint` 2.11.0: + ```nix + inputs.nixpkgs-lint.url = "github:nixos/nixpkgs/0e6cdd5be64608ef630c2e41f8d51d484468492f"; + ``` +3. Use it in the `outputs` function to provide that specific package to your shell. + +## Bumping All Tools (Updating the Lockfile) + +To update all tools (like Go, Node, Terraform) to their latest versions available in the `nixpkgs-unstable` channel, run: + +```bash +nix flake update +``` + +This will fetch the latest commit from the branch and update `flake.lock`. Note that this may update multiple tools at once, so it is recommended to run tests afterwards to ensure compatibility. + +## IDE Integration (VS Code) + +To make VS Code aware of the Nix environment automatically in your integrated terminals, you can use `direnv`. + +> [!WARNING] +> **Do NOT use the VS Code `direnv` extension (`mkhl.direnv`).** There is a known issue (see [direnv #1307](https://github.com/direnv/direnv/issues/1307)) where the extension loads the environment too early, and the shell startup files subsequently clobber the `PATH`. This results in Nix tools missing in new terminals. + +Instead, rely entirely on your host shell's `direnv` hook: + +1. **Install `direnv`** on your host machine. +2. **Hook it into your shell**: + - For **Bash**, add `eval "$(direnv hook bash)"` to your `~/.bashrc`. + - For **Zsh**, add `eval "$(direnv hook zsh)"` to your `~/.zshrc`. +3. **Create `.envrc`**: Create a file named `.envrc` in the project root with the following content: + ```bash + use flake + ``` + _(Note: This file is ignored in `.gitignore` and should not be checked in)._ +4. **Authorize it**: Run `direnv allow` in your terminal. + +Now, whenever you open a new integrated terminal in VS Code, your shell will automatically load the Nix environment _after_ initialization, ensuring all tools are available. + +## Playwright and Browsers + +Playwright manages its own browser binaries and system dependencies. + +When you run `make nix-setup`, it runs `npx playwright install` to download browsers, and `npx playwright install-deps webkit` to install system dependencies for WebKit on the host. + +We only install dependencies for `webkit` because a full `--with-deps` can fail on newer Linux distributions (like Ubuntu 24.04) due to package name changes (e.g., `libasound2` being replaced by `libasound2t64`). + +This ensures that browsers work correctly without needing complex configuration or patching in the Nix flake. + +## Playwright E2E Tests and Docker + +To guarantee 100% font parity for E2E screenshots across different developer machines (and match CI), you can use a remote Docker browser server. + +When you are in the Nix environment, the variable `USE_DOCKER_BROWSER=true` is set by default in `flake.nix`. This tells Playwright to automatically spin up a Docker container (`webstatus-playwright`) and connect to it via WebSocket. + +### Known Limitations + +> [!IMPORTANT] +> The **"Show browsers"** option in the VS Code Playwright extension is currently **broken** when using the Docker server. +> Because the browser is running inside a container without an XServer, requesting a headed browser will cause tests to fail or not display. + +### Workarounds for Debugging + +If you need to debug tests and see the browser or interact with it: + +1. **Use UI Mode**: Run `npx playwright test --ui`. This opens an interactive UI on your host that connects to the remote browser and shows snapshots. +2. **Use Local Browsers**: You can disable the Docker browser temporarily to use your local Nix-patched browsers (which _do_ support showing windows on your host): + ```bash + USE_DOCKER_BROWSER=false npx playwright test --headed + ``` diff --git a/flake.lock b/flake.lock new file mode 100644 index 000000000..3750ab5a3 --- /dev/null +++ b/flake.lock @@ -0,0 +1,78 @@ +{ + "nodes": { + "flake-utils": { + "inputs": { + "systems": "systems" + }, + "locked": { + "lastModified": 1731533236, + "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + "nixpkgs": { + "locked": { + "lastModified": 1775793324, + "narHash": "sha256-omax7atcZbol+6HJ2RLpP+ZCFcPa5bZ65Hn71RufeWQ=", + "owner": "nixos", + "repo": "nixpkgs", + "rev": "9d29d5f667d7467f98efc31881e824fa586c927e", + "type": "github" + }, + "original": { + "owner": "nixos", + "ref": "nixpkgs-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "nixpkgs-lint": { + "locked": { + "lastModified": 1772927210, + "narHash": "sha256-FdRDRoV0jRTiPK5ID22BaUX5P0wdsclpxtIOjaEy9Lo=", + "owner": "nixos", + "repo": "nixpkgs", + "rev": "0e6cdd5be64608ef630c2e41f8d51d484468492f", + "type": "github" + }, + "original": { + "owner": "nixos", + "repo": "nixpkgs", + "rev": "0e6cdd5be64608ef630c2e41f8d51d484468492f", + "type": "github" + } + }, + "root": { + "inputs": { + "flake-utils": "flake-utils", + "nixpkgs": "nixpkgs", + "nixpkgs-lint": "nixpkgs-lint" + } + }, + "systems": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/flake.nix b/flake.nix new file mode 100644 index 000000000..d2a59fbc1 --- /dev/null +++ b/flake.nix @@ -0,0 +1,78 @@ +{ + description = "Dev environment for webstatus.dev"; + + inputs = { + nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable"; + flake-utils.url = "github:numtide/flake-utils"; + # Pinned to nixpkgs commit that provides golangci-lint 2.11.0 + nixpkgs-lint.url = "github:nixos/nixpkgs/0e6cdd5be64608ef630c2e41f8d51d484468492f"; + }; + + outputs = { self, nixpkgs, flake-utils, nixpkgs-lint }: + flake-utils.lib.eachDefaultSystem (system: + let + pkgs = import nixpkgs { + inherit system; + config.allowUnfree = true; + }; + pkgs-lint = import nixpkgs-lint { + inherit system; + }; + in + { + devShells.default = pkgs.mkShell { + buildInputs = with pkgs; [ + go + nodejs + jdk25 + pkgs-lint.golangci-lint + google-cloud-sdk + kubectl + minikube + skaffold + terraform + tflint + shellcheck + gh + jq + netcat + antlr4 + + + ]; + + shellHook = '' + export PS1="[nix-develop:\w]\$ " + export MINIKUBE_PROFILE=webstatus-dev + export DOCKER_BUILDKIT=1 + export ANTLR=antlr4 + export USE_DOCKER_BROWSER=true + # Isolate Playwright browsers to this project + export PLAYWRIGHT_BROWSERS_PATH="$PWD/.nix/browsers" + + echo "Entering Nix environment for webstatus.dev" + + # Print versions + echo "Go version: $(go version)" + echo "Node version: $(node --version)" + echo "Java version: $(java --version | head -n 1)" + echo "Terraform version: $(terraform version | head -n 1)" + echo "TFLint version: $(tflint --version | head -n 1)" + echo "Golangci-lint version: $(golangci-lint --version)" + echo "Skaffold version: $(skaffold version)" + echo "Minikube version: $(minikube version | head -n 1)" + + # Conditional Wayland support + if [ -n "$WAYLAND_DISPLAY" ]; then + echo "Wayland detected. Enabling Wayland support for browsers." + export MOZ_ENABLE_WAYLAND=1 + else + echo "X11 detected. Using standard display settings." + fi + + # Playwright environment variables + export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=1 + ''; + }; + }); +} diff --git a/global-setup.ts b/global-setup.ts new file mode 100644 index 000000000..25a1ecc63 --- /dev/null +++ b/global-setup.ts @@ -0,0 +1,107 @@ +/** + * Copyright 2026 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import {execSync} from 'child_process'; +import fs from 'fs'; +import path from 'path'; + +async function globalSetup() { + if (process.env.USE_DOCKER_BROWSER === 'true') { + console.log( + 'USE_DOCKER_BROWSER is true. Setting up Docker browser server...', + ); + + const port = process.env.PLAYWRIGHT_DOCKER_PORT || '4444'; + + // Read version from package.json + const pkgPath = path.resolve(process.cwd(), 'package.json'); + const pkg = JSON.parse(fs.readFileSync(pkgPath, 'utf-8')); + const playwrightVersion = pkg.devDependencies['@playwright/test'].replace( + '^', + '', + ); + + console.log(`Detected Playwright version: ${playwrightVersion}`); + + // Check if image exists + try { + execSync('docker image inspect webstatus-playwright', {stdio: 'ignore'}); + console.log('Docker image webstatus-playwright already exists.'); + } catch (e) { + console.log( + 'Docker image webstatus-playwright not found. Building it...', + ); + execSync( + 'docker build -f images/playwright.Dockerfile -t webstatus-playwright .', + {stdio: 'inherit'}, + ); + } + + // Check if container is already running + try { + const existing = execSync('docker ps -q --filter name=playwright-server') + .toString() + .trim(); + if (existing) { + console.log('Playwright server container is already running.'); + process.env.PW_TEST_CONNECT_WS_ENDPOINT = `ws://localhost:${port}`; + return; + } + } catch (e) { + // Ignore error + } + + // Start container + console.log('Starting Playwright server container...'); + const runCmd = `docker run -d --name playwright-server --network host webstatus-playwright bash -c "npx playwright@${playwrightVersion} install --with-deps && npx playwright@${playwrightVersion} run-server --port ${port}"`; + + execSync(runCmd, {stdio: 'inherit'}); + + // Wait for port to be ready + const timeout = process.env.PLAYWRIGHT_DOCKER_TIMEOUT + ? parseInt(process.env.PLAYWRIGHT_DOCKER_TIMEOUT, 10) + : 120000; + console.log( + `Waiting for Playwright server to be ready on port ${port} (timeout: ${timeout}ms)...`, + ); + await waitForPort(parseInt(port, 10), timeout); + + process.env.PW_TEST_CONNECT_WS_ENDPOINT = `ws://localhost:${port}`; + console.log('Playwright server is ready!'); + } +} + +async function waitForPort(port: number, timeout = 120000) { + const start = Date.now(); + while (Date.now() - start < timeout) { + try { + const net = await import('net'); + await new Promise((resolve, reject) => { + const socket = net.connect(port, 'localhost', () => { + socket.end(); + resolve(true); + }); + socket.on('error', reject); + }); + return; + } catch (e) { + await new Promise(resolve => setTimeout(resolve, 1000)); + } + } + throw new Error(`Timeout waiting for port ${port}`); +} + +export default globalSetup; diff --git a/images/playwright.Dockerfile b/images/playwright.Dockerfile new file mode 100644 index 000000000..5d8cbaeff --- /dev/null +++ b/images/playwright.Dockerfile @@ -0,0 +1,28 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM mcr.microsoft.com/devcontainers/base:trixie + +# Install Node.js 24.11.1 +RUN apt-get update && apt-get install -y wget xz-utils && \ + wget https://nodejs.org/dist/v24.11.1/node-v24.11.1-linux-x64.tar.xz && \ + tar -xJf node-v24.11.1-linux-x64.tar.xz -C /usr/local --strip-components=1 && \ + rm node-v24.11.1-linux-x64.tar.xz + +WORKDIR /work + +EXPOSE 4444 + +# We don't specify a CMD here because we will pass the command at runtime +# to use the correct Playwright version dynamically. diff --git a/playwright.config.ts b/playwright.config.ts index 7c2e6e4ea..243bf8405 100644 --- a/playwright.config.ts +++ b/playwright.config.ts @@ -28,6 +28,7 @@ import {defineConfig, devices} from '@playwright/test'; export default defineConfig({ testDir: './e2e', outputDir: './e2e/test-results', + globalSetup: './global-setup', /* Run tests in files in parallel */ fullyParallel: true, /* Fail the build on CI if you accidentally left test.only in the source code. */ @@ -40,6 +41,8 @@ export default defineConfig({ reporter: 'html', /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */ use: { + /* Force headless mode when using Docker browser to avoid XServer errors */ + headless: true, /* Base URL to use in actions like `await page.goto('/')`. */ // baseURL: 'http://127.0.0.1:3000', extraHTTPHeaders: {