Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ jobs:
- name: Lint
run: npm run lint

- name: Format
run: npm run format:check

- name: Tests
run: npm test

Expand Down
13 changes: 13 additions & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Generated / vendored — do not auto-format.
node_modules
dist
.output
.nitro
.vercel
.rankproof
coverage
package-lock.json
src/routeTree.gen.ts

# Golden SERP HTML fixtures — whitespace is part of the capture.
src/lib/backlinks/fixtures/**
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ truthfulness, and the packaging a stranger's first clone runs into.
NAT64 and 6to4, plus `ff00::/8`, `100::/64`, `2001:db8::/32` and `2001::/23`.
Regression tests cover the URL-normalised spelling, which is the one that
actually arrives.
- `RANKPROOF_GOOGLE_PROVIDER_URL` (and the legacy OpenVis / SerpRadar aliases)
goes through the same guard before fetch. Loopback, link-local and metadata
addresses are refused, including after DNS resolution.
- `GET /doctor` is rate-limited like the POST endpoints; it runs five sequential
engine probes and previously skipped the limiter entirely. `GET /health` stays
free as a liveness probe.
Expand All @@ -51,6 +54,15 @@ truthfulness, and the packaging a stranger's first clone runs into.

### Fixed

- Unknown `--market`, `--device` and `--depth` values error with the allow-list
instead of falling back. `scan --depth` is the depth the scan uses (10 or 20).
- The CLI exits before `--experimental-strip-types` when Node is older than 22,
with a message that names the floor. `doctor` stays non-zero when unhealthy,
matching HTTP 503.
- Install docs no longer tell people to `npm install rankproof` while the
registry 404s. Clone and `npm ci` is the path that works.
- Dependency overrides clear the high advisories on `fast-uri`, `js-yaml` and
`brace-expansion`. CI runs `prettier --check`.
- **`.env` was never read.** README and `.env.example` both told people to create
one; nothing loaded it, so every variable was silently ignored. The CLI, the
HTTP API and `dev`/`build`/`preview` now load it via `util.parseEnv` — no new
Expand All @@ -71,7 +83,7 @@ truthfulness, and the packaging a stranger's first clone runs into.
`anomaly-modal` puzzle with HTTP 202 and Mojeek a JavaScript challenge with
HTTP 200; neither says "captcha", so `doctor` told users to file an issue
against a working parser. Both now read as `blocked`.
- Conversely, a SERP that merely *mentions* "captcha" or "unusual traffic" in a
- Conversely, a SERP that merely _mentions_ "captcha" or "unusual traffic" in a
snippet is no longer reported as blocked — parsed results settle it first.
- `doctor --format json` returned before the exit code was set, so the CI canary
the docs promise always exited 0.
Expand Down Expand Up @@ -118,6 +130,8 @@ truthfulness, and the packaging a stranger's first clone runs into.
(`prepack` builds it). A clone still runs the TypeScript sources directly, and
sources win when present so a stale `dist/` cannot shadow what you are editing.
CI installs the real tarball into a path with a space and runs it.
- The CLI imports `scripts/node-version.mjs` before any command runs. That file
is now in the published `files` list, so an installed package can start.
- The tarball is 153 kB across 52 files with **no runtime dependencies**, down
from shipping all of `src/` — React routes, styles and tests included — plus
51 dependencies the CLI never imports. Every one of them is consumed by the
Expand Down
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,17 +33,17 @@ Replace `example.com` with a public site you are authorized to audit. Scans make

![Actual local CLI help output, showing available commands without running a website scan.](docs/media/cli.gif)

*Animated excerpt of actual local command output. [Static version](docs/media/cli.png).*
_Animated excerpt of actual local command output. [Static version](docs/media/cli.png)._

## What can I inspect?

| Area | Output |
| --- | --- |
| Search visibility | Positions, keyword ideas, SERP overlap and scan history |
| Backlinks | Discovered links checked against source-page HTML |
| Site structure | Internal links, redirects, canonical issues and orphan candidates |
| Comparison | Link gaps and changes between scans |
| Reporting | JSON, CSV and a standalone HTML report |
| Area | Output |
| ----------------- | ----------------------------------------------------------------- |
| Search visibility | Positions, keyword ideas, SERP overlap and scan history |
| Backlinks | Discovered links checked against source-page HTML |
| Site structure | Internal links, redirects, canonical issues and orphan candidates |
| Comparison | Link gaps and changes between scans |
| Reporting | JSON, CSV and a standalone HTML report |

Bing, DuckDuckGo and Mojeek are enabled by default; Brave is optional. **Google organic results require a separately configured provider.** Search Console supplies data for a property you control. See [providers](docs/providers.md) and [Search Console setup](docs/search-console.md).

Expand All @@ -58,14 +58,14 @@ Read the [metric definitions](docs/metrics.md) before comparing scores.

## Go further

| Task | Guide |
| --- | --- |
| CLI commands and flags | [CLI reference](docs/cli.md) |
| Run the HTTP service | [API reference](docs/api.md) |
| Configure engines and integrations | [.env.example](.env.example) |
| Understand discovery and verification | [Architecture](docs/architecture.md) |
| Deploy beyond localhost | [Production operations](docs/production-ops.md) |
| Contribute or report a vulnerability | [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) |
| Task | Guide |
| ------------------------------------- | --------------------------------------------------------- |
| CLI commands and flags | [CLI reference](docs/cli.md) |
| Run the HTTP service | [API reference](docs/api.md) |
| Configure engines and integrations | [.env.example](.env.example) |
| Understand discovery and verification | [Architecture](docs/architecture.md) |
| Deploy beyond localhost | [Production operations](docs/production-ops.md) |
| Contribute or report a vulnerability | [Contributing](CONTRIBUTING.md) · [Security](SECURITY.md) |

```bash
npm run check
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Reporting a vulnerability

**Do not open a public issue** for a security flaw. Email `hello@cometweb.io` or use GitHub's private vulnerability reporting (*Security* tab → *Report a vulnerability*).
**Do not open a public issue** for a security flaw. Email `hello@cometweb.io` or use GitHub's private vulnerability reporting (_Security_ tab → _Report a vulnerability_).

We respond within 5 working days. If the issue is real, we agree a disclosure timeline together with the reporter.

Expand Down
50 changes: 35 additions & 15 deletions bin/rankproof.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,16 @@ import { spawn } from "node:child_process";
import { fileURLToPath } from "node:url";
import process from "node:process";

import { doctorExitCode, nodeVersionError } from "../scripts/node-version.mjs";

const STRIP = "--experimental-strip-types";

const versionError = nodeVersionError(process.version);
if (versionError) {
console.error(versionError);
process.exit(1);
}

/**
* Where the engine lives.
*
Expand Down Expand Up @@ -105,6 +113,7 @@ async function runScanCommand(options, log) {
market: options.market,
device: options.device,
engines: options.engines.length > 0 ? options.engines : undefined,
depth: options.depth,
skipSiteAudit: options.skipAudit,
});
if (!result.ok) throw new Error(result.error);
Expand Down Expand Up @@ -176,20 +185,16 @@ async function runDoctorCommand(options, log) {
const { runDoctor } = await import(engine("doctor.server"));
log("Probing engines…");
const diagnosis = await runDoctor();
// The non-zero exit is the point of `doctor` in CI, so it has to be set
// before the JSON path returns — not only on the text path below.
if (!diagnosis.healthy) process.exitCode = 1;
// Non-zero when no engine works — same contract as HTTP GET /doctor → 503
// and docs/production-ops.md (cron canary). Set before any return path.
process.exitCode = doctorExitCode(diagnosis.healthy);
if (options.format === "json") return JSON.stringify(diagnosis, null, 2);

const lines = ["RankProof doctor", ""];
lines.push("ENGINES");
for (const engine of diagnosis.engines) {
const mark =
engine.status === "ok"
? "ok "
: engine.status === "not-configured"
? "skip"
: "FAIL";
engine.status === "ok" ? "ok " : engine.status === "not-configured" ? "skip" : "FAIL";
lines.push(
` ${mark} ${engine.engine.padEnd(12)} ${String(engine.hits).padStart(2)} results ${String(engine.ms).padStart(5)} ms ${engine.status}`,
);
Expand Down Expand Up @@ -223,7 +228,9 @@ function renderScanText(report) {
lines.push(`RankProof — ${report.target.host}`);
lines.push(`${report.target.title ?? ""}`.trim());
lines.push("");
lines.push(`VISIBILITY INDEX ${scorecard.index}/100 (${scorecard.grade}) ${bar(scorecard.index)}`);
lines.push(
`VISIBILITY INDEX ${scorecard.index}/100 (${scorecard.grade}) ${bar(scorecard.index)}`,
);
for (const part of scorecard.parts) {
lines.push(
` ${part.label.padEnd(20)} ${String(part.score).padStart(3)}/${String(part.max).padEnd(3)} ${bar(part.score, part.max, 14)}`,
Expand All @@ -237,7 +244,9 @@ function renderScanText(report) {
lines.push(
` lost ${stats.lostLinks} · broken ${stats.brokenLinks} · to disavow ${toxic.disavowCount} · footprint ${report.footprint.score}/100 (${report.footprint.verdict})`,
);
lines.push(` velocity: ${report.velocity.perMonth} new domains/month (${report.velocity.verdict})`);
lines.push(
` velocity: ${report.velocity.perMonth} new domains/month (${report.velocity.verdict})`,
);
lines.push("");
lines.push("SERP");
lines.push(
Expand Down Expand Up @@ -270,7 +279,10 @@ function renderScanText(report) {
if (report.searchConsole?.connected) {
const totals = report.searchConsole.providers
.filter((provider) => provider.connected)
.map((provider) => `${provider.source}: ${provider.totals.clicks} clicks / ${provider.totals.impressions} impressions`)
.map(
(provider) =>
`${provider.source}: ${provider.totals.clicks} clicks / ${provider.totals.impressions} impressions`,
)
.join(" · ");
lines.push("");
lines.push("MEASURED PERFORMANCE");
Expand All @@ -284,13 +296,17 @@ function renderScanText(report) {
lines.push("");
lines.push("MEASUREMENT WARNINGS");
for (const item of unhealthy) {
lines.push(` ${item.engine}: ${item.status} — visibility is based on fewer engines than intended`);
lines.push(
` ${item.engine}: ${item.status} — visibility is based on fewer engines than intended`,
);
}
}
lines.push("");
lines.push(`ACTION PLAN (${plan.items.length} tasks, ${plan.quickWins} quick wins)`);
for (const [index, item] of plan.items.slice(0, 10).entries()) {
lines.push(` ${index + 1}. [${String(item.priority).padStart(3)}] ${item.title} (${item.effort})`);
lines.push(
` ${index + 1}. [${String(item.priority).padStart(3)}] ${item.title} (${item.effort})`,
);
lines.push(` ${item.detail}`);
}
lines.push("");
Expand All @@ -305,7 +321,9 @@ function renderSerpText(snapshot, keywords) {
lines.push(
`Visibility ${snapshot.visibility}/100 · top3 ${snapshot.top3} · top10 ${snapshot.top10} · traffic ~${snapshot.trafficScore}`,
);
lines.push(`Market ${snapshot.market}/${snapshot.device} · engines: ${snapshot.engines.join(", ")}`);
lines.push(
`Market ${snapshot.market}/${snapshot.device} · engines: ${snapshot.engines.join(", ")}`,
);
lines.push("");
for (const row of keywords) {
const positions = row.engines
Expand All @@ -319,7 +337,9 @@ function renderSerpText(snapshot, keywords) {
lines.push("");
lines.push("CANNIBALISATION");
for (const item of snapshot.cannibalization) {
lines.push(` "${item.keyword}" (${item.engine}): ${item.urls.map((u) => `#${u.position}`).join(", ")}`);
lines.push(
` "${item.keyword}" (${item.engine}): ${item.urls.map((u) => `#${u.position}`).join(", ")}`,
);
}
}
return lines.join("\n");
Expand Down
44 changes: 22 additions & 22 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,13 @@ curl -X POST http://127.0.0.1:8787/scan \
-d '{"url":"example.com","market":"us","device":"desktop"}'
```

| Field | Type | Required |
| --- | --- | --- |
| `url` | string | yes |
| `market` | `pl` `us` `gb` `de` `fr` `es` | no |
| `device` | `desktop` `mobile` | no |
| `engines` | string[] | no |
| `skipSiteAudit` | boolean | no |
| Field | Type | Required |
| --------------- | ----------------------------- | -------- |
| `url` | string | yes |
| `market` | `pl` `us` `gb` `de` `fr` `es` | no |
| `device` | `desktop` `mobile` | no |
| `engines` | string[] | no |
| `skipSiteAudit` | boolean | no |

Response: `{ "ok": true, "report": ScanReport }` or `{ "ok": false, "error": "…" }`.

Expand All @@ -61,16 +61,16 @@ The most important fields:

```jsonc
{
"target": { "host": "…", "domainRating": 0 },
"scorecard": { "index": 0, "grade": "C", "parts": [] },
"stats": { "backlinks": 0, "referringDomains": 0, "serpVisibility": 0 },
"backlinks": [],
"serp": { "queries": [], "competitors": [], "clusters": [], "moves": [] },
"toxic": { "domains": [], "disavowCount": 0 },
"plan": { "items": [], "quickWins": 0 },
"brandSerp": null,
"siteAudit": { "score": 0, "orphans": 0, "issues": [] },
"searchConsole": null
"target": { "host": "…", "domainRating": 0 },
"scorecard": { "index": 0, "grade": "C", "parts": [] },
"stats": { "backlinks": 0, "referringDomains": 0, "serpVisibility": 0 },
"backlinks": [],
"serp": { "queries": [], "competitors": [], "clusters": [], "moves": [] },
"toxic": { "domains": [], "disavowCount": 0 },
"plan": { "items": [], "quickWins": 0 },
"brandSerp": null,
"siteAudit": { "score": 0, "orphans": 0, "issues": [] },
"searchConsole": null,
}
```

Expand All @@ -96,9 +96,9 @@ It scans the competitors as well, so a response can take several minutes. Set a

## Response codes

| Code | Meaning |
| --- | --- |
| `200` | Success |
| `400` | Invalid input, or the scan failed |
| Code | Meaning |
| ----- | -------------------------------------------------------------------------- |
| `200` | Success |
| `400` | Invalid input, or the scan failed |
| `405` | Wrong method (`/health` and `/doctor` accept GET; everything else is POST) |
| `429` | Rate limit exceeded |
| `429` | Rate limit exceeded |
10 changes: 5 additions & 5 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,11 @@ When an engine changes its HTML, one parser breaks rather than the whole scan

The same engine serves three interfaces:

| Entry point | File | Use |
| --- | --- | --- |
| Web app | `src/routes/index.tsx` + `scan.ts` (server functions) | interactive audit |
| CLI | `bin/rankproof.mjs` + `cli.ts` (argument parser) | automation, CI, cron |
| HTTP API | `scripts/serve-api.mjs` | integrations, self-hosting |
| Entry point | File | Use |
| ----------- | ----------------------------------------------------- | -------------------------- |
| Web app | `src/routes/index.tsx` + `scan.ts` (server functions) | interactive audit |
| CLI | `bin/rankproof.mjs` + `cli.ts` (argument parser) | automation, CI, cron |
| HTTP API | `scripts/serve-api.mjs` | integrations, self-hosting |

The CLI argument parser is a separate, pure module — tested without running a scan.

Expand Down
34 changes: 17 additions & 17 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,28 +62,28 @@ Exits with code `1` when no engine returns results — useful as a CI canary, be

## Options

| Option | Short | Values | Default |
| --- | --- | --- | --- |
| `--keywords` | `-k` | comma-separated list | — |
| `--competitors` | `-c` | comma-separated list (max 5) | — |
| `--market` | `-m` | `pl` `us` `gb` `de` `fr` `es` | `pl` |
| `--device` | `-d` | `desktop` `mobile` | `desktop` |
| `--engines` | `-e` | `bing` `duckduckgo` `mojeek` `brave` `google` | from configuration |
| `--depth` | | `10` `20` | `10` |
| `--format` | `-f` | `text` `json` `csv` `html` `disavow` | `text` |
| `--out` | `-o` | file path | stdout |
| `--quiet` | `-q` | — | off |
| `--no-audit` | | — | off |
| Option | Short | Values | Default |
| --------------- | ----- | --------------------------------------------- | ------------------ |
| `--keywords` | `-k` | comma-separated list | — |
| `--competitors` | `-c` | comma-separated list (max 5) | — |
| `--market` | `-m` | `pl` `us` `gb` `de` `fr` `es` | `pl` |
| `--device` | `-d` | `desktop` `mobile` | `desktop` |
| `--engines` | `-e` | `bing` `duckduckgo` `mojeek` `brave` `google` | from configuration |
| `--depth` | | `10` `20` | `10` |
| `--format` | `-f` | `text` `json` `csv` `html` `disavow` | `text` |
| `--out` | `-o` | file path | stdout |
| `--quiet` | `-q` | — | off |
| `--no-audit` | | — | off |

Progress messages go to **stderr**, results to **stdout**. `npm run` prints its own banner to stdout, so redirect from the binary rather than through npm — `node bin/rankproof.mjs scan example.com --format json > report.json` — or use `--out report.json`, which npm cannot pollute.

## Exit codes

| Code | Meaning |
| --- | --- |
| `0` | Success |
| `1` | Runtime failure (the scan did not complete) |
| `2` | Invalid arguments |
| Code | Meaning |
| ---- | ------------------------------------------- |
| `0` | Success |
| `1` | Runtime failure (the scan did not complete) |
| `2` | Invalid arguments |

## Keeping it working

Expand Down
Loading
Loading