A Python script that scans a web page of a given URL and validates the links on it.
If the page has a link to the same hostname as the URL given by the user, its destination page will also be scanned. It works like a web crawler, so potentially the whole site will be scanned. Be aware that this may take hours.
All broken links are saved on broken-links-[date]-[time]-[random-ID].txt.
Python 3.10+
uv sync
uv run playwright install chromiumuv run find_broken_links.py
uv run find_broken_links_req.py
uv run find_broken_links_sync.pyOr, after installing with uv sync, you can use the provided entry points:
uv run find-broken-links
uv run find-broken-links-req
uv run find-broken-links-syncThe script scans pages using Playwright to account for links that may be injected via JavaScript. Playwright manages its own browser binaries, so no manual driver installation is needed.
Then, each found link is validated with Requests in a concurrent (multi-thread) fashion.
If you don't care about rendering JavaScript, the find_broken_links_req.py script doesn't use a browser and thus is significantly faster.
A single-thread script is provided for benchmarking.
- Better exception handling
- Validate links in other tags besides
<href a=...>(like<img src=...>) - Timeout
- Link depth limit
- Proxy support
- NTLM support
- Browser parallelization