Skip to content

About

Python script to find broken links on a site

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

Broken Link Finder

A Python script that scans a web page of a given URL and validates the links on it.

If the page has a link to the same hostname as the URL given by the user, its destination page will also be scanned. It works like a web crawler, so potentially the whole site will be scanned. Be aware that this may take hours.

All broken links are saved on broken-links-[date]-[time]-[random-ID].txt.

Requisites

Python 3.10+

uv

Setup

uv sync
uv run playwright install chromium

Usage

uv run find_broken_links.py
uv run find_broken_links_req.py
uv run find_broken_links_sync.py

Or, after installing with uv sync, you can use the provided entry points:

uv run find-broken-links
uv run find-broken-links-req
uv run find-broken-links-sync

Details

The script scans pages using Playwright to account for links that may be injected via JavaScript. Playwright manages its own browser binaries, so no manual driver installation is needed.

Then, each found link is validated with Requests in a concurrent (multi-thread) fashion.

If you don't care about rendering JavaScript, the find_broken_links_req.py script doesn't use a browser and thus is significantly faster.

A single-thread script is provided for benchmarking.

TODO

  • Better exception handling
  • Validate links in other tags besides <href a=...> (like <img src=...>)
  • Timeout
  • Link depth limit
  • Proxy support
  • NTLM support
  • Browser parallelization

About

Python script to find broken links on a site

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages