Skip to content

nsheth12/klaviyo-scrape-workers

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 

Repository files navigation

Klaviyo Snippet Checker

A web-based tool for scanning websites for Klaviyo and Customer Hub code snippets, built on Cloudflare Workers with D1 database and Queues for parallel processing.

Architecture

The application runs entirely on Cloudflare Workers:

  • Web Dashboard — served by the Worker at /, shows runs, progress, and results
  • Dispatcher — triggered by "Run Scan" button, pushes URLs to a Cloudflare Queue
  • Scanner (queue consumer) — processes URLs in parallel batches, checks for snippets, writes results to D1
  • Cron Trigger — runs every minute, detects completed runs and sends email notifications via Resend

Project Structure

workers/                    # Cloudflare Workers project
├── src/
│   ├── index.ts           # Main router (fetch, queue, scheduled handlers)
│   ├── dashboard.ts       # HTML/CSS/JS for the web dashboard
│   ├── dispatcher.ts      # Scan run dispatcher (pushes to queue)
│   ├── scanner.ts         # Queue consumer (fetches URLs, checks snippets)
│   ├── cron.ts            # Completion detection + email notification
│   ├── snippets.ts        # Snippet definitions (shared constants)
│   └── types.ts           # TypeScript type definitions
├── scripts/
│   └── import-urls.ts     # Script to bulk-import URLs from sites.txt
├── schema.sql             # D1 database schema
├── wrangler.toml          # Cloudflare Workers configuration
├── package.json
└── tsconfig.json

D1 Database Schema

  • urls — master list of URLs to scan (id, url, created_at)
  • runs — scan runs (id, status, total_urls, completed_urls, started/completed timestamps, notification_sent)
  • results — per-URL per-run results (9 snippet columns + error + checked_at)

Snippets Checked

  1. klaviyo-email-marketing-sms
  2. customerhub
  3. customer-hub-data
  4. window.customerHub
  5. <a href="#k-hub"
  6. node.href="#k-hub"
  7. customer-hub-replace-links
  8. function replaceAccountLinks()
  9. klaviyo

API Routes

  • GET / — Dashboard HTML
  • GET /api/runs — List all runs with URL count
  • POST /api/run — Start a new scan run
  • GET /api/results/:runId — Get paginated results + stats for a run
  • GET /api/status?runId=... — Get status of a specific run
  • POST /api/urls/upload — Bulk upload URLs (JSON body: {urls: [...]})
  • POST /api/cron/trigger — Manually trigger cron handler (for local dev)

Development

Local dev uses wrangler dev with Miniflare for simulated D1 and Queues.

cd workers
npx wrangler d1 execute snippet-checker-db --local --file=./schema.sql  # Init DB
npx wrangler dev --port 5000 --local --ip 0.0.0.0                       # Start dev server

Deploying to Cloudflare

  1. wrangler login — authenticate with Cloudflare
  2. wrangler d1 create snippet-checker-db — create the D1 database
  3. Update database_id in wrangler.toml with the real ID
  4. wrangler d1 execute snippet-checker-db --remote --file=./schema.sql — run migrations
  5. wrangler secret put RESEND_API_KEY — set email API key
  6. wrangler secret put NOTIFICATION_EMAIL — set notification email
  7. wrangler deploy — deploy the Worker

Dependencies

Node.js (Workers)

  • wrangler — Cloudflare CLI for dev/deploy
  • @cloudflare/workers-types — TypeScript types
  • typescript — TypeScript compiler

Workflow

  • Start application: cd workers && npx wrangler dev --port 5000 --local --ip 0.0.0.0 (webview)

Runtime

  • Node.js 20 (Workers development)

About

Cloudflare Workers-based web scraper for detecting Klaviyo and Customer Hub snippets on websites

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages