Repository navigation
feat: a website's posts are read from the sitemap it names (ADR 0068) - #60
Merged
Merged
Conversation
A website's settings take the address of its sitemap. A background reader reads it, and each page it lists once for its title and publish date, and Settings says how reading went. - Only the website's own hosts, held to the sitemaps.org location rule, and only the public internet: every IANA special-purpose block and Azure's 168.63.129.16 are refused as each connection opens, and no certificate is downloaded to complete a chain. - One request a second per website, robots.txt (RFC 9309) obeyed for every address and redirect, and a User-Agent saying what the reader is and where to read about it. - Pages are read by AngleSharp's tokenizer and no document is ever built. - A claim is a lease on the website's row, every write is fenced on it, and an attempt is written before it is made. - A new sitemap queues as of the moment it is named, a website being read is skipped at both of its names, a website whose account is not being measured is not read, and an engine stopping lets go of the website as due at once. - What a website may hold is bounded in pages and in bytes, and a reading that closes forgets the pages no file vouches for down to twice what a website may list. - landed_path keeps the path a redirected page answered at, for matching visits to posts. - PostgreSQL migration AddSitemapReading: sites.sitemap_address and the tables sitemap_reads, sitemap_parts and site_posts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 of the Insights plan: a website's settings name its sitemap, and the engine reads it, and each page it lists once, for title and publish date. Settings says how reading went. Recorded in ADR 0068.
168.63.129.16are refused. HTTP/1.1 only, no proxy, no cookies, no certificate downloads.Retry-Afterheard, and a User-Agent that names the reader (DewirideAnalytics/{version} (sitemap fetcher; +…)).clock_timestamp(), writes are fenced withFOR KEY SHARE, and attempts are written before they are made. New sitemaps queue as of now, a website being read is skipped at both of its names, and a stopping engine releases the website as due.IMeasurementAllowance) is not read.GET /api/sites/{id}/sitemapreports the state, the problem word, the counts andturnedAway. Partly-read problems are told as being about the files the sitemap lists.AddSitemapReadingaddssites.sitemap_address,sitemap_reads,sitemap_partsandsite_posts(withlanded_path). The release must back up first.Tests
pnpm verify: web 2,092, tracker 84, edition 124 and site 78.🤖 Generated with Claude Code