fix cleaning the wrong top node - #896
Open
idoshamun wants to merge 26 commits into
Open
Conversation
Before this change, `top_node` was cleaned and then copied to the `clean_top_node`. I believe this is not the original intent and should be fixed because it creates confusion and there's no way to get the raw `top_node`
Signed-off-by: Viktor Poluksht <vpoluksht@gmail.com>
Signed-off-by: Viktor Poluksht <vpoluksht@gmail.com>
Signed-off-by: Viktor Poluksht <vpoluksht@gmail.com>
final_url stores the actual url used to fetch the html after redirects and meta refresh
feat: added verify_ssl_cert option to pass to requests for proxy use
Signed-off-by: Viktor Poluksht <viktor@daily.dev>
feat: use og:title if title is absent
Medium serves two <title> tags on article pages, one of which is a bare site name (<title>Medium</title>). In the non-JS HTML our scraper receives that bare tag can come first, so get_title() picked "Medium" as the post title. Instead of blindly using title_element[0], choose the best candidate among all <title> tags: prefer the one whose filtered text contains og:title, otherwise the longest. Single-title pages are unaffected.
fix(title): pick best <title> when a page has multiple (Medium fix)
#12) get_publishing_date was wrong in both directions, and both faults trace to one line: it scanned the whole URL with STRICT_DATE_REGEX, which captures the separators around its match. It missed real dates. A '/2014/04/' path arrived as '2014/04/' and dateutil raises on the trailing slash, so every year-month URL fell through. The comment blamed the missing day specifier and sent everyone looking in the wrong place — date_parser('2014/04') is fine, date_parser('2014/04/') is not. Replayed over a sample of 20 real year-month URLs, the old code parsed 0 of them. And it invented dates that were not there, which is the worse direction: 'kali-linux-2026-1-release' and a '...-2026-07-...' API version both look like dates to a regex that scans anywhere in a string. Those only ever came out right by accident, because the same trailing separator that hid the real dates also hid them. The discriminator is position, not shape: a date in a URL owns the start of its path segment, while a version buried in a slug does not. _get_url_date walks path segments instead of scanning. Metadata extraction gains schema.org JSON-LD datePublished, <time> elements the document marks as the publication date, and three more meta names. Over 22 sampled pages that plainly state a date, the existing 11-tag list found 6. JSON-LD is where most current site generators put it. Both are tried after the existing tags, so no page that has a date today changes its answer. Partial dates are now anchored to an explicit default. Given '2014/04' dateutil fills the missing day from TODAY, so the same page yielded a different date depending on when it was parsed and a re-extraction silently moved the answer. Context: downstream of this, daily.dev's claim ledger holds 11,080 claims dated from the day a page was crawled rather than published, because a page with no recoverable date leaves consumers falling back to crawl time. On an archive import that is wrong by the article's whole age.
The JSON-LD strategy added in #12 could never fire through Article.parse(). parse() handed get_publishing_date the CLEANED document, and the document cleaner's remove_scripts_styles has already stripped every <script> tag — which is where schema.org JSON-LD lives. It failed silently, which is what made it expensive. The URL and <meta> strategies still worked, so a page carrying its date only in JSON-LD came back undated and was indistinguishable from a page that states no date anywhere. Measured on a 300-row prod slice of undated posts: 46 dated, 254 reported as "page states no date" — and the first one inspected by hand, a laravel-news article, had `"datePublished": "2026-02-16T09:00:00-05:00"` sitting in its stored HTML the whole time. Every other strategy reads <head>, which the cleaner leaves alone, so the raw doc is a superset for this purpose. #12's tests could not catch this: they all called get_publishing_date directly with a freshly parsed doc, which is precisely the path that works. The test added here goes through Article.parse() instead, and fails without the one-line change.
The URL was the first date strategy, so once year-month paths started parsing, '/2019/07/should-i-open-source-my-company' beat the page's own article:published_time of 2022-03-25 and the post moved to 2019-07-01 -- a date that is 2019 only because the slug lives under that folder. A year-month URL is strictly coarser than a datePublished the page states about itself: the day is invented, rounded to the 1st. So it now runs last, after the meta tags, JSON-LD and <time>, and only supplies a date when nothing else did. A URL that names the day keeps its place at the front -- it is as precise as the page's claim, and unlike template metadata it is per-article. Measured on the corpus this feeds: 62,245 stored posts have a year-month-only URL alongside a precise date, and would have moved earlier by a median of 15 days on a re-date.
…tamps (#16) Meta names in PUBLISH_DATE_TAGS are matched by SUBSTRING — getElementsByTag builds an xpath `contains(@name, value)`. That is survivable for the long, specific names in the original list and lethal for a short generic one: the `date` entry added in #12 matches `name="last-updated"`, because "updated" contains "date". Measured on a real fixture. A dev.to article whose JSON-LD states `datePublished: 2021-08-02T12:38:11Z` came back as `2024-01-12 13:13:27`, its modification stamp, because the meta strategy runs before JSON-LD and matched the wrong tag. Two other fixtures in the same suite were correct only by accident — they have no modification meta to collide with. A confidently wrong date is worse than no date, which is the whole premise of the ledger this feeds, so the three generic names (`date`, `dc.date`, `dcterms.created`) are removed rather than reordered. They would need equality matching to be safe, and JSON-LD — which now actually runs, after #14 — already covers the pages they were speculatively added for. The three cleaner fixtures that were failing now all produce their true publication dates: dev.to 2021-08-02, blog.suhailkakar.com 2021-08-27, producthunt 2022-04-14.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change,
top_nodewas cleaned and then copied to theclean_top_node.I believe this is not the original intent and should be fixed because it creates confusion and there's no way to get the raw
top_node. It's an issue I'm currently facing myself.Thanks for your awesome work!