Repository navigation
Conversation
`xmltodict` collapses a single-element list into a plain dict, so a feed
carrying exactly one `<item>`, or an item carrying exactly one
`<edgar:xbrlFile>`, made `parse_rss_feed_data` iterate over dict keys
instead of elements. Every item then failed with either:
TypeError: string indices must be integers, not 'str'
AttributeError: 'str' object has no attribute 'get'
Both are caught by the per-item `except`, so the command still exited 0
and wrote a header-only CSV -- no rows and no error status.
Parsing now pins the two repeated elements with `force_list`, which is
xmltodict's built-in for this, so no call site changes.
Latent against `xbrlrss.all.xml`, which normally carries many items each
with several files, but reachable on a quiet feed, on a filing with a
single XBRL file, or for anyone calling `parse_rss_feed_data` directly.
Fixes bellingcat#64
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
|
Hi @rex4539! Thank you for making this PR. I apologize for the wait. I've been out of town. I really appreciate the internal contribution. I will review/merge by tomorrow EOD. |
Author
|
Any update? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #64.
Problem
xmltodictcollapses a single-element list into a plain dict.parse_rss_feed_dataassumesitemandedgar:xbrlFileare always lists, so when the feed carries exactly one of either, the code iterates over dict keys and every item fails to parse:The per-item
exceptswallows both, so the command exits 0 and writes a header-only CSV — no rows, no error status.Fix
Pin the two repeated elements with xmltodict's own
force_list:No call-site changes: the loops already expect lists.
Tests
tests/test_rss_parsing.pyparametrisesparse_rss_feed_dataover the four combinations of many/one<item>× many/one<edgar:xbrlFile>.Against unpatched
main, the three single-element cases fail and themany-items-many-filescontrol passes — so the tests pin this bug specifically rather than merely being red. With the fix, all four pass. Full suite: 788 passed.One thing the tests document: when a filing has a single XBRL file,
xbrl_filesis a bare string rather than a one-element list, becauseunpack_singleton_listunwraps it on the way out. That is existing intended behaviour and this PR does not change it — worth knowing that the two singleton behaviours are deliberate in opposite directions.Relationship to #63
Independent, and branched from
mainrather than stacked, so the two can merge in either order. They touchrss.pyin different functions (parse_rss_feed_datahere,fetch_rss_feedthere) and add separate test files, so no conflict either way.Worth noting the two bugs compound: #63 means the RSS command cannot write rows at all, and this one means it silently parses zero rows on certain feed shapes. Both present as "empty output, exit 0", which is probably why neither surfaced as a bug report.