From 55c7e243e92abf179fe3e718dcd8e5ddea339e42 Mon Sep 17 00:00:00 2001 From: Seb Fox Date: Fri, 17 Jul 2026 10:06:30 +0200 Subject: [PATCH 1/2] Add amazon/order-history domain skill Field-tested pattern for extracting full order history from a logged-in Amazon session: timeFilter/startIndex URL scheme, .order-card extraction JS, and the traps (2024+ React layout returns no .order-card nodes, late hydration, buffered-stdout loss on long scrapes). Co-Authored-By: Claude Opus 4.8 (1M context) --- domain-skills/amazon/order-history.md | 53 +++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 domain-skills/amazon/order-history.md diff --git a/domain-skills/amazon/order-history.md b/domain-skills/amazon/order-history.md new file mode 100644 index 00000000..ee6531b0 --- /dev/null +++ b/domain-skills/amazon/order-history.md @@ -0,0 +1,53 @@ +# Amazon — Order History Extraction + +Field-tested against amazon.co.uk on 2026-07-16 using a logged-in session in the dedicated harness Chrome. Works the same on amazon.com (swap domain). + +## Auth + +Order history requires login. An unauthenticated visit to any orders URL redirects to +`/ap/signin` (title "Amazon Sign-In") — detect this via `page_info()["url"]` and stop for a +human login rather than typing credentials. A one-time headed login persists in the profile; +no re-challenge was observed on later headless runs from the same profile/IP. + +## URL pattern + +```python +goto("https://www.amazon.co.uk/your-orders/orders?timeFilter=year-2019&startIndex=0") +``` + +- `timeFilter` values come from the page's `#time-filter` dropdown: `last30`, `months-3`, + then `year-YYYY` going back to the account's first year (2010 accounts list every year). + Read them rather than guessing: + ```python + js("JSON.stringify(Array.from(document.querySelectorAll('#time-filter option')).map(o => o.value))") + ``` +- Pagination is `startIndex=0,10,20,...` — 10 orders per page. Stop when a page yields + fewer than 10 cards (or zero). + +## Extraction + +Each order renders as an `.order-card`. Titles are the product links inside it; the order +date is loose text matching `\d{1,2} Month \d{4}`: + +```python +data = js("""JSON.stringify(Array.from(document.querySelectorAll('.order-card')).map(c => { + const m = c.innerText.match(/(\\d{1,2}\\s+[A-Za-z]+\\s+\\d{4})/); + const titles = Array.from(new Set(Array.from( + c.querySelectorAll('.yohtmlc-product-title, a.a-link-normal[href*="/dp/"], a.a-link-normal[href*="/gp/product/"]') + ).map(a => a.innerText.trim()).filter(t => t && t.length > 2))); + return {date: m ? m[1] : '', titles}; +}))""") +``` + +## Traps + +- **Recent years render a different (React) layout with no `.order-card` nodes.** On a + 2010-2026 account, `year-2010`…`year-2023` extracted fine; `year-2024`+ returned zero + cards despite orders existing. If you need recent years, inspect the new DOM first — + don't interpret empty results as "no orders". +- **~1.5-2s settle after `wait_for_load()`** before querying cards; the list hydrates late. +- **Digital items** (Kindle/Prime Video) appear alongside physical ones; there is no + reliable in-card type marker — classify by title downstream. +- Long multi-year scrapes are slow (~2-3s/page, grocery-heavy years run 10+ pages). + Checkpoint results to disk after every year and `print(..., flush=True)` — a killed run + with buffered stdout loses everything. From 734a31f8b4a4a2b218e03dd666165fb5f0577d1b Mon Sep 17 00:00:00 2001 From: Seb Fox Date: Wed, 29 Jul 2026 21:54:58 +0100 Subject: [PATCH 2/2] Move skill under agent-workspace and make the scrape self-verifying Addresses the review findings, plus a path bug the review did not catch: the file was added at a new top-level domain-skills/, so the harness (which reads $BH_AGENT_WORKSPACE/domain-skills//) would never have loaded it. It now sits beside amazon/product-search.md. - new_tab() for the first visit, goto_url() after, matching product-search.md (goto() was a local alias and would have raised NameError) - json.loads() around the JSON.stringify payload so the result is a list - date regex accepts "31 December 2024" and "December 31, 2024", so the documented .com support no longer yields empty dates - replace the stop-on-a-short-page rule with .num-orders as an expected count, .a-pagination .a-last for the real end, and an assertion that fails loudly on a short scrape Also corrects the 2024+ "React layout" claim: re-testing on 2026-07-29 found year-2010 through year-2026 all extract with .order-card, so the earlier zero-card reading was a misdiagnosis, not a layout change. The count assertion is what actually guards against it. Verified end-to-end on year-2026: 57 expected, 6 pages, 57 collected, all dated. Co-Authored-By: Claude --- .../domain-skills/amazon/order-history.md | 111 ++++++++++++++++++ domain-skills/amazon/order-history.md | 53 --------- 2 files changed, 111 insertions(+), 53 deletions(-) create mode 100644 agent-workspace/domain-skills/amazon/order-history.md delete mode 100644 domain-skills/amazon/order-history.md diff --git a/agent-workspace/domain-skills/amazon/order-history.md b/agent-workspace/domain-skills/amazon/order-history.md new file mode 100644 index 00000000..9d3a8be2 --- /dev/null +++ b/agent-workspace/domain-skills/amazon/order-history.md @@ -0,0 +1,111 @@ +# Amazon — Order History Extraction + +Field-tested against amazon.co.uk on a 2010-2026 account (1,007 orders), first on +2026-07-16 and re-verified end-to-end on 2026-07-29. Complements +[product-search.md](product-search.md), which covers the public catalogue; this file covers the +logged-in `/your-orders` pages. + +## Auth + +Order history requires login. An unauthenticated visit to any orders URL redirects to +`/ap/signin` (title "Amazon Sign-In") — detect this and stop for a human login rather than +typing credentials: + +```python +if "/ap/signin" in page_info()["url"]: + raise RuntimeError("Amazon session not logged in — do a one-time headed login first") +``` + +A one-time headed login persists in the profile; no re-challenge was observed on later +headless runs from the same profile/IP. + +## Navigation + +First visit uses `new_tab()`, the same gotcha documented in product-search.md; `goto_url()` +is fine for every page after that. + +```python +new_tab("https://www.amazon.co.uk/your-orders/orders?timeFilter=year-2025&startIndex=0") +wait_for_load() +wait(2) # the order list hydrates after readyState=complete +``` + +- `timeFilter` takes `last30`, `months-3`, then `year-YYYY` back to the account's first year. + Read the valid values off the page rather than guessing: + ```python + years = json.loads(js( + "JSON.stringify(Array.from(document.querySelectorAll('#time-filter option')).map(o => o.value))" + )) + ``` +- `startIndex` steps in 10s (`0, 10, 20, ...`) — 10 orders per page. + +## Know the expected count before you paginate + +The page states its own total, so a scrape can verify itself instead of guessing when to +stop. This is the difference between a complete run and a short one that looks complete. + +```python +import re +count_text = js("document.querySelector('.num-orders')?.innerText") # '186 orders' / '1 order' +expected = int(re.sub(r"[^\d]", "", count_text or "0") or 0) +``` + +Stop on the pagination control, not on a short page — `.a-last` carries `a-disabled` on the +final page and the whole `.a-pagination` block is absent when a year fits on one page: + +```python +is_last_page = bool(js(""" + (function () { + const l = document.querySelector('.a-pagination .a-last'); + return !l || l.classList.contains('a-disabled'); + })() +""")) +``` + +Then assert what you collected against what the page promised, and fail loudly on a +mismatch rather than writing a truncated file: + +```python +if len(collected) != expected: + raise RuntimeError(f"year-{year}: got {len(collected)} of {expected} orders — investigate before trusting output") +``` + +Verified on `year-2026`: 57 expected, 6 pages, 57 collected, every order dated. + +## Extraction + +Each order is an `.order-card`. Titles are the product links inside it; the order date is +loose text in the card. `js()` hands back whatever the expression evaluates to, so a +`JSON.stringify` payload needs `json.loads` on the Python side to become a list: + +```python +import json + +page_orders = json.loads(js("""JSON.stringify(Array.from(document.querySelectorAll('.order-card')).map(c => { + const m = c.innerText.match(/(\\d{1,2}\\s+[A-Za-z]+\\s+\\d{4}|[A-Za-z]+\\s+\\d{1,2},\\s+\\d{4})/); + const titles = Array.from(new Set(Array.from( + c.querySelectorAll('.yohtmlc-product-title, a.a-link-normal[href*="/dp/"], a.a-link-normal[href*="/gp/product/"]') + ).map(a => a.innerText.trim()).filter(t => t && t.length > 2))); + return {date: m ? m[1] : '', titles}; +}))""")) +``` + +The date alternation covers both `31 December 2024` (co.uk) and `December 31, 2024` (.com); +a UK-only pattern silently yields `date: ''` for every order on the US site. + +## Gotchas + +- **Wait ~2s after `wait_for_load()`** before querying cards. `readyState=complete` fires + before the list renders, exactly as on search results. +- **Trust `.num-orders`, not an empty page.** A year returning zero cards is a signal to + inspect the DOM, never evidence of "no orders" — an earlier run on this account read + `year-2024`+ as empty and was wrong. Re-verified 2026-07-29: `year-2010` through + `year-2026` all extract with `.order-card`, 10 per page. +- **Don't stop on a short page alone.** The final page is legitimately short (7 of 10 on + `year-2026`), so a short-page-means-done rule is indistinguishable from a page that + under-rendered. Use `.a-last` plus the count assertion. +- **Digital items** (Kindle, Prime Video) sit alongside physical ones with no reliable + in-card type marker — classify by title downstream. +- **Long runs**: ~2-3s per page, and grocery-heavy years run 20+ pages. Checkpoint to disk + after each year and `print(..., flush=True)` — a killed run with buffered stdout loses + everything. diff --git a/domain-skills/amazon/order-history.md b/domain-skills/amazon/order-history.md deleted file mode 100644 index ee6531b0..00000000 --- a/domain-skills/amazon/order-history.md +++ /dev/null @@ -1,53 +0,0 @@ -# Amazon — Order History Extraction - -Field-tested against amazon.co.uk on 2026-07-16 using a logged-in session in the dedicated harness Chrome. Works the same on amazon.com (swap domain). - -## Auth - -Order history requires login. An unauthenticated visit to any orders URL redirects to -`/ap/signin` (title "Amazon Sign-In") — detect this via `page_info()["url"]` and stop for a -human login rather than typing credentials. A one-time headed login persists in the profile; -no re-challenge was observed on later headless runs from the same profile/IP. - -## URL pattern - -```python -goto("https://www.amazon.co.uk/your-orders/orders?timeFilter=year-2019&startIndex=0") -``` - -- `timeFilter` values come from the page's `#time-filter` dropdown: `last30`, `months-3`, - then `year-YYYY` going back to the account's first year (2010 accounts list every year). - Read them rather than guessing: - ```python - js("JSON.stringify(Array.from(document.querySelectorAll('#time-filter option')).map(o => o.value))") - ``` -- Pagination is `startIndex=0,10,20,...` — 10 orders per page. Stop when a page yields - fewer than 10 cards (or zero). - -## Extraction - -Each order renders as an `.order-card`. Titles are the product links inside it; the order -date is loose text matching `\d{1,2} Month \d{4}`: - -```python -data = js("""JSON.stringify(Array.from(document.querySelectorAll('.order-card')).map(c => { - const m = c.innerText.match(/(\\d{1,2}\\s+[A-Za-z]+\\s+\\d{4})/); - const titles = Array.from(new Set(Array.from( - c.querySelectorAll('.yohtmlc-product-title, a.a-link-normal[href*="/dp/"], a.a-link-normal[href*="/gp/product/"]') - ).map(a => a.innerText.trim()).filter(t => t && t.length > 2))); - return {date: m ? m[1] : '', titles}; -}))""") -``` - -## Traps - -- **Recent years render a different (React) layout with no `.order-card` nodes.** On a - 2010-2026 account, `year-2010`…`year-2023` extracted fine; `year-2024`+ returned zero - cards despite orders existing. If you need recent years, inspect the new DOM first — - don't interpret empty results as "no orders". -- **~1.5-2s settle after `wait_for_load()`** before querying cards; the list hydrates late. -- **Digital items** (Kindle/Prime Video) appear alongside physical ones; there is no - reliable in-card type marker — classify by title downstream. -- Long multi-year scrapes are slow (~2-3s/page, grocery-heavy years run 10+ pages). - Checkpoint results to disk after every year and `print(..., flush=True)` — a killed run - with buffered stdout loses everything.