Skip to content

feat(catalog): serve herus13 Apify actors on the platform key with capped billing - #657

Merged
JayZeeDesign merged 42 commits into
mainfrom
feat/apify-herus13-actors
Sep 26, 2026
Merged

JayZeeDesign merged 42 commits into
mainfrom
feat/apify-herus13-actors

Conversation

@JayZeeDesign

@JayZeeDesign JayZeeDesign commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Adds nine Apify tools (six Weibo modes, Google Maps, TikTok Shop, Lazada) and serves them — and the three Apify rows already in the catalog — on the platform key with billing bounded by Apify's own spend cap. Generic apify.web.scrape.job.start stays own-key only: its free price cannot meter an arbitrary actor.

Supersedes #641, #642, #643 and #644. This PR carries the code the rows need as well, so there is nothing to merge first.

Why the billing needed code

Every Apify per-result row settled at its estimate: run-sync-get-dataset-items answers a bare JSON array, which no settlement rule counted. Live runs also showed three things a per-row price alone cannot handle:

  • maxItems does not bind pay-per-event actors whose own input sets the row count (maxItems=1 with body max_items: 3 returned and billed 3 rows). The run option Apify enforces on every billed event is maxTotalChargeUsd.
  • A run stopping at its cap can bill one event it never pushed as a row (observed: 3 events, 2 rows).
  • usageTotalUsd trails a finished run by minutes (0.00005 at SUCCEEDED, 0.01505 45 s later), so it cannot be settlement evidence.

What changed

Call path (application/call/resolve.py, settle.py)

  • platform_request can pin queryParams.* singleton values; query values parse as plain ASCII only.
  • An Apify per_result call on the platform key must send maxTotalChargeUsd (above 0, at most $1, at least call_fee + three rows) and timeout (at most 90 s, and 30 s under call_timeout_s, so the run and its rows arrive before any wait gives up). Only maxTotalChargeUsd, maxItems, memory and timeout are accepted, each once; dataset-view options (limit, format, unwind) are refused because they would make returned rows disagree with billed events.
  • Hold = cap + cost.call_fee. Settlement = returned rows × row price + call_fee; within two rows of the hold, the hold (the caller's cap was reached).
  • Own-key calls are untouched: no guard, no metering.

Catalog (apify.yaml)

  • cost.call_fee (validated: Apify, per-result, USD) carries the per-run charge: Weibo start $0.00005 and Maps start $0.01 at pinned 1 GB, TikTok Shop two start events at its forced 2 GB ($0.004), LinkedIn jobs actor-start ($0.001 per job title × location searched: cost.call_fee_per names the body arrays that multiply the fee). Lazada has none: since the actor's 2026-09-25 pricing it bills no start event and no run compute to the caller.
  • Pins on the platform key: memory and timeout for the new rows; TikTok Shop scrape_product_details: false plus body_allowlist (keeps the pricier product-detail event out); meta-ads enrichWithEcommerceData: false (an extra event per ad).
  • LinkedIn jobs accepts only its declared body fields (body_allowlist, on every credential tier): the actor bills an actor-start per job title × location, which call_fee_per counts, and per geoIds entry, so geoIds is refused and places go in locations. Filters that do not multiply starts (company, industry, salary, workplace/employment/experience, Easy Apply) stay declared.
  • Notes name maxTotalChargeUsd as the spend limit. Lazada's actor, since 2026-09-25, refuses caps below $0.05 and returns at least 10 products.

Routing (adapters.yaml)

  • treg.google.serp.maps and treg.weibo.post.detail now include the Apify rows as candidates, the only two of these capabilities with a routing contract. The adapters fix the run's spend cap, timeout and memory so the child call passes the platform guard: Maps caps a routed call at 20 places (cap $0.25, no contact lookup), Weibo at one post (cap $0.05).

Tests and CI

  • Async ownership tests (tests/test_asynctasks.py, tests/test_pinned_read_scope.py) now exercise Bright Data and CompanyEnrich, because apify.web.scrape.job.start no longer serves on the platform key; a new test proves it never reaches the relay.
  • Frontend CI uploads Playwright traces and screenshots on failure.

Docs: docs/context/architecture/money.md ("Apify dataset-row settlement"), catalog.md.

Pricing on tiered actors

Google Maps, TikTok Shop and Meta ads price per Apify subscription tier. The catalog charges platform callers Apify's published default-plan (or, for Meta ads, BRONZE) rate, which is at or above what any paid plan pays, so the platform never bills below cost; own-key callers pay Apify directly at their own tier. This is deliberate list pricing, not an error.

Residual loss, bounded

A run that times out, fails or aborts answers 400 with no rows and releases; so does a caller disconnect or an answer over the 8 MiB evidence limit. Apify may still have billed events up to the cap, so the platform absorbs at most $1 per such call. Billing a timed-out run at the hold was tried and reverted: a run that timed out after its start event cost $0.00005 and would have billed the whole cap.

Operator precondition: the Apify account behind the platform key must keep general resource access Restricted. The 400 body names the run, and with public access anyone could read that unbilled run's dataset by id.

Evidence

Prices, 2026-09-26: every Apify per-result row's test_request ran on the platform key and Apify's chargedEventCounts matched its rate card; all twelve rows are now source: observed, confidence: verified.

Row Events billed Apify charge
Weibo (6 modes) 1 row event + start $0.00505 each
Google Maps 1 place + start rate-card price for the key's tier
TikTok Shop 1 search result + 2 starts at 2 GB rate-card price for the key's tier
Lazada 10 products (the actor's floor) $0.05, no platform usage
Meta ads 1 ad rate-card price for the key's tier
TikTok ads 1 ad $0.003
LinkedIn jobs 1 job + actor-start; 3 titles → 3 jobs + 3 actor-starts $0.002; $0.006 (treg billed $0.006)

TikTok ads was marked upstream-broken (empty datasets); it returns rows again and is verified with a captured example. Routed calls through the adapters: treg.google.serp.maps returned 20 places and billed $0.19; treg.weibo.post.detail returned the post and billed $0.00505.

Live on the Apify API:

  • Rate cards (pricingInfos) match every row's price, event and start fee.
  • Through a local server on the platform key, all twelve Apify per-result rows returned 201 and settled at rows × price + fee; in every run the caller was billed at least what Apify charged (e.g. Weibo $0.00505 vs $0.00505, a capped Weibo run $0.02005 vs $0.01505).
  • Refused before relay: missing cap, cap above $1 or below three rows, limit, format=csv, non-ASCII digits, repeated keys, timeout over 90, meta-ads enrichment on, LinkedIn geoIds, TikTok Shop product_urls/detail mode.

Reviewed in independent rounds with live verification; each blocking finding was fixed and re-reviewed.

Validation

  • Catalog validator: no Apify errors (the three openrouter.ai-judge.decide errors are on main).
  • Import contracts: 14 kept.
  • Full suite: the same 13 failures as main (dashboard assets not built locally, hub, MCP OAuth, referrals, SEO, single-user); no failures specific to this branch.

Follow-ups

  • TikTok ads: an undeclared only_total returns one free count row that settlement bills as a result (caller overpays $0.003; no platform loss). Declare it or allowlist the row.

  • Settle failed runs exactly from chargedEventCounts through a deferred settle.

  • A reconcile check that each custom-event actor still bills one event per returned row.

  • Accept an absent pinned field when the actor default equals the pin.

🤖 Generated with Claude Code

JayZeeDesign and others added 5 commits September 24, 2026 20:59
apify.web.scrape.job.start is priced free because the run bills later by the
actor's own pricing, and nothing on the shared-key path meters that run. On
treg's key it let any caller run any actor on treg's Apify account at no
charge. It now needs the team's own Apify key; job.status and job.results
stay open because per-team ownership already scopes them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ctors

Combines #641-#644 into one catalog change: nine run-sync entries over four
herus13 actors, plus the lazada platform. The six Weibo entries serve on
treg's key at a price observed on it. Lazada, Google Maps and TikTok Shop are
own-key only: their actors bill run compute or a per-GB start fee that a
per-result price cannot meter.

Co-authored-by: Herus13 <bootforge.ai@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the area:docs Documentation & design fragments label Sep 25, 2026
@JayZeeDesign JayZeeDesign changed the title feat(catalog): Apify actors for Weibo, TikTok Shop, Lazada, Google Maps; keep actor starts off treg's key feat(catalog): add own-key Apify actors and block unmetered starts Sep 25, 2026
@github-actions github-actions Bot added the area:ci CI / workflows / repo tooling label Sep 25, 2026
JayZeeDesign and others added 22 commits September 25, 2026 21:28
Some upstreams take their spend bounds as query options rather than body
fields (Apify's maxTotalChargeUsd, memory and timeout run options). A
queryParams pin must be sent exactly once and is compared as the pinned
value's type; own credentials keep the upstream contract.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
run-sync-get-dataset-items answers a bare array, so every Apify per_result
call settled at its estimate whatever it returned. Count the rows, add an
optional per-row-independent call_fee for the actor's start or compute
charge, and require maxItems (1-200) on the platform key so the hold is the
worst case. Apify's usageTotalUsd lags a finished run by minutes, so the
response body is the settlement evidence.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Weibo, Google Maps, TikTok Shop and Lazada now settle on the platform key by
counted dataset rows plus a flat call_fee (actor start, or Lazada's run
compute). memory and timeout are pinned so the fee is fixed and a run ends
before Apify's synchronous wait; TikTok Shop is held to keyword search.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
maxItems does not bind pay-per-event actors whose own input sets the row
count, so a one-row hold could settle thousands of rows. Require the
maxTotalChargeUsd run option Apify enforces (at most $1), hold it plus
call_fee, accept only the run options each once in plain ASCII, and bill the
hold when a run reaches its cap. Pin meta-ads enrichment off and bill the
LinkedIn actor-start event on treg's key.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Row notes name maxTotalChargeUsd as the enforced spend limit; Lazada's
compute fee is 0.015 under a 180-second timeout pin.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…orm runs

Past Apify's 300-second synchronous wait a run answers 408 and keeps billing,
so every Apify per_result platform call now names timeout <= 280. A cap under
call_fee plus three rows would bill an empty answer in full, so it is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
treg's upstream read timeout (call_timeout_s, 180 s) and the MCP client's
120 s end the call before a 280 s run finishes, releasing the hold unbilled
while the run keeps billing. Bound timeout to 90 s and 30 s under
call_timeout_s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s key

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A run that outlives its own timeout answers 400 run-failed with no rows, but
Apify billed its events up to the caller's cap and the run id in that body
reads the dataset. The caller chose the run's size and timeout, so the hold
settles instead of releasing. The minimum-cap check compares micro-USD.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ricted

Billing the whole cap on a TIMED-OUT 400 overcharged callers: a run that
timed out after its start event cost Apify $0.00005 and would have billed the
full cap. The loss treg absorbs stays bounded by the $1 cap, and keeping the
Apify account's resource access Restricted stops anyone reading the unbilled
run's rows by id. Examples now show the 90-second platform timeout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JayZeeDesign and others added 4 commits September 26, 2026 01:27
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tors-2

# Conflicts:
#	frontend/e2e/landing.spec.ts
#	tests/test_asynctasks.py
#	tests/test_marketplace_call.py
…s the merge dropped

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JayZeeDesign and others added 11 commits September 26, 2026 11:43
…tors-2

# Conflicts:
#	plugins/minimax/skills/treg/SKILL.md
#	plugins/treg/skills/treg/SKILL.md
#	skills/treg/SKILL.md
Each row's test_request ran on 2026-09-26 and Apify's chargedEventCounts
matched its rate card. The TikTok ad library actor returns rows again, so it
is verified with a captured example.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adapters let treg.google.serp.maps and treg.weibo.post.detail choose the
Apify actors, with the run's spend cap, timeout and memory fixed so the
platform guard accepts the child call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…of notes

The actor's 2026-09-25 pricing no longer bills run compute to the caller (a
79-second run showed no platform usage), so its call_fee over-charged every
call. Notes describe observations by price tier, not by the account that
made them, and carry no run ids.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tier

Notes for plan-tiered actors record the events billed and that they matched
the rate card for the key's tier, not the dollar figure that would name it;
the repeated Weibo observation and the owner-meter notes go.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The LinkedIn jobs actor bills its actor-start event for every job title x
location searched, so a flat call_fee under-billed any multi-query call.
cost.call_fee_per names the body arrays whose lengths multiply the fee. The
apify.yaml header no longer names a plan or calls the TikTok actor broken.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The actor bills an actor-start for every geo id it searches, and geoIds was
undeclared, so it passed through unbilled. The row now takes its declared
filters only (salary, easyApply, under10Applicants and industryIds added);
places go in locations, which call_fee_per counts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A run whose charges land exactly on maxTotalChargeUsd can be aborted by
Apify and answer 400 with no rows, which releases the hold. Lazada's
10-product floor costs exactly its 0.05 minimum cap, so its example now uses
0.06 and the notes say to set the cap above the expected spend.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@JayZeeDesign
JayZeeDesign merged commit b4ed925 into main Sep 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ci CI / workflows / repo tooling area:docs Documentation & design fragments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant