Skip to content

fix: alienvault multipage disk read + rate limit skip - #67

Merged
Runaho merged 1 commit into
developfrom
fix/alienvault-multipage-disk-read
May 28, 2026
Merged

Runaho merged 1 commit into
developfrom
fix/alienvault-multipage-disk-read

Conversation

@Runaho

@Runaho Runaho commented May 28, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes three issues in the AlienVault multipage fetch:

1. Disk read on resume without rate limit

When ResumePageNumber is greater than 1, pages 1..N are now read directly from disk and parsed, instead of re-fetching from the network. The rate limit sleep is skipped for disk reads.

2. Store NextPageURL in PageInfo metadata

SavePageData now accepts a nextPageURL parameter and stores it in the PageInfo struct. This allows accurate resume: on restart, the last saved next_page_url is used as the currentURL for the next network fetch, avoiding redundant re-fetching of already-saved pages.

3. Correct last-page detection for resumed fetches

Previously, a disk page with next_url empty could be mistaken for "no more pages". Now, when resuming (pageCount greater than or equal to startPage AND currentURL not equal to SourceURL), an empty response.Next is correctly identified as the last page of a previously interrupted run.

Changes

  • internal/utils/multipage.go: Added NextPageURL field to PageInfo; updated SavePageData signature to accept and persist nextPageURL
  • features/providers/alienvault/provider.go: Rewrote FetchPages resume logic to read and parse disk pages (skipping rate limit), then resume network fetch from the stored next_page_url

Copilot AI review requested due to automatic review settings May 28, 2026 08:15

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@Runaho
Runaho merged commit 21a7a21 into develop May 28, 2026
1 check failed
@Runaho
Runaho deleted the fix/alienvault-multipage-disk-read branch May 28, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants