Skip to content

[registry] Make Artifact Hub rate limit configurable#1058

Open
KumarNirupam1 wants to merge 2 commits into
meshery:masterfrom
KumarNirupam1:fix/artifact-hub-rate-limit-11034
Open

[registry] Make Artifact Hub rate limit configurable#1058
KumarNirupam1 wants to merge 2 commits into
meshery:masterfrom
KumarNirupam1:fix/artifact-hub-rate-limit-11034

Conversation

@KumarNirupam1

Copy link
Copy Markdown

Description

Relates to meshery/meshery #11034

Fixes the meshkit side of meshery#11034 — rate-limited model generation during mesheryctl registry generate.

Issue

The daily model generator CI runs registry generate over many Artifact Hub models. When too many requests hit artifacthub.io at once, the API returns HTTP 429 and those models fail:

status code 429 for https://artifacthub.io/api/v1/packages/search?...

meshkit already had basic rate limiting in RateLimitArtifactHub() — sleep 5 minutes after every 100 Artifact Hub model starts — but both values were hardcoded globals. mesheryctl and the CI workflow had no way to configure a longer wait or a lower request limit when 429s show up.

Solution (this PR)

Make the existing rate limit configurable through GenerationOptions, without changing default behavior.

registry/helpers.go

  • ArtifactHubRequestLimit — sleep after this many Artifact Hub model starts (default: 100)
  • ArtifactHubWaitDuration — how long to sleep when the limit is hit (default: 5m)
  • Defaults set in DefaultGenerationOptions()

registry/model.go

  • RateLimitArtifactHub(requestLimit, waitDuration) uses the passed values (falls back to 100 / 5m if zero)
  • InvokeGenerationFromSheetWithOptions passes opts.ArtifactHubRequestLimit and opts.ArtifactHubWaitDuration

If no options are set, behavior is identical to before. Callers (mesheryctl, once wired) can tune values for CI, e.g. limit 20 and wait 10m to sleep sooner and reduce 429s.

Out of scope (follow-up)

  • meshery: mesheryctl flags and model-generator.yml workflow inputs — needed to close meshery#11034 fully
  • meshkit: HTTP 429 retry in generators/artifacthub/scanner.go — separate improvement (#11161)
  • meshkit: changing parallel worker count (semaphore.NewWeighted(20))

Notes for Reviewers

  • 2 files changed; same mutex + counter logic, only parameterized
  • meshery PR to follow after meshkit release: bump meshkit + expose CLI/workflow knobs

Signed commits

  • Yes, I signed my commits.

Signed-off-by: Kumar Nirupam <kumar.nirupam24@gmail.com>
Signed-off-by: Kumar Nirupam <kumar.nirupam24@gmail.com>
@welcome

welcome Bot commented Jul 6, 2026

Copy link
Copy Markdown

Yay, your first pull request! 👍 A contributor will be by to give feedback soon. In the meantime, you can find updates in the #github-notifications channel in the community Slack.
Be sure to double-check that you have signed your commits. Here are instructions for making signing an implicit activity while performing a commit.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request makes the Artifact Hub rate limiting parameters configurable by adding ArtifactHubRequestLimit and ArtifactHubWaitDuration to GenerationOptions and updating RateLimitArtifactHub to accept these parameters. The review feedback highlights a concurrency issue where holding the artifactHubMutex lock during time.Sleep blocks other goroutines, and suggests adding a TODO comment to address this in a follow-up task.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread registry/model.go
@yi-nuo426 yi-nuo426 added the pr/on hold PR/Issue on hold label Jul 7, 2026
@yi-nuo426

Copy link
Copy Markdown
Contributor

I'm placing this PR on-hold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr/on hold PR/Issue on hold

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants