[registry] Make Artifact Hub rate limit configurable#1058
[registry] Make Artifact Hub rate limit configurable#1058KumarNirupam1 wants to merge 2 commits into
Conversation
Signed-off-by: Kumar Nirupam <kumar.nirupam24@gmail.com>
Signed-off-by: Kumar Nirupam <kumar.nirupam24@gmail.com>
|
Yay, your first pull request! 👍 A contributor will be by to give feedback soon. In the meantime, you can find updates in the #github-notifications channel in the community Slack. |
There was a problem hiding this comment.
Code Review
This pull request makes the Artifact Hub rate limiting parameters configurable by adding ArtifactHubRequestLimit and ArtifactHubWaitDuration to GenerationOptions and updating RateLimitArtifactHub to accept these parameters. The review feedback highlights a concurrency issue where holding the artifactHubMutex lock during time.Sleep blocks other goroutines, and suggests adding a TODO comment to address this in a follow-up task.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
I'm placing this PR on-hold. |
Description
Relates to meshery/meshery #11034
Fixes the meshkit side of meshery#11034 — rate-limited model generation during
mesheryctl registry generate.Issue
The daily model generator CI runs
registry generateover many Artifact Hub models. When too many requests hit artifacthub.io at once, the API returns HTTP 429 and those models fail:status code 429 for https://artifacthub.io/api/v1/packages/search?...
meshkit already had basic rate limiting in
RateLimitArtifactHub()— sleep 5 minutes after every 100 Artifact Hub model starts — but both values were hardcoded globals. mesheryctl and the CI workflow had no way to configure a longer wait or a lower request limit when 429s show up.Solution (this PR)
Make the existing rate limit configurable through
GenerationOptions, without changing default behavior.registry/helpers.goArtifactHubRequestLimit— sleep after this many Artifact Hub model starts (default:100)ArtifactHubWaitDuration— how long to sleep when the limit is hit (default:5m)DefaultGenerationOptions()registry/model.goRateLimitArtifactHub(requestLimit, waitDuration)uses the passed values (falls back to 100 / 5m if zero)InvokeGenerationFromSheetWithOptionspassesopts.ArtifactHubRequestLimitandopts.ArtifactHubWaitDurationIf no options are set, behavior is identical to before. Callers (mesheryctl, once wired) can tune values for CI, e.g. limit
20and wait10mto sleep sooner and reduce 429s.Out of scope (follow-up)
model-generator.ymlworkflow inputs — needed to close meshery#11034 fullygenerators/artifacthub/scanner.go— separate improvement (#11161)semaphore.NewWeighted(20))Notes for Reviewers
Signed commits