Skip to content

fix(Endpoints CLI): Ensure compatibility when deploying with specific model configs - #591

Open
blainekasten wants to merge 2 commits into
mainfrom
blaine/eng-93331-fix-bug-with-deploying-lora-configs
Open

blainekasten wants to merge 2 commits into
mainfrom
blaine/eng-93331-fix-bug-with-deploying-lora-configs

Conversation

@blainekasten

Copy link
Copy Markdown
Collaborator

No description provided.

@broly-code-security-scanner

Copy link
Copy Markdown

Broly Security Scan

Note

Clean scan
No vulnerabilities detected in this PR.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (GLM-5.3-Flash) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

Ask in #security-engineering   Powered by Together AI

fallback = await _resolve_explicit_config_for_public_model(
config,
public_model_name=public_model.name,
reference_model_id=public_model.base_model_id or _profile_model_id(candidate_profiles[0]),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why candidate_profiles[0]?

model with per-quantization profiles can point at different model ids (BF16 → ml_pub, FP8 → ml_fp8), so when base_model_id is unset we only search configs under whichever profile the API happened to list first. A legitimate LoRA config whose reference model is ml_fp8 would come back empty and the user gets "Config X is not valid for model Y" — the same misleading error we're fixing, just in a narrower case. Could we prefer public_model.base_model (the resource path, like resolve_model_reference does) and/or query the distinct profile model ids rather than just the first?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants