Skip to content

fix: use image path placeholders for text-only models - #1750

Open
wangwingzero wants to merge 1 commit into
BigPizzaV3:mainfrom
wangwingzero:feat/image-path-placeholder-for-text-models
Open

fix: use image path placeholders for text-only models#1750
wangwingzero wants to merge 1 commit into
BigPizzaV3:mainfrom
wangwingzero:feat/image-path-placeholder-for-text-models

Conversation

@wangwingzero

Copy link
Copy Markdown

问题

纯文本模型(如 DeepSeek V4 Flash)不支持图片输入。旧逻辑在以下场景存在两个问题:

  1. strip images 只把图片替换成 [图片已省略],模型完全看不到图片,也无法调用读图 MCP 读取本地文件。
  2. VLM 配置不完整或 VLM 分析失败时,image_url 会被原样发给纯文本模型,导致 OpenAI-compatible 上游报错:
    unknown variant 'image_url', expected 'text'

改动

参考 cursor-byok 的图片路径占位实现,把文本模型的图片块替换为本地路径占位:

  • 新增 replace_images_with_path_placeholders,图片块替换为:
    [图片文件: 绝对路径] 用户刚刚发送了这张图片。请使用你可用的读图工具读取该文件路径来查看图片内容...
  • 让模型自行查询并调用可用的读图 MCP 工具(如 vision-luna 提供的 analyze_image 等),不再写死工具名。
  • 优先复用 Codex 桌面端已写入的 %TEMP%\codex-clipboard-*.png 路径。
  • data URL 解码后写入 %TEMP%\codex-plus-vision-images\;http(s) 图片下载后落盘;本地路径 canonicalize 为绝对路径。
  • strip 模式、VLM 配置不完整、VLM 当前轮失败均改为路径占位,不再向上游发送 image_url
  • 临时文件名加入进程内自增序号,避免并发请求写临时图片时撞名。

测试

  • cargo test -p codex-plus-core --lib:206 passed
  • cargo test -p codex-plus-core --test protocol_proxy:51 passed
  • cargo fmt --all -- --check / git diff --check:通过
  • npm run check / npm run vite:build:通过
  • cargo build --release:通过
  • 本地实际发送图片到 DeepSeek 成功,模型收到路径占位并调用看图 MCP。

涉及文件

  • crates/codex-plus-core/src/vision.rs
  • crates/codex-plus-core/src/protocol_proxy.rs
  • crates/codex-plus-core/tests/protocol_proxy.rs

Replace stripped image blocks with local file path placeholders so text-only models (e.g. DeepSeek V4 Flash) can invoke an image-reading MCP tool instead of receiving an unsupported image_url block.

- Prefer Codex clipboard paths under %TEMP% when present
- Decode data URLs and download http(s) images into %TEMP%\codex-plus-vision-images
- Fall back to path placeholders when VLM config is incomplete or VLM analysis fails
- Add integration tests for strip mode and incomplete VLM config
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant