mineru-vl-utils

A Python package for interacting with the MinerU Vision-Language Model.

It's a lightweight wrapper that simplifies the process of sending requests and handling responses from the MinerU Vision-Language Model.

About Backends

We provides 7 different backends(deployment modes):

1. http-client: A HTTP client for interacting with the OpenAI-compatible model server.
2. transformers: A backend for using HuggingFace Transformers models. (slow but simple to install)
3. mlx-engine: A backend for using Apple Silicon devices with macOS.
4. lmdeploy-engine: A backend for using the LmDeploy engine.
5. vllm-engine: A backend for using the VLLM synchronous batching engine.
6. vllm-async-engine: A backend for using the VLLM asynchronous engine. (requires async programming)
7. llama-cpp-engine: A backend for using in-process llama.cpp VLM inference via mineru-llama-cpp — no HTTP server, no subprocess. Install with pip install mineru-vl-utils[llama-cpp].

Optional block error isolation

Ordinary batch prediction raises errors by default. To retain successful blocks when an individual inference request fails, pass isolate_block_errors=True to MinerUClient or the supported backend client. Failed blocks return an empty string and their backend, input index and exception are logged; successful results retain their input order. A batch in which every request failed raises the first input's original exception. A successfully generated empty string still counts as success.

The option supports HTTP, llama.cpp and LMDeploy synchronous/asynchronous batches, and vLLM asynchronous batches. Other backends reject an enabled option before loading models. Single predictions, scored predictions and streaming iterator methods keep their strict error behavior. Authentication, invalid endpoint, quota and HTTP server errors, closed clients, cancellation, memory errors and programming errors are not converted to empty results.

With LMDeploy, enabling isolation switches synchronous batches to independently scheduled requests through the existing asynchronous path, which can change batching throughput. Keep the default for the existing bulk inference behavior.

client = MinerUClient(
    backend="http-client",
    server_url="http://127.0.0.1:8000/v1",
    isolate_block_errors=True,
)

About Output Format

MinerU Vision-Language Model can handle document layout detection and text/table/equation recognition tasks in a same model.

The output of the model is a list of ContentBlock objects, each representing a detected block in the document with its content recognition results.

Each ContentBlock contains the following attributes:

• type (str): The type of the block, e.g., 'text', 'image', 'table', 'equation'.
  • For a complete list of supported block types, please refer to structs.py.
• bbox (list of floats): The bounding box of the block in the format [xmin, ymin, xmax, ymax], with coordinates normalized to the range [0, 1].
• angle (int or None): The rotation angle of the block, can be one of [0, 90, 180, 270].
  • 0 means upward.
  • 90 means rightward.
  • 180 means upside down.
  • 270 means leftward.
  • None means the angle is not specified.
• content (str or None): The recognized content of the block, if applicable.
  • For 'text' blocks, this is the recognized text.
  • For 'table' blocks, this is the recognized table in HTML format.
  • For 'equation' blocks, this is the recognized LaTeX code.
  • For 'image' blocks, this is None.

Installation

For http-client backend, just install the package via pip:

pip install -U mineru-vl-utils

The 2.0 development line requires Transformers 5.10.1 or newer within 5.x. Supported Python versions are 3.10–3.14. LMDeploy uses >=0.17.0,<0.18, vLLM uses >=0.19.1,<0.29.0, and MLX-VLM uses >=0.7.0,<0.8.0. MLX-VLM requires Transformers >=5.14; vLLM 0.19.1 excludes Python 3.14, which selects a newer release in the supported range.

Synchronous vLLM clients pass raw prompts to LLM.generate; asynchronous clients use rendered engine inputs. MLX and LMDeploy cancellation waits for in-flight synchronous work before releasing the caller’s model lease.

For transformers backend, install the package with the transformers extra:

pip install -U "mineru-vl-utils[transformers]"

For vllm-engine and vllm-async-engine backend, install the package with the vllm extra:

pip install -U "mineru-vl-utils[vllm]"

For mlx-engine backend, install the package with the mlx extra:

pip install -U "mineru-vl-utils[mlx]"

For lmdeploy-engine backend, install the package with the lmdeploy extra:

pip install -U "mineru-vl-utils[lmdeploy]"

For llama-cpp-engine backend, install the package with the llama-cpp extra:

pip install -U "mineru-vl-utils[llama-cpp]"

Note

For using the http-client backend, you still need to have another vllm(or other LLM deployment tool) environment to serve the model as a http server.

Serving the Model (Optional)

This is only needed if you want to use the http-client backend.

You can use vllm or another LLM deployment tool to serve the model. Here we only demonstrate how to use vllm to serve the model.

With vllm>=0.10.1, you can use following command to serve the model. The logits processor is used to support no_repeat_ngram_size sampling param, which can help the model to avoid generating repeated content.

vllm serve opendatalab/MinerU2.5-2509-1.2B --host 127.0.0.1 --port 8000 \
  --logits-processors mineru_vl_utils:MinerULogitsProcessor

If you are using vllm<0.10.1, no_repeat_ngram_size sampling param is not supported. You still can serve the model without logits processor:

vllm serve opendatalab/MinerU2.5-2509-1.2B --host 127.0.0.1 --port 8000

Using MinerUClient by Code

Now you can use the MinerUClient class to interact with the model. Following are examples of using different backends.

http-client Example

from PIL import Image
from mineru_vl_utils import MinerUClient

client = MinerUClient(
    backend="http-client",
    server_url="http://127.0.0.1:8000"
)

image = Image.open("/path/to/the/test/image.png")
extracted_blocks = client.two_step_extract(image)
print(extracted_blocks)

transformers Example

from mineru_vl_utils.transformers_loading import load_transformers_model, load_transformers_processor
from PIL import Image
from mineru_vl_utils import MinerUClient

# Requires transformers>=5.10.1,<6
model = load_transformers_model(
    "opendatalab/MinerU2.5-2509-1.2B",
    device_map="auto"
)

processor = load_transformers_processor("opendatalab/MinerU2.5-2509-1.2B")

client = MinerUClient(
    backend="transformers",
    model=model,
    processor=processor
)

image = Image.open("/path/to/the/test/image.png")
extracted_blocks = client.two_step_extract(image)
print(extracted_blocks)

mlx-engine Example

from mlx_vlm import load as mlx_load
from PIL import Image
from mineru_vl_utils import MinerUClient

model, processor = mlx_load("opendatalab/MinerU2.5-2509-1.2B")

client = MinerUClient(
    backend="mlx-engine",
    model=model,
    processor=processor
)

image = Image.open("/path/to/the/test/image.png")
extracted_blocks = client.two_step_extract(image)
print(extracted_blocks)

lmdeploy-engine Example

Version 2.0 accepts the public Pipeline from LMDeploy 0.17. The parameter name remains lmdeploy_engine. The context manager closes the engine after inference.

from lmdeploy import pipeline
from mineru_vl_utils import MinerUClient
from PIL import Image

if __name__ == "__main__":
    with pipeline("opendatalab/MinerU2.5-2509-1.2B") as lmdeploy_engine:
        client = MinerUClient(backend="lmdeploy-engine", lmdeploy_engine=lmdeploy_engine)
        image = Image.open("/path/to/the/test/image.png")
        print(client.two_step_extract(image))

For the PyTorch engine on CUDA:

from lmdeploy import PytorchEngineConfig, pipeline
from mineru_vl_utils import MinerUClient
from PIL import Image

if __name__ == "__main__":
    with pipeline(
        "opendatalab/MinerU2.5-2509-1.2B",
        backend_config=PytorchEngineConfig(device_type="cuda"),
    ) as lmdeploy_engine:
        client = MinerUClient(backend="lmdeploy-engine", lmdeploy_engine=lmdeploy_engine)
        image = Image.open("/path/to/the/test/image.png")
        print(client.two_step_extract(image))

Async calls execute Pipeline.infer in worker threads. Cancellation waits for an in-flight call to finish before releasing its concurrency slot, so callers can safely close the shared engine afterward. Legacy accelerator images remain on their previously validated 1.x stack until separately migrated.

vllm-engine Example

from vllm import LLM
from PIL import Image
from mineru_vl_utils import MinerUClient
from mineru_vl_utils import MinerULogitsProcessor  # if vllm>=0.10.1

llm = LLM(
    model="opendatalab/MinerU2.5-2509-1.2B",
    logits_processors=[MinerULogitsProcessor]  # if vllm>=0.10.1
)

client = MinerUClient(
    backend="vllm-engine",
    vllm_llm=llm
)

image = Image.open("/path/to/the/test/image.png")
extracted_blocks = client.two_step_extract(image)
print(extracted_blocks)

vllm-async-engine Example

import io
import asyncio
import aiofiles

from vllm.v1.engine.async_llm import AsyncLLM
from vllm.engine.arg_utils import AsyncEngineArgs
from PIL import Image
from mineru_vl_utils import MinerUClient
from mineru_vl_utils import MinerULogitsProcessor  # if vllm>=0.10.1

async_llm = AsyncLLM.from_engine_args(
    AsyncEngineArgs(
        model="opendatalab/MinerU2.5-2509-1.2B",
        logits_processors=[MinerULogitsProcessor]  # if vllm>=0.10.1
    )
)

client = MinerUClient(
  backend="vllm-async-engine",
  vllm_async_llm=async_llm,
)

async def main():
    image_path = "/path/to/the/test/image.png"
    async with aiofiles.open(image_path, "rb") as f:
        image_data = await f.read()
    image = Image.open(io.BytesIO(image_data))
    extracted_blocks = await client.aio_two_step_extract(image)
    print(extracted_blocks)

asyncio.run(main())

async_llm.shutdown()

llama-cpp-engine Example

This backend uses the in-process mineru_llama_cpp.Engine — no HTTP server needed. The caller creates and manages the Engine lifecycle.

from PIL import Image
from mineru_llama_cpp import Engine
from mineru_vl_utils import MinerUClient

with Engine("/path/to/model.gguf", "/path/to/mmproj.gguf") as engine:
    client = MinerUClient(backend="llama-cpp-engine", llama_cpp_engine=engine)
    image = Image.open("/path/to/the/test/image.png")
    extracted_blocks = client.two_step_extract(image)
    print(extracted_blocks)

Other APIs

Besides the two_step_extract method, MinerUClient also provides other APIs for interacting with the model. Following are the main APIs:

class MinerUClient:

    def layout_detect(self, image: Image.Image) -> list[ContentBlock]:
        ...

    def batch_layout_detect(self, images: list[Image.Image]) -> list[list[ContentBlock]]:
        ...

    async def aio_layout_detect(self, image: Image.Image) -> list[ContentBlock]:
        ...

    async def aio_batch_layout_detect(self, images: list[Image.Image]) -> list[list[ContentBlock]]:
        ...

    def two_step_extract(self, image: Image.Image) -> list[ContentBlock]:
        ...

    def batch_two_step_extract(self, images: list[Image.Image]) -> list[list[ContentBlock]]:
        ...

    async def aio_two_step_extract(self, image: Image.Image) -> list[ContentBlock]:
        ...

    async def aio_batch_two_step_extract(self, images: list[Image.Image]) -> list[list[ContentBlock]]:
        ...

Inference progress

MinerUClient(..., use_tqdm=True) enables inference progress by default, including local LMDeploy and remote HTTP batch extraction. Set use_tqdm=False to disable it. Progress is written to the terminal of the process running the client; it is not an HTTP response field.

Synchronous backend batch methods use the client instance setting. Supported asynchronous backend batch methods accept their own use_tqdm (default False) and tqdm_desc arguments; high-level MinerUClient batch methods pass the instance setting to the appropriate stage. MLX and Transformers honor these per-call settings without changing shared client state.

Default prediction bars and external-layout extraction use the same VLM Predict label across backends. Explicit tqdm_desc values still take precedence where supported. Synchronous vLLM keeps its native counters and speed display through a named progress factory. LMDeploy uses complete responses from its public streaming API when progress is enabled, updating once per successful request and restoring input order; response validation errors are raised after draining the batch. Both retain existing batch boundaries. HTTP extraction counts completed requests and preserves input order. Concurrent two-step extraction retains its page-level Two Step Extraction bar, with internal requests kept silent. Empty inference batches do not display progress, and skipped blocks do not count as inference requests. Loading models and preparing inputs are separate from inference completion.

Limitations

The transformers backend is slow and not suitable for production use.

The MinerUClient only supports standalone image(s) as input. PDF and DOCX files are not planned to be supported. Cross-page and cross-document operations are not planned to be supported, too.

For production use cases, please use MinerU, which is a more complete toolkit for document analyzing and data extraction.

MLX model path lifecycle

mineru_vl_utils.mlx_compat.prepare_mlx_model_path (2.0.1+) resolves a local path or Hugging Face repo and prepares Qwen compatibility configuration without modifying source weights:

from mineru_vl_utils.mlx_compat import prepare_mlx_model_path

with prepare_mlx_model_path("/path/to/model") as model_path:
    # Keep this context open for the entire server lifetime.
    run_server(model_path)

The context removes only its own temporary directory on exit, including failure paths. load_mlx_model uses the same preparation logic for local inference.

MLX engine batches

MLX engine batches use the public mlx-vlm BatchGenerator while preserving the existing MinerU chat template and image position:

client = MinerUClient("mlx-engine", model_path="/path/to/model", batch_size=8)
results = client.batch_content_extract(images)

batch_size=0 (the default) selects 8; batch_size=1 retains single-image generation. Inputs are grouped by effective sampling parameters and image/text modality, then sorted by pixel count. Results retain input order. Batches are limited by both the requested count and a 9,000,000-pixel input budget; an oversized image runs alone without resizing. This is a batching budget, not a hard memory limit: long contexts and outputs still consume KV cache memory.

Generation remains serialized with the same lock, including asynchronous calls; batching does not enable simultaneous GPU calls from multiple threads. Cancellation waits for in-flight work before releasing resources. Each sample gets independent penalty processors. As before, MLX does not implement no_repeat_ngram_size or priority scheduling. Batch decoding can produce small whitespace differences compared with sequential decoding.

The pixel budget accommodates eight 1036×1036 layout inputs (8,586,368 pixels). The default batch is 8; users can explicitly select batch_size=4, 2, or 1 to reduce memory use. Increasing the budget does not enable concurrent GPU calls or remove per-batch limits.

Asynchronous HTTP lifecycle

Reuse a MinerUClient on a running event loop to reuse its HTTP connection pool. Finish, or cancel and await, all outstanding extraction tasks before closing the client with await client.aclose(). Close it before its owning event loop stops. The existing synchronous HTTP batch methods close their temporary-loop connections before returning and can be called repeatedly.

from PIL import Image
from mineru_vl_utils import MinerUClient
from mineru_vl_utils.structs import ExtractResult

async def extract_document(client: MinerUClient, images: list[Image.Image]) -> list[ExtractResult]:
    """完成抽取并在所属事件循环结束之前释放 HTTP 连接。"""
    try:
        return await client.aio_batch_two_step_extract(images)
    finally:
        await client.aclose()

MinerUClient.aclose() closes owned HTTP connections. Engines and executors supplied by the caller remain caller-owned. Batch failures and cancellation drain child tasks and synchronous image preparation before returning, so callers can then release their input images. Canceling an HTTP request does not guarantee that the remote server immediately aborts generation.