Multimodal Model Selection Guide

Multimodal LLMs are evolving from "can see images" to "can perform multi-step tasks around images." Multiple Chinese open families now offer image-text understanding, but their positioning and toolchains differ significantly. This guide compares four families side by side and shows how to choose by scenario.

Four families' image-text product lines

Four dimensions for selection

1. Input format. Different models accept different input types: single image, multiple images, video frames, RGBA. If your task requires video-frame processing or multi-image correlation, confirm support first.

2. Editing versus understanding. Some models focus on "understanding the image and answering questions" (understanding type); others focus on "modifying image content" (editing type). Qwen-Image-2.1 and MiniMax H3 have editing capability, but they are image generation models, not multimodal understanding LLMs. Confirm whether your task is understanding or generating.

3. Deployment toolchain. Multimodal models typically load through the transformers library, but some require additional vision processors or specialized inference engines. Confirm your framework supports the architecture before deploying.

4. OCR versus general VL. If your task is precise document OCR (table extraction, receipt recognition), a specialized OCR model (like ERNIE's Unlimited-OCR or DeepSeek-OCR-2) beats general VL for both accuracy and cost. General VL suits "look at the image and reason" scenarios.

How to benchmark

Same principle as embedding selection — use your own data:

1. Prepare test images. Pick 10-20 representative images from your real scenario (different types, qualities, languages). Write 2-3 questions you expect the model to answer for each.

2. Run inference. Send the same images and questions to each candidate model and evaluate answers manually. Focus on: correct identification of key information, spatial reasoning, and useful accurate responses.

3. Compare deployment cost. Record inference latency and VRAM usage for each model to assess hardware feasibility.

Understanding input formats

Multimodal models differ significantly in what input types they accept:

• Single image + text: the most common baseline, supported by all four families
• Multiple images + text: needed for comparing images or referencing multiple sources in one query; not all models support this
• Video frames: some models accept sequential frames for temporal reasoning; this is less common in open models
• RGBA/transparent images: only Qwen-Image-2.1 supports native RGBA generation; other models process opaque images only

Before committing to a model, list every input type your workload will actually send. A model that handles single images well but does not support multi-image correlation will fail on your specific use case, even if it scores well on general benchmarks.

The OCR specialist versus general VL tradeoff

This is the most common decision point in multimodal model selection:

If your workload is 80% document extraction, use an OCR specialist. If it is 80% visual reasoning, use a general VL. If it is mixed, use both and route by task type — this is the same task-routing principle described in the agent framework article.

A concrete example: receipt extraction

To make the selection concrete, consider a receipt-extraction task:

1. The task: extract merchant name, date, items, quantities, and total from receipt photos.
2. Best fit: an OCR specialist (ERNIE's Unlimited-OCR or DeepSeek-OCR-2) — this is exactly what they are trained for.
3. If follow-up reasoning is needed: send the extracted structured data to a general text LLM for interpretation or categorization.

Using a general VL model for this task would work but be slower and less accurate. Using a general text LLM on the raw image would be worse still. This is why matching model type to task type matters — the OCR specialist is not a lesser model, it is a purpose-built one.

If your system also needs to answer free-form questions about receipts ("why was this receipt flagged as unusual?"), a general VL model handles that while the OCR specialist handles extraction. Routing between them by task type gives the best of both.

Practical benchmarking steps

Beyond the four selection dimensions, here is a concrete method to evaluate multimodal models:

1. Build a test image set. Pick 10-20 images from your real scenario: screenshots, charts, diagrams, handwritten notes, product photos, and documents with mixed languages. Cover edge cases like low resolution, unusual aspect ratios, and multi-page documents.

2. Write evaluation questions. For each image, write 2-3 questions that matter to your use case. Include both factual questions ("What is the total in this receipt?") and reasoning questions ("Why is this chart showing a decline?").

3. Score manually. Run each model on all images and questions, then score each answer on a simple scale (correct / partially correct / wrong / refused). This takes an hour but gives you real data instead of marketing claims.

4. Compare across dimensions. Look at which model handles which image type best. Some models are better at charts; others at natural photos; others at documents with mixed languages. Match the model to your dominant image type.

5. Test deployment. For models that pass the quality bar, test actual deployment: VRAM usage, inference latency, and whether your inference framework supports the architecture.

Local deployment notes

Multimodal local deployment is more complex than pure text:

• Vision processor: you need to load an additional image processor, not just the tokenizer
• VRAM: multimodal models typically use 20-50% more VRAM than same-parameter-count text models
• Inference framework: confirm your framework (vLLM, SGLang) supports the model's multimodal architecture
• Preprocessing: different models have different image preprocessing requirements (resolution, normalization)

FAQ

Related reading

• Running Qwen-Image-2.1 Locally — image generation model deployment
• MiniMax Family Overview — M3 multimodal line background
• Baidu ERNIE Family Overview — ERNIE OCR/VL product lines
• Embedding Model Selection Guide — multimodal vector retrieval
• Model Catalog — currently available models