hey! this is a static git repo viewer. commit diffs, file history, older commits and other revisions won't load without javascript.

you are still able to view source files on branch/tag tips, but the experience is degraded.

char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Xuan-Son Nguyenmtmd: refactor preprocessor, add mtmd_image_preproc_out (#24736)24bba7b982026-06-18

renderedcode

Multimodal

llama.cpp supports multimodal input via libmtmd. Currently, there are 2 tools support this feature:

llama-cli
llama-server via OpenAI-compatible /chat/completions API
llama-mtmd-cli, for testing and development

Currently, we support image, audio and video input.

To enable it, you can use one of the 2 methods below:

Use -hf option with a supported model (see a list of pre-quantized model below)
- To load a model using -hf while disabling multimodal, use --no-mmproj
- To load a model using -hf while using a custom mmproj file, use --mmproj local_file.gguf
Use -m model.gguf option with --mmproj file.gguf to specify text and multimodal projector respectively

By default, multimodal projector will be offloaded to GPU. To disable this, add --no-mmproj-offload

For example:

# simple usage with CLI
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF

# simple usage with server
llama-server -hf ggml-org/gemma-3-4b-it-GGUF

# using local file
llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf

# no GPU offload
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload

[!IMPORTANT]

OCR models are trained with specific prompt and input structure, please refer to these discussions for more info:

PaddleOCR-VL: https://github.com/ggml-org/llama.cpp/pull/18825

GLM-OCR: https://github.com/ggml-org/llama.cpp/pull/19677

Deepseek-OCR: https://github.com/ggml-org/llama.cpp/pull/17400

Dots.OCR: https://github.com/ggml-org/llama.cpp/pull/17575

HunyuanOCR: https://github.com/ggml-org/llama.cpp/pull/21395

Pre-quantized models

These are ready-to-use models, most of them come with Q4_K_M quantization by default. They can be found at the Hugging Face page of the ggml-org: https://huggingface.co/collections/ggml-org/multimodal-ggufs-68244e01ff1f39e5bebeeedc

Replaces the (tool_name) with the name of binary you want to use. For example, llama-mtmd-cli or llama-server

NOTE: some models may require large context window, for example: -c 8192

Vision models:

# Gemma 3
(tool_name) -hf ggml-org/gemma-3-4b-it-GGUF
(tool_name) -hf ggml-org/gemma-3-12b-it-GGUF
(tool_name) -hf ggml-org/gemma-3-27b-it-GGUF

# SmolVLM
(tool_name) -hf ggml-org/SmolVLM-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM-256M-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM-500M-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM2-256M-Video-Instruct-GGUF
(tool_name) -hf ggml-org/SmolVLM2-500M-Video-Instruct-GGUF

# Pixtral 12B
(tool_name) -hf ggml-org/pixtral-12b-GGUF

# Qwen 2 VL
(tool_name) -hf ggml-org/Qwen2-VL-2B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2-VL-7B-Instruct-GGUF

# Qwen 2.5 VL
(tool_name) -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2.5-VL-32B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen2.5-VL-72B-Instruct-GGUF

# Mistral Small 3.1 24B (IQ2_M quantization)
(tool_name) -hf ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF

# InternVL 2.5 and 3
(tool_name) -hf ggml-org/InternVL2_5-1B-GGUF
(tool_name) -hf ggml-org/InternVL2_5-4B-GGUF
(tool_name) -hf ggml-org/InternVL3-1B-Instruct-GGUF
(tool_name) -hf ggml-org/InternVL3-2B-Instruct-GGUF
(tool_name) -hf ggml-org/InternVL3-8B-Instruct-GGUF
(tool_name) -hf ggml-org/InternVL3-14B-Instruct-GGUF

# Llama 4 Scout
(tool_name) -hf ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUF

# Moondream2 20250414 version
(tool_name) -hf ggml-org/moondream2-20250414-GGUF

# Gemma 4
(tool_name) -hf ggml-org/gemma-4-E2B-it-GGUF
(tool_name) -hf ggml-org/gemma-4-E4B-it-GGUF
(tool_name) -hf ggml-org/gemma-4-26B-A4B-it-GGUF
(tool_name) -hf ggml-org/gemma-4-31B-it-GGUF

Audio models:

# Ultravox 0.5
(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF
(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_1-8b-GGUF

# Qwen2-Audio and SeaLLM-Audio
# note: no pre-quantized GGUF this model, as they have very poor result
# ref: https://github.com/ggml-org/llama.cpp/pull/13760

# Mistral's Voxtral
(tool_name) -hf ggml-org/Voxtral-Mini-3B-2507-GGUF

# Qwen3-ASR
(tool_name) -hf ggml-org/Qwen3-ASR-0.6B-GGUF
(tool_name) -hf ggml-org/Qwen3-ASR-1.7B-GGUF

Mixed modalities:

# Qwen2.5 Omni
# Capabilities: audio input, vision input
(tool_name) -hf ggml-org/Qwen2.5-Omni-3B-GGUF
(tool_name) -hf ggml-org/Qwen2.5-Omni-7B-GGUF

# Qwen3 Omni
# Capabilities: audio input, vision input
(tool_name) -hf ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF
(tool_name) -hf ggml-org/Qwen3-Omni-30B-A3B-Thinking-GGUF

# Gemma 4
# Capabilities: audio input, vision input
(tool_name) -hf ggml-org/gemma-4-E2B-it-GGUF
(tool_name) -hf ggml-org/gemma-4-E4B-it-GGUF

Finding more models:

GGUF models on Huggingface with vision capabilities can be found here: https://huggingface.co/models?pipeline_tag=image-text-to-text&sort=trending&search=gguf

1# Multimodal
2
3llama.cpp supports multimodal input via `libmtmd`. Currently, there are 2 tools support this feature:
4- [llama-cli](../tools/cli/README.md)
5- [llama-server](../tools/server/README.md) via OpenAI-compatible `/chat/completions` API
6- [llama-mtmd-cli](../tools/mtmd/README.md), for testing and development
7
8Currently, we support **image**, **audio** and **video** input.
9
10To enable it, you can use one of the 2 methods below:
11
12- Use `-hf` option with a supported model (see a list of pre-quantized model below)
13    - To load a model using `-hf` while disabling multimodal, use `--no-mmproj`
14    - To load a model using `-hf` while using a custom mmproj file, use `--mmproj local_file.gguf`
15- Use `-m model.gguf` option with `--mmproj file.gguf` to specify text and multimodal projector respectively
16
17By default, multimodal projector will be offloaded to GPU. To disable this, add `--no-mmproj-offload`
18
19For example:
20
21```sh
22# simple usage with CLI
23llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF
24
25# simple usage with server
26llama-server -hf ggml-org/gemma-3-4b-it-GGUF
27
28# using local file
29llama-server -m gemma-3-4b-it-Q4_K_M.gguf --mmproj mmproj-gemma-3-4b-it-Q4_K_M.gguf
30
31# no GPU offload
32llama-server -hf ggml-org/gemma-3-4b-it-GGUF --no-mmproj-offload
33```
34
35> [!IMPORTANT]
36>
37> OCR models are trained with specific prompt and input structure, please refer to these discussions for more info:
38> - PaddleOCR-VL: https://github.com/ggml-org/llama.cpp/pull/18825
39> - GLM-OCR: https://github.com/ggml-org/llama.cpp/pull/19677
40> - Deepseek-OCR: https://github.com/ggml-org/llama.cpp/pull/17400
41> - Dots.OCR: https://github.com/ggml-org/llama.cpp/pull/17575
42> - HunyuanOCR: https://github.com/ggml-org/llama.cpp/pull/21395
43
44## Pre-quantized models
45
46These are ready-to-use models, most of them come with `Q4_K_M` quantization by default. They can be found at the Hugging Face page of the ggml-org: https://huggingface.co/collections/ggml-org/multimodal-ggufs-68244e01ff1f39e5bebeeedc
47
48Replaces the `(tool_name)` with the name of binary you want to use. For example, `llama-mtmd-cli` or `llama-server`
49
50NOTE: some models may require large context window, for example: `-c 8192`
51
52**Vision models**:
53
54```sh
55# Gemma 3
56(tool_name) -hf ggml-org/gemma-3-4b-it-GGUF
57(tool_name) -hf ggml-org/gemma-3-12b-it-GGUF
58(tool_name) -hf ggml-org/gemma-3-27b-it-GGUF
59
60# SmolVLM
61(tool_name) -hf ggml-org/SmolVLM-Instruct-GGUF
62(tool_name) -hf ggml-org/SmolVLM-256M-Instruct-GGUF
63(tool_name) -hf ggml-org/SmolVLM-500M-Instruct-GGUF
64(tool_name) -hf ggml-org/SmolVLM2-2.2B-Instruct-GGUF
65(tool_name) -hf ggml-org/SmolVLM2-256M-Video-Instruct-GGUF
66(tool_name) -hf ggml-org/SmolVLM2-500M-Video-Instruct-GGUF
67
68# Pixtral 12B
69(tool_name) -hf ggml-org/pixtral-12b-GGUF
70
71# Qwen 2 VL
72(tool_name) -hf ggml-org/Qwen2-VL-2B-Instruct-GGUF
73(tool_name) -hf ggml-org/Qwen2-VL-7B-Instruct-GGUF
74
75# Qwen 2.5 VL
76(tool_name) -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF
77(tool_name) -hf ggml-org/Qwen2.5-VL-7B-Instruct-GGUF
78(tool_name) -hf ggml-org/Qwen2.5-VL-32B-Instruct-GGUF
79(tool_name) -hf ggml-org/Qwen2.5-VL-72B-Instruct-GGUF
80
81# Mistral Small 3.1 24B (IQ2_M quantization)
82(tool_name) -hf ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF
83
84# InternVL 2.5 and 3
85(tool_name) -hf ggml-org/InternVL2_5-1B-GGUF
86(tool_name) -hf ggml-org/InternVL2_5-4B-GGUF
87(tool_name) -hf ggml-org/InternVL3-1B-Instruct-GGUF
88(tool_name) -hf ggml-org/InternVL3-2B-Instruct-GGUF
89(tool_name) -hf ggml-org/InternVL3-8B-Instruct-GGUF
90(tool_name) -hf ggml-org/InternVL3-14B-Instruct-GGUF
91
92# Llama 4 Scout
93(tool_name) -hf ggml-org/Llama-4-Scout-17B-16E-Instruct-GGUF
94
95# Moondream2 20250414 version
96(tool_name) -hf ggml-org/moondream2-20250414-GGUF
97
98# Gemma 4
99(tool_name) -hf ggml-org/gemma-4-E2B-it-GGUF
100(tool_name) -hf ggml-org/gemma-4-E4B-it-GGUF
101(tool_name) -hf ggml-org/gemma-4-26B-A4B-it-GGUF
102(tool_name) -hf ggml-org/gemma-4-31B-it-GGUF
103```
104
105**Audio models**:
106
107```sh
108# Ultravox 0.5
109(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF
110(tool_name) -hf ggml-org/ultravox-v0_5-llama-3_1-8b-GGUF
111
112# Qwen2-Audio and SeaLLM-Audio
113# note: no pre-quantized GGUF this model, as they have very poor result
114# ref: https://github.com/ggml-org/llama.cpp/pull/13760
115
116# Mistral's Voxtral
117(tool_name) -hf ggml-org/Voxtral-Mini-3B-2507-GGUF
118
119# Qwen3-ASR
120(tool_name) -hf ggml-org/Qwen3-ASR-0.6B-GGUF
121(tool_name) -hf ggml-org/Qwen3-ASR-1.7B-GGUF
122```
123
124**Mixed modalities**:
125
126```sh
127# Qwen2.5 Omni
128# Capabilities: audio input, vision input
129(tool_name) -hf ggml-org/Qwen2.5-Omni-3B-GGUF
130(tool_name) -hf ggml-org/Qwen2.5-Omni-7B-GGUF
131
132# Qwen3 Omni
133# Capabilities: audio input, vision input
134(tool_name) -hf ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF
135(tool_name) -hf ggml-org/Qwen3-Omni-30B-A3B-Thinking-GGUF
136
137# Gemma 4
138# Capabilities: audio input, vision input
139(tool_name) -hf ggml-org/gemma-4-E2B-it-GGUF
140(tool_name) -hf ggml-org/gemma-4-E4B-it-GGUF
141```
142
143## Finding more models:
144
145GGUF models on Huggingface with vision capabilities can be found here: https://huggingface.co/models?pipeline_tag=image-text-to-text&sort=trending&search=gguf