char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Xuan-Son Nguyenmtmd : Support Pixtral 12B (#13065)ecda2ec4b

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
1.1 KiB51 linesraw

Gemma 3 vision

[!IMPORTANT]

This is very experimental, only used for demo purpose.

Quick started

You can use pre-quantized model from ggml-org's Hugging Face account

# build
cmake -B build
cmake --build build --target llama-mtmd-cli

# alternatively, install from brew (MacOS)
brew install llama.cpp

# run it
llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF
llama-mtmd-cli -hf ggml-org/gemma-3-12b-it-GGUF
llama-mtmd-cli -hf ggml-org/gemma-3-27b-it-GGUF

# note: 1B model does not support vision

How to get mmproj.gguf?

Simply to add --mmproj in when converting model via convert_hf_to_gguf.py:

cd gemma-3-4b-it
python ../llama.cpp/convert_hf_to_gguf.py --outfile model.gguf --outtype f16 --mmproj .
# output file: mmproj-model.gguf

How to run it?

What you need:

  • The text model GGUF, can be converted using convert_hf_to_gguf.py
  • The mmproj file from step above
  • An image file
# build
cmake -B build
cmake --build build --target llama-mtmd-cli

# run it
./build/bin/llama-mtmd-cli -m {text_model}.gguf --mmproj mmproj.gguf --image your_image.jpg
1# Gemma 3 vision
2
3> [!IMPORTANT]
4>
5> This is very experimental, only used for demo purpose.
6
7## Quick started
8
9You can use pre-quantized model from [ggml-org](https://huggingface.co/ggml-org)'s Hugging Face account
10
11```bash
12# build
13cmake -B build
14cmake --build build --target llama-mtmd-cli
15
16# alternatively, install from brew (MacOS)
17brew install llama.cpp
18
19# run it
20llama-mtmd-cli -hf ggml-org/gemma-3-4b-it-GGUF
21llama-mtmd-cli -hf ggml-org/gemma-3-12b-it-GGUF
22llama-mtmd-cli -hf ggml-org/gemma-3-27b-it-GGUF
23
24# note: 1B model does not support vision
25```
26
27## How to get mmproj.gguf?
28
29Simply to add `--mmproj` in when converting model via `convert_hf_to_gguf.py`:
30
31```bash
32cd gemma-3-4b-it
33python ../llama.cpp/convert_hf_to_gguf.py --outfile model.gguf --outtype f16 --mmproj .
34# output file: mmproj-model.gguf
35```
36
37## How to run it?
38
39What you need:
40- The text model GGUF, can be converted using `convert_hf_to_gguf.py`
41- The mmproj file from step above
42- An image file
43
44```bash
45# build
46cmake -B build
47cmake --build build --target llama-mtmd-cli
48
49# run it
50./build/bin/llama-mtmd-cli -m {text_model}.gguf --mmproj mmproj.gguf --image your_image.jpg
51```