char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Xuan-Son Nguyenmtmd : rename llava directory to mtmd (#13311)9b61acf06

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
1.5 KiB43 linesraw

GLMV-EDGE

Currently this implementation supports glm-edge-v-2b and glm-edge-v-5b.

Usage

Build the llama-mtmd-cli binary.

After building, run: ./llama-mtmd-cli to see the usage. For example:

./llama-mtmd-cli -m model_path/ggml-model-f16.gguf --mmproj model_path/mmproj-model-f16.gguf

note: A lower temperature like 0.1 is recommended for better quality. add --temp 0.1 to the command to do so. note: For GPU offloading ensure to use the -ngl flag just like usual

GGUF conversion

  1. Clone a GLMV-EDGE model (2B or 5B). For example:
git clone https://huggingface.co/THUDM/glm-edge-v-5b or https://huggingface.co/THUDM/glm-edge-v-2b
  1. Use glmedge-surgery.py to split the GLMV-EDGE model to LLM and multimodel projector constituents:
python ./tools/mtmd/glmedge-surgery.py -m ../model_path
  1. Use glmedge-convert-image-encoder-to-gguf.py to convert the GLMV-EDGE image encoder to GGUF:
python ./tools/mtmd/glmedge-convert-image-encoder-to-gguf.py -m ../model_path --llava-projector ../model_path/glm.projector --output-dir ../model_path
  1. Use examples/convert_hf_to_gguf.py to convert the LLM part of GLMV-EDGE to GGUF:
python convert_hf_to_gguf.py ../model_path

Now both the LLM part and the image encoder are in the model_path directory.

1# GLMV-EDGE
2
3Currently this implementation supports [glm-edge-v-2b](https://huggingface.co/THUDM/glm-edge-v-2b) and [glm-edge-v-5b](https://huggingface.co/THUDM/glm-edge-v-5b).
4
5## Usage
6Build the `llama-mtmd-cli` binary.
7
8After building, run: `./llama-mtmd-cli` to see the usage. For example:
9
10```sh
11./llama-mtmd-cli -m model_path/ggml-model-f16.gguf --mmproj model_path/mmproj-model-f16.gguf
12```
13
14**note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so.
15**note**: For GPU offloading ensure to use the `-ngl` flag just like usual
16
17## GGUF conversion
18
191. Clone a GLMV-EDGE model ([2B](https://huggingface.co/THUDM/glm-edge-v-2b) or [5B](https://huggingface.co/THUDM/glm-edge-v-5b)). For example:
20
21```sh
22git clone https://huggingface.co/THUDM/glm-edge-v-5b or https://huggingface.co/THUDM/glm-edge-v-2b
23```
24
252. Use `glmedge-surgery.py` to split the GLMV-EDGE model to LLM and multimodel projector constituents:
26
27```sh
28python ./tools/mtmd/glmedge-surgery.py -m ../model_path
29```
30
314. Use `glmedge-convert-image-encoder-to-gguf.py` to convert the GLMV-EDGE image encoder to GGUF:
32
33```sh
34python ./tools/mtmd/glmedge-convert-image-encoder-to-gguf.py -m ../model_path --llava-projector ../model_path/glm.projector --output-dir ../model_path
35```
36
375. Use `examples/convert_hf_to_gguf.py` to convert the LLM part of GLMV-EDGE to GGUF:
38
39```sh
40python convert_hf_to_gguf.py ../model_path
41```
42
43Now both the LLM part and the image encoder are in the `model_path` directory.