hey! this is a static git repo viewer. commit diffs, file history, older commits and other revisions won't load without javascript.

you are still able to view source files on branch/tag tips, but the experience is degraded.

char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Xuan-Son Nguyenmodel: use ggml_rope_set_offset() (#27382)873e5d8e32026-08-21

renderedcode

Add a new model architecture to `llama.cpp`

Adding a model requires few steps:

Convert the model to GGUF
Define the model architecture in llama.cpp
Build the GGML graph implementation
Optional: Add multimodal encoder implementation

After following these steps, you can open PR.

Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially:

1. Convert the model to GGUF

This step is done in python with a convert script using the gguf library. Depending on the model architecture, you can use either convert_hf_to_gguf.py or examples/convert_legacy_llama.py (for llama/llama2 models in .pth format).

The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors.

The required steps to implement for an HF model are:

Define the model ModelBase.register annotation in a new TextModel or MmprojModel subclass in the conversion folder, example:

@ModelBase.register("MyModelForCausalLM")
@ModelBase.example("user/model")
class MyModel(TextModel):
    model_arch = gguf.MODEL_ARCH.MYMODEL

@ModelBase.register("MyModelForConditionalGeneration")
@ModelBase.example("user/model")
class MyModel(MmprojModel):
    model_arch = gguf.MODEL_ARCH.MYMODEL

The example should point to a valid Hugging Face model that will be used for testing. You can add multiple models if necessary. Prefer a non-gated model, or tiny random weights if no such model exists.

Define the layout of the GGUF tensors in constants.py

Add an enum entry in MODEL_ARCH, the model human friendly name in MODEL_ARCH_NAMES and the GGUF tensor names in MODEL_TENSORS.

NOTE: Pick the GGUF arch string (and the matching src/models/<name>.cpp filename, see section 3) carefully up front, following existing naming conventions. Once GGUF files are published under a given arch string, renaming it later breaks the community's existing files, so this is not something to leave for cleanup in a follow-up PR.

Example for falcon model:

    MODEL_ARCH.FALCON: [
        MODEL_TENSOR.TOKEN_EMBD,
        MODEL_TENSOR.OUTPUT_NORM,
        MODEL_TENSOR.OUTPUT,
        MODEL_TENSOR.ATTN_NORM,
        MODEL_TENSOR.ATTN_NORM_2,
        MODEL_TENSOR.ATTN_QKV,
        MODEL_TENSOR.ATTN_OUT,
        MODEL_TENSOR.FFN_DOWN,
        MODEL_TENSOR.FFN_UP,
    ]

Map the original tensor names to the standardize equivalent in GGUF

As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist.

Once you have found the GGUF tensor name equivalent, add it to the tensor_mapping.py file.

If the tensor name is part of a repetitive layer/block, the key word bid substitutes it.

Example for the normalization tensor in attention layers:

block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {
        # Attention norm
        MODEL_TENSOR.ATTN_NORM: (
            "gpt_neox.layers.{bid}.input_layernorm",                # gptneox
            "transformer.h.{bid}.ln_1",                             # gpt2 gpt-j refact qwen
            "transformer.blocks.{bid}.norm_1",                      # mpt
            ...
        )
}

transformer.blocks.{bid}.norm_1 will be mapped to blk.{bid}.attn_norm in GGUF.

Depending on the model configuration, tokenizer, code and tensors layout, you will have to override:

TextModel#set_gguf_parameters
MmprojModel#set_gguf_parameters
ModelBase#set_vocab
ModelBase#modify_tensors

NOTE: Tensor names must end with .weight or .bias suffixes, that is the convention and several tools like quantize expect this to proceed the weights.

2. Define the model architecture in `llama.cpp`

The model params and tensors layout must be defined in llama.cpp source files:

Define a new llm_arch enum value in src/llama-arch.h.
In src/llama-arch.cpp:
- Add the architecture name to the LLM_ARCH_NAMES map.
- You may also need to update LLM_KV_NAMES, LLM_TENSOR_NAMES and LLM_TENSOR_INFOS
Add any non-standard metadata loading in the llama_model_loader constructor in src/llama-model-loader.cpp.
If the model has a RoPE operation, add a case for the architecture in llama_model_rope_type function in src/llama-model.cpp.
Check for other places that switch/iterate over every llm_arch value, e.g. src/llama-model-saver.cpp and any mandatory-hparam lists (such as which archs require MoE metadata). Grep for LLM_ARCH_ usages to find them. Missing one of these is a common cause of CI test failures (e.g. test-llama-archs) after adding a new arch.

NOTE: The dimensions in ggml are typically in the reverse order of the pytorch dimensions.

3. Build the GGML graph implementation

This is the funniest part, you have to provide the inference graph implementation of the new model architecture in src/llama-model.cpp:

Create a new struct that inherits from llama_model_base.
Implement the graph-building logic in its build_arch_graph method.
The build_arch_graph method should return a constructed graph (inherited from llm_graph_context). Have a look at existing implementations like llama_model_llama, llama_model_dbrx or llama_model_bert.
Then, in the llama_model_mapping function, add a case for your architecture to instantiate your new graph-building struct.

Some ggml backends do not support all operations. Backend implementations can be added in a separate PR.

Note: to debug the inference graph: you can use llama-eval-callback.

4. Optional: Add multimodal encoder implementation

If the new model supports multimodal inputs, you will need to add a new encoder definition in libmtmd. You can find more information about llama.cpp's multimodal support in the docs and in the tools/mtmd source directory.

In the conversion script, make sure you add a subclass that extends MmprojModel or another class that inherits from the same base class.
Add the encoder definition in clip.cpp.
Implement the preprocessor in mtmd.cpp. In most cases, you can reuse an existing preprocessor.
Implement the encoder GGML graph, either in a dedicated file if the model is truly different from existing ones, or by reusing an existing implementation (for example: siglip, pixtral, or qwen) and adding a model-specific projector.

Note:

Many multimodal encoders are based on models that are already supported. Make sure to read the existing encoder definitions in tools/mtmd/models before adding a new one. In libmtmd, it is generally better to extend an existing model than to duplicate code.
To debug the multimodal preprocessor and encoder, you can use llama-mtmd-debug.
Adding a model-specific API or CLI is an anti-pattern in libmtmd. The goal of libmtmd is to provide an easy-to-use, model-agnostic library for multimodal pipeline.
In most cases, llama-mtmd-cli should not be modified. If a model requires a specific prompt, either let the user provide it or bake it into the Jinja chat template.
For audio generation models, see tools/mtmd/README-dev.md

Tips and tricks

Prefer conversion-time tensor modifications over graph-time ones

If the model contains constant modifications of tensors in the graph (for example, norm(1 + weight)) or performs tensor permutations/chunking, perform the modifications during conversion rather than in the graph code. This keeps the inference graph simpler and avoids extra runtime ops.

Examples:

Gemma 3 folds the 1 + of its norm(1 + weight) normalization into the weights at conversion time, so the graph just does a plain RMS norm.
Qwen3-Next applies its tensor permutation during conversion (in modify_tensors), so the graph can consume the already-permuted weights directly.

Exception: a plain weight * scale with a constant scale is usually better left to inference time rather than folded into the weight at conversion. The scale conceptually applies to the activation, not the weight, so folding it into the weight can hurt numerical stability, and it shifts the weight's value range in a way that can make quantization worse. In this case, write the scale to GGUF as its own metadata key (e.g. %s.attention.output_scale, %s.attention.value_scale, %s.embedding_scale) and apply it in the graph, instead of pre-multiplying the weight tensor during conversion.

Working with ggml_rope_ext

PyTorch implementations usually prefer explicitly calculating freq_cis/sin/cos components. However, in llama.cpp, most RoPE operations can be handled via ggml_rope_ext, which does not require a sin/cos matrix. This saves memory while allowing the GGML RoPE kernel to be fused with other ops.

However, since ggml_rope_ext only provides a subset of the RoPE implementations that models use, converting models from PyTorch to llama.cpp may require some creative adaptations.

For more information about ggml_rope_ext, please refer to the in-code documentation in ggml.h.

Examples:

libmtmd implements 2D RoPE with GGML_ROPE_TYPE_NORMAL ordering by splitting the input tensor in half, applying ggml_rope_ext separately to each half, then joining them back together using ggml_concat.
The Kimi-K2.5 vision encoder uses vision RoPE with interleaved frequencies. The weights must be permuted during conversion in order to reuse the build_rope_2d() function.
Gemma 4 uses "proportional" RoPE. We employ a trick where rope_freqs is set to a very large value in the last dimensions to prevent those dimensions from being rotated. See the Gemma4Model class in convert_hf_to_gguf.py.
Some models require scaling the input position. For example, [0, 1, 2, ...] becomes [0, 0.5, 1, ...]. In this case, you can provide the scaling via freq_scale = 0.5f.
Some models use learned RoPE frequencies instead of relying on powf(freq_base, -2.0 * i / n_dims). In this case, you can provide the learned frequencies via the rope_freqs tensor (corresponding to the c argument in ggml_rope_ext), then set freq_base = 1.0f. An important note is that rope_freqs in GGML is the inverse (theta = pos[i] / rope_freqs), so you may need to invert rope_freqs during conversion.

Rotating only a part of the head

Many models rotate only a part of each head and leave the rest untouched (often called the "nope" part). Do not build this with views plus ggml_concat, it's not efficient. Both layouts can be done with a single RoPE op:

[rope|nope], rotated dims first: pass n_dims smaller than the head size to ggml_rope_ext. Dims from n_dims to the end are copied as-is.
[nope|rope], rotated dims last: call ggml_rope_set_offset(cur, n_offs) on the result of the RoPE, where n_offs is the size of the leading untouched part. Dims outside [n_offs, n_offs + n_dims) are copied as-is.

n_offs must be even, n_offs + n_dims must fit in the row, and vision RoPE is not supported. Note that the frequencies are computed relative to the rotated window.

Example: DeepSeek-V4 uses [nope|rope] for its query, key and compressed KV tensors, so src/models/deepseek4.cpp ropes the whole tensor and then calls ggml_rope_set_offset(cur, n_embd_head_nope).

Exception: some models apply an extra op to the nope part, for example deepseek32.cpp, and may not use this optimization. While RoPE can be applied selectively to a part of the head, the extra op may not, so these models still need views plus ggml_concat.

GGUF specification

https://github.com/ggml-org/ggml/blob/master/docs/gguf.md

Resources

YaRN RoPE scaling https://github.com/ggml-org/llama.cpp/pull/2268
support Baichuan serial models https://github.com/ggml-org/llama.cpp/pull/3009
support attention bias https://github.com/ggml-org/llama.cpp/pull/4283
Mixtral support https://github.com/ggml-org/llama.cpp/pull/4406
BERT embeddings https://github.com/ggml-org/llama.cpp/pull/5423
Grok-1 support https://github.com/ggml-org/llama.cpp/pull/6204
Command R Plus support https://github.com/ggml-org/llama.cpp/pull/6491
support arch DBRX https://github.com/ggml-org/llama.cpp/pull/6515
How to convert HuggingFace model to GGUF format https://github.com/ggml-org/llama.cpp/discussions/2948

1# Add a new model architecture to `llama.cpp`
2
3Adding a model requires few steps:
4
51. Convert the model to GGUF
62. Define the model architecture in `llama.cpp`
73. Build the GGML graph implementation
84. Optional: Add multimodal encoder implementation
9
10After following these steps, you can open PR.
11
12Also, it is important to check that the examples and main ggml backends (CUDA, METAL, CPU) are working with the new architecture, especially:
13- [cli](/tools/cli/)
14- [completion](/tools/completion/)
15- [imatrix](/tools/imatrix/)
16- [quantize](/tools/quantize/)
17- [server](/tools/server/)
18
19### 1. Convert the model to GGUF
20
21This step is done in python with a `convert` script using the [gguf](https://pypi.org/project/gguf/) library.
22Depending on the model architecture, you can use either [convert_hf_to_gguf.py](/convert_hf_to_gguf.py) or [examples/convert_legacy_llama.py](/examples/convert_legacy_llama.py) (for `llama/llama2` models in `.pth` format).
23
24The convert script reads the model configuration, tokenizer, tensor names+data and converts them to GGUF metadata and tensors.
25
26The required steps to implement for an HF model are:
27
281. Define the model `ModelBase.register` annotation in a new `TextModel` or `MmprojModel` subclass in the [conversion](/conversion) folder, example:
29
30```python
31@ModelBase.register("MyModelForCausalLM")
32@ModelBase.example("user/model")
33class MyModel(TextModel):
34    model_arch = gguf.MODEL_ARCH.MYMODEL
35```
36
37or
38
39```python
40@ModelBase.register("MyModelForConditionalGeneration")
41@ModelBase.example("user/model")
42class MyModel(MmprojModel):
43    model_arch = gguf.MODEL_ARCH.MYMODEL
44```
45
46The `example` should point to a valid Hugging Face model that will be used for testing. You can add multiple models if necessary. Prefer a non-gated model, or tiny random weights if no such model exists.
47
482. Define the layout of the GGUF tensors in [constants.py](/gguf-py/gguf/constants.py)
49
50Add an enum entry in `MODEL_ARCH`, the model human friendly name in `MODEL_ARCH_NAMES` and the GGUF tensor names in `MODEL_TENSORS`.
51
52NOTE: Pick the GGUF arch string (and the matching `src/models/<name>.cpp` filename, see section 3) carefully up front, following existing naming conventions. Once GGUF files are published under a given arch string, renaming it later breaks the community's existing files, so this is not something to leave for cleanup in a follow-up PR.
53
54Example for `falcon` model:
55```python
56    MODEL_ARCH.FALCON: [
57        MODEL_TENSOR.TOKEN_EMBD,
58        MODEL_TENSOR.OUTPUT_NORM,
59        MODEL_TENSOR.OUTPUT,
60        MODEL_TENSOR.ATTN_NORM,
61        MODEL_TENSOR.ATTN_NORM_2,
62        MODEL_TENSOR.ATTN_QKV,
63        MODEL_TENSOR.ATTN_OUT,
64        MODEL_TENSOR.FFN_DOWN,
65        MODEL_TENSOR.FFN_UP,
66    ]
67```
68
693. Map the original tensor names to the standardize equivalent in GGUF
70
71As a general rule, before adding a new tensor name to GGUF, be sure the equivalent naming does not already exist.
72
73Once you have found the GGUF tensor name equivalent, add it to the [tensor_mapping.py](/gguf-py/gguf/tensor_mapping.py) file.
74
75If the tensor name is part of a repetitive layer/block, the key word `bid` substitutes it.
76
77Example for the normalization tensor in attention layers:
78
79```python
80block_mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {
81        # Attention norm
82        MODEL_TENSOR.ATTN_NORM: (
83            "gpt_neox.layers.{bid}.input_layernorm",                # gptneox
84            "transformer.h.{bid}.ln_1",                             # gpt2 gpt-j refact qwen
85            "transformer.blocks.{bid}.norm_1",                      # mpt
86            ...
87        )
88}
89```
90
91`transformer.blocks.{bid}.norm_1` will be mapped to `blk.{bid}.attn_norm` in GGUF.
92
93Depending on the model configuration, tokenizer, code and tensors layout, you will have to override:
94- `TextModel#set_gguf_parameters`
95- `MmprojModel#set_gguf_parameters`
96- `ModelBase#set_vocab`
97- `ModelBase#modify_tensors`
98
99NOTE: Tensor names must end with `.weight` or `.bias` suffixes, that is the convention and several tools like `quantize` expect this to proceed the weights.
100
101### 2. Define the model architecture in `llama.cpp`
102
103The model params and tensors layout must be defined in `llama.cpp` source files:
1041. Define a new `llm_arch` enum value in `src/llama-arch.h`.
1052. In `src/llama-arch.cpp`:
106    - Add the architecture name to the `LLM_ARCH_NAMES` map.
107    - You may also need to update `LLM_KV_NAMES`, `LLM_TENSOR_NAMES` and `LLM_TENSOR_INFOS`
1083. Add any non-standard metadata loading in the `llama_model_loader` constructor in `src/llama-model-loader.cpp`.
1094. If the model has a RoPE operation, add a case for the architecture in `llama_model_rope_type` function in `src/llama-model.cpp`.
1105. Check for other places that switch/iterate over every `llm_arch` value, e.g. `src/llama-model-saver.cpp` and any mandatory-hparam lists (such as which archs require MoE metadata). Grep for `LLM_ARCH_` usages to find them. Missing one of these is a common cause of CI test failures (e.g. `test-llama-archs`) after adding a new arch.
111
112NOTE: The dimensions in `ggml` are typically in the reverse order of the `pytorch` dimensions.
113
114### 3. Build the GGML graph implementation
115
116This is the funniest part, you have to provide the inference graph implementation of the new model architecture in `src/llama-model.cpp`:
1171. Create a new struct that inherits from `llama_model_base`.
1182. Implement the graph-building logic in its `build_arch_graph` method.
1193. The `build_arch_graph` method should return a constructed graph (inherited from `llm_graph_context`). Have a look at existing implementations like `llama_model_llama`, `llama_model_dbrx` or `llama_model_bert`.
1204. Then, in the `llama_model_mapping` function, add a case for your architecture to instantiate your new graph-building struct.
121
122Some `ggml` backends do not support all operations. Backend implementations can be added in a separate PR.
123
124Note: to debug the inference graph: you can use [llama-eval-callback](/examples/eval-callback/).
125
126### 4. Optional: Add multimodal encoder implementation
127
128If the new model supports multimodal inputs, you will need to add a new encoder definition in `libmtmd`. You can find more information about llama.cpp's multimodal support in [the docs](../multimodal.md) and in the `tools/mtmd` source directory.
129
1301. In the conversion script, make sure you add a subclass that extends `MmprojModel` or another class that inherits from the same base class.
1312. Add the encoder definition in `clip.cpp`.
1323. Implement the preprocessor in `mtmd.cpp`. In most cases, you can reuse an existing preprocessor.
1334. Implement the encoder GGML graph, either in a dedicated file if the model is truly different from existing ones, or by reusing an existing implementation (for example: siglip, pixtral, or qwen) and adding a model-specific projector.
134
135Note:
136- Many multimodal encoders are based on models that are already supported. Make sure to read the existing encoder definitions in `tools/mtmd/models` before adding a new one. In `libmtmd`, it is generally better to extend an existing model than to duplicate code.
137- To debug the multimodal preprocessor and encoder, you can use [llama-mtmd-debug](tools/mtmd/debug/mtmd-debug.cpp).
138- Adding a model-specific API or CLI is an anti-pattern in `libmtmd`. The goal of `libmtmd` is to provide an easy-to-use, model-agnostic library for multimodal pipeline.
139- In most cases, `llama-mtmd-cli` should not be modified. If a model requires a specific prompt, either let the user provide it or bake it into the Jinja chat template.
140- For audio generation models, see `tools/mtmd/README-dev.md`
141
142## Tips and tricks
143
144### Prefer conversion-time tensor modifications over graph-time ones
145
146If the model contains constant modifications of tensors in the graph (for example, `norm(1 + weight)`) or performs tensor permutations/chunking, perform the modifications during conversion rather than in the graph code. This keeps the inference graph simpler and avoids extra runtime ops.
147
148Examples:
149- Gemma 3 folds the `1 +` of its `norm(1 + weight)` normalization into the weights at conversion time, so the graph just does a plain RMS norm.
150- Qwen3-Next applies its tensor permutation during conversion (in `modify_tensors`), so the graph can consume the already-permuted weights directly.
151
152Exception: a plain `weight * scale` with a constant scale is usually better left to inference time rather than folded into the weight at conversion. The scale conceptually applies to the activation, not the weight, so folding it into the weight can hurt numerical stability, and it shifts the weight's value range in a way that can make quantization worse. In this case, write the scale to GGUF as its own metadata key (e.g. `%s.attention.output_scale`, `%s.attention.value_scale`, `%s.embedding_scale`) and apply it in the graph, instead of pre-multiplying the weight tensor during conversion.
153
154### Working with ggml_rope_ext
155
156PyTorch implementations usually prefer explicitly calculating `freq_cis`/`sin`/`cos` components. However, in llama.cpp, most RoPE operations can be handled via `ggml_rope_ext`, which does not require a sin/cos matrix. This saves memory while allowing the GGML RoPE kernel to be fused with other ops.
157
158However, since `ggml_rope_ext` only provides a subset of the RoPE implementations that models use, converting models from PyTorch to llama.cpp may require some creative adaptations.
159
160For more information about `ggml_rope_ext`, please refer to the in-code documentation in `ggml.h`.
161
162Examples:
163- `libmtmd` implements 2D RoPE with `GGML_ROPE_TYPE_NORMAL` ordering by splitting the input tensor in half, applying `ggml_rope_ext` separately to each half, then joining them back together using `ggml_concat`.
164- The [Kimi-K2.5](https://github.com/ggml-org/llama.cpp/pull/19170) vision encoder uses vision RoPE with interleaved frequencies. The weights must be permuted during conversion in order to reuse the `build_rope_2d()` function.
165- [Gemma 4](https://github.com/ggml-org/llama.cpp/pull/21309) uses "proportional" RoPE. We employ a trick where `rope_freqs` is set to a very large value in the last dimensions to prevent those dimensions from being rotated. See the `Gemma4Model` class in `convert_hf_to_gguf.py`.
166- Some models require scaling the input position. For example, `[0, 1, 2, ...]` becomes `[0, 0.5, 1, ...]`. In this case, you can provide the scaling via `freq_scale = 0.5f`.
167- Some models use learned RoPE frequencies instead of relying on `powf(freq_base, -2.0 * i / n_dims)`. In this case, you can provide the learned frequencies via the `rope_freqs` tensor (corresponding to the `c` argument in `ggml_rope_ext`), then set `freq_base = 1.0f`. An important note is that `rope_freqs` in GGML is the **inverse** (`theta = pos[i] / rope_freqs`), so you may need to invert `rope_freqs` during conversion.
168
169### Rotating only a part of the head
170
171Many models rotate only a part of each head and leave the rest untouched (often called the "nope" part). Do not build this with views plus `ggml_concat`, it's not efficient. Both layouts can be done with a single RoPE op:
172
173- `[rope|nope]`, rotated dims first: pass `n_dims` smaller than the head size to `ggml_rope_ext`. Dims from `n_dims` to the end are copied as-is.
174- `[nope|rope]`, rotated dims last: call `ggml_rope_set_offset(cur, n_offs)` on the result of the RoPE, where `n_offs` is the size of the leading untouched part. Dims outside `[n_offs, n_offs + n_dims)` are copied as-is.
175
176`n_offs` must be even, `n_offs + n_dims` must fit in the row, and vision RoPE is not supported. Note that the frequencies are computed relative to the rotated window.
177
178Example: DeepSeek-V4 uses `[nope|rope]` for its query, key and compressed KV tensors, so `src/models/deepseek4.cpp` ropes the whole tensor and then calls `ggml_rope_set_offset(cur, n_embd_head_nope)`.
179
180Exception: some models apply an extra op to the `nope` part, for example `deepseek32.cpp`, and may not use this optimization. While RoPE can be applied selectively to a part of the head, the extra op may not, so these models still need views plus `ggml_concat`.
181
182## GGUF specification
183
184https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
185
186## Resources
187
188- YaRN RoPE scaling https://github.com/ggml-org/llama.cpp/pull/2268
189- support Baichuan serial models https://github.com/ggml-org/llama.cpp/pull/3009
190- support attention bias https://github.com/ggml-org/llama.cpp/pull/4283
191- Mixtral support https://github.com/ggml-org/llama.cpp/pull/4406
192- BERT embeddings https://github.com/ggml-org/llama.cpp/pull/5423
193- Grok-1 support https://github.com/ggml-org/llama.cpp/pull/6204
194- Command R Plus support https://github.com/ggml-org/llama.cpp/pull/6491
195- support arch DBRX https://github.com/ggml-org/llama.cpp/pull/6515
196- How to convert HuggingFace model to GGUF format https://github.com/ggml-org/llama.cpp/discussions/2948

Add a new model architecture to llama.cpp