hey! this is a static git repo viewer. commit diffs, file history, older commits and other revisions won't load without javascript.

you are still able to view source files on branch/tag tips, but the experience is degraded.

char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Georgi Gerganovreadme : refresh (#26280)6b36c23052026-07-30

renderedcode

Obtaining and quantizing models

The Hugging Face platform hosts thousands of models compatible with llama.cpp:

Trending

You can use any llama.cpp-compatible model from Hugging Face using this CLI argument: -hf <user>/<model>[:quant]. For example:

llama cli -hf ggml-org/gemma-3-1b-it-GGUF

You can use the same CLI invocation to download from other sites, by pointing the MODEL_ENDPOINT environment variable to an endpoint compatible with the Hugging Face API. llama.cpp can also run models you have downloaded locally to your filesystem.

After downloading a model, use the CLI tools to run it locally - see below.

llama.cpp requires the model to be stored in the GGUF file format. Models in other data formats can be converted to GGUF using the convert_*.py Python scripts in this repo. To learn more about model quantization, read this documentation

The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with llama.cpp:

Use the GGUF-my-repo space to convert to GGUF format and quantize model weights to smaller sizes
Use the GGUF-my-LoRA space to convert LoRA adapters to GGUF format (more info: https://github.com/ggml-org/llama.cpp/discussions/10123)
Use the GGUF-editor space to edit GGUF meta data in the browser (more info: https://github.com/ggml-org/llama.cpp/discussions/9268)
Use the Inference Endpoints to directly host llama.cpp in the cloud (more info: https://github.com/ggml-org/llama.cpp/discussions/9669)

1# Obtaining and quantizing models
2
3The [Hugging Face](https://huggingface.co) platform hosts [thousands of models](https://huggingface.co/models?library=gguf&sort=trending) compatible with `llama.cpp`:
4
5- [Trending](https://huggingface.co/models?library=gguf&sort=trending)
6
7You can use any `llama.cpp`-compatible model from [Hugging Face](https://huggingface.co/) using this CLI argument: `-hf <user>/<model>[:quant]`. For example:
8
9```sh
10llama cli -hf ggml-org/gemma-3-1b-it-GGUF
11```
12
13You can use the same CLI invocation to download from other sites, by pointing the `MODEL_ENDPOINT` environment variable to an endpoint compatible with the Hugging Face API.
14`llama.cpp` can also run models you have downloaded locally to your filesystem.
15
16After downloading a model, use the CLI tools to run it locally - see below.
17
18`llama.cpp` requires the model to be stored in the [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) file format. Models in other data formats can be converted to GGUF using the `convert_*.py` Python scripts in this repo.
19To learn more about model quantization, [read this documentation](../tools/quantize/README.md)
20
21The Hugging Face platform provides a variety of online tools for converting, quantizing and hosting models with `llama.cpp`:
22
23- Use the [GGUF-my-repo space](https://huggingface.co/spaces/ggml-org/gguf-my-repo) to convert to GGUF format and quantize model weights to smaller sizes
24- Use the [GGUF-my-LoRA space](https://huggingface.co/spaces/ggml-org/gguf-my-lora) to convert LoRA adapters to GGUF format (more info: https://github.com/ggml-org/llama.cpp/discussions/10123)
25- Use the [GGUF-editor space](https://huggingface.co/spaces/CISCai/gguf-editor) to edit GGUF meta data in the browser (more info: https://github.com/ggml-org/llama.cpp/discussions/9268)
26- Use the [Inference Endpoints](https://ui.endpoints.huggingface.co/) to directly host `llama.cpp` in the cloud (more info: https://github.com/ggml-org/llama.cpp/discussions/9669)