← back

char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Charlotte Somenc/dec in server context8300f1411

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
27 folders31 files
languages

files

.devops/
.gemini/
.github/
.pi/gg/
app/
benches/
ci/
cmake/
common/
conversion/
docs/
examples/
ggml/
gguf-py/
grammars/
include/
licenses/
media/
models/
pocs/
requirements/
scripts/
skills/
src/
tests/
tools/
vendor/
.clang-format4.8 KiB
.clang-tidy931 B
.dockerignore261 B
.ecrc97 B
.editorconfig1.2 KiB
.flake8565 B
.gitignore1.7 KiB
.gitmodules0 B
.pre-commit-config.yaml447 B
AGENTS.md11.9 KiB
AUTHORS87.2 KiB
CLAUDE.md106 B
CMakeLists.txt10.7 KiB
CMakePresets.json4.5 KiB
CODEOWNERS6.4 KiB
CONTRIBUTING.md12.5 KiB
LICENSE1.1 KiB
Makefile257 B
README.md7.2 KiB
SECURITY.md7.3 KiB
build-xcframework.sh24.3 KiB
convert_hf_to_gguf.py12.7 KiB
convert_hf_to_gguf_update.py27.9 KiB
convert_llama_ggml_to_gguf.py18.7 KiB
convert_lora_to_gguf.py22.7 KiB
flake.nix7.1 KiB
mypy.ini163 B
pyproject.toml1.9 KiB
pyrightconfig.json696 B
requirements.txt551 B
ty.toml643 B

llama.cpp

llama

<div align="center">

<b>LLM inference in C/C++</b>

License: MIT Release Nightly Server Docker Winget

ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API

</div>

Quick start

A few options to get llama.cpp installed on your machine:

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
<table align="center"> <tr> <td align="center" width=50%> <img width="1310" height="888" alt="VLM session with `llama cli`" src="https://github.com/user-attachments/assets/88726b48-1713-48aa-a525-95a02e78afc4" /> <i>VLM session with <b>llama cli</b></i> </td> <td align="center"> <img width="1392" height="958" alt="Built-in web UI against `llama serve` running Qwen 3.6" src="https://github.com/user-attachments/assets/b402f972-2e32-4def-8771-8d849f08cf2e" /> <i>Built-in web UI against <b>llama serve</b></i> </td> </tr> <table>

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

The llama.cpp project is build on top of the ggml library.

Supported backends

BackendTarget devices
BLASAll
BLISAll
CANNAscend NPU
CUDANvidia GPU
HIPAMD GPU
HexagonSnapdragon
IBM zDNNIBM Z & LinuxONE
MUSAMoore Threads GPU
MetalApple Silicon
OpenCLAdreno GPU
OpenVINO [In Progress]Intel CPUs, GPUs, and NPUs
RPCAll
SYCLIntel GPU
VirtGPUVirtGPU APIR
VulkanGPU
WebGPUAll
ZenDNNAMD CPU

Documentation

Tools

Development

Contributing

Acknowledgements