char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Charlotte Somenc/dec in server context8300f1411

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
4 folders26 files

files

debug/
legacy-models/
models/
tests/
CMakeLists.txt5.5 KiB
README-dev.md6.1 KiB
README.md5.5 KiB
clip-graph.h5.3 KiB
clip-impl.h44.0 KiB
clip-model.h32.7 KiB
clip.cpp311.2 KiB
clip.h4.8 KiB
deprecation-warning.cpp652 B
mtmd-audio.cpp57.7 KiB
mtmd-audio.h6.8 KiB
mtmd-cli.cpp20.4 KiB
mtmd-helper-common.h6.5 KiB
mtmd-helper-gen.cpp39.4 KiB
mtmd-helper.cpp34.9 KiB
mtmd-helper.h12.8 KiB
mtmd-image.cpp71.2 KiB
mtmd-image.h11.8 KiB
mtmd-internal.h576 B
mtmd.cpp104.4 KiB
mtmd.h21.9 KiB
requirements.txt262 B
test-1.jpeg121.2 KiB
test-2.mp3136.8 KiB
test-3.mp4646.3 KiB
tests.sh7.8 KiB

Multimodal Support in llama.cpp

This directory provides multimodal capabilities for llama.cpp. Initially intended as a showcase for running LLaVA models, its scope has expanded significantly over time to include various other vision-capable models. As a result, LLaVA is no longer the only multimodal architecture supported.

[!IMPORTANT]

Multimodal support can be viewed as a sub-project within llama.cpp. It is under very heavy development, and breaking changes are expected.

The naming and structure related to multimodal support have evolved, which might cause some confusion. Here's a brief timeline to clarify:

Pre-quantized models

See the list of pre-quantized model here

How it works and what is mmproj?

Multimodal support in llama.cpp works by encoding images into embeddings using a separate model component, and then feeding these embeddings into the language model.

This approach keeps the multimodal components distinct from the core libllama library. Separating these allows for faster, independent development cycles. While many modern vision models are based on Vision Transformers (ViTs), their specific pre-processing and projection steps can vary significantly. Integrating this diverse complexity directly into libllama is currently challenging.

Consequently, running a multimodal model typically requires two GGUF files:

  1. The standard language model file.
  2. A corresponding multimodal projector (mmproj) file, which handles the image encoding and projection.

What is libmtmd?

As outlined in the history, libmtmd is the modern library designed to replace the original llava.cpp implementation for handling multimodal inputs.

Built upon clip.cpp (similar to llava.cpp), libmtmd offers several advantages:

How to obtain mmproj

Multimodal projector (mmproj) files are specific to each model architecture.

For the following models, you can use convert_hf_to_gguf.py with --mmproj flag to get the mmproj file:

For older models, please refer to the relevant guide for instructions on how to obtain or create them:

NOTE: conversion scripts are located under tools/mtmd/legacy-models