hey! this is a static git repo viewer. commit diffs, file history, older commits and other revisions won't load without javascript.

you are still able to view source files on branch/tag tips, but the experience is degraded.

char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Zijun YuOpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (#27843)511f9c1372026-08-28

renderedcode

OpenVINO Backend for llama.cpp

[!NOTE] Performance and memory optimizations, accuracy validation, broader quantization coverage, broader operator and model support are work in progress.

OpenVINO is an open-source toolkit for optimizing and deploying high-performance AI inference, specifically designed for Intel hardware, including CPUs, GPUs, and NPUs, in the cloud, on-premises, and on the edge. OpenVINO backend for llama.cpp enables hardware-accelerated inference on Intel® CPUs, GPUs, and NPUs while remaining compatible with the existing GGUF model ecosystem. The backend translates GGML compute graphs into OpenVINO graphs and leverages graph compilation, kernel fusion, and device-specific optimizations to improve inference performance on supported Intel hardware.

The OpenVINO backend is implemented in ggml/src/ggml-openvino and provides a translation layer for core GGML operations. The OpenVINO backend replaces the standard GGML graph execution path with Intel's OpenVINO inference engine. This approach allows the same GGUF model file to run on Intel CPUs, Intel GPUs (integrated and discrete), and Intel NPUs without changes to the model or the rest of the llama.cpp stack. When a ggml_cgraph is dispatched to OpenVINO backend, it:

Walks the GGML graph and identifies inputs, outputs, weights, and KV cache tensors.
Translates the GGML operations into an ov::Model using OpenVINO's frontend API.
Compiles and caches the model for the target device.
Binds GGML tensor memory to OpenVINO inference tensors and runs inference.

Supported Devices
Supported Model Precisions
Supported Llama.cpp Tools
Validated Models
Build Instructions
GGML OpenVINO Backend Runtime Configurations
Known Limitations
Work in Progress

Supported Devices

OpenVINO backend supports the following hardware:

Intel CPUs
Intel GPUs (integrated and discrete)
Intel NPUs

Although OpenVINO supports a wide range of Intel hardware, the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 1 and Series 2.

Supported Model Precisions

FP16
BF16 (on Intel Xeon)
Q8_0
Q4_0
Q4_1
Q4_K
Q4_K_M
Q5_K (converted to Q8_0_C at runtime)
Q6_K (converted to Q8_0_C at runtime)

[!NOTE] Accuracy validation and performance optimizations for quantized models are a work in progress.

CPU and GPU Quantization Details:

Q5_K and Q6_K tensors are converted to Q8_0_C

NPU Quantization Details:

Primary supported quantization scheme is Q4_0
Q6_K tensors are requantized to Q4_0_128 in general. For embedding weights, Q6_K tensors are requantized to Q8_0_C except for the token embedding matrix which is dequantized to fp16

Additional Notes:

Both Q4_0 and Q4_1 models use Q6_K for the token embedding tensor and the final matmul weight tensor (often the same tensor)
Q4_0 models may produce some Q4_1 tensors if an imatrix is provided during quantization using llama-quantize
Q4_K_M models may include both Q6_K and Q5_K tensors (observed in Phi-3)
Q5_1 tensors are dequantized natively (weights, scales, and zero-points extracted directly)

Supported Llama.cpp Tools

The OpenVINO backend integrates with the standard llama.cpp tools listed below. However, all the tools coverage across all devices is not uniform and exhaustive validation is work in progress.

llama-bench
llama-cli
llama-completion
llama-embedding
llama-perplexity
llama-run
llama-server
llama-simple

Validated Models

Although, the validated models below were tested with llama-cli using the Q4_K_M quantization format on Intel® Core™ Ultra Series 2 (Lunar Lake), the OpenVINO backend is expected to work across a broader range of Intel hardware, supported model precisions, supported llama.cpp tools and additional model architectures.

[!NOTE] Extensive accuracy validation, performance optimizations, and broader architecture coverage are work in progress.

Legend & Test Configuration:

Status: ✓ = Passed | ✗ = Failed or Unsupported
Execution Modes:
- SL = Stateless (GGML_OPENVINO_STATEFUL_EXECUTION=0)
- SF = Stateful (GGML_OPENVINO_STATEFUL_EXECUTION=1)
- Note: The NPU operates in stateless mode only.
Validation system: Intel® Core™ Ultra 5 238V (Lunar Lake) | 32 GB RAM | Ubuntu 24.04 | Intel OpenCL GPU Driver 26.31.39395.13-0 | Intel NPU Driver 1.35.0.
See Known Limitations for context on observed failures.

Model	CPU (SL / SF)	GPU (SL / SF)	NPU (SL)
bartowski/Llama-3.2-1B-Instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/Llama-3.2-3B-Instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/Meta-Llama-3.1-8B-Instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓

Qwen/qwen2.5-1.5b-instruct-q4_k_m	✓ / ✓	✓ / ✓	✓
Qwen/qwen2.5-coder-7b-instruct-q4_k_m	✓ / ✓	✓ / ✓	✓
bartowski/Qwen_Qwen3-0.6B-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/Qwen_Qwen3-1.7B-Q4_K_M	✓ / ✓	✓ / ✓	✓
Qwen/Qwen3-4B-Q4_K_M	✓ / ✓	✓ / ✓	✓
lm-kit/Qwen3-8B-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M	✓ / ✗	✓ / ✗	✗
bartowski/Qwen_Qwen3.5-2B-Q4_K_M	✓ / ✗	✓ / ✗	✗
bartowski/Qwen_Qwen3.5-4B-Q4_K_M	✓ / ✗	✓ / ✗	✗
lmstudio-community/Qwen3.5-9B-Q4_K_M	✓ / ✗	✓ / ✗	✗

unsloth/gemma-3-4b-it-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/google_gemma-4-E2B-it-Q4_K_M	✓ / ✗	✓ / ✗	✗
bartowski/google_gemma-4-E4B-it-Q4_K_M	✓ / ✗	✓ / ✗	✓
bartowski/gemma-4-12B-it-Q4_K_M	✓ / ✗	✓ / ✗	✓

bartowski/Phi-3-mini-4k-instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/Phi-3.5-mini-instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/microsoft_Phi-4-mini-instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓

bartowski/Mistral-7B-Instruct-v0.3-Q4_K_M	✓ / ✓	✓ / ✓	✓
QuantFactory/Ministral-3b-instruct.Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/Ministral-8B-Instruct-2410-Q4_K_M	✓ / ✓	✓ / ✓	✓

bartowski/DeepSeek-R1-Distill-Llama-8B-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M	✓ / ✓	✓ / ✓	✓

ibm-granite/granite-4.0-350m-Q4_K_M	✓ / ✓	✗ / ✗	✓
ibm-granite/granite-4.0-micro-Q4_K_M	✓ / ✓	✓ / ✓	✓
ibm-granite/granite-4.0-1b-Q4_K_M	✓ / ✓	✗ / ✗	✗
ibm-research/granite-3.2-8b-instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓

HuggingFaceTB/smollm2-1.7b-instruct-q4_k_m	✓ / ✓	✓ / ✓	✓
openbmb/MiniCPM-V-2_6-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/tencent_Hunyuan-7B-Instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓
LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-Q4_K_M	✓ / ✓	✓ / ✓	✓
bartowski/prism-ml_Bonsai-8B-unpacked-Q4_K_M	✓ / ✓	✓ / ✓	✓

gpustack/bge-m3-Q4_K_M.gguf	✓	✗	✗

Build Instructions

0. Prerequisites

Linux or Windows system with Intel hardware (CPU, GPU, or NPU)
For Intel GPU or NPU Usage: Install the appropriate hardware drivers for your Intel GPU or NPU. For detailed instructions, see: Additional Configurations for Hardware Acceleration.

Linux:

Git, CMake, and Ninja software tools are needed for building.

  sudo apt-get update
  sudo apt-get install -y build-essential libcurl4-openssl-dev libtbb12 cmake ninja-build python3-pip curl wget tar

OpenCL

  sudo apt install ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd

Windows:

Download and install Microsoft Visual Studio 2022 Build Tools. During installation, select the "Desktop development with C++" workload.

Install required tools:

# Windows PowerShell
winget install Git.Git
winget install GNU.Wget
winget install Ninja-build.Ninja

Install OpenCL using vcpkg:

# Windows PowerShell
cd C:\
git clone https://github.com/microsoft/vcpkg
cd vcpkg
.\bootstrap-vcpkg.bat
.\vcpkg install opencl
# Optional but recommended: Integrate vcpkg with Visual Studio / CMake:
.\vcpkg integrate install

1. Install OpenVINO Runtime

Follow the guide to install OpenVINO Runtime from an archive file: Linux | Windows
Verify OpenVINO is initialized properly:
```
echo $OpenVINO_DIR
```

2. Build llama.cpp with OpenVINO Backend

Clone llama.cpp repo and build :

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

Linux:

source /opt/intel/openvino/setupvars.sh
cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
cmake --build build/ReleaseOV --parallel

Windows: Open x64 Native Tools Command Prompt for VS (so the MSVC toolchain is on PATH), then run:

C:\Intel\openvino\setupvars.bat
cmake -B build\ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
cmake --build build\ReleaseOV --parallel

[!NOTE] The Windows install path is C:\Intel\openvino (no spaces) to avoid quoting problems some CMake/Ninja toolchains have with C:\Program Files (x86)\.... Adjust to wherever you installed OpenVINO Runtime. From cmd, run C:\Intel\openvino\setupvars.bat; from PowerShell, run & "C:\Intel\openvino\setupvars.ps1" instead. Once the build is finished you can launch the binaries from any cmd or PowerShell window after sourcing the matching setupvars script for that shell.

Ubuntu Build Script

For Ubuntu24 users, the following shell script automates the prerequisite installs (build tools, OpenCL ICD), the OpenVINO Runtime download/extract/setup, and the Ninja-based llama.cpp build. Save the following as build-llamacpp-ov.sh next to where you want the llama.cpp folder to land, then run it:

chmod +x build-llamacpp-ov.sh
./build-llamacpp-ov.sh

<details> <summary>Click to expand <code>build-llamacpp-ov.sh</code></summary>

#!/usr/bin/env bash
# ============================================
# llama.cpp OpenVINO Build Script (Ninja)
# ============================================
set -euo pipefail

OPENVINO_VERSION_MAJOR="2026.3.1"
OPENVINO_VERSION_FULL="2026.3.1.22476.56d9685302d"

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OPENVINO_INSTALL_DIR="/opt/intel/openvino_${OPENVINO_VERSION_MAJOR}"
OPENVINO_LINK_DIR="/opt/intel/openvino"
OPENVINO_TGZ="${SCRIPT_DIR}/openvino.tgz"
OPENVINO_URL="https://storage.openvinotoolkit.org/repositories/openvino/packages/${OPENVINO_VERSION_MAJOR}/linux/openvino_toolkit_ubuntu24_${OPENVINO_VERSION_FULL}_x86_64.tgz"

echo "============================================"
echo "Installing prerequisites (apt)..."
echo "============================================"
sudo apt-get update
sudo apt-get install -y \
    build-essential libcurl4-openssl-dev libtbb12 \
    cmake ninja-build python3-pip \
    curl wget tar git

echo "============================================"
echo "Installing OpenCL runtime + headers..."
echo "============================================"
sudo apt-get install -y \
    ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd

cd "${SCRIPT_DIR}"

# ============================================
# Clone llama.cpp if missing
# ============================================
if [[ ! -f "llama.cpp/CMakeLists.txt" ]]; then
    echo "Cloning llama.cpp..."
    git clone https://github.com/ggml-org/llama.cpp
fi

# ============================================
# Setup OpenVINO: download & extract to /opt/intel/openvino_${OPENVINO_VERSION_MAJOR},
# then point /opt/intel/openvino at it via symlink so the active version is swappable.
# ============================================
if [[ -f "${OPENVINO_INSTALL_DIR}/setupvars.sh" ]]; then
    echo "OpenVINO ${OPENVINO_VERSION_MAJOR} already installed at ${OPENVINO_INSTALL_DIR}. Skipping download."
else
    echo "OpenVINO not found at ${OPENVINO_INSTALL_DIR}. Starting download..."
    curl -L -o "${OPENVINO_TGZ}" "${OPENVINO_URL}"

    echo "Extracting OpenVINO to ${OPENVINO_INSTALL_DIR}..."
    sudo mkdir -p "${OPENVINO_INSTALL_DIR}"
    sudo tar -xzf "${OPENVINO_TGZ}" -C "${OPENVINO_INSTALL_DIR}" --strip-components=1
    rm -f "${OPENVINO_TGZ}"
fi

# Refresh symlink: /opt/intel/openvino -> /opt/intel/openvino_${OPENVINO_VERSION_MAJOR}
sudo ln -sfn "${OPENVINO_INSTALL_DIR}" "${OPENVINO_LINK_DIR}"

OPENVINO_ROOT="${OPENVINO_LINK_DIR}"
echo "OpenVINO Ready: ${OPENVINO_ROOT} -> ${OPENVINO_INSTALL_DIR}"

# Install OpenVINO's own runtime dependencies (one-time per system).
if [[ -x "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh" ]]; then
    echo "============================================"
    echo "Installing OpenVINO runtime dependencies..."
    echo "============================================"
    echo "Y" | sudo -E "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh"
fi

# ============================================
# Clean old build cache
# ============================================
cd "${SCRIPT_DIR}/llama.cpp"
if [[ -d "build/ReleaseOV" ]]; then
    echo "Removing old build directory..."
    rm -rf "build/ReleaseOV"
fi

echo "============================================"
echo "Configuring with CMake..."
echo "============================================"
set +u
source "${OPENVINO_ROOT}/setupvars.sh"
set -u

cmake -B build/ReleaseOV -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_OPENVINO=ON

cmake --build build/ReleaseOV --parallel

echo "============================================"
echo "Build completed successfully!"
echo "============================================"
echo "Binaries: $(pwd)/build/ReleaseOV/bin"
echo
echo "NOTE: To run, source setupvars.sh and pick a device:"
echo "  source /opt/intel/openvino/setupvars.sh"
echo "  export GGML_OPENVINO_DEVICE=CPU   # or GPU / NPU"
echo "  ./build/ReleaseOV/bin/llama-cli -m model.gguf"

[!NOTE] The script pins OpenVINO 2026.3.1 via the OPENVINO_VERSION_MAJOR / OPENVINO_VERSION_FULL variables at the top — edit them to track a different release.

</details>

Windows Build Script

For Windows users, the following .bat script automates the prerequisite installs (Git, Ninja, CMake, Visual Studio 2022 Build Tools, vcpkg + OpenCL), the OpenVINO Runtime download/extract, and the Ninja-based llama.cpp build. Save the following as build-llamacpp-ov.bat next to where you want the llama.cpp to land, then run it from either Command Prompt or PowerShell:

:: Command Prompt
build-llamacpp-ov.bat

# PowerShell
.\build-llamacpp-ov.bat

<details> <summary>Click to expand <code>build-llamacpp-ov.bat</code></summary>

@echo off
setlocal enabledelayedexpansion

REM ============================================
REM llama.cpp OpenVINO Build Script (Ninja)
REM ============================================

set "OPENVINO_VERSION_MAJOR=2026.3.1"
set "OPENVINO_VERSION_FULL=2026.3.1.22476.56d9685302d"

set "SCRIPT_DIR=%~dp0"
set "VCPKG_DIR=C:\vcpkg"
set "OPENVINO_INSTALL_DIR=C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%"
set "OPENVINO_LINK_DIR=C:\Intel\openvino"
set "OPENVINO_ZIP=%SCRIPT_DIR%openvino.zip"
set "OPENVINO_EXTRACT_TMP=%SCRIPT_DIR%openvino_extract_tmp"
set "OPENVINO_URL=https://storage.openvinotoolkit.org/repositories/openvino/packages/%OPENVINO_VERSION_MAJOR%/windows/openvino_toolkit_windows_%OPENVINO_VERSION_FULL%_x86_64.zip"

echo ============================================
echo Installing prerequisites...
echo ============================================
winget install --id Git.Git -e --accept-source-agreements --accept-package-agreements 2>nul
winget install --id Ninja-build.Ninja -e --accept-source-agreements --accept-package-agreements 2>nul
winget install --id Kitware.CMake -e --accept-source-agreements --accept-package-agreements 2>nul

REM Ensure Visual Studio Build Tools are installed.
echo Checking for Visual Studio Build Tools...
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
set "VS_INSTALLED="
if exist "%VSWHERE%" (
    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath 2^>nul`) do (
        set "VS_INSTALLED=%%i"
    )
)
if defined VS_INSTALLED (
    echo Visual Studio with VC++ x86/x64 tools already present at "!VS_INSTALLED!". Skipping winget install.
) else (
    winget install --id Microsoft.VisualStudio.2022.BuildTools -e --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended" --accept-source-agreements --accept-package-agreements
    if errorlevel 1 (
        echo WARNING: winget could not install Visual Studio Build Tools automatically.
        echo Install manually from https://aka.ms/vs/17/release/vs_BuildTools.exe ^(select the "Desktop development with C++" workload^)
        echo and re-run this script from a "Developer Command Prompt for VS 2022".
    )
)

echo ============================================
echo Installing OpenCL via vcpkg...
echo ============================================
if not exist "%VCPKG_DIR%" (
    git clone https://github.com/microsoft/vcpkg "%VCPKG_DIR%"
    cd /d "%VCPKG_DIR%"
    call bootstrap-vcpkg.bat
    call vcpkg integrate install
)
cd /d "%VCPKG_DIR%"
call vcpkg install opencl

cd /d "%SCRIPT_DIR%"

REM ============================================
REM Clone llama.cpp if missing
REM ============================================
if not exist "llama.cpp\CMakeLists.txt" (
    echo Cloning llama.cpp...
    git clone https://github.com/ggml-org/llama.cpp
)

cd /d "llama.cpp"
set "SCRIPT_DIR=%CD%"

REM ============================================
REM Setup OpenVINO: download & extract to C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%,
REM then point C:\Intel\openvino at it via a directory junction (mklink /J).
REM ============================================

if exist "%OPENVINO_INSTALL_DIR%\setupvars.bat" (
    echo OpenVINO %OPENVINO_VERSION_MAJOR% already installed at "%OPENVINO_INSTALL_DIR%". Skipping download.
) else (
    echo OpenVINO not found at "%OPENVINO_INSTALL_DIR%". Starting download...

    curl -L -o "%OPENVINO_ZIP%" "%OPENVINO_URL%"
    if errorlevel 1 (
        echo ERROR: Download failed.
        exit /b 1
    )

    echo Extracting OpenVINO...
    if exist "%OPENVINO_EXTRACT_TMP%" rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
    mkdir "%OPENVINO_EXTRACT_TMP%"
    tar -xf "%OPENVINO_ZIP%" -C "%OPENVINO_EXTRACT_TMP%"
    if errorlevel 1 (
        echo ERROR: Extraction failed.
        exit /b 1
    )

    REM Move the single top-level folder contents into the versioned install dir.
    set "OPENVINO_EXTRACTED="
    for /d %%i in ("%OPENVINO_EXTRACT_TMP%\*") do set "OPENVINO_EXTRACTED=%%i"
    if not defined OPENVINO_EXTRACTED (
        echo ERROR: Could not locate extracted OpenVINO folder under "%OPENVINO_EXTRACT_TMP%".
        exit /b 1
    )
    if not exist "%OPENVINO_INSTALL_DIR%" mkdir "%OPENVINO_INSTALL_DIR%"
    xcopy /e /i /y /q "!OPENVINO_EXTRACTED!\*" "%OPENVINO_INSTALL_DIR%\" >nul
    if errorlevel 1 (
        echo ERROR: Failed to copy OpenVINO from "!OPENVINO_EXTRACTED!" to "%OPENVINO_INSTALL_DIR%".
        echo Re-run this script from an elevated Command Prompt ^(Run as administrator^) if access is denied.
        exit /b 1
    )

    rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
    del "%OPENVINO_ZIP%"
)

REM Refresh junction: C:\Intel\openvino -> C:\Intel\openvino_<version>.
REM `mklink /J` creates a directory junction (no admin / Developer Mode required).
if exist "%OPENVINO_LINK_DIR%" rmdir "%OPENVINO_LINK_DIR%"
mklink /J "%OPENVINO_LINK_DIR%" "%OPENVINO_INSTALL_DIR%" >nul
if errorlevel 1 (
    echo ERROR: Failed to create junction "%OPENVINO_LINK_DIR%" -^> "%OPENVINO_INSTALL_DIR%".
    echo If "%OPENVINO_LINK_DIR%" already exists as a regular non-empty folder, remove it manually and re-run.
    exit /b 1
)

set "OPENVINO_ROOT=%OPENVINO_LINK_DIR%"
echo OpenVINO Ready: %OPENVINO_ROOT% -^> %OPENVINO_INSTALL_DIR%


echo ============================================
echo Setting up compiler environment...
echo ============================================
REM Locate Visual Studio Build Tools vcvars64.bat
set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
if exist "%VSWHERE%" (
    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products Microsoft.VisualStudio.Product.BuildTools -property installationPath`) do (
        set "VS_PATH=%%i"
    )
)
if defined VS_PATH (
    call "%VS_PATH%\VC\Auxiliary\Build\vcvars64.bat" >nul
) else (
    echo WARNING: Visual Studio Build Tools not found. Compiler may be missing.
)

REM ============================================
REM Clean old build cache
REM ============================================
if exist "build\ReleaseOV" (
    echo Removing old build directory ...
    rmdir /s /q "build\ReleaseOV"
)

echo ============================================
echo Configuring with CMake...
echo ============================================
call "%OPENVINO_ROOT%\setupvars.bat" >nul 2>nul

cmake -B build\ReleaseOV -G Ninja ^
    -DCMAKE_BUILD_TYPE=Release ^
    -DGGML_OPENVINO=ON ^
    -DCMAKE_TOOLCHAIN_FILE="%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake"

if errorlevel 1 (
    echo If you continue to face CMAKE errors, make sure to install:
    echo   winget install Microsoft.VisualStudio.2022.BuildTools
    echo   Then run the "Developer Command Prompt for VS 2022" and launch this script from there.
    exit /b 1
)

cmake --build build\ReleaseOV --config Release
if errorlevel 1 exit /b 1

echo ============================================
echo Build completed successfully!
echo ============================================
echo Binaries: %CD%\build\ReleaseOV\bin
echo.
echo NOTE: To run, source setupvars.bat and pick a device:
echo   call "C:\Intel\openvino\setupvars.bat"
echo   set GGML_OPENVINO_DEVICE=CPU   ^&^& REM or GPU / NPU
echo   build\ReleaseOV\bin\llama-cli.exe -m model.gguf
echo.

endlocal

[!NOTE] The script pins OpenVINO 2026.3.1 via the OPENVINO_VERSION_MAJOR / OPENVINO_VERSION_FULL variables at the top — edit them to track a different release. From any new shell, source the matching setupvars script via the junction — call "C:\Intel\openvino\setupvars.bat" from cmd, or & "C:\Intel\openvino\setupvars.ps1" from PowerShell. If winget cannot register Visual Studio Build Tools on first run, install them once manually and re-run the script from an elevated Developer Command Prompt for VS 2022.

</details>

3. Download Sample Model

Download sample model for testing.

# Linux
mkdir -p ~/models/
wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
     -O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# Windows PowerShell
mkdir C:\models
Invoke-WebRequest -Uri https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -OutFile C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf

# Windows Command Line
mkdir C:\models
curl -L https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -o C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf

4. Run Inference with OpenVINO Backend

When using the OpenVINO backend, the first inference token may have slightly higher latency due to on-the-fly conversion to the OpenVINO graph. Subsequent tokens and runs will be faster.

[!NOTE] Default context size is set to the model training context, which may be very large. For example, 131072 for Llama 3.2 1B, which may result in lower performance, especially on edge/laptop devices. Use -c to limit context size in supported llama.cpp tools for better performance. For example, -c 512.

# If device is unset or unavailable, defaults to CPU.
# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.

# Linux
export GGML_OPENVINO_DEVICE=GPU
# Optional: enable stateful execution for improved GPU performance (recommended).
export GGML_OPENVINO_STATEFUL_EXECUTION=1
# To run llama-simple:
./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
# To run in chat mode:
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024
# To run llama-bench, -fa 1 is needed
GGML_OPENVINO_STATEFUL_EXECUTION=1 GGML_OPENVINO_DEVICE=GPU ./build/ReleaseOV/bin/llama-bench -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -fa 1

# NPU: keep context small to avoid failures from very large model context windows.
export GGML_OPENVINO_DEVICE=NPU
./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 512

# Windows Command Line
set GGML_OPENVINO_DEVICE=GPU
# Optional: enable stateful execution for improved GPU performance (recommended).
set GGML_OPENVINO_STATEFUL_EXECUTION=1
# Windows PowerShell
$env:GGML_OPENVINO_DEVICE = "GPU"
$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"

# To run llama-simple
build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
# To run in chat mode:
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 1024
# To run llama-bench, -fa 1 is needed
build\ReleaseOV\bin\llama-bench.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -fa 1

# NPU: keep context small to avoid failures from very large model context windows.
# Windows Command Line
set GGML_OPENVINO_DEVICE=NPU
# Windows PowerShell
$env:GGML_OPENVINO_DEVICE = "NPU"
build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 512

[!NOTE] On systems with multiple GPUs, use GPU.0 or GPU.1 to explicitly target specific GPU. See OpenVINO GPU Device for more details.

5. Docker Build

You can build and run llama.cpp with OpenVINO backend using Docker.

# Build the base runtime image with compiled shared libraries and minimal dependencies.
docker build -t llama-openvino:base -f .devops/openvino.Dockerfile .

# Build the complete image with all binaries, Python tools, gguf-py library, and model conversion utilities.
docker build --target=full -t llama-openvino:full -f .devops/openvino.Dockerfile .

# Build a minimal CLI-only image containing just the llama-cli executable.
docker build --target=light -t llama-openvino:light -f .devops/openvino.Dockerfile .

# Builds a server-only image with llama-server executable, health check endpoint, and REST API support.
docker build --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .

# If you are behind a proxy:
docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .

Run llama.cpp with OpenVINO backend Docker container. Save sample models in ~/models as shown above. It will be mounted to the container in the examples below.

#  Run Docker container
docker run --rm -it -v ~/models:/models llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# With Intel GPU access (iGPU or dGPU)
docker run --rm -it -v ~/models:/models \
--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
--env=GGML_OPENVINO_DEVICE=GPU --env=GGML_OPENVINO_STATEFUL_EXECUTION=1 \
llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

# With Intel NPU access
docker run --rm -it -v ~/models:/models \
--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
--env=GGML_OPENVINO_DEVICE=NPU \
llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf

Run Llama.cpp Server with OpenVINO Backend.

[!NOTE] llama-server with OpenVINO backend supports only one chat session/thread, when GGML_OPENVINO_STATEFUL_EXECUTION=1 is enabled.

# Run the llama-openvino:server Docker container (CPU)
docker run --rm -it -p 8080:8080 -v ~/models:/models llama-openvino:server --no-warmup -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024 --host 0.0.0.0

# Run the llama-openvino:server Docker container with Intel GPU access (iGPU or dGPU)
docker run --rm -it -v ~/models:/models \
--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
-p 8080:8080 --env=GGML_OPENVINO_DEVICE=GPU  \
llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0

# Run the llama-openvino:server Docker container with Intel NPU access
docker run --rm -it -v ~/models:/models \
--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
-p 8080:8080 --env=GGML_OPENVINO_DEVICE=NPU \
llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0

# Or Using llama-server executable
./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024

# Option 1: Open your browser to http://localhost:8080 to access the web UI for the llama.cpp server.
# Option 2: In a NEW terminal, test the server with curl

# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost
export NO_PROXY=localhost,127.0.0.1

# Test health endpoint
curl -f http://localhost:8080/health

# Test with a simple prompt
curl -X POST "http://localhost:8080/v1/chat/completions" -H "Content-Type: application/json" \
 -d '{"messages":[{"role":"user","content":"Write a poem about OpenVINO"}],"max_tokens":100}' | jq .

GGML OpenVINO Backend Runtime Configurations

The OpenVINO backend can be configured using the following environment variables at runtime to control device selection, caching, debugging, and profiling behavior. Boolean flags follow a uniform convention: set to a positive integer (e.g. 1) to enable; unset, empty, 0, negative, or non-numeric values are treated as disabled.

Variable	Type	Default	Description
`GGML_OPENVINO_DEVICE`	String	`CPU`	Specify the target device (CPU, GPU, NPU). On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. See OpenVINO GPU Device. When set to NPU, static compilation mode is enabled for optimal performance.
`GGML_OPENVINO_CACHE_DIR`	String	`not set`	Directory for OpenVINO model caching (recommended: `/tmp/ov_cache`). Enables model caching when set. Not supported on NPU devices.
`GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR`	String	`not set`	Directory for the frontend compiled-model cache. When set, OpenVINO compiled models are exported as blobs and imported on later runs to skip weight requantization, graph conversion, and compilation for matching single-graph models.
`GGML_OPENVINO_PREFILL_CHUNK_SIZE`	Integer	`256`	Token chunk size for NPU prefill (NPU-only; ignored on CPU/GPU). Must be a positive integer; otherwise the default is used.
`GGML_OPENVINO_NPU_COMPILE_CONFIG`	String	`not set`	NPU-only compiler mode parameters forwarded to OpenVINO as `NPU_COMPILATION_MODE_PARAMS`, for example `optimization-level=3`.
`GGML_OPENVINO_STATEFUL_EXECUTION`	Boolean	`0`	Enable stateful KV cache for better performance. Recommended on CPU, GPU.
`GGML_OPENVINO_DISABLE_CACHE`	Boolean	`0`	Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable.
`GGML_OPENVINO_DISABLE_KV_SLICE`	Boolean	`0`	Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable.
`GGML_OPENVINO_MANUAL_GQA_ATTN`	Boolean	device-based	Tri-state. When unset, manual GQA attention is enabled by default on `GPU` and disabled on other devices. Set to a positive integer to force-enable, or `0` to force-disable.
`GGML_OPENVINO_MEMORY_OPTIMIZE`	Boolean	`0`	Umbrella switch for compile-time memory reductions. Enables `GGML_OPENVINO_REDUCE_COMPILE_MEM` and, on GPU, `GGML_OPENVINO_RELEASE_WEIGHTS` unless those fine-grained variables are explicitly set.
`GGML_OPENVINO_REDUCE_COMPILE_MEM`	Boolean	inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE`	Reduce compile-time host memory use by streaming weight requantization and avoiding extra weight-node materialization where possible. Set explicitly to override the umbrella switch.
`GGML_OPENVINO_RELEASE_WEIGHTS`	Boolean	inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` on GPU	GPU-only. Release host weight buffers after the compiled model cache can reuse the device/plugin copy. Requires stable graph shapes; dynamic workloads that need recompilation should leave this disabled.
`GGML_OPENVINO_PROFILING`	Boolean	`0`	Enable execution-time profiling.
`GGML_OPENVINO_DUMP_CGRAPH`	Boolean	`0`	Dump the GGML compute graph to `cgraph_ov.txt`.
`GGML_OPENVINO_DUMP_IR`	Boolean	`0`	Serialize OpenVINO IR files with timestamps.
`GGML_OPENVINO_DEBUG_INPUT`	Boolean	`0`	Enable input debugging and print input tensor info.
`GGML_OPENVINO_DEBUG_OUTPUT`	Boolean	`0`	Enable output debugging and print output tensor info.
`GGML_OPENVINO_PRINT_CGRAPH_TENSOR_ADDRESS`	Boolean	`0`	Print tensor address map once.
`GGML_OPENVINO_LOG_UNSUPPORTED_OPS`	Boolean	`0`	Log warning messages with tensor details and rejection reasons for any ops not supported by the OpenVINO backend. Emits at `WARN` level (requires `--log-verbosity >= 2`, enabled by default).

[!NOTE]

GGML_OPENVINO_STATEFUL_EXECUTION is an Experimental feature to allow stateful execution for managing the KV cache internally inside the OpenVINO model, improving performance on CPUs and GPUs. Stateful execution is not effective on NPUs, and not all models currently support this feature. This feature is experimental and has been validated only with the llama-simple, llama-cli, llama-bench, and llama-run applications and is recommended to enable for the best performance. Other applications, such as llama-server and llama-perplexity, are not yet supported.

GGML_OPENVINO_LOG_UNSUPPORTED_OPS emits logs at WARN level (GGML_LOG_WARN), which requires application log verbosity --log-verbosity >= 2 (or -lv 2).

Example Usage

GPU Inference with Profiling

# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.

# Linux
export GGML_OPENVINO_CACHE_DIR=/tmp/ov_cache
export GGML_OPENVINO_PROFILING=1
export GGML_OPENVINO_DEVICE=GPU
export GGML_OPENVINO_STATEFUL_EXECUTION=1

./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "

# Windows Command Line
set GGML_OPENVINO_CACHE_DIR=C:\tmp\ov_cache
set GGML_OPENVINO_PROFILING=1
set GGML_OPENVINO_DEVICE=GPU
set GGML_OPENVINO_STATEFUL_EXECUTION=1

# Windows PowerShell
$env:GGML_OPENVINO_CACHE_DIR = "C:\tmp\ov_cache"
$env:GGML_OPENVINO_PROFILING = "1"
$env:GGML_OPENVINO_DEVICE = "GPU"
$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"

build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "

Known Limitations

General (all devices)

Llama.cpp OpenVINO backend currently supports a subset of GGML ops and text-only models. Unsupported ops or unsupported op shapes/cases fail during OpenVINO translation.
Multimodal features (audio/image/video) are a work in progress.
Limited Embedding and Reranking model support.
Llama.cpp tool coverage across CPU/GPU/NPU is not uniform.

Tool-specific

llama-bench: requires -fa 1 (flash-attention).
llama-cli --context-shift: stateless only (GGML_OPENVINO_STATEFUL_EXECUTION=0). In stateful mode the KV cache is owned by the OpenVINO model and cannot be shifted externally.
llama-server: only one chat session/thread when GGML_OPENVINO_STATEFUL_EXECUTION=1.

GPU-specific

llama-server -np > 1: concurrent requests are batched together, which may slightly reduce per-request throughput.

NPU-specific

Default context resolves to the model's training context (e.g. 131072 for Llama 3.2 1B), which can OOM or fail or degrade performance on NPU. Inspect the resolved value with -lv 3.
- Workaround: Pass an explicit -c <N>, e.g. -c 1024.
NPU device uses a static graph with a fixed prefill chunk size (defaults to 256), configurable with GGML_OPENVINO_PREFILL_CHUNK_SIZE. Large prefill/batch settings may need tuning.
llama-server -np > 1 (multiple parallel sequences) is not supported.
llama-perplexity: requires -b 512 or smaller.

[!NOTE] The OpenVINO backend is actively under development. Fixes and improvements are underway, and this document will continue to be updated.

Work in Progress

Performance and memory optimizations
Accuracy validation
Broader quantization coverage
Support for additional model architectures

1# OpenVINO Backend for llama.cpp
2
3> [!NOTE]
4> Performance and memory optimizations, accuracy validation, broader quantization coverage, broader operator and model support are work in progress.
5
6[OpenVINO](https://docs.openvino.ai/) is an open-source toolkit for optimizing and deploying high-performance AI inference, specifically designed for Intel hardware, including CPUs, GPUs, and NPUs, in the cloud, on-premises, and on the edge. [OpenVINO backend for llama.cpp](../../ggml/src/ggml-openvino) enables hardware-accelerated inference on **Intel® CPUs, GPUs, and NPUs** while remaining compatible with the existing **GGUF model ecosystem**. The backend translates GGML compute graphs into OpenVINO graphs and leverages graph compilation, kernel fusion, and device-specific optimizations to improve inference performance on supported Intel hardware.
7
8The OpenVINO backend is implemented in `ggml/src/ggml-openvino` and provides a translation layer for core GGML operations. The OpenVINO backend replaces the standard GGML graph execution path with Intel's OpenVINO inference engine. This approach allows the same GGUF model file to run on Intel CPUs, Intel GPUs (integrated and discrete), and Intel NPUs without changes to the model or the rest of the llama.cpp stack. When a `ggml_cgraph` is dispatched to OpenVINO backend, it:
9
10- Walks the GGML graph and identifies inputs, outputs, weights, and KV cache tensors.
11- Translates the GGML operations into an `ov::Model` using OpenVINO's frontend API.
12- Compiles and caches the model for the target device.
13- Binds GGML tensor memory to OpenVINO inference tensors and runs inference.
14
15## Contents
16
17- [Supported Devices](#supported-devices)
18- [Supported Model Precisions](#supported-model-precisions)
19- [Supported Llama.cpp Tools](#supported-llamacpp-tools)
20- [Validated Models](#validated-models)
21- [Build Instructions](#build-instructions)
22  - [0. Prerequisites](#0-prerequisites)
23  - [1. Install OpenVINO Runtime](#1-install-openvino-runtime)
24  - [2. Build llama.cpp with OpenVINO Backend](#2-build-llamacpp-with-openvino-backend)
25    - [Ubuntu Build Script](#ubuntu-build-script)
26    - [Windows Build Script](#windows-build-script)
27  - [3. Download Sample Model](#3-download-sample-model)
28  - [4. Run Inference with OpenVINO Backend](#4-run-inference-with-openvino-backend)
29  - [5. Docker Build](#5-docker-build)
30- [GGML OpenVINO Backend Runtime Configurations](#ggml-openvino-backend-runtime-configurations)
31- [Known Limitations](#known-limitations)
32- [Work in Progress](#work-in-progress)
33
34## Supported Devices
35
36OpenVINO backend supports the following hardware:
37
38- Intel CPUs
39- Intel GPUs (integrated and discrete)
40- Intel NPUs
41
42Although OpenVINO supports a wide range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), the llama.cpp OpenVINO backend has been validated specifically on AI PCs such as the Intel® Core™ Ultra Series 1 and Series 2.
43
44## Supported Model Precisions
45
46- `FP16`
47- `BF16` (on Intel Xeon)
48- `Q8_0`
49- `Q4_0`
50- `Q4_1`
51- `Q4_K`
52- `Q4_K_M`
53- `Q5_K` (converted to `Q8_0_C` at runtime)
54- `Q6_K` (converted to `Q8_0_C` at runtime)
55
56> [!NOTE]
57> Accuracy validation and performance optimizations for quantized models are a work in progress.
58
59**CPU and GPU Quantization Details:**
60- `Q5_K` and `Q6_K` tensors are converted to `Q8_0_C`
61
62**NPU Quantization Details:**
63- Primary supported quantization scheme is `Q4_0`
64- `Q6_K` tensors are requantized to `Q4_0_128` in general. For embedding weights, `Q6_K` tensors are requantized to `Q8_0_C` except for the token embedding matrix which is dequantized to fp16
65
66**Additional Notes:**
67- Both `Q4_0` and `Q4_1` models use `Q6_K` for the token embedding tensor and the final matmul weight tensor (often the same tensor)
68- `Q4_0` models may produce some `Q4_1` tensors if an imatrix is provided during quantization using `llama-quantize`
69- `Q4_K_M` models may include both `Q6_K` and `Q5_K` tensors (observed in Phi-3)
70- `Q5_1` tensors are dequantized natively (weights, scales, and zero-points extracted directly)
71
72## Supported Llama.cpp Tools
73
74The OpenVINO backend integrates with the standard llama.cpp tools listed below.
75However, all the tools coverage across all devices is not uniform and exhaustive validation is work in progress.
76
77- llama-bench
78- llama-cli
79- llama-completion
80- llama-embedding
81- llama-perplexity
82- llama-run
83- llama-server
84- llama-simple
85
86## Validated Models
87
88Although, the validated models below were tested with `llama-cli` using the `Q4_K_M` quantization format on Intel® Core™ Ultra Series 2 (Lunar Lake), the OpenVINO backend is expected to work across a broader range of [Intel hardware](https://docs.openvino.ai/2026/about-openvino/release-notes-openvino/system-requirements.html), [supported model precisions](#supported-model-precisions), [supported llama.cpp tools](#supported-llamacpp-tools) and additional model architectures.
89
90> [!NOTE]
91> Extensive accuracy validation, performance optimizations, and broader architecture coverage are work in progress.
92
93**Legend & Test Configuration:**
94- **Status:** ✓ = Passed | ✗ = Failed or Unsupported
95- **Execution Modes:**
96  - **SL** = Stateless (`GGML_OPENVINO_STATEFUL_EXECUTION=0`)
97  - **SF** = Stateful (`GGML_OPENVINO_STATEFUL_EXECUTION=1`)
98  - Note: The NPU operates in stateless mode only.
99- **Validation system:** Intel® Core™ Ultra 5 238V (Lunar Lake) | 32 GB RAM | Ubuntu 24.04 | Intel OpenCL GPU Driver 26.31.39395.13-0 | Intel NPU Driver 1.35.0.
100- See [Known Limitations](#known-limitations) for context on observed failures.
101
102| Model | CPU (SL / SF) | GPU (SL / SF) | NPU (SL) |
103| :--- | :---: | :---: | :---: |
104| [bartowski/Llama-3.2-1B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
105| [bartowski/Llama-3.2-3B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
106| [bartowski/Meta-Llama-3.1-8B-Instruct-Q4_K_M](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
107|  |  |  |  |
108| [Qwen/qwen2.5-1.5b-instruct-q4_k_m](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
109| [Qwen/qwen2.5-coder-7b-instruct-q4_k_m](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
110| [bartowski/Qwen_Qwen3-0.6B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-0.6B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
111| [bartowski/Qwen_Qwen3-1.7B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3-1.7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
112| [Qwen/Qwen3-4B-Q4_K_M](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
113| [lm-kit/Qwen3-8B-Q4_K_M](https://huggingface.co/lm-kit/qwen-3-8b-instruct-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |
114| [bartowski/Qwen_Qwen3.5-0.8B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-0.8B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |
115| [bartowski/Qwen_Qwen3.5-2B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-2B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |
116| [bartowski/Qwen_Qwen3.5-4B-Q4_K_M](https://huggingface.co/bartowski/Qwen_Qwen3.5-4B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |
117| [lmstudio-community/Qwen3.5-9B-Q4_K_M](https://huggingface.co/lmstudio-community/Qwen3.5-9B-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |
118|  |  |  |  |
119| [unsloth/gemma-3-4b-it-Q4_K_M](https://huggingface.co/unsloth/gemma-3-4b-it-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
120| [bartowski/google_gemma-4-E2B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF) | ✓ / ✗ | ✓ / ✗ | ✗ |
121| [bartowski/google_gemma-4-E4B-it-Q4_K_M](https://huggingface.co/bartowski/google_gemma-4-E4B-it-GGUF) | ✓ / ✗ | ✓ / ✗ | ✓ |
122| [bartowski/gemma-4-12B-it-Q4_K_M](https://huggingface.co/bartowski/gemma-4-12B-it-GGUF) | ✓ / ✗ | ✓ / ✗ | ✓ |
123|  |  |  |  |
124| [bartowski/Phi-3-mini-4k-instruct-Q4_K_M](https://huggingface.co/bartowski/Phi-3-mini-4k-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
125| [bartowski/Phi-3.5-mini-instruct-Q4_K_M](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
126| [bartowski/microsoft_Phi-4-mini-instruct-Q4_K_M](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
127|  |  |  |  |
128| [bartowski/Mistral-7B-Instruct-v0.3-Q4_K_M](https://huggingface.co/bartowski/Mistral-7B-Instruct-v0.3-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
129| [QuantFactory/Ministral-3b-instruct.Q4_K_M](https://huggingface.co/QuantFactory/Ministral-3b-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
130| [bartowski/Ministral-8B-Instruct-2410-Q4_K_M](https://huggingface.co/bartowski/Ministral-8B-Instruct-2410-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
131|  |  |  |  |
132| [bartowski/DeepSeek-R1-Distill-Llama-8B-Q4_K_M](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
133| [bartowski/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
134|  |  |  |  |
135| [ibm-granite/granite-4.0-350m-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-350m-GGUF) | ✓ / ✓ | ✗ / ✗ | ✓ |
136| [ibm-granite/granite-4.0-micro-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-micro-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
137| [ibm-granite/granite-4.0-1b-Q4_K_M](https://huggingface.co/ibm-granite/granite-4.0-1b-GGUF) | ✓ / ✓ | ✗ / ✗ | ✗ |
138| [ibm-research/granite-3.2-8b-instruct-Q4_K_M](https://huggingface.co/ibm-research/granite-3.2-8b-instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
139|  |  |  |  |
140| [HuggingFaceTB/smollm2-1.7b-instruct-q4_k_m](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
141| [openbmb/MiniCPM-V-2_6-Q4_K_M](https://huggingface.co/openbmb/MiniCPM-V-2_6-gguf) | ✓ / ✓ | ✓ / ✓ | ✓ |
142| [bartowski/tencent_Hunyuan-7B-Instruct-Q4_K_M](https://huggingface.co/bartowski/tencent_Hunyuan-7B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
143| [LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-Q4_K_M](https://huggingface.co/LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
144| [bartowski/prism-ml_Bonsai-8B-unpacked-Q4_K_M](https://huggingface.co/bartowski/prism-ml_Bonsai-8B-unpacked-GGUF) | ✓ / ✓ | ✓ / ✓ | ✓ |
145|  |  |  |  |
146| [gpustack/bge-m3-Q4_K_M.gguf](https://huggingface.co/gpustack/bge-m3-GGUF) | ✓ | ✗ | ✗ |
147
148## Build Instructions
149
150### 0. Prerequisites
151
152- Linux or Windows system with Intel hardware (CPU, GPU, or NPU)
153- **For Intel GPU or NPU Usage**: Install the appropriate hardware drivers for your Intel GPU or NPU. For detailed instructions, see: [Additional Configurations for Hardware Acceleration](https://docs.openvino.ai/2026/get-started/install-openvino/configurations.html).
154
155- **Linux:**
156    - Git, CMake, and Ninja software tools are needed for building.
157    ```bash
158      sudo apt-get update
159      sudo apt-get install -y build-essential libcurl4-openssl-dev libtbb12 cmake ninja-build python3-pip curl wget tar
160    ```
161    - OpenCL
162    ```bash
163      sudo apt install ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
164    ```
165
166- **Windows:**
167  - Download and install [Microsoft Visual Studio 2022 Build Tools](https://aka.ms/vs/17/release/vs_BuildTools.exe). During installation, select the **"Desktop development with C++"** workload.
168
169  - Install required tools:
170    ```powershell
171    # Windows PowerShell
172    winget install Git.Git
173    winget install GNU.Wget
174    winget install Ninja-build.Ninja
175    ```
176
177  - Install **OpenCL** using **vcpkg**:
178    ```powershell
179    # Windows PowerShell
180    cd C:\
181    git clone https://github.com/microsoft/vcpkg
182    cd vcpkg
183    .\bootstrap-vcpkg.bat
184    .\vcpkg install opencl
185    # Optional but recommended: Integrate vcpkg with Visual Studio / CMake:
186    .\vcpkg integrate install
187    ```
188
189### 1. Install OpenVINO Runtime
190
191- Follow the guide to install OpenVINO Runtime from an archive file: [Linux](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-linux.html) | [Windows](https://docs.openvino.ai/2026/get-started/install-openvino/install-openvino-archive-windows.html)
192
193- Verify OpenVINO is initialized properly:
194  ```bash
195  echo $OpenVINO_DIR
196  ```
197
198### 2. Build llama.cpp with OpenVINO Backend
199
200Clone llama.cpp repo and build :
201
202```bash
203git clone https://github.com/ggml-org/llama.cpp
204cd llama.cpp
205```
206
207- **Linux:**
208```bash
209source /opt/intel/openvino/setupvars.sh
210cmake -B build/ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
211cmake --build build/ReleaseOV --parallel
212```
213
214- **Windows:** Open **x64 Native Tools Command Prompt for VS** (so the MSVC toolchain is on `PATH`), then run:
215
216```cmd
217C:\Intel\openvino\setupvars.bat
218cmake -B build\ReleaseOV -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake
219cmake --build build\ReleaseOV --parallel
220```
221
222> [!NOTE]
223> The Windows install path is `C:\Intel\openvino` (no spaces) to avoid quoting problems some CMake/Ninja toolchains have with `C:\Program Files (x86)\...`. Adjust to wherever you installed OpenVINO Runtime. From `cmd`, run `C:\Intel\openvino\setupvars.bat`; from PowerShell, run `& "C:\Intel\openvino\setupvars.ps1"` instead. Once the build is finished you can launch the binaries from any `cmd` or `PowerShell` window after sourcing the matching `setupvars` script for that shell.
224
225#### Ubuntu Build Script
226
227For Ubuntu24 users, the following shell script automates the prerequisite installs (build tools, OpenCL ICD), the OpenVINO Runtime download/extract/setup, and the Ninja-based llama.cpp build.
228Save the following as `build-llamacpp-ov.sh` next to where you want the `llama.cpp` folder to land, then run it:
229
230```bash
231chmod +x build-llamacpp-ov.sh
232./build-llamacpp-ov.sh
233```
234
235<details>
236<summary>Click to expand <code>build-llamacpp-ov.sh</code></summary>
237
238```bash
239#!/usr/bin/env bash
240# ============================================
241# llama.cpp OpenVINO Build Script (Ninja)
242# ============================================
243set -euo pipefail
244
245OPENVINO_VERSION_MAJOR="2026.3.1"
246OPENVINO_VERSION_FULL="2026.3.1.22476.56d9685302d"
247
248SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
249OPENVINO_INSTALL_DIR="/opt/intel/openvino_${OPENVINO_VERSION_MAJOR}"
250OPENVINO_LINK_DIR="/opt/intel/openvino"
251OPENVINO_TGZ="${SCRIPT_DIR}/openvino.tgz"
252OPENVINO_URL="https://storage.openvinotoolkit.org/repositories/openvino/packages/${OPENVINO_VERSION_MAJOR}/linux/openvino_toolkit_ubuntu24_${OPENVINO_VERSION_FULL}_x86_64.tgz"
253
254echo "============================================"
255echo "Installing prerequisites (apt)..."
256echo "============================================"
257sudo apt-get update
258sudo apt-get install -y \
259    build-essential libcurl4-openssl-dev libtbb12 \
260    cmake ninja-build python3-pip \
261    curl wget tar git
262
263echo "============================================"
264echo "Installing OpenCL runtime + headers..."
265echo "============================================"
266sudo apt-get install -y \
267    ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers intel-opencl-icd
268
269cd "${SCRIPT_DIR}"
270
271# ============================================
272# Clone llama.cpp if missing
273# ============================================
274if [[ ! -f "llama.cpp/CMakeLists.txt" ]]; then
275    echo "Cloning llama.cpp..."
276    git clone https://github.com/ggml-org/llama.cpp
277fi
278
279# ============================================
280# Setup OpenVINO: download & extract to /opt/intel/openvino_${OPENVINO_VERSION_MAJOR},
281# then point /opt/intel/openvino at it via symlink so the active version is swappable.
282# ============================================
283if [[ -f "${OPENVINO_INSTALL_DIR}/setupvars.sh" ]]; then
284    echo "OpenVINO ${OPENVINO_VERSION_MAJOR} already installed at ${OPENVINO_INSTALL_DIR}. Skipping download."
285else
286    echo "OpenVINO not found at ${OPENVINO_INSTALL_DIR}. Starting download..."
287    curl -L -o "${OPENVINO_TGZ}" "${OPENVINO_URL}"
288
289    echo "Extracting OpenVINO to ${OPENVINO_INSTALL_DIR}..."
290    sudo mkdir -p "${OPENVINO_INSTALL_DIR}"
291    sudo tar -xzf "${OPENVINO_TGZ}" -C "${OPENVINO_INSTALL_DIR}" --strip-components=1
292    rm -f "${OPENVINO_TGZ}"
293fi
294
295# Refresh symlink: /opt/intel/openvino -> /opt/intel/openvino_${OPENVINO_VERSION_MAJOR}
296sudo ln -sfn "${OPENVINO_INSTALL_DIR}" "${OPENVINO_LINK_DIR}"
297
298OPENVINO_ROOT="${OPENVINO_LINK_DIR}"
299echo "OpenVINO Ready: ${OPENVINO_ROOT} -> ${OPENVINO_INSTALL_DIR}"
300
301# Install OpenVINO's own runtime dependencies (one-time per system).
302if [[ -x "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh" ]]; then
303    echo "============================================"
304    echo "Installing OpenVINO runtime dependencies..."
305    echo "============================================"
306    echo "Y" | sudo -E "${OPENVINO_ROOT}/install_dependencies/install_openvino_dependencies.sh"
307fi
308
309# ============================================
310# Clean old build cache
311# ============================================
312cd "${SCRIPT_DIR}/llama.cpp"
313if [[ -d "build/ReleaseOV" ]]; then
314    echo "Removing old build directory..."
315    rm -rf "build/ReleaseOV"
316fi
317
318echo "============================================"
319echo "Configuring with CMake..."
320echo "============================================"
321set +u
322source "${OPENVINO_ROOT}/setupvars.sh"
323set -u
324
325cmake -B build/ReleaseOV -G Ninja \
326    -DCMAKE_BUILD_TYPE=Release \
327    -DGGML_OPENVINO=ON
328
329cmake --build build/ReleaseOV --parallel
330
331echo "============================================"
332echo "Build completed successfully!"
333echo "============================================"
334echo "Binaries: $(pwd)/build/ReleaseOV/bin"
335echo
336echo "NOTE: To run, source setupvars.sh and pick a device:"
337echo "  source /opt/intel/openvino/setupvars.sh"
338echo "  export GGML_OPENVINO_DEVICE=CPU   # or GPU / NPU"
339echo "  ./build/ReleaseOV/bin/llama-cli -m model.gguf"
340```
341
342> [!NOTE]
343> The script pins OpenVINO `2026.3.1` via the `OPENVINO_VERSION_MAJOR` / `OPENVINO_VERSION_FULL` variables at the top — edit them to track a different release.
344
345</details>
346
347#### Windows Build Script
348
349For Windows users, the following `.bat` script automates the prerequisite installs (Git, Ninja, CMake, Visual Studio 2022 Build Tools, vcpkg + OpenCL), the OpenVINO Runtime download/extract, and the Ninja-based llama.cpp build.
350Save the following as `build-llamacpp-ov.bat` next to where you want the `llama.cpp` to land, then run it from either **Command Prompt** or **PowerShell**:
351
352```cmd
353:: Command Prompt
354build-llamacpp-ov.bat
355```
356
357```powershell
358# PowerShell
359.\build-llamacpp-ov.bat
360```
361
362<details>
363<summary>Click to expand <code>build-llamacpp-ov.bat</code></summary>
364
365```bat
366@echo off
367setlocal enabledelayedexpansion
368
369REM ============================================
370REM llama.cpp OpenVINO Build Script (Ninja)
371REM ============================================
372
373set "OPENVINO_VERSION_MAJOR=2026.3.1"
374set "OPENVINO_VERSION_FULL=2026.3.1.22476.56d9685302d"
375
376set "SCRIPT_DIR=%~dp0"
377set "VCPKG_DIR=C:\vcpkg"
378set "OPENVINO_INSTALL_DIR=C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%"
379set "OPENVINO_LINK_DIR=C:\Intel\openvino"
380set "OPENVINO_ZIP=%SCRIPT_DIR%openvino.zip"
381set "OPENVINO_EXTRACT_TMP=%SCRIPT_DIR%openvino_extract_tmp"
382set "OPENVINO_URL=https://storage.openvinotoolkit.org/repositories/openvino/packages/%OPENVINO_VERSION_MAJOR%/windows/openvino_toolkit_windows_%OPENVINO_VERSION_FULL%_x86_64.zip"
383
384echo ============================================
385echo Installing prerequisites...
386echo ============================================
387winget install --id Git.Git -e --accept-source-agreements --accept-package-agreements 2>nul
388winget install --id Ninja-build.Ninja -e --accept-source-agreements --accept-package-agreements 2>nul
389winget install --id Kitware.CMake -e --accept-source-agreements --accept-package-agreements 2>nul
390
391REM Ensure Visual Studio Build Tools are installed.
392echo Checking for Visual Studio Build Tools...
393set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
394set "VS_INSTALLED="
395if exist "%VSWHERE%" (
396    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products * -requires Microsoft.VisualStudio.Component.VC.Tools.x86.x64 -property installationPath 2^>nul`) do (
397        set "VS_INSTALLED=%%i"
398    )
399)
400if defined VS_INSTALLED (
401    echo Visual Studio with VC++ x86/x64 tools already present at "!VS_INSTALLED!". Skipping winget install.
402) else (
403    winget install --id Microsoft.VisualStudio.2022.BuildTools -e --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended" --accept-source-agreements --accept-package-agreements
404    if errorlevel 1 (
405        echo WARNING: winget could not install Visual Studio Build Tools automatically.
406        echo Install manually from https://aka.ms/vs/17/release/vs_BuildTools.exe ^(select the "Desktop development with C++" workload^)
407        echo and re-run this script from a "Developer Command Prompt for VS 2022".
408    )
409)
410
411echo ============================================
412echo Installing OpenCL via vcpkg...
413echo ============================================
414if not exist "%VCPKG_DIR%" (
415    git clone https://github.com/microsoft/vcpkg "%VCPKG_DIR%"
416    cd /d "%VCPKG_DIR%"
417    call bootstrap-vcpkg.bat
418    call vcpkg integrate install
419)
420cd /d "%VCPKG_DIR%"
421call vcpkg install opencl
422
423cd /d "%SCRIPT_DIR%"
424
425REM ============================================
426REM Clone llama.cpp if missing
427REM ============================================
428if not exist "llama.cpp\CMakeLists.txt" (
429    echo Cloning llama.cpp...
430    git clone https://github.com/ggml-org/llama.cpp
431)
432
433cd /d "llama.cpp"
434set "SCRIPT_DIR=%CD%"
435
436REM ============================================
437REM Setup OpenVINO: download & extract to C:\Intel\openvino_%OPENVINO_VERSION_MAJOR%,
438REM then point C:\Intel\openvino at it via a directory junction (mklink /J).
439REM ============================================
440
441if exist "%OPENVINO_INSTALL_DIR%\setupvars.bat" (
442    echo OpenVINO %OPENVINO_VERSION_MAJOR% already installed at "%OPENVINO_INSTALL_DIR%". Skipping download.
443) else (
444    echo OpenVINO not found at "%OPENVINO_INSTALL_DIR%". Starting download...
445
446    curl -L -o "%OPENVINO_ZIP%" "%OPENVINO_URL%"
447    if errorlevel 1 (
448        echo ERROR: Download failed.
449        exit /b 1
450    )
451
452    echo Extracting OpenVINO...
453    if exist "%OPENVINO_EXTRACT_TMP%" rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
454    mkdir "%OPENVINO_EXTRACT_TMP%"
455    tar -xf "%OPENVINO_ZIP%" -C "%OPENVINO_EXTRACT_TMP%"
456    if errorlevel 1 (
457        echo ERROR: Extraction failed.
458        exit /b 1
459    )
460
461    REM Move the single top-level folder contents into the versioned install dir.
462    set "OPENVINO_EXTRACTED="
463    for /d %%i in ("%OPENVINO_EXTRACT_TMP%\*") do set "OPENVINO_EXTRACTED=%%i"
464    if not defined OPENVINO_EXTRACTED (
465        echo ERROR: Could not locate extracted OpenVINO folder under "%OPENVINO_EXTRACT_TMP%".
466        exit /b 1
467    )
468    if not exist "%OPENVINO_INSTALL_DIR%" mkdir "%OPENVINO_INSTALL_DIR%"
469    xcopy /e /i /y /q "!OPENVINO_EXTRACTED!\*" "%OPENVINO_INSTALL_DIR%\" >nul
470    if errorlevel 1 (
471        echo ERROR: Failed to copy OpenVINO from "!OPENVINO_EXTRACTED!" to "%OPENVINO_INSTALL_DIR%".
472        echo Re-run this script from an elevated Command Prompt ^(Run as administrator^) if access is denied.
473        exit /b 1
474    )
475
476    rmdir /s /q "%OPENVINO_EXTRACT_TMP%"
477    del "%OPENVINO_ZIP%"
478)
479
480REM Refresh junction: C:\Intel\openvino -> C:\Intel\openvino_<version>.
481REM `mklink /J` creates a directory junction (no admin / Developer Mode required).
482if exist "%OPENVINO_LINK_DIR%" rmdir "%OPENVINO_LINK_DIR%"
483mklink /J "%OPENVINO_LINK_DIR%" "%OPENVINO_INSTALL_DIR%" >nul
484if errorlevel 1 (
485    echo ERROR: Failed to create junction "%OPENVINO_LINK_DIR%" -^> "%OPENVINO_INSTALL_DIR%".
486    echo If "%OPENVINO_LINK_DIR%" already exists as a regular non-empty folder, remove it manually and re-run.
487    exit /b 1
488)
489
490set "OPENVINO_ROOT=%OPENVINO_LINK_DIR%"
491echo OpenVINO Ready: %OPENVINO_ROOT% -^> %OPENVINO_INSTALL_DIR%
492
493
494echo ============================================
495echo Setting up compiler environment...
496echo ============================================
497REM Locate Visual Studio Build Tools vcvars64.bat
498set "VSWHERE=%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe"
499if exist "%VSWHERE%" (
500    for /f "usebackq tokens=*" %%i in (`"%VSWHERE%" -latest -products Microsoft.VisualStudio.Product.BuildTools -property installationPath`) do (
501        set "VS_PATH=%%i"
502    )
503)
504if defined VS_PATH (
505    call "%VS_PATH%\VC\Auxiliary\Build\vcvars64.bat" >nul
506) else (
507    echo WARNING: Visual Studio Build Tools not found. Compiler may be missing.
508)
509
510REM ============================================
511REM Clean old build cache
512REM ============================================
513if exist "build\ReleaseOV" (
514    echo Removing old build directory ...
515    rmdir /s /q "build\ReleaseOV"
516)
517
518echo ============================================
519echo Configuring with CMake...
520echo ============================================
521call "%OPENVINO_ROOT%\setupvars.bat" >nul 2>nul
522
523cmake -B build\ReleaseOV -G Ninja ^
524    -DCMAKE_BUILD_TYPE=Release ^
525    -DGGML_OPENVINO=ON ^
526    -DCMAKE_TOOLCHAIN_FILE="%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake"
527
528if errorlevel 1 (
529    echo If you continue to face CMAKE errors, make sure to install:
530    echo   winget install Microsoft.VisualStudio.2022.BuildTools
531    echo   Then run the "Developer Command Prompt for VS 2022" and launch this script from there.
532    exit /b 1
533)
534
535cmake --build build\ReleaseOV --config Release
536if errorlevel 1 exit /b 1
537
538echo ============================================
539echo Build completed successfully!
540echo ============================================
541echo Binaries: %CD%\build\ReleaseOV\bin
542echo.
543echo NOTE: To run, source setupvars.bat and pick a device:
544echo   call "C:\Intel\openvino\setupvars.bat"
545echo   set GGML_OPENVINO_DEVICE=CPU   ^&^& REM or GPU / NPU
546echo   build\ReleaseOV\bin\llama-cli.exe -m model.gguf
547echo.
548
549endlocal
550```
551
552> [!NOTE]
553> The script pins OpenVINO `2026.3.1` via the `OPENVINO_VERSION_MAJOR` / `OPENVINO_VERSION_FULL` variables at the top — edit them to track a different release. From any new shell, source the matching `setupvars` script via the junction — `call "C:\Intel\openvino\setupvars.bat"` from `cmd`, or `& "C:\Intel\openvino\setupvars.ps1"` from PowerShell. If `winget` cannot register Visual Studio Build Tools on first run, install them once manually and re-run the script from an elevated **Developer Command Prompt for VS 2022**.
554
555</details>
556
557
558### 3. Download Sample Model
559
560Download sample model for testing.
561
562```bash
563# Linux
564mkdir -p ~/models/
565wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf \
566     -O ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
567
568# Windows PowerShell
569mkdir C:\models
570Invoke-WebRequest -Uri https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -OutFile C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
571
572# Windows Command Line
573mkdir C:\models
574curl -L https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf -o C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf
575```
576
577### 4. Run Inference with OpenVINO Backend
578
579When using the OpenVINO backend, the first inference token may have slightly higher latency due to on-the-fly conversion to the OpenVINO graph. Subsequent tokens and runs will be faster.
580
581> [!NOTE]
582> Default context size is set to the model training context, which may be very large. For example, 131072 for Llama 3.2 1B, which may result in lower performance, especially on edge/laptop devices. Use `-c` to limit context size in supported llama.cpp tools for better performance. For example, `-c 512`.
583
584```bash
585# If device is unset or unavailable, defaults to CPU.
586# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.
587
588# Linux
589export GGML_OPENVINO_DEVICE=GPU
590# Optional: enable stateful execution for improved GPU performance (recommended).
591export GGML_OPENVINO_STATEFUL_EXECUTION=1
592# To run llama-simple:
593./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
594# To run in chat mode:
595./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024
596# To run llama-bench, -fa 1 is needed
597GGML_OPENVINO_STATEFUL_EXECUTION=1 GGML_OPENVINO_DEVICE=GPU ./build/ReleaseOV/bin/llama-bench -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -fa 1
598
599# NPU: keep context small to avoid failures from very large model context windows.
600export GGML_OPENVINO_DEVICE=NPU
601./build/ReleaseOV/bin/llama-cli -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 512
602
603# Windows Command Line
604set GGML_OPENVINO_DEVICE=GPU
605# Optional: enable stateful execution for improved GPU performance (recommended).
606set GGML_OPENVINO_STATEFUL_EXECUTION=1
607# Windows PowerShell
608$env:GGML_OPENVINO_DEVICE = "GPU"
609$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"
610
611# To run llama-simple
612build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
613# To run in chat mode:
614build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 1024
615# To run llama-bench, -fa 1 is needed
616build\ReleaseOV\bin\llama-bench.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -fa 1
617
618# NPU: keep context small to avoid failures from very large model context windows.
619# Windows Command Line
620set GGML_OPENVINO_DEVICE=NPU
621# Windows PowerShell
622$env:GGML_OPENVINO_DEVICE = "NPU"
623build\ReleaseOV\bin\llama-cli.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -c 512
624```
625> [!NOTE]
626> On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. See [OpenVINO GPU Device](https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html) for more details.
627
628### 5. Docker Build
629
630You can build and run llama.cpp with OpenVINO backend using Docker.
631
632```bash
633# Build the base runtime image with compiled shared libraries and minimal dependencies.
634docker build -t llama-openvino:base -f .devops/openvino.Dockerfile .
635
636# Build the complete image with all binaries, Python tools, gguf-py library, and model conversion utilities.
637docker build --target=full -t llama-openvino:full -f .devops/openvino.Dockerfile .
638
639# Build a minimal CLI-only image containing just the llama-cli executable.
640docker build --target=light -t llama-openvino:light -f .devops/openvino.Dockerfile .
641
642# Builds a server-only image with llama-server executable, health check endpoint, and REST API support.
643docker build --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .
644
645# If you are behind a proxy:
646docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --target=server -t llama-openvino:server -f .devops/openvino.Dockerfile .
647```
648
649Run llama.cpp with OpenVINO backend Docker container.
650Save sample models in `~/models` as [shown above](#3-download-sample-model). It will be mounted to the container in the examples below.
651
652
653```bash
654#  Run Docker container
655docker run --rm -it -v ~/models:/models llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
656
657# With Intel GPU access (iGPU or dGPU)
658docker run --rm -it -v ~/models:/models \
659--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
660--env=GGML_OPENVINO_DEVICE=GPU --env=GGML_OPENVINO_STATEFUL_EXECUTION=1 \
661llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
662
663# With Intel NPU access
664docker run --rm -it -v ~/models:/models \
665--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
666--env=GGML_OPENVINO_DEVICE=NPU \
667llama-openvino:light --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf
668```
669
670Run Llama.cpp Server with OpenVINO Backend.
671> [!NOTE]
672> `llama-server` with OpenVINO backend supports only one chat session/thread, when `GGML_OPENVINO_STATEFUL_EXECUTION=1` is enabled.
673
674```bash
675# Run the llama-openvino:server Docker container (CPU)
676docker run --rm -it -p 8080:8080 -v ~/models:/models llama-openvino:server --no-warmup -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -c 1024 --host 0.0.0.0
677
678# Run the llama-openvino:server Docker container with Intel GPU access (iGPU or dGPU)
679docker run --rm -it -v ~/models:/models \
680--device=/dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
681-p 8080:8080 --env=GGML_OPENVINO_DEVICE=GPU  \
682llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0
683
684# Run the llama-openvino:server Docker container with Intel NPU access
685docker run --rm -it -v ~/models:/models \
686--device=/dev/accel --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -u $(id -u):$(id -g) \
687-p 8080:8080 --env=GGML_OPENVINO_DEVICE=NPU \
688llama-openvino:server --no-warmup -c 1024 -m /models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --host 0.0.0.0
689
690# Or Using llama-server executable
691./build/ReleaseOV/bin/llama-server -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf --port 8080 -c 1024
692
693# Option 1: Open your browser to http://localhost:8080 to access the web UI for the llama.cpp server.
694# Option 2: In a NEW terminal, test the server with curl
695
696# If you are behind a proxy, make sure to set NO_PROXY to avoid proxy for localhost
697export NO_PROXY=localhost,127.0.0.1
698
699# Test health endpoint
700curl -f http://localhost:8080/health
701
702# Test with a simple prompt
703curl -X POST "http://localhost:8080/v1/chat/completions" -H "Content-Type: application/json" \
704 -d '{"messages":[{"role":"user","content":"Write a poem about OpenVINO"}],"max_tokens":100}' | jq .
705```
706
707## GGML OpenVINO Backend Runtime Configurations
708
709The OpenVINO backend can be configured using the following environment variables at runtime to control device selection, caching, debugging, and profiling behavior.
710Boolean flags follow a uniform convention: set to a **positive integer** (e.g. `1`) to enable; unset, empty, `0`, negative, or non-numeric values are treated as disabled.
711
712| Variable                          | Type      | Default    | Description                                                                                                 |
713|-----------------------------------|-----------|------------|-------------------------------------------------------------------------------------------------------------|
714| `GGML_OPENVINO_DEVICE`            | String    | `CPU`      | Specify the target device (CPU, GPU, NPU). On systems with multiple GPUs, use `GPU.0` or `GPU.1` to explicitly target specific GPU. See [OpenVINO GPU Device](https://docs.openvino.ai/2026/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html). When set to **NPU**, static compilation mode is enabled for optimal performance. |
715| `GGML_OPENVINO_CACHE_DIR`         | String    | `not set`  | Directory for OpenVINO model caching (recommended: `/tmp/ov_cache`). Enables model caching when set. **Not supported on NPU devices.** |
716| `GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR` | String | `not set` | Directory for the frontend compiled-model cache. When set, OpenVINO compiled models are exported as blobs and imported on later runs to skip weight requantization, graph conversion, and compilation for matching single-graph models. |
717| `GGML_OPENVINO_PREFILL_CHUNK_SIZE`| Integer   | `256`      | Token chunk size for **NPU** prefill (NPU-only; ignored on CPU/GPU). Must be a positive integer; otherwise the default is used. |
718| `GGML_OPENVINO_NPU_COMPILE_CONFIG` | String | `not set` | NPU-only compiler mode parameters forwarded to OpenVINO as `NPU_COMPILATION_MODE_PARAMS`, for example `optimization-level=3`. |
719| `GGML_OPENVINO_STATEFUL_EXECUTION`| Boolean   | `0`        | Enable stateful KV cache for better performance. Recommended on CPU, GPU.                                   |
720| `GGML_OPENVINO_DISABLE_CACHE`     | Boolean   | `0`        | Disable the in-process compiled-model / decoder cache (cache is on by default). Set to `1` to disable.      |
721| `GGML_OPENVINO_DISABLE_KV_SLICE`  | Boolean   | `0`        | Disable the KV-cache input-tensor slicing optimization (slicing is on by default on CPU/GPU). Set to `1` to disable. |
722| `GGML_OPENVINO_MANUAL_GQA_ATTN`   | Boolean   | device-based | Tri-state. When **unset**, manual GQA attention is enabled by default on `GPU` and disabled on other devices. Set to a positive integer to force-enable, or `0` to force-disable. |
723| `GGML_OPENVINO_MEMORY_OPTIMIZE`   | Boolean   | `0`        | Umbrella switch for compile-time memory reductions. Enables `GGML_OPENVINO_REDUCE_COMPILE_MEM` and, on GPU, `GGML_OPENVINO_RELEASE_WEIGHTS` unless those fine-grained variables are explicitly set. |
724| `GGML_OPENVINO_REDUCE_COMPILE_MEM`| Boolean   | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` | Reduce compile-time host memory use by streaming weight requantization and avoiding extra weight-node materialization where possible. Set explicitly to override the umbrella switch. |
725| `GGML_OPENVINO_RELEASE_WEIGHTS`   | Boolean   | inherits from `GGML_OPENVINO_MEMORY_OPTIMIZE` on GPU | GPU-only. Release host weight buffers after the compiled model cache can reuse the device/plugin copy. Requires stable graph shapes; dynamic workloads that need recompilation should leave this disabled. |
726| `GGML_OPENVINO_PROFILING`         | Boolean   | `0`        | Enable execution-time profiling.                                                                            |
727| `GGML_OPENVINO_DUMP_CGRAPH`       | Boolean   | `0`        | Dump the GGML compute graph to `cgraph_ov.txt`.                                                             |
728| `GGML_OPENVINO_DUMP_IR`           | Boolean   | `0`        | Serialize OpenVINO IR files with timestamps.                                                                |
729| `GGML_OPENVINO_DEBUG_INPUT`       | Boolean   | `0`        | Enable input debugging and print input tensor info.                                                         |
730| `GGML_OPENVINO_DEBUG_OUTPUT`      | Boolean   | `0`        | Enable output debugging and print output tensor info.                                                       |
731| `GGML_OPENVINO_PRINT_CGRAPH_TENSOR_ADDRESS` | Boolean | `0` | Print tensor address map once.                                                                           |
732| `GGML_OPENVINO_LOG_UNSUPPORTED_OPS`| Boolean   | `0`        | Log warning messages with tensor details and rejection reasons for any ops not supported by the OpenVINO backend. Emits at `WARN` level (requires `--log-verbosity >= 2`, enabled by default). |
733
734> [!NOTE]
735> - `GGML_OPENVINO_STATEFUL_EXECUTION` is an **Experimental** feature to allow stateful execution for managing the KV cache internally inside the OpenVINO model, improving performance on CPUs and GPUs. Stateful execution is not effective on NPUs, and not all models currently support this feature. This feature is experimental and has been validated only with the llama-simple, llama-cli, llama-bench, and llama-run applications and is recommended to enable for the best performance. Other applications, such as llama-server and llama-perplexity, are not yet supported.
736> - `GGML_OPENVINO_LOG_UNSUPPORTED_OPS` emits logs at `WARN` level (`GGML_LOG_WARN`), which requires application log verbosity `--log-verbosity >= 2` (or `-lv 2`).
737
738### Example Usage
739
740#### GPU Inference with Profiling
741
742```bash
743# If the system has multiple GPUs, use GPU.0 or GPU.1 to explicitly target a specific GPU.
744
745# Linux
746export GGML_OPENVINO_CACHE_DIR=/tmp/ov_cache
747export GGML_OPENVINO_PROFILING=1
748export GGML_OPENVINO_DEVICE=GPU
749export GGML_OPENVINO_STATEFUL_EXECUTION=1
750
751./build/ReleaseOV/bin/llama-simple -m ~/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf -n 50 "The story of AI is "
752
753# Windows Command Line
754set GGML_OPENVINO_CACHE_DIR=C:\tmp\ov_cache
755set GGML_OPENVINO_PROFILING=1
756set GGML_OPENVINO_DEVICE=GPU
757set GGML_OPENVINO_STATEFUL_EXECUTION=1
758
759# Windows PowerShell
760$env:GGML_OPENVINO_CACHE_DIR = "C:\tmp\ov_cache"
761$env:GGML_OPENVINO_PROFILING = "1"
762$env:GGML_OPENVINO_DEVICE = "GPU"
763$env:GGML_OPENVINO_STATEFUL_EXECUTION = "1"
764
765build\ReleaseOV\bin\llama-simple.exe -m "C:\models\Llama-3.2-1B-Instruct-Q4_K_M.gguf" -n 50 "The story of AI is "
766
767```
768
769## Known Limitations
770
771**General (all devices)**
772
773- Llama.cpp OpenVINO backend currently supports a subset of GGML ops and text-only models. Unsupported ops or unsupported op shapes/cases fail during OpenVINO translation.
774- Multimodal features (audio/image/video) are a work in progress.
775- Limited Embedding and Reranking model support.
776- Llama.cpp tool coverage across CPU/GPU/NPU is not uniform.
777
778**Tool-specific**
779
780- `llama-bench`: requires `-fa 1` (flash-attention).
781- `llama-cli --context-shift`: stateless only (`GGML_OPENVINO_STATEFUL_EXECUTION=0`). In stateful mode the KV cache is owned by the OpenVINO model and cannot be shifted externally.
782- `llama-server`: only one chat session/thread when `GGML_OPENVINO_STATEFUL_EXECUTION=1`.
783
784**GPU-specific**
785
786- `llama-server -np > 1`: concurrent requests are batched together, which may slightly reduce per-request throughput.
787
788**NPU-specific**
789
790- Default context resolves to the model's training context (e.g. 131072 for Llama 3.2 1B), which can OOM or fail or degrade performance on NPU. Inspect the resolved value with `-lv 3`.
791  - **Workaround:** Pass an explicit `-c <N>`, e.g. `-c 1024`.
792- NPU device uses a static graph with a fixed prefill chunk size (defaults to 256), configurable with `GGML_OPENVINO_PREFILL_CHUNK_SIZE`. Large prefill/batch settings may need tuning.
793- `llama-server -np > 1` (multiple parallel sequences) is not supported.
794- `llama-perplexity`: requires `-b 512` or smaller.
795
796> [!NOTE]
797> The OpenVINO backend is actively under development. Fixes and improvements are underway, and this document will continue to be updated.
798
799## Work in Progress
800
801- Performance and memory optimizations
802- Accuracy validation
803- Broader quantization coverage
804- Support for additional model architectures