char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Vishal Singhggml-zendnn : add ZenDNN backend for AMD CPUs (#17690)017761daf

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
2.2 KiB66 linesraw

llama.cpp for IBM zDNN Accelerator

[!WARNING] Note: zDNN is not the same as ZenDNN.

  • zDNN (this page): IBM's Deep Neural Network acceleration library for IBM Z & LinuxONE Mainframes
  • ZenDNN: AMD's deep learning library for AMD EPYC CPUs (see ZenDNN documentation)

Background

IBM zDNN (Z Deep Neural Network) is a hardware acceleration library designed specifically to leverage the IBM NNPA (Neural Network Processor Assist) accelerator located within IBM Telum I and II processors. It provides significant performance improvements for neural network inference operations.

Llama.cpp + IBM zDNN

The llama.cpp zDNN backend is designed to enable llama.cpp on IBM z17 and later systems via the IBM zDNN hardware acceleration library.

Software & Hardware Support

Hardware LevelStatusVerified
IBM z17 / LinuxONE 5SupportedRHEL 9.6, IBM z17, 40 IFLs
IBM z16 / LinuxONE 4Not Supported

Data Types Supported

Data TypeStatus
F32Supported
F16Supported
BF16Supported

CMake Options

The IBM zDNN backend has the following CMake options that control the behaviour of the backend.

CMake OptionDefault ValueDescription
GGML_ZDNNOFFCompile llama.cpp with zDNN support
ZDNN_ROOT""Override zDNN library lookup

1. Install zDNN Library

Note: Using the zDNN library provided via apt or yum may not work correctly as reported in #15772. It is preferred that you compile from source.

git clone --recurse-submodules https://github.com/IBM/zDNN
cd zDNN

autoreconf .
./configure --prefix=/opt/zdnn-libs

make build
sudo make install

2. Build llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -S . -G Ninja -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_ZDNN=ON \
    -DZDNN_ROOT=/opt/zdnn-libs
cmake --build build --config Release -j$(nproc)
1# llama.cpp for IBM zDNN Accelerator
2
3> [!WARNING]
4> **Note:** zDNN is **not** the same as ZenDNN.
5> - **zDNN** (this page): IBM's Deep Neural Network acceleration library for IBM Z & LinuxONE Mainframes
6> - **ZenDNN**: AMD's deep learning library for AMD EPYC CPUs ([see ZenDNN documentation](ZenDNN.md))
7
8## Background
9
10IBM zDNN (Z Deep Neural Network) is a hardware acceleration library designed specifically to leverage the IBM NNPA (Neural Network Processor Assist) accelerator located within IBM Telum I and II processors. It provides significant performance improvements for neural network inference operations.
11
12### Llama.cpp + IBM zDNN
13
14The llama.cpp zDNN backend is designed to enable llama.cpp on IBM z17 and later systems via the IBM zDNN hardware acceleration library.
15
16## Software & Hardware Support
17
18| Hardware Level       | Status        | Verified                   |
19| -------------------- | ------------- | -------------------------- |
20| IBM z17 / LinuxONE 5 | Supported     | RHEL 9.6, IBM z17, 40 IFLs |
21| IBM z16 / LinuxONE 4 | Not Supported |                            |
22
23## Data Types Supported
24
25| Data Type | Status    |
26| --------- | --------- |
27| F32       | Supported |
28| F16       | Supported |
29| BF16      | Supported |
30
31## CMake Options
32
33The IBM zDNN backend has the following CMake options that control the behaviour of the backend.
34
35| CMake Option | Default Value | Description                         |
36| ------------ | ------------- | ----------------------------------- |
37| `GGML_ZDNN`  | `OFF`         | Compile llama.cpp with zDNN support |
38| `ZDNN_ROOT`  | `""`          | Override zDNN library lookup        |
39
40## 1. Install zDNN Library
41
42Note: Using the zDNN library provided via `apt` or `yum` may not work correctly as reported in [#15772](https://github.com/ggml-org/llama.cpp/issues/15772). It is preferred that you compile from source.
43
44```sh
45git clone --recurse-submodules https://github.com/IBM/zDNN
46cd zDNN
47
48autoreconf .
49./configure --prefix=/opt/zdnn-libs
50
51make build
52sudo make install
53```
54
55## 2. Build llama.cpp
56
57```sh
58git clone https://github.com/ggml-org/llama.cpp
59cd llama.cpp
60
61cmake -S . -G Ninja -B build \
62    -DCMAKE_BUILD_TYPE=Release \
63    -DGGML_ZDNN=ON \
64    -DZDNN_ROOT=/opt/zdnn-libs
65cmake --build build --config Release -j$(nproc)
66```