char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Georgi Gerganovexamples : add llama-eval (#21152)fde69a360

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
662 B26 linesraw

llama-eval

Simple evaluation tool for llama.cpp with support for multiple datasets.

For a full description, usage examples, and sample results, see:

Quick start

# Single server
python3 llama-eval.py \
  --server http://localhost:8033 \
  --model my-model \
  --dataset gsm8k --n_cases 100 \
  --grader-type regex --threads 32

# Multiple servers (comma-separated URLs and thread counts)
python3 llama-eval.py \
  --server http://server1:8033,http://server2:8033 \
  --server-name server1,server2 \
  --threads 16,16 \
  --dataset aime2025 --n_cases 240 \
  --grader-type regex
1# llama-eval
2
3Simple evaluation tool for llama.cpp with support for multiple datasets.
4
5For a full description, usage examples, and sample results, see:
6
7- [PR 21152](https://github.com/ggml-org/llama.cpp/pull/21152)
8
9## Quick start
10
11```bash
12# Single server
13python3 llama-eval.py \
14  --server http://localhost:8033 \
15  --model my-model \
16  --dataset gsm8k --n_cases 100 \
17  --grader-type regex --threads 32
18
19# Multiple servers (comma-separated URLs and thread counts)
20python3 llama-eval.py \
21  --server http://server1:8033,http://server2:8033 \
22  --server-name server1,server2 \
23  --threads 16,16 \
24  --dataset aime2025 --n_cases 240 \
25  --grader-type regex
26```