char-fork/llama.cpp

git clone https://git.t4t.associates/char-fork/llama.cpp

Georgi Gerganovbenches : add Nemotron 3 Nano on DGX Spark (#20652)9b342d0a9

char/aliceai-t5-moe
brancheschar/aliceai-t5-moeall refs →
8.8 KiB117 linesraw

NVIDIA DGX Spark

System info

uname --all
Linux spark-17ed 6.11.0-1016-nvidia #16-Ubuntu SMP PREEMPT_DYNAMIC Sun Sep 21 16:52:46 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

g++ --version
g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0

nvidia-smi
Fri Mar  6 11:39:45 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
| N/A   52C    P0             13W /  N/A  | Not Supported          |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

ggml-org/Nemotron-3-Super-120B-GGUF

Model: https://huggingface.co/ggml-org/Nemotron-3-Super-120B-GGUF

  • llama-batched-bench

main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 99, n_threads = 20, n_threads_batch = 20

PPTGBN_KVT_PP sS_PP t/sT_TG sS_TG t/sT sS t/s
5123215441.094468.051.62119.742.715200.37
51232210881.463700.162.43726.263.900279.01
51232421762.647773.764.04331.666.689325.29
51232843525.291774.146.15141.6211.442380.37
5123216870410.603772.6210.38549.3020.987414.72
51232321740821.231771.6918.23556.1639.466441.09
409632141285.340767.051.61619.816.956593.47
4096322825610.673767.552.45426.0813.127628.94
40963241651221.348767.464.07231.4425.420649.57
40963283302442.714767.156.27740.7848.991674.08
409632166604885.385767.5410.59648.3295.981688.14
40963232132096170.819767.3218.61955.00189.437697.31
8192321822410.690766.321.61919.7612.310668.10
81923221644821.382766.242.46725.9423.850689.65
81923243289642.782765.924.09831.2346.881701.69
81923286579285.582765.776.36840.2091.951715.52
81923216131584171.066766.2110.77447.52181.840723.62
81923232263168342.140766.1918.96953.98361.109728.78
  • llama-bench
modelsizeparamsbackendn_ubatchfatestt/s
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481pp2048768.84 ± 0.90
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481tg3219.94 ± 0.16
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481pp2048 @ d4096764.51 ± 0.50
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481tg32 @ d409619.95 ± 0.18
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481pp2048 @ d8192759.53 ± 0.71
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481tg32 @ d819219.83 ± 0.18
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481pp2048 @ d16384747.98 ± 1.58
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481tg32 @ d1638419.84 ± 0.18
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481pp2048 @ d32768724.40 ± 2.70
nemotron 120B.A12B Q4_K65.10 GiB120.67 BCUDA20481tg32 @ d3276819.45 ± 0.18

build: 04a65daab (8268)

ggml-org/Nemotron-3-Nano-4B-GGUF

Model: https://huggingface.co/ggml-org/Nemotron-3-Nano-4B-GGUF

  • llama-batched-bench

main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 99, n_threads = 20, n_threads_batch = 20

PPTGBN_KVT_PP sS_PP t/sT_TG sS_TG t/sT sS t/s
5123215440.1523371.610.59753.640.748726.90
51232210880.3193208.680.85774.661.176924.89
51232421760.7202843.561.32396.782.0431065.18
51232843521.4282867.962.311110.763.7391163.82
512321687042.8572866.944.203121.827.0601232.82
5123232174085.7092869.767.964128.5813.6731273.14
409632141281.4582809.760.60552.922.0622001.52
409632282562.9052819.950.87573.123.7802183.95
4096324165125.7902829.741.36194.077.1512309.17
40963283302411.5982825.322.378107.6513.9762362.89
409632166604823.2082823.884.348117.7627.5562396.89
4096323213209646.5152817.858.279123.6954.7942410.79
819232182242.9502776.950.61751.893.5672305.75
8192322164485.9212767.320.89671.456.8162413.05
81923243289611.8422767.211.40191.3413.2432484.03
81923286579223.7262762.172.461104.0326.1872512.38
8192321613158447.7772743.434.577111.8652.3542513.36
8192323226316896.6912711.168.772116.73105.4632495.36
  • llama-bench
modelsizeparamsbackendn_ubatchfatestt/s
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481pp20482761.90 ± 19.31
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481tg3252.85 ± 0.12
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481pp2048 @ d40962687.07 ± 21.84
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481tg32 @ d409652.32 ± 0.23
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481pp2048 @ d81922564.52 ± 57.69
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481tg32 @ d819251.27 ± 0.34
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481pp2048 @ d163842334.02 ± 37.83
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481tg32 @ d1638449.71 ± 0.14
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481pp2048 @ d327682041.46 ± 40.45
nemotron 4B Q8_03.94 GiB3.97 BCUDA20481tg32 @ d3276846.71 ± 0.13

build: 1bbec6a75 (8382)

1# NVIDIA DGX Spark
2
3## System info
4
5```bash
6uname --all
7Linux spark-17ed 6.11.0-1016-nvidia #16-Ubuntu SMP PREEMPT_DYNAMIC Sun Sep 21 16:52:46 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
8
9g++ --version
10g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
11
12nvidia-smi
13Fri Mar  6 11:39:45 2026
14+-----------------------------------------------------------------------------------------+
15| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
16+-----------------------------------------+------------------------+----------------------+
17| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
18| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
19|                                         |                        |               MIG M. |
20|=========================================+========================+======================|
21|   0  NVIDIA GB10                    On  |   0000000F:01:00.0 Off |                  N/A |
22| N/A   52C    P0             13W /  N/A  | Not Supported          |      0%      Default |
23|                                         |                        |                  N/A |
24+-----------------------------------------+------------------------+----------------------+
25```
26
27## ggml-org/Nemotron-3-Super-120B-GGUF
28
29Model: https://huggingface.co/ggml-org/Nemotron-3-Super-120B-GGUF
30
31- `llama-batched-bench`
32
33main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 99, n_threads = 20, n_threads_batch = 20
34
35|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
36|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
37|   512 |     32 |    1 |    544 |    1.094 |   468.05 |    1.621 |    19.74 |    2.715 |   200.37 |
38|   512 |     32 |    2 |   1088 |    1.463 |   700.16 |    2.437 |    26.26 |    3.900 |   279.01 |
39|   512 |     32 |    4 |   2176 |    2.647 |   773.76 |    4.043 |    31.66 |    6.689 |   325.29 |
40|   512 |     32 |    8 |   4352 |    5.291 |   774.14 |    6.151 |    41.62 |   11.442 |   380.37 |
41|   512 |     32 |   16 |   8704 |   10.603 |   772.62 |   10.385 |    49.30 |   20.987 |   414.72 |
42|   512 |     32 |   32 |  17408 |   21.231 |   771.69 |   18.235 |    56.16 |   39.466 |   441.09 |
43|  4096 |     32 |    1 |   4128 |    5.340 |   767.05 |    1.616 |    19.81 |    6.956 |   593.47 |
44|  4096 |     32 |    2 |   8256 |   10.673 |   767.55 |    2.454 |    26.08 |   13.127 |   628.94 |
45|  4096 |     32 |    4 |  16512 |   21.348 |   767.46 |    4.072 |    31.44 |   25.420 |   649.57 |
46|  4096 |     32 |    8 |  33024 |   42.714 |   767.15 |    6.277 |    40.78 |   48.991 |   674.08 |
47|  4096 |     32 |   16 |  66048 |   85.385 |   767.54 |   10.596 |    48.32 |   95.981 |   688.14 |
48|  4096 |     32 |   32 | 132096 |  170.819 |   767.32 |   18.619 |    55.00 |  189.437 |   697.31 |
49|  8192 |     32 |    1 |   8224 |   10.690 |   766.32 |    1.619 |    19.76 |   12.310 |   668.10 |
50|  8192 |     32 |    2 |  16448 |   21.382 |   766.24 |    2.467 |    25.94 |   23.850 |   689.65 |
51|  8192 |     32 |    4 |  32896 |   42.782 |   765.92 |    4.098 |    31.23 |   46.881 |   701.69 |
52|  8192 |     32 |    8 |  65792 |   85.582 |   765.77 |    6.368 |    40.20 |   91.951 |   715.52 |
53|  8192 |     32 |   16 | 131584 |  171.066 |   766.21 |   10.774 |    47.52 |  181.840 |   723.62 |
54|  8192 |     32 |   32 | 263168 |  342.140 |   766.19 |   18.969 |    53.98 |  361.109 |   728.78 |
55
56- `llama-bench`
57
58| model                   |       size |     params | backend    | n_ubatch | fa |            test |                  t/s |
59| ----------------------- | ---------: | ---------: | ---------- | -------: | -: | --------------: | -------------------: |
60| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |          pp2048 |        768.84 ± 0.90 |
61| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |            tg32 |         19.94 ± 0.16 |
62| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |  pp2048 @ d4096 |        764.51 ± 0.50 |
63| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |    tg32 @ d4096 |         19.95 ± 0.18 |
64| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |  pp2048 @ d8192 |        759.53 ± 0.71 |
65| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |    tg32 @ d8192 |         19.83 ± 0.18 |
66| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 | pp2048 @ d16384 |        747.98 ± 1.58 |
67| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |   tg32 @ d16384 |         19.84 ± 0.18 |
68| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 | pp2048 @ d32768 |        724.40 ± 2.70 |
69| nemotron 120B.A12B Q4_K |  65.10 GiB |   120.67 B | CUDA       |     2048 |  1 |   tg32 @ d32768 |         19.45 ± 0.18 |
70
71build: 04a65daab (8268)
72
73## ggml-org/Nemotron-3-Nano-4B-GGUF
74
75Model: https://huggingface.co/ggml-org/Nemotron-3-Nano-4B-GGUF
76
77- `llama-batched-bench`
78
79main: n_kv_max = 303104, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = 99, n_threads = 20, n_threads_batch = 20
80
81|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
82|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
83|   512 |     32 |    1 |    544 |    0.152 |  3371.61 |    0.597 |    53.64 |    0.748 |   726.90 |
84|   512 |     32 |    2 |   1088 |    0.319 |  3208.68 |    0.857 |    74.66 |    1.176 |   924.89 |
85|   512 |     32 |    4 |   2176 |    0.720 |  2843.56 |    1.323 |    96.78 |    2.043 |  1065.18 |
86|   512 |     32 |    8 |   4352 |    1.428 |  2867.96 |    2.311 |   110.76 |    3.739 |  1163.82 |
87|   512 |     32 |   16 |   8704 |    2.857 |  2866.94 |    4.203 |   121.82 |    7.060 |  1232.82 |
88|   512 |     32 |   32 |  17408 |    5.709 |  2869.76 |    7.964 |   128.58 |   13.673 |  1273.14 |
89|  4096 |     32 |    1 |   4128 |    1.458 |  2809.76 |    0.605 |    52.92 |    2.062 |  2001.52 |
90|  4096 |     32 |    2 |   8256 |    2.905 |  2819.95 |    0.875 |    73.12 |    3.780 |  2183.95 |
91|  4096 |     32 |    4 |  16512 |    5.790 |  2829.74 |    1.361 |    94.07 |    7.151 |  2309.17 |
92|  4096 |     32 |    8 |  33024 |   11.598 |  2825.32 |    2.378 |   107.65 |   13.976 |  2362.89 |
93|  4096 |     32 |   16 |  66048 |   23.208 |  2823.88 |    4.348 |   117.76 |   27.556 |  2396.89 |
94|  4096 |     32 |   32 | 132096 |   46.515 |  2817.85 |    8.279 |   123.69 |   54.794 |  2410.79 |
95|  8192 |     32 |    1 |   8224 |    2.950 |  2776.95 |    0.617 |    51.89 |    3.567 |  2305.75 |
96|  8192 |     32 |    2 |  16448 |    5.921 |  2767.32 |    0.896 |    71.45 |    6.816 |  2413.05 |
97|  8192 |     32 |    4 |  32896 |   11.842 |  2767.21 |    1.401 |    91.34 |   13.243 |  2484.03 |
98|  8192 |     32 |    8 |  65792 |   23.726 |  2762.17 |    2.461 |   104.03 |   26.187 |  2512.38 |
99|  8192 |     32 |   16 | 131584 |   47.777 |  2743.43 |    4.577 |   111.86 |   52.354 |  2513.36 |
100|  8192 |     32 |   32 | 263168 |   96.691 |  2711.16 |    8.772 |   116.73 |  105.463 |  2495.36 |
101
102- `llama-bench`
103
104| model                   |       size |     params | backend    | n_ubatch | fa |            test |                  t/s |
105| ----------------------- | ---------: | ---------: | ---------- | -------: | -: | --------------: | -------------------: |
106| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |          pp2048 |      2761.90 ± 19.31 |
107| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |            tg32 |         52.85 ± 0.12 |
108| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |  pp2048 @ d4096 |      2687.07 ± 21.84 |
109| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |    tg32 @ d4096 |         52.32 ± 0.23 |
110| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |  pp2048 @ d8192 |      2564.52 ± 57.69 |
111| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |    tg32 @ d8192 |         51.27 ± 0.34 |
112| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 | pp2048 @ d16384 |      2334.02 ± 37.83 |
113| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |   tg32 @ d16384 |         49.71 ± 0.14 |
114| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 | pp2048 @ d32768 |      2041.46 ± 40.45 |
115| nemotron 4B Q8_0        |   3.94 GiB |     3.97 B | CUDA       |     2048 |  1 |   tg32 @ d32768 |         46.71 ± 0.13 |
116
117build: 1bbec6a75 (8382)