llama-bench : add test measuring token generation rate at given prompt length

[fairydreaming]

I needed a test that would measure token generation rate after processing a prompt of given length, so I decided to add a new kind of test to the llama-bench tool.

This PR adds -gp <pp,tg> option that allows to specify a prompt length and number of tokens generated after processing the prompt. This new test works almost the same way as old -pg test, but it doesn't take into account the prompt length and prompt processing time when calculating result, only the token generation rate is reported.

Test results are labeled in a different way to avoid confusion with -pg test results, I used @ character to emphasize that the result indicates the token generation rate AT given prompt length.

Example:

$ ./bin/llama-bench --numa distribute -t 32 -m /mnt/md0/models/deepseek-v3-Q4_K_S.gguf -p 0 -n 0 -gp 128,32 -gp 256,32 -r 3

model	size	params	backend	threads	test	t/s
deepseek2 671B Q4_K - Small	353.90 GiB	671.03 B	CPU	32	tg32@pp128	8.94 ± 0.06
deepseek2 671B Q4_K - Small	353.90 GiB	671.03 B	CPU	32	tg32@pp256	8.35 ± 0.01

Hopefully this is more intuitive compared to averaged prompt processing + token generation rate in -pg test results.

[slaren]

The other printers (sql, json, etc) would also need to be updated.

[fairydreaming]

The other printers (sql, json, etc) would also need to be updated.

@slaren Can you be more specific?

[slaren]

The test type needs to be exported in these printers as well, since n_prompt and n_gen is no longer enough to tell the difference. Another option would be to get rid of test_kind_type and just record and report timings for the prompt and generation steps separately, then the -pg test would be enough.

[fairydreaming]

The test type needs to be exported in these printers as well, since n_prompt and n_gen is no longer enough to tell the difference. Another option would be to get rid of test_kind_type and just record and report timings for the prompt and generation steps separately, then the -pg test would be enough.

I guess another option is to add a "test" column in all printers with the same values as displayed in default console output. Any specific reason it's not included there?

[slaren]

Yes, that's what I meant when I said that the test type would need to be exported in these printers. There isn't a test column/field at the moment because it is not necessary.

[slaren]

This formatting should only be applied to the markdown printer. The other printers are intended to be used programmatically, so it should be a simple enum that can be parsed easily, without the token counts. The token counts can be obtained from the n_prompt and n_gen parameters already.