Init prompts are truncated to --batch-size (max 512 tokens)

[MystCaster]

Prompt should not have a max size of --batch-size.
if ((int) embd.size() >= params.n_batch) { break; } seems to have no particular usage and should be removed to allow prompting more than 512 tokens.

A side effect is that with the current version, initial prompts are truncated to 512 tokens no matter the --ctx_size or the --batch-size parameter.

[slaren]

This seems to be a misunderstanding of what the batch size means. The prompt isn't truncated, it is just processed in chunks of batch-size tokens. This is meant to improve performance, but it requires more memory, which is why it is limited to 512.

[MystCaster]

Removing that break does not interfer with the processing of llama_eval by batches of --batch-size tokens.

Currently an initial prompt of more than --batch-size (maxed out at 512 in common.cpp by the way...) give back control to the user before the initial prompt is fully processed.

You can reproduce the issue by increasing the init prompt size of the Miku.sh example above 512 tokens or by lowering it's batch-size parameter below the actual init prompt token count.

[slaren]

The break is there to prevent using batch sizes larger than n_batch. Using batch sizes larges than 512 currently is not safe because the amount of memory allocated assumes a max batch size of 512.

I am not really sure if I understand what issue you are seeing, If you have a reliable way of reproducing it, please post clear step-by-step instructions.

[MystCaster]

Yes: configure batch-size of Miku.sh example to 100. You'll get the control back before the end of the initial prompt.

[slaren]

I am not able to reproduce this issue.

[MystCaster]

I am not able to reproduce this issue.

Ok, I tested it again and I get it now, I just didn't have enough patience : I forgot each batch takes time to process...
I can confirm that long initial prompts are not truncated, one just have to wait for every batches to process.
Note that I'm still not sure about the relevance of that "break" since it seems to only affect display and not evaluation, but it is another question.

Thanks for your patience, I'll close the issue which is not one :-)