Does the Current Implementation Break Full Attention When block_length < gen_length?

[ashun989]

Thank you for your excellent work. I would like to confirm my understanding of the current design:

According to my understanding, the current library seems to be suitable only for dLLMs that natively support block attention. For dLLMs using full attention (e.g. LLaDA), if block_length < gen_length, then during the generation of the current block, the model cannot access information from the subsequent blocks in the prompt. In this case, does the attention mechanism no longer satisfy the requirements of full attention?

[Labman42]

There are some misunderstanding here.
The model will always be able to access all the information from the prompt since the whole prompt is processed at once during the prefilling.
For most full attention model, the block_length means the context length. since these model can not continue to generate new blocks due to their training paradigm. (you can checkout the dllm-Var which behaves differently)
The engine currently do not support semi-ar style remasking strategy for full attention model, but it is quite easy to implement it is just a way of remasking.
The current engine does not support those approximate kv cache tricks (e.g. dllm-cache, suffix cache from fast dllm) for full attention model like Llada.