Correct and clarify the handling of empty/zero-length Docs during training and inference

[shadeMe]

Description

This PR addresses two issues:

• When zero-length inputs passed to the transformer pipe, the outputs yielded by an attached listener has the wrong shape in the representation width dimension. This is more of a question of correctness than a practical concern due to the lack of tokens in such Docs, but we explicitly clarify why we use zero-width outputs.
• Backprop callbacks and gradients were not correctly aligned to their corresponding model outputs (outputs from the transformer pipe) when empty documents occurred in the middle of a training batch. The existing tests for this corner-case were inadvertently passing due to the behaviour of zip, i.e., the test always added the empty document to the end of the batch, and the subsequent zip operation would implicitly skip the last backprop/gradient due to the latter list being shorter than the list of model outputs.

It also adds comments about the pre-conditions to clarify the circumstances under which zero-length inputs are passed to the model.

Types of change

Bug Fix

Checklist

• I confirm that I have the right to submit this contribution under the project's MIT license.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[danieldk]

Assigned to you Sofie, since you added the handling of zero-length docs, so it would be good get an additional sanity check.

[svlandeg]

Nice digging, looks great!