How to use videos as input for HiLo？

[q958287831]

Dear author, I would like to input the data dimensions [2,64,300,64,64] into HiLo, which represent dimension, number of channels, video sequence, height, and width, respectively. But the acceptable data for HiLo is [batch_size, batch_size,hidden_dimension]， I don't know how to convert it to a suitable dimension. I am a beginner in the field of artificial intelligence, and your answer is crucial to me!

[HubHop]

Hi @q958287831, thank you for your interest! A video clip can be thought of as a list of frames. Initially, for HiLo, we have an input tensor in the shape of [B, N, D], where:

• B: Batch size
• N: Sequence length (image tokens for one image)
• D: Hidden dimension

With a video clip, you will have an additional dimension:

• T: Temporal length

Now, your video data might be represented as a tensor in the shape of [B, T, N, D]. Next, we can reshape this data into [B * T, N, D], which effectively combines all frames of different video clips into the batch size. This allows HiLo to process the data.

However, note that HiLo does not handle the dependencies in the temporal dimension. There could be a promising extension in future work.