Seems video token isn't used in the model during video inference

[Virtualexistence]

Great work! I just have a question/doubt on using the <video> token in the video inference.
The constants defined don't seem to recognise any video token but only image tokens:

VILA/llava/constants.py

Line 27 in 0085724

DEFAULT_IMAGE_TOKEN = "<image>"

Am I missing something or does the model take <video> as just text context while the image token is added automatically N times i.e. length of frames to the query?

VILA/llava/eval/run_vila.py

Line 66 in 0085724

if IMAGE_PLACEHOLDER in qs:

python -W ignore llava/eval/run_vila.py \ --model-path Efficient-Large-Model/VILA1.5-3b \ --conv-mode vicuna_v1 \ --query "<video>\n Please describe this video." \ --video-file "demo.mp4"

[Lyken17]

You are right, current open source version does not handle the

[Virtualexistence]

Aw, it's still a great repository. Thank you for reply!