Hello, why do I only generate one frame of video using sv3d_u and sv3d_p

[kaikaizhangdragonplus]

执行结果：python scripts/sampling/simple_video_sample.py --input_path assets/test_image.png --version sv3d_u
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
Initialized embedder #0: FrozenOpenCLIPImagePredictionEmbedder with 683800065 params. Trainable: False
Initialized embedder #1: VideoPredictionEmbedderWithEncoder with 83653863 params. Trainable: False
Initialized embedder #2: ConcatTimestepEmbedderND with 0 params. Trainable: False
Restored from checkpoints/sv3d_u.safetensors with 0 missing and 0 unexpected keys
/opt/dragonplus/generative-models/venv/lib/python3.10/site-packages/torch/utils/checkpoint.py:31: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn("None of the inputs have requires_grad=True. Gradients will be None")

或者：
streamlit run scripts/demo/video_sampling.py

Collecting usage statistics. To deactivate, set browser.gatherUsageStats to False.

You can now view your Streamlit app in your browser.

Network URL: http://xxx
External URL: http://xxx

VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
VideoTransformerBlock is using checkpointing
Initialized embedder #0: FrozenOpenCLIPImagePredictionEmbedder with 683800065 params. Trainable: False
Initialized embedder #1: VideoPredictionEmbedderWithEncoder with 83653863 params. Trainable: False
Initialized embedder #2: ConcatTimestepEmbedderND with 0 params. Trainable: False
Loading model from checkpoints/sv3d_u.safetensors
576 576 None
2024-03-21 18:52:16.771 Uncaught app exception
Traceback (most recent call last):
File "/opt/dragonplus/generative-models/venv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 542, in _run_script
exec(code, module.dict)
File "/opt/dragonplus/generative-models/scripts/demo/video_sampling.py", line 192, in
value_dict["cond_frames"] = img + cond_aug * torch.randn_like(img)
TypeError: randn_like(): argument 'input' (position 1) must be Tensor, not NoneType

[JinSeoung-Oh]

@kaikaizhangdragonplus
Hi, I don't know my answer is helpful,
but it seems that the issue mostly arises due to the imageio library.
Therefore, instead of imageio, you can save frame by frame using cv2.
However, looking at your error logs... it seems there might be a need to check the input data.
The fact that a NoneType was passed as input data implies that either the input data is corrupted or the path was incorrectly set, resulting in failure to read it.

Thanks!

[zoldaten]

its imagio issue. seek for it here.

sould be something like this:

                #imageio.mimwrite(video_path, vid)
                #cv2.imwrite(video_path, vid)
                frame0 = vid[0, :, :, :].squeeze()
                out = cv2.VideoWriter(
                    video_path,
                    cv2.VideoWriter_fourcc(*"MP4V"),
                    20.0,
                    (frame0.shape[1], frame0.shape[0]),
                )
                for frame in vid:
                    out.write(frame[:, :, ::-1])
                out.release()

[wyweiyangweiGh]

Same issue.

[wyweiyangweiGh]

problem solved?

[lydonchandra]

its imagio issue. seek for it here.

sould be something like this:

                #imageio.mimwrite(video_path, vid)
                #cv2.imwrite(video_path, vid)
                frame0 = vid[0, :, :, :].squeeze()
                out = cv2.VideoWriter(
                    video_path,
                    cv2.VideoWriter_fourcc(*"MP4V"),
                    20.0,
                    (frame0.shape[1], frame0.shape[0]),
                )
                for frame in vid:
                    out.write(frame[:, :, ::-1])
                out.release()

Thank you very much, confirmed this worked for me.
This video is playable in VLC Desktop, but not here on github web, not sure what is the issue with the codec.

cat-000007.mp4