Add yolov5-youtube example

[dsuess]

addresses #1130

This is work in progress. I opened the PR to discuss some details.

Here's what this example does:

• you send a request with a youtube URL
• the predictor runs a YoloV5 model in ONNX format over it
• the overlayed video is returned to the user

This demonstrates both, how to load any Yolo model from the ultralytics repo and how to process video pretty easily.

I am using ffmpeg-python (instead of the more standard opencv/scikkit-video) since it allows you to do large parts of the pre-processing (resizing & padding) in a separate process. The reason I am using youtube-dl (instead of passing in a video directly) is that we don't have to provide an example video ourselves.

Would this be a good example? Any suggestions for improvements? How do you want to store the ONNX file?

Here's what still needs to be done:

• document how to export the ONNX model from the ultralytics repo. There's a minor change that makes our life much easier that I'll need to document.
• improve the overlay to show class names
• document & clean up the python code (e.g. delete intermediate files)
• update the documentation on the website
• a bit of performance optimization
• async the video-download?

---

checklist:

• run make test and make lint
• test manually (i.e. build/push all images, restart operator, and re-deploy APIs)
• update examples
• update docs and add any new files to summary.md (view in gitbook after merging)
• cherry-pick into release branches if applicable
• alert the dev team if the dev environment changed

[deliahu]

@dsuess this is really awesome!! I like how simple the API is: send a youtube link, and get back and annotated video!

I am using ffmpeg-python (instead of the more standard opencv/scikkit-video) since it allows you to do large parts of the pre-processing (resizing & padding) in a separate process.

Does this help improve latency or just throughput? Are you running the API with processes_per_replica=1 and threads_per_process=1? (note that processes_per_replica refers just to the uvicorn processes; ffmpeg will spawn additional processes). I'd be curious to understand how all of the threading/parallelism interplays, but also, there's no need to spend a lot of time on it, since it's just an example and not running in production :)

The reason I am using youtube-dl (instead of passing in a video directly) is that we don't have to provide an example video ourselves.

Yes, I agree, this makes for a nice example!

Would this be a good example? Any suggestions for improvements?

Yes, I think this is a great example! I can't think of any suggestions for improvements at the moment.

How do you want to store the ONNX file?

I will add it to our cortex-examples bucket; feel free to send me the model when it's ready (via google drive, dropbox, etc), and I can upload it there.

async the video-download?

This is also an interesting one, similar to the discussion about ffmpeg. My intuition says that downloading the video async could increase throughput (and not latency) assuming processes_per_replica>1 and/or threads_per_process>1. Like before, there's no need to spend a lot of time on this for the example's sake.

Thanks again for adding this!

[dsuess]

Does this help improve latency or just throughput? Are you running the API with processes_per_replica=1 and threads_per_process=1? (note that processes_per_replica refers just to the uvicorn processes; ffmpeg will spawn additional processes). I'd be curious to understand how all of the threading/parallelism interplays, but also, there's no need to spend a lot of time on it, since it's just an example and not running in production :)

To be honest, the main reason for using ffmpeg-python was that I didn't want to implement the resize-padding for the hundredth time 😄

I think it improves both (especially when you run on a GPU) since it allows you to run model inference and data preprocessing (decoding, resizing) in parallel. But I haven't done any tests and there are definitely better solutions if you actually need to optimize either performance metric.

async the video-download?

This is also an interesting one, similar to the discussion about ffmpeg. My intuition says that downloading the video async could increase throughput (and not latency) assuming processes_per_replica>1 and/or threads_per_process>1. Like before, there's no need to spend a lot of time on this for the example's sake.

You're right on this one, this would only benefit throughput. And just increasing either process_per_replica or `threads_per_replica will probably be the better solution anyway

[dsuess]

OK, I've send through the ONNX file and fixed the things that needed fixing. So if someone with the necessary permissions can a) upload the ONNX file and change the path in the config and b) fix the failing test, we're good to go IMO.

[RobertLucian]

@dsuess Thank you a lot for adding this!

I tried pushing to your branch, but apparently, I don't have write permissions. I think this might be because Allow edits from maintainer hasn't been checked. This option should be somewhere on the right-hand side of the PR I think.

Also, I'd want to test this over the weekend. For that, do you have a public link to your ONNX model I could use?

[dsuess]

The box is ticked as far as I can tell, so not sure what's going on.

For the ONNX file: https://drive.google.com/file/d/1p0nbSHUFpZhp6RxR2scNFaMk0ANHu_1e/view?usp=sharing

I've also included the steps how to create that ONNX file from the original repo, so you could also try to reproduce those setps

[RobertLucian]

@dsuess thanks for the link. As for the write access, that was a false alarm (an old peculiarity of my setup). It now works!

[deliahu]

This is a really great example, thanks again!

[deliahu]

@RobertLucian @dsuess I've uploaded the model to s3://cortex-examples/onnx/yolov5-youtube/yolov5s.onnx for now, and I can move it depending on what we decide regarding the API name

[RobertLucian]

This looks well to me! I took the liberty of modifying a few things here and there:

1. Read the output video file as bytes and deleted the file before returning the bytes as the response's payload.
2. Fix a bug with ffmpeg where the latest version of it from conda, which was released on the 6th of July, lead to this error: ffmpeg: relocation error: /opt/conda/envs/env/bin/../lib/./libgnutls.so.30: symbol mpn_add_1 version H OGWEED_4 not defined in file libhogweed.so.4 with link time reference. Reverting it to its previous version 4.2.3 made it work again - which is most likely the version @dsuess used.
3. Use a context manager when instantiating an object of FrameWriter's class.
4. Move functions to the utils module.
5. Decrease the line thickness by 4 times.
6. Create GIF using the sample YT video.

[deliahu]

Everything looks and works great, thanks again for adding this example!

@dsuess is it ready to merge from your perspective? If so, we can go ahead and merge it now.

[dsuess]

Yes, all good from my side