SNAP: Shortclip Navigator and Auto Producer

SNAP helps to create viral shorts from long YouTube videos by utilizing multimodal information.

• DEMO: currently shut down
• PAPER: will be released

Contents of Table

• Demo
• Architecture
• Install
• Evaluation
• Examples
• Citation
• Acknowledgement
• License
• Release

Demo

The official demo service was closed as of March 21, 2024. If you want to check the detailed service code, please refer to the production branch.

Below is an image demonstrating the demo service.

Please note: This model comes with certain limitations:

• Allowed categories: Comedy, Education, Entertainment, News & Politics
  • More categories will be added soon
    
• Language support: English
  

How can I check my youtube category?

Model Architecture

Service Architecture

Structure

The folder structure should be organized as follows before launching.

SNAP-Shortclip-Navigator-and-Auto-Producer
├── clip4clip
│   ├── models
│   │   └── models--openai-clip-vit-base-patch32
│   ├── Dockerfile
│   ├── poetry.lock
│   ├── pyproject.toml
│   ├── settings.py
│   └── visual_score.py
├── llm
│   ├── audio2text.py
│   ├── Dockerfile
│   ├── file
│   │   └── config.json
│   ├── models
│   │   ├── models--mistralai-Mistral-7B-Instruct-v0.2
│   │   └── models--openai-whisper-large-v3
│   ├── pipeline4eval.py
│   ├── pipeline.py
│   ├── poetry.lock
│   ├── pyproject.toml
│   ├── results
│   ├── settings.py
│   ├── text2summ.py
│   ├── timestamp.py
│   ├── utils
│   │   ├── audioUtils.py
│   │   ├── crawlers.py
│   │   ├── domainFlow.py
│   │   ├── get_src.py
│   │   ├── llmUtils.py
│   │   ├── preprocess.py
│   │   ├── prompts.py
│   │   ├── scores.py
│   │   └── videoUtils.py
│   └── youtube2audio.py
├── docker-compose.yml
└── snap.sh

Install

Please follow the instructions below

Clone this repository

git clone https://github.com/90stcamp/SNAP-Shortclip-Navigator-and-Auto-Producer.git

Create a Docker Compose setup that builds two Docker images

• One for the language model (LLM) and another for vision processing
• Automatically install dependencies using Poetry.

Run snap.sh

• Put your youtube link and category in .env
  

Evaluation

• Will be released soon.

Examples

youtube_link: https://www.youtube.com/watch?v=KOEfDvr4DcQ
category: Entertainment

Release

• MAR, 23, 2024 Release the initial version of the SNAP model publicly.