frame interpolation pipeline after fixing go api bindings issue #130
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… with creating go api bindings because of openapi json sceme having a null option.
…i-worker into frame-interpolation
testing workflow
* update to the frame interpolation pipeline, there is some minor issue with creating go api bindings because of openapi json sceme having a null option. * minor changes to requirements * update to requrements to fetch from --index-url * simple patch to solve the go api bindings issue * checking if it works in my system --------- Co-authored-by: Jason Stone <ganesh.panth2012@gmail.com>
Bumps the go_modules group with 1 update in the / directory: [github.com/docker/docker](https://github.com/docker/docker). Updates `github.com/docker/docker` from 24.0.7+incompatible to 24.0.9+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v24.0.7...v24.0.9) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* update to the frame interpolation pipeline, there is some minor issue with creating go api bindings because of openapi json sceme having a null option. * minor changes to requirements * update to requrements to fetch from --index-url * simple patch to solve the go api bindings issue * checking if it works in my system --------- Co-authored-by: Jason Stone <ganesh.panth2012@gmail.com>
* update to the frame interpolation pipeline, there is some minor issue with creating go api bindings because of openapi json sceme having a null option. * minor changes to requirements * update to requrements to fetch from --index-url * simple patch to solve the go api bindings issue * checking if it works in my system * test-examples for frame-interpolation * update to sfast optimization to i2i and t2i and upscale pipelines * changes to extra files * added git ignore to the files to remove unnecessary files * files not removed checking again * still in test phase * test-test * Update .gitignore * Delete runner/app/tests-examples directory * update to directory reader as it now reads almost any naming convention --------- Co-authored-by: Jason Stone <ganesh.panth2012@gmail.com>
…i-worker into frame-interpolation
Bumps the go_modules group with 1 update in the / directory: [github.com/docker/docker](https://github.com/docker/docker). Updates `github.com/docker/docker` from 24.0.9+incompatible to 26.1.4+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v24.0.9...v26.1.4) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the go_modules group with 1 update in the / directory: [github.com/docker/docker](https://github.com/docker/docker). Updates `github.com/docker/docker` from 26.1.4+incompatible to 26.1.5+incompatible - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.4...v26.1.5) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* changes to mod to be able to pull from go get * update from to the branch for test compatibility * update for bug, random frames in interpolated video * patch_1 * patch_2 * patch_3
This was referenced Aug 24, 2024
* feat(model): add Realistic Vision model T2I support (livepeer#136) This commit ensures that the https://huggingface.co/SG161222/Realistic_Vision_V6.0_B1_noVAE model is supported in the T2I pipeline. * ci: add JS/TS SDK update trigger (livepeer#138) This commit adds a update trigger to the OpenAPI sync action that triggers a update of the JS/TS SDK. * ci: add TS/JS SDK OpenAPI spec update trigger (livepeer#139) This commit addes a trigger to update the OpenAPI spec in https://github.com/livepeer/ai-sdk-js. Furhter it improves the OpenAPI spec upstream sync action to forward more information. * refactor: add T2I parameter annotations (livepeer#141) This commit adds parameter annotations to the T2I pipeline similar to how it is done in the rest of the pipelines. Descriptions will be added in a subsequenty commit. * refactor: sort imports using isort (livepeer#142) This commit sorts the python imports using the https://pypi.org/project/isort package. * ci: update OpenAPI spec trigger repos (livepeer#143) This commit ensures the right upstream repos are triggered in the trigger upstream OpenAPI sync action. * feat: improve prompt splitter (livepeer#146) This commit ensures that an empty dict is returned by the prompt splitter when no valid prompt was found. * feat(T2I): add Black Forrest Labs Flux 1 support (livepeer#147) This commit adds support for the [Black Forrest Labs Flux 1 Schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell) model to the T2I pipeline. It is important to note that this model can only run on GPUs with more than 33 GB or VRAM. * refactor: fix unet load deprecation warnings (livepeer#151) This commit fixes some unet deprecation warnings by chaning the way the stable diffusion base model is loaded. * refactor: resolve CLIPFeatureExtractor deprecation warning (livepeer#152) This commit resolves a CLIPFeatureExtractor deprecation warning thrown by the NSFW check logic. * refactor: added descriptions to the pipeline parameters. (livepeer#144) * Added descriptions to the parameters. All parameters needing descriptions across: A2T, I2I, I2V, T2I, and Upscale have had their descriptions added. * The descriptions have been updated to better apply to the current implementation. * refactor: shorten parameter descriptions This commit shortens some of the parameter descriptions since the longer description is found on huggingface. * chore: update OpenAPI spec and golang bindings This commit ensures that the OpenAPI spec and golang bindings are updated with the new descriptions. --------- Co-authored-by: Rick Staa <rick.staa@outlook.com> * chore: apply black formatter (livepeer#153) This commit applies the black formatter to the codebase to ensure the code formatting is consistent. * feat: improve OpenAPI spec generation and naming (livepeer#155) This commit improves the naming and generation for the OpenAPI spec so that they are easier to work with. * chore: remove redundant OpenAPI spec (livepeer#156) This commit removes a redundant OpenAPI spec that was introduced some time ago. * Revert "chore: remove redundant OpenAPI spec (livepeer#156)" (livepeer#157) This reverts commit 41fa3f4. * chore: remove redundant OpenAPI spec (livepeer#158) This commit removes a redundant OpenAPI spec file that was introduced some time ago. * refactor: cleanup Gateway OpenAPI spec (livepeer#160) This commit removes the health endpoint schema from the generated Gateway OpenAPI spec. * chore: fix flake8 errors (livepeer#159) This commit fixes the flake8 errors that were introduced into the codebase in the last months. * refactor: remove old OpenAPI spec (livepeer#161) This commit removes the old unused OpenAPI spec. * feat: update ByteDance/SDXL-Lighting to default to 8step (livepeer#162) * update ByteDance/SDXL-Lightning to default to 8 step unet * update I2I to 8step default for ByteDance/SDXL-Lightning model * feat: apply git release version to OpenAPI spec (livepeer#164) This commit ensures that the latest git release flag is applied to the OpenAPI spec. * refactor: add pipeline descriptions (livepeer#169) This commit adds pipeline descriptions so that each pipeline is clearly explained on the docs. * refactor(openapi): replace json with yaml (livepeer#170) This commit replaces the default OpenAPI spec with yaml. * refactor: add response type descriptions (livepeer#171) This commit ensures that descriptions show up for the route response types in the docs. * chore(worker): update go bindings (livepeer#172) This commit updates the go bindings to include the right docstrings. * ci: fix OpenAPI spec check action (livepeer#173) This commit fixes the OpenAPI spec check action. This action can be used to ensure the OpenAPI spec and go bindings are up to date. * ci: remove manual SDK/Docs update trigger (livepeer#174) This commit replaces the manual update trigger for the docs and SDKs by Speakeasy actions. * refactor: type gen_openapi file (livepeer#175) This commit ensures that the functions in the gen_openapi file are typed. * chore: remove redundant OpenAPI specs (livepeer#177) This commit removes the JSON versions of the OpanAPI spec since they are no longer used. * refactor: rename A2T pipeline attribute (livepeer#179) This commit renames the self.ldm (Latent Diffusion Model) to self.tm (Transformer model) to make the distinction clearer. * chore: update make go-bindings generation command (livepeer#180) This commit ensures that the make file uses the right OpenAPI spec to generate the go bindings. * add studio api url (livepeer#178) * feat: add Studio Gateway This commit adds the studio Gateway to the list of servers. * chore: update OpenAPI spec This commit updates the OpenAPI spec to add the Studio gateway to the list of servers and thus the documentation. * feat: enable multiple containers for pipeline/model_id (livepeer#148) This commit makes the container map more unique providing users the case of running multiple pipelines behind one external endpoint. Co-authored-by: Rick Staa <rick.staa@outlook.com> * feat: add OpenAPI gen version arg (livepeer#184) This commit provides developers with a `--version` argument they can use when generating the OpenAPI spec using the `gen_openapi.py` script`. * Segment anything 2 pipeline image (livepeer#185) * feat(pipeline): add SAM2 image segmentation prototype This commit introduces a prototype implementation of the [Segment Anything v2](https://github.com/facebookresearch/segment-anything-2) (SAM2) pipeline within the AI worker. The prototype demonstrates the basic functionality needed to perform segmentation on an image. Note that video segmentation is not yet implemented. Additionally, the dependencies were updated quickly, which may temporarily break other pipelines. * revert Dockerfile, requirements, add sam2 Dockerfile * refactor: enhance SAM2 input handling and error management This commit allows nested arrays to be supplied as JSON strings for SAM2 input. It also implements robust error handling to return a 400 error with a descriptive message when incorrect parameters are provided. * refactor: improve SAM2 return time This commit ensures that we return the masks, iou_predictions and low_res_masks in json format. * Sam2 -> SegmentAnything2 * update go bindings * update multipart.go binding with NewSegmentAnything2Writer * update worker and multipart methods * predictions -> scores, mask -> logits * add sam2 specific multipartwriter fields * add segment-anything-2 to containerHostPorts * fix pipeline name in worker.go * revert Dockerfile, requirements, add sam2 Dockerfile * Sam2 -> SegmentAnything2 * predictions -> scores, mask -> logits * feat: replace JSON.dump with str This commit replaces the JSON.dump method with a simple str method since it is highly unlikely that the string contains invalid data. Co-authored-by: Peter Schroedl <peter_schroedl@me.com> * move pipeline-specific dockerfile * update openapi yaml * add segment anything specific readme * update go bindings * refactor: move SAM2 docker This commit moves the SAM2 docker file inside the docker container. * refactor: add FastAPI descriptions This commit cleansup the codebase and adds FastAPI parameter and pipeline descriptions. * refactor: improve sam2 route function name This commit improves the sam2 route function name so that it is more pythonic and shows up nicer in the OpenAPI spec pipeline summary. * chore(worker): update golang bindings This commit updates the golang bindings so that the runner changes are reflected. * refactor(runner): add media_type This commit adds the media type content MIME type to the segment anything 2 pipeline. * chore(worker): remove debug patch This commit removes the debug patch which was accidentally added to the code. * feat(runnner): add SAM2 model download command This commit adds the SAM2 model download command so that orchestrators can pre-download the model. * refactor(worker): change SAM2 multipart reader param order This commit ensures that the parameters are in the same order as the pipeline parameters. * determine docker image in createContainer * fix: fix examples This commit fixes the example scripts. --------- Co-authored-by: Rick Staa <rick.staa@outlook.com> Co-authored-by: Elite Encoder <john@eliteencoder.net> Co-authored-by: Peter Schroedl <peter@livepeer.org> * fix(pipeline): add FLUX.1-dev and disable negative_prompt on flux (livepeer#167) This commit adds the black-forest-labs/FLUX.1 model download commands. The dev model is placed under the `--restricted` flag since it can not be used for commercial purposes. Co-authored-by: Rick Staa <rick.staa@outlook.com> * chore: update OpenAPI spec version This commit updates the version set in the OpenAPI spec. * ci(docker): add ai-runner base Docker tag (livepeer#194) This commit ensures that the main Docker container is also tagged as the base container so that it can be used as the base for the pipeline specific containers. Co-authored-by: ad-astra-video <99882368+ad-astra-video@users.noreply.github.com> * ci(docker): add workflow dispatch (livepeer#195) This commit ensures that developers can trigger docker image building. * ci(docker): ensure docker ci dispatch works (livepeer#196) This commit ensures that the workflow dispatch triggers the docker job. * ci: add pipeline docker ci (livepeer#193) * chore(docker): add 'base' tag and segment-anything-2 docker image build * update segment-anything-2 to dynamic base image * make more space on runner * refactor(ci): split Docker CI This commit ensures that the pipeline docker build ci is found in a seperate action from the base. * ci(docker): enable pipeline docker workflow dispatch This commit ensures that maintainers can trigger the pipeline specific Docker action using a workflow dispatch. * ci(docker): fix out of space error This commit switches to the oxfort runner to see if it can fis the OS storage error. * ci: cleanup hosted runner This commit cleans up the hosted runner so that we don't run into OS storage issues when trying to build the container. --------- Co-authored-by: Brad P <brad@ad-astra.video> * ci(docker): add sam2 docker tags (livepeer#197) This commit ensures that the SAM2 docker has the right tags. * ci(docker): enable pipeline docker workflow dispatch (livepeer#198) This commit ensures that maintainers can manually run the pipeline docker ci. * feat(sdks): implement SDK-specific API customizations (livepeer#191) * feat(sdks): implement SDK-specific API customizations This commit introduces several SDK-specific OpenAPI configurations to the runner API configuration. These customizations will enhance the SDKs we are planning to release. Co-authored-by: Victor Elias <victorgelias@gmail.com> * feat: enable speakeasy retries (livepeer#201) This commit enables the [speakeasy retries](https://www.speakeasy.com/docs/customize-sdks/retries#global-retries) feature for the SDKs. * Revert "feat: enable speakeasy retries (livepeer#201)" (livepeer#202) This reverts commit caa4bb7. * chore: release v0.5.0 This commit releases a new minor version since we had to revert the SDK related changes. * chore: update alpha to beta phase (livepeer#203) This commit updates the code and documentation to signal we are entering the Beta phase of the AI network journey. * fix(runner): improve 'num_inference_steps' logic (livepeer#205) This commit prevents a Key Error from being thrown when the pipelines are called directly. * fix(runner): fix benchmarking script (livepeer#206) This commit removes the 'batch_size' argument from the benchmarking script since our current pipeliens don't support batching requests due to us not having a way to estimate VRAM and preventing out of memory errors. For more information see livepeer#66. We can add this option back in when we have solved this. * readme: update with note (livepeer#207) * docs: remove AI Realtime Video note from main branch This commit removes the AI Realtime video warning note from the mainbranch as it should have been on the https://github.com/livepeer/ai-worker/tree/realtime-ai-experimental branch. --------- Co-authored-by: Rick Staa <rick.staa@outlook.com> Co-authored-by: ea_superstar <135424109+EAsuperstar@users.noreply.github.com> Co-authored-by: ad-astra-video <99882368+ad-astra-video@users.noreply.github.com> Co-authored-by: PSchroedl <peter_schroedl@me.com> Co-authored-by: Elite Encoder <john@eliteencoder.net> Co-authored-by: Peter Schroedl <peter@livepeer.org> Co-authored-by: Brad P <brad@ad-astra.video> Co-authored-by: Victor Elias <victorgelias@gmail.com> Co-authored-by: Emran M <3213391+emranemran@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
changed optional argument in routes to resolve the openapi and go api bindings.
added the frame interpolation pipeline and its corresponding routes, tested in the uvicorn server.
added utility functions for directory reader and writer as the pipeline expects frames in a very specific way. (the model expects the images in the directory to be indexed such that when ordering by sort they should lineup frames by frame)
i have tons of examples of usage of the pipeline i can create pull request for those if they will be useful.
minor correction to upscaling.py with incorrect naming in info.
added a line in the dl_checkpoinsts.sh script to download the model from djes github repo, might need to update it later to make it dynamic.
the input to the pipeline can be either two images or directory path and the output will be interpolated frames depending upon the value of inter_frames. I was unsure if the pipeline should return a video or frames so, there is a utility function included that can be used convert the frames to video if needed. Also since the input can also be a video instead of frame or directory so there is another utility function that will take a video and convert it to frames. Let me know so I can implement it accordingly.