[Hackaton] AI painting project

[eaidova]

Task description

Synthesis of AI photo-realistic image based on an exemplar image sketch semantic using CocosNet and OpenVINO. In this task, contributors will explore how deep learning can be used to create an interactive python notebook for image semantic translation. The notebook should utilize CoCosNet model from Open Model Zoo.

As a result, we expect to see the demo where users can draw sketch using interactive canvas and get a realistic photo based on provided semantic drawings. Similar to images below:

For image translation

Background

OpenVINO notebooks

A collection of ready-to-run Jupyter notebooks for learning and experimenting with the OpenVINO™ Toolkit. The notebooks provide an introduction to OpenVINO basics and teach developers how to leverage our API for optimized deep learning inference.

Open Model Zoo

Open Model Zoo is part of OpenVINO Toolkit and includes optimized deep learning models and a set of demos to expedite development of high-performance deep learning inference applications.

CoCosNet

CoCosNet is a general framework for exemplar-based image translation, which synthesizes a photo-realistic image from the input in a distinct domain firstly described in Cross-domain Correspondence Learning for Exemplar-based Image Translation paper. The output has the style (e.g., color, texture) in consistency with the semantically corresponding objects in the exemplar

Desired solution

The result of the contribution will be an interactive python notebook that utilizes Deep Learning models, OpenVINO and an interactive user interface for painting. The contribution should follow openvino_notebooks contribution guide.
In this demonstration, the user should provide a reference image and semantic (or source image with mask) for exemplar painting.
For creating semantic source UI elements for drawing should be used.
Image Translation demo from Open Model Zoo can be considered as a reference on how to perform image translation using CoCosNet and OpenVINO.
As a possible solution for obtaining drawing capabilities for the notebook following solutions can be suggested:

• Ipycanvas is the library with widgets for drawing in Jupyter notebooks
• gradio - easy-to-use building blocks for the creation of a web-based GUI / demo around a machine learning model (or any Python function) in a few lines of code, for example, see sketch recognition tutorial

[sususama]

Hi, can I use this issue you wrote as my RFC proposal for PaddlePaddle Hackathon 4?

[openvino-dev-samples]

Hi @sususama If you can implement a pretrained model in/from Paddle Framework, I think you can create a RFC for this case.

[ntk97]

Hi, @AnishaUdayakumar. Can you rename the Issue for hackathon with more descriptive ?

[ntk97]

PR raised for this issue
#1026