Neural Style Transfer /Mixed Neural Style Transfer

Introduction

This project involves the implementation of Neural Style Transfer (NST) and the creation of a website that provides users with an easy way to apply style transfer to their images. Initially, my plan was to implement Leon Gatys’ paper titled “A Neural Algorithm of Artistic Style” for Neural Style Transfer. However, during this period, I explored other papers related to Neural Style Transfer, one of which was another paper by Leon Gatys titled " Controlling Perceptual Factors in Neural Style Transfer " I incorporated additional features from this paper into my project website.

Features

• Neural Style Transfer (NST) : Users can input a content image and a style image for style transfer.
• Style Transfer only on Background of Image : This option applies NST exclusively to the background of an image, useful for scenarios like styling the background of a portrait image.
• Color Preserving Style Transfer :NST occurs, but the color of the style image does not transfer to the content image; only paint strokes or texture are applied.
• High Resolution Style Transfer : This feature not only applies style transfer but also increases the resolution of the final image.
• Mixed Neural Style Transfer (MNST): Users can provide two style images—one major and one minor. The generated image showcases the style of both images. This feature also supports options like MNST on the background only, MNST with color preservation, and High-Resolution MNST.

In addition to this there is also a text to image generator that will be used style images.

website Demo

Installation

Prerequisites

1. Make sure first to create a virtual environment
2. Then open the virtual environment
3. Download some prequistes libraries :

pip install torch
pip install torchvision
pip install numpy 
pip install matplotlib
pip install skimage
pip install --upgrade diffusers transformers -q
pip install flask 

• If still there is any issue regarding the import of any library then install those

Do the following in the Virtual environment:

First open the terminal :

git clone https://github.com/rtb-12/Neural-Style-Transfer.git

then

cd Neural-Style-Transfer

Now create a Models folder in this directory and paste the models file there which can be downloaded from the following link: https://drive.google.com/drive/folders/15OVwtPRnxwxZwKD6qPrKo6RbfsPt-lhA?usp=sharing

NOTE : For Text to image generation one requires a hugging face auth token which can be generated on hugging face webiste and then paste it the code where auth token is written (Currently on auth token is already there for testing purposes and will be soon removed)

Here then go to > App folder > Run main.py

Results

• Neural Style Transfer:
• Color Preserving Style Transfer :

• Background only Style Transfer :

Generated Image:

• High Resolution Style Transfer :
• Mixed Style Transfer : Generated Image:

NOTE:

Currently the number of steps for image generation from prompt is kept :10 for style transfer :200(size of one dimension of image :256) and for hig res transfer :25 (size of one dimension of image:1024) these are kept less for testing on low end laptop but can be modified in code as required based on the hardware of the device.

One can Follow the following report for more understanding of the model implemented in the lab:

https://hackmd.io/@nZL7vfaUTsqsyyZrlF9LXw/S1bx76756