Skip to content

alexandonian/paint-by-word

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Paint By Word

teaser

We provide a PyTorch implementation of our Paint By Word Method presented in this paper.

Prerequisites

  • Linux or macOS
  • Python 3.6+
  • CPU or NVIDIA GPU + CUDA CuDNN

Table of Contents:

  1. Setup
  2. Painting

Setup

  • Clone this repo:
git clone https://github.com/alexandonian/paint-by-word.git
cd paint-by-word
  • Create python virtual environment

  • Install a recent version of PyTorch and other dependencies specified below.

We highly recommend that you install additional dependencies in an isolated python virtual environment (of your choosing). For Conda+pip users, you can create a new conda environment and then pip install dependencies with the following snippet:

conda env create -f scripts/environment.yml
conda activate paintbyword
pip install -e .
./scripts/setup_env.sh

Painting

Free form painting with words

from paintbyword import StyleganPainter
from paintbyword.utils import show, pilim

# Create instance of painter class
painter = StyleganPainter(pretrained='birds')

seed_index = 0
z, image, loss_history, im_history = painter.paint(
  seed_index,
  'A photo of a yellow bird with black colored wings',
  optim_method='cma + adam'
)

# Show the painted output image.
show(pilim(image[0]))

should produce an image similar to the one below on the right with the optimization history on the left:

yellow bird yellow bird

Masked painting with words

from paintbyword import StyleganMaskedPainter
from paintbyword.utils import show, pilim

# Create instance of masked painter class
painter = StyleganMaskedPainter(pretrained='bedroom')

# Display grid of seed images to choose from
painter.show_seed_images(batch_size=32)

choice = 30  # Choose index of desired seed image.
painter.mask_seed_image(choice)  # Scribble mask on seed image.

masked painting

result = painter.paint(
    choice,
    description='A photo of a rustic bed',
    optim_method='cma + adam'
)
# Show the painted output image.
show(pilim(image[0]))

painted image

For complete working examples, please see the juypter notebooks in the notebooks directory.

Citation

If you use this code for your research, please cite our paper.

@misc{bau2021paintbyword,
title={Paint by Word},
author={Alex Andonian and Sabrina Osmany and Audrey Cui and YeonHwan Park and Ali Jahanian and Antonio Torralba and David Bau},
year={2021},
eprint={arXiv:2103.10951},
}
}

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published