Skip to content

Optim-wip: Add Neural Style Transfer tutorial & pre-trained VGG-16 model from Caffe #677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

ProGamerGov
Copy link
Contributor

@ProGamerGov ProGamerGov commented Jun 7, 2021

  • Added the pretrained VGG-16 model from Caffe used by the Visual Geometry Group for the ImageNet ILSVRC-2014 Challenge.
  • Added neural style transfer (nst) tutorial notebook for InceptionV1 and VGG models.

The InceptionV1 portion of the tutorial may fail until the input range changes from #655 are implemented. I updated the transform_input function in this PR to match the changes made in the other PR, so #655 is no longer required for this PR.

@ProGamerGov
Copy link
Contributor Author

ProGamerGov commented Nov 1, 2021

Some of the style images in the tutorial are currently failing to download as Wikipedia is now requiring we supply a user agent with a download request. We'll need to add a user agent header to the ImageTensor.open function to resolve the issue, or we could host the images ourselves.

@aobo-y aobo-y self-assigned this Nov 30, 2021
@aobo-y aobo-y self-requested a review November 30, 2021 21:20
@aobo-y
Copy link
Contributor

aobo-y commented Nov 30, 2021

Hi @ProGamerGov, learned a lot by going through the tutorial! But do you mind providing me more context? What is the goal and why do you believe Captum needs a neural style transfer tutorial? I can see it demonstrates how to leverage some newly added utility & loss functions to redo a similar PyTorch tutorial. However, why can't we show them in a tutorial with stronger taste of model interpretability?

@ProGamerGov
Copy link
Contributor Author

ProGamerGov commented Dec 3, 2021

Hi @aobo-y, thank you for reviewing this tutorial and it's associated code!

In terms of the benefits of including an NST tutorial, I have the following:

  • Show users how to use a custom optimization function for optimizers that require closure functions. No other tutorial is able to cover this at the moment.
  • Illustrate the usage and benefits of FFT image parameterization and transform robustness and show how they can conflict with architectural robustness (like in VGG models for example).
  • Show users how to work with custom input ranges.
  • Demonstrate the use of ImageTensor’s open() class function to load images.
  • Lucid’s NST tutorial is seemingly one of it’s most popular tutorials and thus having a Captum equivalent will aid in user migrations.

@ProGamerGov
Copy link
Contributor Author

We can probably skip this PR for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants