Convert "Visualization utilities" notebook into a sphinx-gallery example #3774
Conversation
|
Yes @NicolasHug I had written the notebook 😄 Linked with #3688 I think People often ask about this in dicusssion forums stackoveflow And even in some codebases So for legacy purpose. I think we can keep it. Edit: Here is the page I think this is slightly intrusive. Can we set
|
There was a problem hiding this comment.
Few thoughts from my side @NicolasHug 😄
Btw torchvision gallery is awesome way to add documentation.
…into visualization_example
gallery/plot_visualization_utils.py
Outdated
| show(result) | ||
|
|
||
| ##################################### | ||
| # We can adjust alpha to show the masks with a different transparency level: |
There was a problem hiding this comment.
I think another thing which would be good to illustrate in the example is how to get the output of Mask R-CNN to be plotted with draw_segmentation_masks. People have asked it in the past and I think would be a good thing to showcase as well, as the output format is slightly different
There was a problem hiding this comment.
I'm getting poor results with maskrcnn_resnet50_fpn, is this expected?
There was a problem hiding this comment.
Chatted offline with Nicolas, this is current an issue / limitation of the draw_segmentation_masks which we will be looking into fixing soon
There was a problem hiding this comment.
Yes. There are limitations 🤕 IIRC the limitation was because Mask RCNN returns a boolean tensor of masks which isn't handled by the utility.
There was a problem hiding this comment.
Sure, no worries, thanks for all your work and contributions @oke-aditya !
gallery/plot_visualization_utils.py
Outdated
| model = fasterrcnn_resnet50_fpn(pretrained=True, progress=False) | ||
| model = model.eval() | ||
|
|
||
| outputs = model(racoon.unsqueeze(0)) |
There was a problem hiding this comment.
Can we use an image which contains categories present in COCO, like dog or people? The current predictions seems a bit arbitrary.
Also, I think it would be good to showcase how to add labels (containing for example the class name and score).
gallery/plot_visualization_utils.py
Outdated
|
|
||
| model = fcn_resnet50(pretrained=True, progress=False) | ||
| model = model.eval() | ||
| output = model(people.unsqueeze(0)) |
There was a problem hiding this comment.
This is not fully correct I believe, as we need to normalize with the imagenet mean / std, see
vision/references/segmentation/presets.py
Line 28 in d6fee5a
The only model that doesn't require this is Faster-RCNN / Mask-RCNN, but classification and segmentation models require it
There was a problem hiding this comment.
thanks for the note @fmassa , does this also apply to the boxes on the racoon (which I'll change) above?
Oh they use Faster-RCNN so no, sorry ^^
There was a problem hiding this comment.
The boxes for the racoon uses Faster R-CNN, which only requires the image to be in 0-1, so it should be fine.
gallery/plot_visualization_utils.py
Outdated
| output = model(people.unsqueeze(0)) | ||
| masks = output['out'].squeeze(0) | ||
|
|
||
| people_int = T.ConvertImageDtype(dtype=torch.uint8)(people) |
There was a problem hiding this comment.
In general, I would maybe recommend users using the functional API in situations like this, so something like
people_int = convert_image_dtype(people, dtype=torch.uint8)
|
Thanks for the reviews! I'll add instructions for the Mask-RCNN model in a follow up PR |
…lery example (#3774) Reviewed By: cpuhrsch Differential Revision: D28538756 fbshipit-source-id: 9903ad98e78f7efb4db8dc0a0b90ce65173d0963
This PR converts https://github.com/pytorch/vision/blob/master/examples/python/visualization_utils.ipynb into a sphinx-gallery example.
I modified it a bit to keep it short and to the point. I also chose not to showcase
make_gridbecause I don't see a compelling use-case for it: users can just create matplotlib grid themselves, the mpl API has gotten a lot better in the last few years. Also it's impossible to plot images with different dimensions wit hit. (No strong opinion, feel free to disagree, I can put it back).Eventually I think we'll want to remove the original notebook but I'm keeping it there for now.
Image used is https://www.pexels.com/photo/smiling-women-and-men-sitting-on-green-grass-1231230/
CC @oke-aditya as I think you wrote the original notebook?