RCN example #96

shrinuKushagra · 2021-11-19T03:07:26Z

This PR contains an example implementation of RCN using the pgmax package.
We load a pre-trained RCN model on a very small subset of mnist (20 examples) and test on a small subset of mnist (20 examples).
The reported accuracy = 0.80.

codecov-commenter · 2021-11-19T03:09:26Z

Codecov Report

Merging #96 (49c5ca0) into master (2c81ff8) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master       #96   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            8         8           
  Lines          633       633           
=========================================
  Hits           633       633

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2c81ff8...49c5ca0. Read the comment docs.

StannisZhou

Several major issues:

We should aim for a single notebook file (with one data file containing the trained models in examples_data)
There seems to be many redundant/random pieces
Since this is intended as a tutorial, please make sure this can actually be run by someone. That means:
- Make sure the data is accessible
- Make sure you have properly updated the relevant dependicies
- Make sure you have detailed docstrings/type hints for all your functions, and relevant comments when needed

examples/rcn/load_data.py

pgmax/fg/graph.py

examples/rcn/preproc.py

examples/rcn/mnist_small.py

shrinuKushagra · 2021-11-19T23:42:04Z

Addressed the comments.

However, I could not find a principled solution to the Hashable int problem.
Seems like there might be some issues with the library itself.
python/mypy#2412

StannisZhou

Made some initially comments after trying to run it. Will make more later

examples/rcn.py

pgmax/fg/graph.py

StannisZhou

Made more comments

examples/rcn.py

shrinuKushagra · 2021-11-30T19:26:51Z

All the comments are now addressed.
Let me know once you run it and other comments you have.

StannisZhou · 2021-12-01T07:55:57Z

Want to mention that not all comments are addressed. There were 3 unaddressed comments (with reminders), eg. #96 (comment)

But I addressed them in my commits. Left a few more comments from playing with the notebook. Once these are addressed we can merge the PR.

StannisZhou

Hopefully the final round of comments

examples/rcn.py

swaroopgj

Thanks for adding. I have left comments regarding readability since these are intended for understanding.
If possible I would add a short text at the top to describe RCN model with reference to the paper and the Science implementation available.

examples/rcn.py

swaroopgj · 2021-12-01T20:41:39Z

examples/rcn.py

+# # 6. Compute metrics (accuracy)
+
+# %%
+best_model_idx = np.argmax(scores, axis=1)


what if we have two classes that score the same?

argmax will chose the one with smaller value.

I know but that is wrongly reducing the accuracy.

If two templates have the same score, I am not sure what we can do in this case. It amounts to an arbitrarily selecting between the two (or more) of them.

shrinuKushagra · 2021-12-02T01:41:07Z

Want to mention that not all comments are addressed. There were 3 unaddressed comments (with reminders), eg. #96 (comment)

But I addressed them in my commits. Left a few more comments from playing with the notebook. Once these are addressed we can merge the PR.

Hi Stannis,
Sorry about this. I went back and checked which of the comments were not addressed. It seems that the following comments were missed.

Can we instead use np.unravel_index/ravel_multi_index for these? -- In the context of the valid_configs function.
Changing the for loop in valid_config construction
3 This can be omitted (it would be automatically initialized to all 0 if omitted) -- In the context of valid_configs

I went over the Conversations tab multiple times and these comments were not there. But looking back at the github emails, I do see that you had made these comments and also reminders for this. The only possible explanation seems that, these comments were present in the Files tab and not in the conversations tab. I just realized that sometimes there are comments that only show in the Files tab and not in Conversations tab. (this is in the context of another pr).

To conclude, I want to apologize for missing them. It was genuine over-sight (from not knowing about a github feature) and I will be more careful in future. Also, thanks for addressing the comments.

swaroopgj · 2021-12-02T17:50:20Z

examples/rcn.py

+# %% [markdown]
+# RCN is a neuroscience based, data-efficient, interpretable object detection/vision system. It is a probabilistic graphical model (PGM) that trains in minutes and generalizes well (for certain traits). It can reason about, and is robust to, occlusion. It can also imagine and interpret scenes.
+#
+# Please refer to the [Science](https://www.cs.jhu.edu/~ayuille/JHUcourses/ProbabilisticModelsOfVisualCognition2020/Lec21/A%20generative%20vision%20model%20that%20trains%20with%20high%20data%20efficiency%20and%20breaks%20text-based%20CAPTCHAs.pdf) paper for more details. In this notebook, we implement a two-level RCN model on a small subset of the MNIST dataset.


🙄
Can you use this link please?
https://www.science.org/doi/10.1126/science.aag2612

@lazarox do we have open-access link to share?

I would also link to the earlier repo https://github.com/vicariousinc/science_rcn since you used that to train the model I assume.

Changed the link.

And added new link where the model is loaded.

swaroopgj · 2021-12-02T17:57:31Z

examples/rcn.py

+
+
+# %%
+def fetch_mnist_dataset(test_size: int, seed: int = 5) -> Tuple[np.ndarray, np.ndarray]:


Suggested change

def fetch_mnist_dataset(test_size: int, seed: int = 5) -> Tuple[np.ndarray, np.ndarray]:

def fetch_mnist_dataset(test_size: int, seed: int = 5) -> tuple[np.ndarray, np.ndarray]:

And remove import

Seems that using tuple was enabled in python 3.9?
https://www.python.org/dev/peps/pep-0585/

I was trying to use the suggestion but keep getting TypeError: 'type' object is not subscriptable. The project is using python version of 3.7.5, maybe thats why?

Ahh - my bad. You need to add:

from __future__ import annotations

to the top of the file - it'll be default in future python version
You often want that for other reasons, as various other typing things will fail otherwise.

Feel free to ignore (or fix), as you prefer

swaroopgj · 2021-12-02T17:59:08Z

examples/rcn.py

+    data["suppression_masks"],
+    data["filters"],
+)
+


Add a comment here to indicate these are RCN model params

Added detailed description of the parameters.
Moved max_perturb_radius to the relevant cell as it not central to the model and only used to save computation.

swaroopgj · 2021-12-02T18:01:01Z

examples/rcn.py

+hps, vps = 12, 12  # Horizontal and vertical pool sizes respectively for RCN models.
+num_orients = filters.shape[0]
+brightness_diff_threshold = 40.0
+max_perturb_radius = 22


I would re-group them and explain every variable defined here, with either an inline comment or a comment above it, like you did for pool sizes.
num_orients and bdt are related to preproc to run edge detection.
different from the core PGM model.

It also won't hurt to expand the pool size to indicate what it exactly is, is it half the window size, is it radius or full size?

StannisZhou

Left some minor comments. LGTM for the most part. Will take a final look after the comments are addressed. Thanks for patiently addressing all the comments!

examples/rcn.py

StannisZhou · 2021-12-03T03:09:30Z

examples/rcn.py

+# %% [markdown]
+# We load the following variables (that have been pre-computed).
+# - train_set and train_labels - A sample of MNIST train dataset containing 100 train images and their labels.
+# - frcs and edges - Used to represent the learned rcn graphical models.


Link to the learning module in science RCN repo?

examples/rcn.py

StannisZhou · 2021-12-03T03:14:03Z

examples/rcn.py

+
+
+# %% [markdown]
+# ## 5.2 Run maximum a-posteriori probability (map) inference on all test images


I believe the word probability is redundant here, and map is usually capitalized.

My earlier comment was intended for potential typo in usage of "map product inference".
But you can just use MAP inference here without having to expand.

Changed to say MAP inference.

pyproject.toml

examples/rcn.py

StannisZhou

LGTM. Excited to have this example ready!

examples/rcn.py

Co-authored-by: Swaroop Guntupalli <swaroopgj@gmail.com>

StannisZhou suggested changes Nov 19, 2021

View reviewed changes

shrinuKushagra requested a review from StannisZhou November 19, 2021 23:43

StannisZhou reviewed Nov 21, 2021

View reviewed changes

examples/rcn.py Outdated Show resolved Hide resolved

examples/rcn.py Outdated Show resolved Hide resolved

examples/rcn.py Outdated Show resolved Hide resolved

pgmax/fg/graph.py Outdated Show resolved Hide resolved

StannisZhou reviewed Nov 22, 2021

View reviewed changes

examples/rcn.py Outdated Show resolved Hide resolved

StannisZhou reviewed Nov 24, 2021

View reviewed changes

examples/rcn.py Show resolved Hide resolved

shrinuKushagra force-pushed the rcn_notebook branch from f985834 to 457cec7 Compare November 29, 2021 14:52

shrinuKushagra requested a review from StannisZhou November 29, 2021 15:01

shrinuKushagra force-pushed the rcn_notebook branch from abb614f to e803909 Compare November 30, 2021 19:23

StannisZhou reviewed Dec 1, 2021

View reviewed changes

StannisZhou requested review from swaroopgj and wlehrach December 1, 2021 08:05

wlehrach reviewed Dec 1, 2021

View reviewed changes

examples/rcn.py Outdated Show resolved Hide resolved

swaroopgj reviewed Dec 1, 2021

View reviewed changes

shrinuKushagra force-pushed the rcn_notebook branch from a1162cf to e9b9f52 Compare December 2, 2021 15:24

shrinuKushagra requested a review from StannisZhou December 2, 2021 15:25

Shrinu Kushagra and others added 9 commits December 2, 2021 07:27

notebook for running rcn using pgmax

57a0e4d

notebook for running rcn using pgmax

9ad5ce4

addressing PR comments. making it a single notebook

ada5d94

minor edits

e00fb3c

fixing type hints

e62aa15

adding docustrings to the public functions

067a514

visualization changes

16c23b3

addressing all comments. adding support for training with 100 templates

693c912

Updates

6fef8c4

StannisZhou and others added 3 commits December 2, 2021 07:29

More tweaks

1a7fe29

Edits

1729f12

addressed comments around styling

d1886d5

shrinuKushagra force-pushed the rcn_notebook branch from e9b9f52 to d1886d5 Compare December 2, 2021 15:29

swaroopgj reviewed Dec 2, 2021

View reviewed changes

added more explanations

1318f5b

shrinuKushagra requested a review from swaroopgj December 2, 2021 20:51

StannisZhou reviewed Dec 3, 2021

View reviewed changes

Shrinu Kushagra added 2 commits December 3, 2021 07:51

adding joblib to dependency and other comments

f3b614c

comment

c4298a3

shrinuKushagra requested a review from StannisZhou December 3, 2021 08:40

Fix MAP inference

3a91eec

StannisZhou approved these changes Dec 3, 2021

View reviewed changes

swaroopgj reviewed Dec 3, 2021

View reviewed changes

examples/rcn.py Outdated Show resolved Hide resolved

Update examples/rcn.py

49c5ca0

Co-authored-by: Swaroop Guntupalli <swaroopgj@gmail.com>

StannisZhou merged commit d8f80b8 into vicariousinc:master Dec 3, 2021

StannisZhou linked an issue Dec 3, 2021 that may be closed by this pull request

Experiments on RCN #79

Closed

NishanthJKumar mentioned this pull request Jan 20, 2022

RCN implementation on a small train and test set #93

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RCN example #96

RCN example #96

shrinuKushagra commented Nov 19, 2021 •

edited

Loading

codecov-commenter commented Nov 19, 2021 •

edited

Loading

StannisZhou left a comment •

edited

Loading

shrinuKushagra commented Nov 19, 2021

StannisZhou left a comment

StannisZhou left a comment

shrinuKushagra commented Nov 30, 2021

StannisZhou commented Dec 1, 2021

StannisZhou left a comment

swaroopgj left a comment

swaroopgj Dec 1, 2021

shrinuKushagra Dec 2, 2021

swaroopgj Dec 2, 2021

shrinuKushagra Dec 3, 2021

shrinuKushagra commented Dec 2, 2021

swaroopgj Dec 2, 2021

swaroopgj Dec 2, 2021

shrinuKushagra Dec 2, 2021

shrinuKushagra Dec 3, 2021

swaroopgj Dec 2, 2021

shrinuKushagra Dec 2, 2021

wlehrach Dec 2, 2021 •

edited

Loading

shrinuKushagra Dec 3, 2021

swaroopgj Dec 2, 2021

shrinuKushagra Dec 2, 2021

swaroopgj Dec 2, 2021

swaroopgj Dec 2, 2021

shrinuKushagra Dec 2, 2021

StannisZhou left a comment

StannisZhou Dec 3, 2021

shrinuKushagra Dec 3, 2021

StannisZhou Dec 3, 2021 •

edited

Loading

swaroopgj Dec 3, 2021

shrinuKushagra Dec 3, 2021

StannisZhou left a comment



		# %%
		def fetch_mnist_dataset(test_size: int, seed: int = 5) -> Tuple[np.ndarray, np.ndarray]:



		# %% [markdown]
		# ## 5.2 Run maximum a-posteriori probability (map) inference on all test images

RCN example #96

RCN example #96

Conversation

shrinuKushagra commented Nov 19, 2021 • edited Loading

codecov-commenter commented Nov 19, 2021 • edited Loading

Codecov Report

StannisZhou left a comment • edited Loading

Choose a reason for hiding this comment

shrinuKushagra commented Nov 19, 2021

StannisZhou left a comment

Choose a reason for hiding this comment

StannisZhou left a comment

Choose a reason for hiding this comment

shrinuKushagra commented Nov 30, 2021

StannisZhou commented Dec 1, 2021

StannisZhou left a comment

Choose a reason for hiding this comment

swaroopgj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shrinuKushagra commented Dec 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wlehrach Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StannisZhou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StannisZhou Dec 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StannisZhou left a comment

Choose a reason for hiding this comment

shrinuKushagra commented Nov 19, 2021 •

edited

Loading

codecov-commenter commented Nov 19, 2021 •

edited

Loading

StannisZhou left a comment •

edited

Loading

wlehrach Dec 2, 2021 •

edited

Loading

StannisZhou Dec 3, 2021 •

edited

Loading