add naming tutorial #10511

piiswrong · 2018-04-11T19:50:36Z

Description

(Brief description on what this PR is about)

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

anirudh2290 · 2018-04-11T21:32:16Z

docs/tutorials/gluon/naming.md

+    dense0_
+
+
+When you create more Blocks of the same kind, they will be named differetly to avoid collision:


differently

Would it make sense to mention that number appended to the name would be incremented to avoid collision ?

anirudh2290 · 2018-04-11T21:56:23Z

docs/tutorials/gluon/naming.md

+
+
+```python
+model0 = Model()


should we use names like zeroth_model and first_model to get the point across.

I think model0 is fine. It matches the prefix

ThomasDelteil · 2018-04-12T01:38:43Z

docs/tutorials/gluon/naming.md

+```
+
+    Dense(None -> 100, linear) alexnet0_dense3_
+


can you please add  to enable the notebook download of your tutorial?

ThomasDelteil · 2018-04-12T01:52:35Z

docs/tutorials/gluon/naming.md

+
+For example, the alexnet in model zoo has 1000 output dimensions, but maybe you only have 100 classes in your application.
+
+To see how to do this, we first load an pretrained alexnet.


a pretrained*

maybe alexnet if you want to refer to it now as an object instead of a model from the zoo, where it would be AlexNet.

ThomasDelteil · 2018-04-12T01:57:03Z

docs/tutorials/gluon/naming.md

+
+```python
+with alexnet.name_scope():
+    alexnet.output = gluon.nn.Dense(100)


Maybe it is worth putting here that this works only because the models in model zoo are inheriting HybridBlock and have been built in a way so that their final output layer is associated with the .output property.

I can see people building their own network usingHybridSequential for example and trying to update the output layer by calling output on their network.

ThomasDelteil · 2018-04-12T01:58:04Z

Great tutorial, it was much needed and clarifies what is going on under the hood with the naming scopes 👍

" the PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes) " 😄

aaronmarkham

I have some minor changes requested and some suggestions for clarity.

aaronmarkham · 2018-04-12T17:12:06Z

docs/tutorials/gluon/naming.md

+
+To manage the names of nested Blocks, each Block has a `name_scope` attached to it. All Blocks created within a name scope will have its parent Block's prefix prepended to its name.
+
+Let's demonstrate this by first define a simple neural net:


aaronmarkham · 2018-04-12T17:24:37Z

docs/tutorials/gluon/naming.md

+model0 = Model()
+model0.initialize()
+model0(mx.nd.zeros((1, 20)))
+print(model0.prefix, model0.dense0.prefix, model0.dense1.prefix, model0.mydense.prefix)


This print all runs together, and... maybe a I'm a little dense ;) but I'm not quite catching the point. Plus I have a hard time breaking it up when the Jupyter font uses an 'l' that looks like a '1', so I see mode10, not model0, and run together it's incomprehensible.

aaronmarkham · 2018-04-12T17:29:43Z

docs/tutorials/gluon/naming.md

+    model1_ model1_dense0_ model1_dense1_ model1_mydense_
+
+
+**It is recommended that you manually specify prefix for the top level Block (i.e. `model = Model(prefix='mymodel_')`) to avoid potential confusions in naming**


specify a prefix

add a period at the end

And why not show that code sample if it's so important?

aaronmarkham · 2018-04-12T17:30:20Z

docs/tutorials/gluon/naming.md

+
+**It is recommended that you manually specify prefix for the top level Block (i.e. `model = Model(prefix='mymodel_')`) to avoid potential confusions in naming**
+
+The same principle also applies to container blocks like Sequantial. `name_scope` can be used inside `__init__` as well as out side of `__init__`:


spelling... Sequential

aaronmarkham · 2018-04-12T17:34:13Z

docs/tutorials/gluon/naming.md

+
+
+```python
+mydense = gluon.nn.Dense(100, prefix='mydense_')


You can also choose to not use an underscore and name it what you want. Maybe explain why you want to use an underscore.

aaronmarkham · 2018-04-12T17:45:39Z

docs/tutorials/gluon/naming.md

+    )
+
+
+As a result if you try to save parameters from model0 and load it with model1, you'll get an error due to unmatching names:


aaronmarkham · 2018-04-12T17:46:20Z

docs/tutorials/gluon/naming.md

+model0.collect_params().save('model.params')
+try:
+    model1.collect_params().load('model.params', mx.cpu())
+except Exception, e:


Not python 3 compatible syntax , --> as

aaronmarkham · 2018-04-12T17:48:25Z

docs/tutorials/gluon/naming.md

+
+Sometimes you may want to load a pretrained model, and replace certain Blocks in it for fine-tuning.
+
+For example, the alexnet in model zoo has 1000 output dimensions, but maybe you only have 100 classes in your application.


the AlexNet model from the model zoo (add link?)

1,000

aaronmarkham · 2018-04-12T17:50:00Z

docs/tutorials/gluon/naming.md

+
+For example, the alexnet in model zoo has 1000 output dimensions, but maybe you only have 100 classes in your application.
+
+To see how to do this, we first load an pretrained alexnet.


maybe alexnet if you want to refer to it now as an object instead of a model from the zoo, where it would be AlexNet.

aaronmarkham · 2018-04-12T17:53:34Z

docs/tutorials/gluon/naming.md

+
+To change the output to 100 dimension, we replace it with a new block.
+
+- Note that it's important to do this in alexnet's name_scope, otherwise you will have unmatching names when you try to save and load your model.


I ran into similar issues and just blanked out the prefix. Here's an example that I used.
https://gist.github.com/aaronmarkham/1017664fe683596c614961112a867145#gistcomment-2336981
Can you explain a bit more? This is all really helpful, and I wonder how you'd explain what I ran into and better ways to fix it.

Like how I used ignore_extra to get around the errors but was still worried that I hadn't really loaded the model properly.

make a fix so that it won't be a problem anymore

szha · 2018-04-13T04:21:45Z

python/mxnet/gluon/block.py

+                                    name=name, type1=type(existing), type2=type(value)))
+
+        if isinstance(value, Block):
+            self.register_child(value, name)


Removing the block update logic can cause having stale blocks in _children.

_children is OrderedDict now

Just noticed. This means it's finally possible to print the model repr in order.

zhanghang1989 · 2018-04-13T18:11:06Z

Still have problem save and load. #10544

* add naming tutorial * fix * Update naming.md * Update index.md * fix save load * fix * fix * fix * fix

piiswrong requested a review from szha as a code owner April 11, 2018 19:50

anirudh2290 reviewed Apr 11, 2018

View reviewed changes

ThomasDelteil reviewed Apr 12, 2018

View reviewed changes

aaronmarkham suggested changes Apr 12, 2018

View reviewed changes

piiswrong force-pushed the msg branch from eacac76 to ff7b468 Compare April 13, 2018 02:20

szha reviewed Apr 13, 2018

View reviewed changes

piiswrong and others added 9 commits April 12, 2018 22:59

add naming tutorial

dc8ce6d

fix

8e063bd

Update naming.md

9585a30

Update index.md

45e458e

fix save load

da2caa6

fix

b02043d

fix

89b8c34

fix

d5805ff

fix

89d71de

piiswrong force-pushed the msg branch from 02362b1 to 89d71de Compare April 13, 2018 05:59

ThomasDelteil approved these changes Apr 13, 2018

View reviewed changes

piiswrong merged commit b929892 into apache:master Apr 13, 2018

luoyetx mentioned this pull request Apr 27, 2018

SymbolBlock has no _reg_params which cause save/load params fails #10695

Closed

rahul003 pushed a commit to rahul003/mxnet that referenced this pull request Jun 4, 2018

add naming tutorial (apache#10511)

7db3786

* add naming tutorial * fix * Update naming.md * Update index.md * fix save load * fix * fix * fix * fix

zheng-da pushed a commit to zheng-da/incubator-mxnet that referenced this pull request Jun 28, 2018

add naming tutorial (apache#10511)

b243fe7

* add naming tutorial * fix * Update naming.md * Update index.md * fix save load * fix * fix * fix * fix

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add naming tutorial #10511

add naming tutorial #10511

piiswrong commented Apr 11, 2018

anirudh2290 Apr 11, 2018

anirudh2290 Apr 11, 2018

anirudh2290 Apr 11, 2018

piiswrong Apr 12, 2018

ThomasDelteil Apr 12, 2018

ThomasDelteil Apr 12, 2018

aaronmarkham Apr 12, 2018

ThomasDelteil Apr 12, 2018

ThomasDelteil commented Apr 12, 2018 •

edited

Loading

aaronmarkham left a comment

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

aaronmarkham Apr 12, 2018

piiswrong Apr 13, 2018

szha Apr 13, 2018

piiswrong Apr 13, 2018

szha Apr 13, 2018

zhanghang1989 commented Apr 13, 2018

		dense0_


		When you create more Blocks of the same kind, they will be named differetly to avoid collision:


		For example, the alexnet in model zoo has 1000 output dimensions, but maybe you only have 100 classes in your application.

		To see how to do this, we first load an pretrained alexnet.


		To manage the names of nested Blocks, each Block has a `name_scope` attached to it. All Blocks created within a name scope will have its parent Block's prefix prepended to its name.

		Let's demonstrate this by first define a simple neural net:

		model1_ model1_dense0_ model1_dense1_ model1_mydense_


		It is recommended that you manually specify prefix for the top level Block (i.e. `model = Model(prefix='mymodel_')`) to avoid potential confusions in naming


		It is recommended that you manually specify prefix for the top level Block (i.e. `model = Model(prefix='mymodel_')`) to avoid potential confusions in naming

		The same principle also applies to container blocks like Sequantial. `name_scope` can be used inside `__init__` as well as out side of `__init__`:

		)


		As a result if you try to save parameters from model0 and load it with model1, you'll get an error due to unmatching names:


		Sometimes you may want to load a pretrained model, and replace certain Blocks in it for fine-tuning.

		For example, the alexnet in model zoo has 1000 output dimensions, but maybe you only have 100 classes in your application.


		To change the output to 100 dimension, we replace it with a new block.

		- Note that it's important to do this in alexnet's name_scope, otherwise you will have unmatching names when you try to save and load your model.



		```python
		mydense = gluon.nn.Dense(100, prefix='mydense_')

add naming tutorial #10511

add naming tutorial #10511

Conversation

piiswrong commented Apr 11, 2018

Description

Checklist

Essentials

Changes

Comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasDelteil commented Apr 12, 2018 • edited Loading

aaronmarkham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhanghang1989 commented Apr 13, 2018

ThomasDelteil commented Apr 12, 2018 •

edited

Loading