Add an inference script providing both accuracy and benchmark result for original wide_n_deep example #13895

juliusshufan · 2019-01-16T03:47:14Z

Description

Add a script for inference based on the saved model and parameter files achieved during training.
This script can provide either accuracy or benchmark results with specified parameters.
@TaoLv @pengzhao-intel

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

a new script file example/sparse/wide_deep/inference.py
Readme also slightly revised.

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

pengzhao-intel · 2019-01-16T05:03:56Z

example/sparse/wide_deep/inference.py

+                    help='the model prefix')
+
+# Related to feature engineering, please see preprocess in data.py
+ADULT = {


this is common parts with training. Could we put in some util/model file?

move to config.py

TaoLv · 2019-01-16T14:23:01Z

example/sparse/wide_deep/inference.py

+                    help='loading the params of the corresponding training epoch.')
+parser.add_argument('--batch-size', type=int, default=100,
+                    help='number of examples per batch')
+parser.add_argument('--accuracy', action='store_true', default=False,


I would like to have accuracy=True by default. Use --benchmark to do performance benchmarking as what we do in other scripts.

thanks for the comments, modified accordingly.

TaoLv · 2019-01-16T14:23:56Z

example/sparse/wide_deep/inference.py

+parser.add_argument('--verbose', action='store_true', default=False,
+                    help='accurcy for each batch will be logged if set')
+parser.add_argument('--cuda', action='store_true', default=False,
+                    help='Train on GPU with CUDA')


Train? BTW, we use --gpu in other examples.

thanks, now use '--gpu', same change also applied to train.py

stu1130 · 2019-01-16T17:27:47Z

@mxnet-label-bot add [pr-awaiting-review]
Thanks for your contribution @juliusshufan

larroy · 2019-01-17T12:44:33Z

example/sparse/wide_deep/inference.py

@@ -0,0 +1,121 @@
+# Licensed to the Apache Software Foundation (ASF) under one


Can we add python3 shebang?

Hi Larroy,

thanks for review, I think this script can work with Python2, it might be fine without python3 shebang?

Thanks.

larroy · 2019-01-17T12:45:38Z

example/sparse/wide_deep/inference.py

+    'hidden_units': [8, 50, 100],
+}
+
+if __name__ == '__main__':


could we wrap it into a main function as:

if __name__ == '__main__': sys.exit(main())

Hi Larroy,

Seems sys only supported by python3, which may introduce incompatibility to python2, meanwhile, I just follow the "structure" of exiting train.py to implement the inference.py. thanks

juliusshufan · 2019-01-30T02:23:15Z

@szha and @TaoLv @pengzhao-intel @larroy may I have your approval or comments on this PR?
thanks.

TaoLv · 2019-01-30T13:41:36Z

example/sparse/wide_deep/inference.py

+                    help='number of examples per batch')
+parser.add_argument('--accuracy', action='store_true', default=True,
+                    help='run the script for inference accuracy, not set for benchmark.')
+parser.add_argument('--benchmark', action='store_true', default=False,


How about only have --benchmark here and remove --accuracy? If --benchmark is given in the command line, it will do benchmarking with dummy data, otherwise the script will run with real data and print the final accuracy.

thanks for review and agree, revised accordingly.

TaoLv · 2019-01-30T13:49:37Z

@ZiyueHuang Could you help to review the change?

vandanavk

Please execute pylint on these files using ci/other/pylintrc in incubator-mxnet folder.
Something like this
pylint --rcfile=ci/other/pylintrc --ignore-patterns="..so$$,..dll$$,._.dylib$$" example/sparse/wide_deep/*.py

vandanavk · 2019-02-08T17:20:18Z

example/sparse/wide_deep/inference.py

+    data_iter = iter(eval_data)
+    if benchmark:
+        logging.info('Inference benchmark started ...')
+        nbatch = 0


nit: nbatch =0 can be moved before the if...else

Thanks, revised accordingly

pengzhao-intel

LGTM

@juliusshufan please resolve other reviewers' comments.

TaoLv

@juliusshufan Thank you. LGTM now.

TaoLv · 2019-02-13T09:59:01Z

@vandanavk @larroy Please confirm if your comments are addressed.

ankkhedia · 2019-02-15T22:50:24Z

@vandanavk @larroy Could you please check if your comments are resolved.

ankkhedia

LGTM

TaoLv · 2019-02-16T11:06:20Z

Seems there is no other comments. Merging now.

…for original wide_n_deep example (apache#13895) * Add a inference script can provide both accuracy and benchmark result * minor changes * minor fix to use keep similar coding style as other examples * fix typo * remove code redundance and other minor changes * Addressing review comments and minor pylint fix * remove parameter 'accuracy' to make logic simple

Add a inference script can provide both accuracy and benchmark result

5acc379

juliusshufan requested a review from szha as a code owner January 16, 2019 03:47

juliusshufan changed the title ~~Add a inference script can provide both accuracy and benchmark result~~ Add a inference script providing both accuracy and benchmark result for original wide_n_deep example Jan 16, 2019

minor changes

13feab2

pengzhao-intel reviewed Jan 16, 2019

View reviewed changes

TaoLv reviewed Jan 16, 2019

View reviewed changes

marcoabreu added the pr-awaiting-review PR is waiting for code review label Jan 16, 2019

larroy reviewed Jan 17, 2019

View reviewed changes

juliusshufan added 3 commits January 22, 2019 22:04

minor fix to use keep similar coding style as other examples

4ecb0f7

fix typo

a2157b8

remove code redundance and other minor changes

9c1b331

juliusshufan changed the title ~~Add a inference script providing both accuracy and benchmark result for original wide_n_deep example~~ Add an inference script providing both accuracy and benchmark result for original wide_n_deep example Jan 30, 2019

TaoLv reviewed Jan 30, 2019

View reviewed changes

vandanavk reviewed Feb 8, 2019

View reviewed changes

pengzhao-intel approved these changes Feb 11, 2019

View reviewed changes

juliusshufan added 2 commits February 11, 2019 20:56

Addressing review comments and minor pylint fix

e695ab6

remove parameter 'accuracy' to make logic simple

a5da23b

TaoLv approved these changes Feb 12, 2019

View reviewed changes

ankkhedia approved these changes Feb 15, 2019

View reviewed changes

TaoLv merged commit 8bfbb7d into apache:master Feb 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an inference script providing both accuracy and benchmark result for original wide_n_deep example #13895

Add an inference script providing both accuracy and benchmark result for original wide_n_deep example #13895

juliusshufan commented Jan 16, 2019 •

edited

Loading

pengzhao-intel Jan 16, 2019

juliusshufan Jan 23, 2019

TaoLv Jan 16, 2019

juliusshufan Jan 23, 2019

TaoLv Jan 16, 2019

juliusshufan Jan 23, 2019

stu1130 commented Jan 16, 2019

larroy Jan 17, 2019

juliusshufan Jan 23, 2019 •

edited

Loading

larroy Jan 17, 2019

juliusshufan Jan 30, 2019 •

edited

Loading

juliusshufan commented Jan 30, 2019

TaoLv Jan 30, 2019

juliusshufan Feb 11, 2019

TaoLv commented Jan 30, 2019

vandanavk left a comment

vandanavk Feb 8, 2019

juliusshufan Feb 11, 2019

pengzhao-intel left a comment

TaoLv left a comment

TaoLv commented Feb 13, 2019

ankkhedia commented Feb 15, 2019

ankkhedia left a comment

TaoLv commented Feb 16, 2019

		@@ -0,0 +1,121 @@
		# Licensed to the Apache Software Foundation (ASF) under one

Add an inference script providing both accuracy and benchmark result for original wide_n_deep example #13895

Add an inference script providing both accuracy and benchmark result for original wide_n_deep example #13895

Conversation

juliusshufan commented Jan 16, 2019 • edited Loading

Description

Checklist

Essentials

Changes

Comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stu1130 commented Jan 16, 2019

Choose a reason for hiding this comment

juliusshufan Jan 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliusshufan Jan 30, 2019 • edited Loading

Choose a reason for hiding this comment

juliusshufan commented Jan 30, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TaoLv commented Jan 30, 2019

vandanavk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pengzhao-intel left a comment

Choose a reason for hiding this comment

TaoLv left a comment

Choose a reason for hiding this comment

TaoLv commented Feb 13, 2019

ankkhedia commented Feb 15, 2019

ankkhedia left a comment

Choose a reason for hiding this comment

TaoLv commented Feb 16, 2019

juliusshufan commented Jan 16, 2019 •

edited

Loading

juliusshufan Jan 23, 2019 •

edited

Loading

juliusshufan Jan 30, 2019 •

edited

Loading