Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

fix BucketSentenceIter bug related to #11430 #11580

Merged
merged 2 commits into from
Jul 18, 2018
Merged

Conversation

Soonhwan-Kwon
Copy link
Contributor

Description

BucketSentenceIter has bug
when there are buckets with no item.
It keeps invalid buckets as self.buckets
but self.data contains only valid buckets and
it leads to the mismatch with sentence length of symbol and actual data shape.
I fixed the problem by checking valid buckets and handed it to self.buckets.
This fixes issue #11430.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@Soonhwan-Kwon Soonhwan-Kwon requested a review from szha as a code owner July 6, 2018 10:33
Copy link
Member

@szha szha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind adding a small test for this change?

@Soonhwan-Kwon
Copy link
Contributor Author

Soonhwan-Kwon commented Jul 6, 2018

Of course not thank you for the review.

@Soonhwan-Kwon
Copy link
Contributor Author

Soonhwan-Kwon commented Jul 8, 2018

I meant you can absolutely feel free to write a small test. I'm sorry if the answer I wrote before confuses you.

@Soonhwan-Kwon
Copy link
Contributor Author

@szha I added test case , by adding smallest empty bucket, and it reproduces error in previous rnn/io.py version and it works good in patched version.

below is the reproduced error.
Traceback (most recent call last):
File "test_bucketing.py", line 110, in
test_bucket_module()
File "test_bucketing.py", line 102, in test_bucket_module
batch_end_callback=mx.callback.Speedometer(batch_size, 50))
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/base_module.py", line 515, in fit
self.forward_backward(data_batch)
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/base_module.py", line 194, in forward_backward
self.forward(data_batch, is_train=True)
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/bucketing_module.py", line 455, in forward
data_batch.provide_label)
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/bucketing_module.py", line 376, in switch_bucket
force_rebind=False, shared_module=self._buckets[self._default_bucket_key])
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/module.py", line 430, in bind
state_names=self._state_names)
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/executor_group.py", line 279, in init
self.bind_exec(data_shapes, label_shapes, shared_group)
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/executor_group.py", line 375, in bind_exec
shared_group))
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/module/executor_group.py", line 662, in bind_ith_exec
shared_buffer=shared_data_arrays, **input_shapes)
File "/home/ubuntu/anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet-1.3.0-py2.7.egg/mxnet/symbol/symbol.py", line 1528, in simple_bind
raise RuntimeError(error_msg)
RuntimeError: simple_bind error. Arguments:
data: (128, 20L)
softmax_label: (128, 20L)
Error in operator split0: [06:10:03] src/operator/./slice_channel-inl.h:216: Check failed: ishape[real_axis] == static_cast<size_t>(param
.num_outputs) (20 vs. 10) If squeeze axis is True, the size of the sliced axis must be the same as num_outputs. Input shape=[128,20,25], axis=1, num_outputs=10.

@Soonhwan-Kwon
Copy link
Contributor Author

I rebased pr.

@szha szha merged commit 904052d into apache:master Jul 18, 2018
KellenSunderland pushed a commit to KellenSunderland/incubator-mxnet that referenced this pull request Jul 21, 2018
* fix BucketSentenceIter bug

* add test case for smallest empty bucket
XinYao1994 pushed a commit to XinYao1994/incubator-mxnet that referenced this pull request Aug 29, 2018
* fix BucketSentenceIter bug

* add test case for smallest empty bucket
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants