Fix for low accuracy in Tensorflow training #167
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Indexed_file_loader.h performs a sharded read of the index files supplied alongside the tfrecords for training. Thus, ReadSample() is called from multiple threads with different shard_id_. When all indexes have been read, we proceed to reset current_index_ to start again at the beginning of the shard. This commit fixes a defect in the detection that the shard has arrived to the end, by comparing the value of current_index to the size of all the indexes. The fix consists in comparing against the end of the shard instead of the end of the indices. Before the fix, the code would work correctly for 1 GPU when there is only 1 shard, but with >1 GPUs, it would result in a loss in accuracy due to---most likely---redundant reads. Signed-off-by: Pablo Ribalta Lorenzo <pribalta@nvidia.com>
JanuszL
reviewed
Sep 13, 2018
| @@ -94,6 +94,9 @@ class IndexedFileLoader : public Loader<CPUBackend> { | |||
| ReadIndexFile(index_uris); | |||
| size_t num_indices = indices_.size(); | |||
| current_index_ = num_indices/num_shards_ * shard_id_; | |||
| max_index_ = (shard_id_ != num_shards_ - 1) | |||
| ? num_indices/num_shards_ * (shard_id_ + 1) | |||
There was a problem hiding this comment.
How about just:
max_index_ = num_indices * (shard_id_ + 1) / num_shards_
Signed-off-by: Pablo Ribalta Lorenzo <pribalta@nvidia.com>
pribalta
force-pushed
the
pribalta_fix_sharding_indexed_reader
branch
from
6875724 to
0e27633
Compare
JanuszL
approved these changes
Sep 13, 2018
|
Waiting till another run finishes and reports good results and I'll be merging this guy |
|
I agree that it is a bug (good job finding it :-) ) but I do not agree fully with the fix, because it is not consistent with the rest of readers. You should instead change the reset function to set the current index to 0 |
Another way to fix this issue is to make the different threads wrap around the number of shards and and only initializing them to their corresponding offset. This solution is consistent with other readers. Signed-off-by: Pablo Ribalta Lorenzo <pribalta@nvidia.com>
ptrendx
approved these changes
Sep 14, 2018
JanuszL
pushed a commit
that referenced
this pull request
Sep 17, 2018
* Fix for low accuracy in Tensorflow training Indexed_file_loader.h performs a sharded read of the index files supplied alongside the tfrecords for training. Thus, ReadSample() is called from multiple threads with different shard_id_. When all indexes have been read, we proceed to reset current_index_ to start again at the beginning of the shard. This commit fixes a defect in the detection that the shard has arrived to the end, by comparing the value of current_index to the size of all the indexes. The fix consists in enabling the shard to wrap around the indices by making reset set current_index_ to zero. Before the fix, the code would work correctly for 1 GPU when there is only 1 shard, but with >1 GPUs, it would result in a loss in accuracy due to shards unevenly loading the dataset. Signed-off-by: Pablo Ribalta Lorenzo <pribalta@nvidia.com>
pribalta
added a commit
to pribalta/DALI
that referenced
this pull request
Oct 1, 2018
* Fix for low accuracy in Tensorflow training Indexed_file_loader.h performs a sharded read of the index files supplied alongside the tfrecords for training. Thus, ReadSample() is called from multiple threads with different shard_id_. When all indexes have been read, we proceed to reset current_index_ to start again at the beginning of the shard. This commit fixes a defect in the detection that the shard has arrived to the end, by comparing the value of current_index to the size of all the indexes. The fix consists in enabling the shard to wrap around the indices by making reset set current_index_ to zero. Before the fix, the code would work correctly for 1 GPU when there is only 1 shard, but with >1 GPUs, it would result in a loss in accuracy due to shards unevenly loading the dataset. Signed-off-by: Pablo Ribalta Lorenzo <pribalta@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Indexed_file_loader.h performs a sharded read of the index files supplied alongside the tfrecords for training. Thus, ReadSample() is called from multiple threads with different shard_id_. When all indexes have been read, we proceed to reset current_index_ to start again at the beginning of the shard.
This commit fixes a defect in the detection that the shard has arrived to the end, by comparing the value of current_index to the size of all the indexes. The fix consists in comparing against the end of the shard instead of the end of the indices.
Before the fix, the code would work correctly for 1 GPU when there is only 1 shard, but with >1 GPUs, it would result in a loss in accuracy due to---most likely---redundant reads.
Signed-off-by: Pablo Ribalta Lorenzo pribalta@nvidia.com