wait for non-master workers to finish reading vocab before master wor…

…ker saves it (#4274)

* wait for non-master workers to finish reading

* update CHANGELOG

* typo

CHANGELOG.md

@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Tons of docstring inconsistencies resolved.
 - Nightly builds no longer run on forks.
 - Distributed training now automatically figures out which worker should see which instances
+- A race condition bug in distributed training caused from saving the vocab to file from the master process while other processing might be reading those files.
 
 ### Added
 

allennlp/commands/train.py

@@ -632,7 +632,12 @@ def from_partial_objects(
 
         # Initializing the model can have side effect of expanding the vocabulary.
         # Save the vocab only in the master. In the degenerate non-distributed
-        # case, we're trivially the master.
+        # case, we're trivially the master. But in the distributed case we need to be careful
+        # to avoid a race condition where we might be writing the vocab from the master
+        # process while another process is reading it. So we use a barrier here
+        # to wait for the other processes to finish reading.
+        if common_util.is_distributed():
+            dist.barrier()
         if common_util.is_master():
             vocabulary_path = os.path.join(serialization_dir, "vocabulary")
             vocabulary_.save_to_files(vocabulary_path)