NetCDF thread safety take two

docs/src/common_links.inc

@@ -41,6 +41,7 @@
 .. _issues on GitHub: https://github.com/SciTools/iris/issues?q=is%3Aopen+is%3Aissue+sort%3Areactions-%2B1-desc
 .. _python-stratify: https://github.com/SciTools/python-stratify
 .. _iris-esmf-regrid: https://github.com/SciTools-incubator/iris-esmf-regrid
+.. _netCDF4: https://github.com/Unidata/netcdf4-python
 
 
 .. comment

docs/src/userguide/glossary.rst

@@ -1,3 +1,5 @@
+.. include:: ../common_links.inc
+
 .. _glossary:
 
 Glossary
@@ -125,7 +127,7 @@ Glossary
         of formats.
 
         | **Related:** :term:`CartoPy` **|** :term:`NumPy`
-        | **More information:** `Matplotlib <https://scitools.org.uk/cartopy/docs/latest/>`_
+        | **More information:**  `matplotlib`_
         |
 
    Metadata
@@ -143,9 +145,11 @@ Glossary
         When Iris loads this format, it also especially recognises and interprets data
         encoded according to the :term:`CF Conventions`.
 
+        __ `NetCDF4`_
+
         | **Related:** :term:`Fields File (FF) Format`
          **|** :term:`GRIB Format` **|** :term:`Post Processing (PP) Format`
-        | **More information:** `NetCDF-4 Python Git <https://github.com/Unidata/netcdf4-python>`_
+        | **More information:** `NetCDF-4 Python Git`__
         |
 
    NumPy

docs/src/whatsnew/2.1.rst

@@ -1,3 +1,5 @@
+.. include:: ../common_links.inc
+
 v2.1 (06 Jun 2018)
 ******************
 
@@ -67,7 +69,7 @@ Incompatible Changes
   as an alternative.
 
 * This release of Iris contains a number of updated metadata translations.
-  See this 
+  See this
   `changelist <https://github.com/SciTools/iris/commit/69597eb3d8501ff16ee3d56aef1f7b8f1c2bb316#diff-1680206bdc5cfaa83e14428f5ba0f848>`_
   for further information.
 
@@ -84,14 +86,16 @@ Internal
   calendar.
 
 * Iris updated its time-handling functionality from the
-  `netcdf4-python <http://unidata.github.io/netcdf4-python/>`_
+  `netcdf4-python`__
   ``netcdftime`` implementation to the standalone module
   `cftime <https://github.com/Unidata/cftime>`_.
   cftime is entirely compatible with netcdftime, but some issues may
   occur where users are constructing their own datetime objects.
   In this situation, simply replacing ``netcdftime.datetime`` with
   ``cftime.datetime`` should be sufficient.
 
+__ `netCDF4`_
+
 * Iris now requires version 2 of Matplotlib, and ``>=1.14`` of NumPy.
   Full requirements can be seen in the `requirements <https://github.com/SciTools/iris/>`_
   directory of the Iris' the source.

docs/src/whatsnew/latest.rst

@@ -40,7 +40,8 @@ This document explains the changes made to Iris for this release
 🐛 Bugs Fixed
 =============
 
-#. N/A
+#. `@trexfeathers`_ and `@pp-mo`_ made Iris' use of the `netCDF4`_ library
+   thread-safe. (:pull:`5095`)
 
 
 💣 Incompatible Changes

lib/iris/experimental/ugrid/load.py

@@ -209,7 +209,8 @@ def load_meshes(uris, var_name=None):
 
     result = {}
     for source in valid_sources:
-        meshes_dict = _meshes_from_cf(CFUGridReader(source))
+        with CFUGridReader(source) as cf_reader:
+            meshes_dict = _meshes_from_cf(cf_reader)
         meshes = list(meshes_dict.values())
         if var_name is not None:
             meshes = list(filter(lambda m: m.var_name == var_name, meshes))

lib/iris/fileformats/cf.py

@@ -20,10 +20,10 @@
 import re
 import warnings
 
-import netCDF4
 import numpy as np
 import numpy.ma as ma
 
+from iris.fileformats.netcdf import _thread_safe_nc
 import iris.util
 
 #
@@ -1050,7 +1050,9 @@ def __init__(self, filename, warn=False, monotonic=False):
         #: Collection of CF-netCDF variables associated with this netCDF file
         self.cf_group = self.CFGroup()
 
-        self._dataset = netCDF4.Dataset(self._filename, mode="r")
+        self._dataset = _thread_safe_nc.DatasetWrapper(
+            self._filename, mode="r"
+        )
 
         # Issue load optimisation warning.
         if warn and self._dataset.file_format in [
@@ -1068,6 +1070,19 @@ def __init__(self, filename, warn=False, monotonic=False):
         self._build_cf_groups()
         self._reset()
 
+    def __enter__(self):
+        # Enable use as a context manager
+        # N.B. this **guarantees* closure of the file, when the context is exited.
+        # Note: ideally, the class would not do so much work in the __init__ call, and
+        # would do all that here, after acquiring necessary permissions/locks.
+        # But for legacy reasons, we can't do that.  So **effectively**, the context
+        # (in terms of access control) alreday started, when we created the object.
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        # When used as a context-manager, **always** close the file on exit.
+        self._close()
+
     @property
     def filename(self):
         """The file that the CFReader is reading."""
@@ -1294,10 +1309,15 @@ def _reset(self):
         for nc_var_name in self._dataset.variables.keys():
             self.cf_group[nc_var_name].cf_attrs_reset()
 
-    def __del__(self):
+    def _close(self):
         # Explicitly close dataset to prevent file remaining open.
         if self._dataset is not None:
             self._dataset.close()
+            self._dataset = None
+
+    def __del__(self):
+        # Be sure to close dataset when CFReader is destroyed / garbage-collected.
+        self._close()
 
 
 def _getncattr(dataset, attr, default=None):