BUG: Reference de-duplication isn't parallel-safe

[astrofrog]

Currently Numpydoc is marked as a parallel-safe sphinx plugin, but the renaming of duplicate references in rename_references is not parallel-safe because reference_offset will be [0] in each process at the start of the function call.

I'm running into conflicts when building the Astropy docs:

/Users/tom/Dropbox/Code/Astropy/astropy/astropy/stats/biweight.py:docstring of astropy.stats.biweight_midcorrelation:70: WARNING: duplicate citation R11, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/modeling/index.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/stats/biweight.py:docstring of astropy.stats.biweight_midcovariance:138: WARNING: duplicate citation R33, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.coordinates.BaseCoordinateFrame.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/stats/biweight.py:docstring of astropy.stats.biweight_midvariance:101: WARNING: duplicate citation R56, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.coordinates.EarthLocation.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/stats/biweight.py:docstring of astropy.stats.biweight_midvariance:103: WARNING: duplicate citation R66, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.coordinates.EarthLocation.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/stats/bayesian_blocks.py:docstring of astropy.stats.FitnessFunc:7: WARNING: Unknown target name: "scargle2012".
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/stats/bayesian_blocks.py:docstring of astropy.stats.FitnessFunc:26: WARNING: Unknown target name: "scargle2012".
docstring of astropy.stats.LombScargle.false_alarm_probability:62: WARNING: duplicate citation R1313, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.coordinates.SkyCoord.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/modeling/polynomial.py:docstring of astropy.modeling.polynomial.SIP:55: WARNING: duplicate citation R11, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.stats.biweight_midcorrelation.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/modeling/functional_models.py:docstring of astropy.modeling.functional_models.AiryDisk2D:91: WARNING: duplicate citation R11, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.stats.biweight_midcorrelation.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/modeling/functional_models.py:docstring of astropy.modeling.functional_models.Gaussian2D:133: WARNING: duplicate citation R33, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.stats.biweight_midcovariance.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/modeling/optimizers.py:docstring of astropy.modeling.optimizers.SLSQP:19: WARNING: duplicate citation R99, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.coordinates.Galactic.rst
/Users/tom/Dropbox/Code/Astropy/astropy/astropy/modeling/optimizers.py:docstring of astropy.modeling.optimizers.Simplex:18: WARNING: duplicate citation R1111, other instance in /Users/tom/Dropbox/Code/Astropy/astropy/docs/api/astropy.stats.LombScargle.rst

[stefanv]

This is correct: numpydoc reference renumbering is not parallel safe.

[jnothman]

Should we consider making the reference renumbering a separate extension, marked unsafe? It would seem to me to be completely separate from the rest of numpydoc.

[pv]

IIRC the numpydoc citation format is not RST citation but RST footnotes, so not so clear all mangling can be removed.

[stefanv]

At the cost of some nastier looking reference names, we could hash the reference text itself to generate the link IDs.

[stefanv]

(This should solve the parallel problem;)

[jnothman]

ideally we wouldn't change the displayed reference names at all, but generate unique IDs for linking. I wonder if there's some way to do that.

[stefanv]

We can probably just generate internal links?

[jnothman]

I.e. work around the reST citation mechanism? I suppose we could.

[jnothman]

It would be hard to get citation formatting matching the default in both web and PDF using :ref: links.

[jnothman]

(I.e. in PDF the bibliography appears as an endnote)

[jnothman]

At the cost of some nastier looking reference names, we could hash the reference text itself to generate the link IDs.

I think this could be acceptable. If the same hash appears multiple times in a doc, the first will be the link target, but maybe we can live with this. Alternatively, we can use numbered refs prefixed by the docstring object name.

There seem to be a number of issues with reference mangling. It appears to be untested. It should be possible to disable, and perhaps should be an extension of its own with parallel_read_safe=False. Unfortunately, metadata such as parallel_read_safe cannot be a function of config, so we are forced to set parallel_read_safe=False as long as an unsafe solution is possible within the numpydoc extension.

I'm inclined to one of the parallel-safe solutions above, but perhaps still making the mangling possible to disable...