NEURON 9.0 release

[pramodk]

Action items

Pre-release

• Make sur to look out for ModelDB regressions by launching analyzing nrn-modeldb-ci last version vs nightly reports

Sanity checks

• Create release/x.y branch and make sure GitHub, Azure and CircleCI builds pass
• Run nrn-build-ci for the respective Azure build; see Azure drop guide
• Activate ReadTheDocs build for release/x.y & make it hidden. Check docs are fine after build is done.
• Run a test wheel build WITHOUT upload for release/x.y to ensure all the wheels build (see details)
• Run BBP Simulation Stack & other relevant tests

Releasing

• Update semantic version in CMakeLists.txt
• Update changelog below and agree on it with everyone; then commit it to docs/changelog (copy structure as-is)
• Update docs/index.rst accordingly with the new .pkg and .exe links for PKG installer and Windows Installer
• Run the ReadTheDocs build again for release-x.y, make sure the build passes and inspect the Changelog page.

Important: we print the following message from nrnivmodl on error. Make sure the URL is right one!

NOTE: If you are encountering MOD file compilation errors only with NEURON version 9.0 or later
refer to the C++ migration guide at https://nrn.readthedocs.io/en/9.0.0/guide/porting_mechanisms_to_cpp.html

• Create new release+tag on GitHub via release workflow. Note that the GitHub release will be marked as pre-release and will contain the full-src-package and the Windows installer at the end of the release workflow.
• Build release wheels but WITHOUT upload (see details)
• Create, test and upload manual artifacts
  • MacOS package installer (manual task, ask Michael)
  • arm64 wheels (manual task, check with Erik, Goran or Pramod)
  • aarch64 wheels (create a release/x.y-aarch64 branch for this, see guide)
• Publish the x.y.z wheels on Pypi; see wheel publishing instructions
• Once wheels are published, activate the x.y.z tag on ReadTheDocs
• Rename the Windows installer in the GitHub release to match the new version and the supported python versions (i.e. nrn-8.2.2.w64-mingw-py-37-38-39-310-311-setup.exe
  )
• Publish release on GitHub (edit https://github.com/neuronsimulator/nrn/releases/tag/x.y.z and un-tick the pre-release checkbox)

Post-release

• To mark the start of a new development cycle, tag master as follows:
  • minor version: x.(y+1).dev
  • major version: (x+1).0.dev
• Deactivate ReadTheDocs build for release/x.y
• Go to ReadTheDocs advanced settings and set Default version to x.y.z
• Let people know 🚀
• Cherrypick changelog and installer links to master
• Update the changelog for the release on GitHub

Changelog

NEURON X.Y

[x.y.z]

Release Date : DD-MM-YYYY

What's New

• [List new features/support added]
• .....

Breaking Changes

• [List the changes that aren't backward compatible]
• ...

Deprecations

• [List the features that are deprecated]
• ...

Bug Fixes

• [List the important bug fixes]
• ...

Improvements / Other Changes

• [List the improvements made in the new release and any other changes]
• ...

Upgrade Steps

• [Describe how to migrate from previous NEURON Version]
• ...

For the complete list of features and bug fixes, see the list in GitHub Issue #[GH_no.]

ReadTheDocs sneak peek

• https://nrn.readthedocs.io/en/release-x.y

Commits going into x.y.z

[given a.b.c is the last release:]

Since [a.b.c], with:

git log --pretty=format:"%h : %s" a.b.c..release/x.y

we get:

• commit 1
• commit 2
• ...

[JCGoran]

Currently there are some failures in modelDB for version 9.0, as seen here: https://github.com/neuronsimulator/nrn-modeldb-ci/actions/runs/11586042419/job/32255970874
The culprit appears to be this PR: #3093
I tested with a patched version of release 8.2, and the differences between the two (9.0 nightly and patched 8.2) disappear, so the failures are actually a non-issue.

[JCGoran]

Somehow #3143 (see #3143 (comment) for reproducer) broke the arm64 build on Linux: https://app.circleci.com/pipelines/github/neuronsimulator/nrn/9533/workflows/f44568f7-581f-4a2b-b9bf-fdf643ba0c14/jobs/4696
so the release is currently blocked.
UPDATE: same PR seems to break on MacOS on arm as well: https://github.com/neuronsimulator/nrn/actions/runs/11632330511/job/32395225512
The "good" news is, if we revert the commit, it passes: https://github.com/neuronsimulator/nrn/actions/runs/11651457741/job/32441452028

[JCGoran]

It also seems that this commit (from #3135): 510b9bc
causes some modelDB models to fail: https://github.com/neuronsimulator/nrn-modeldb-ci/actions/runs/11682619896/job/32530121626
I've verified that the parent commit works: https://github.com/neuronsimulator/nrn-modeldb-ci/actions/runs/11682464986/job/32529644621

The issue in #3137 (comment) appears to be unrelated to this one (tested locally by building a wheel on MacOS in a Docker container that runs Linux under arm64).

[nrnhines]

I've looked at the gout for

runmodels --virtual --workdir=8.2.6-g078a34a9d 126467 113997 256388 149100 144520 3800
runmodels --virtual --workdir=9.0a-390-g68b193d8d 126467 113997 256388 149100 144520 3800

and, yes, they all differ. However looking at the trajectories with diffgout they are more similar than I can ever get them to correspond for versions 2 and 3 of cvode at any tolerance. This has been an issue since 1996. And it has always forced me to do manual checking. See #1328 (comment)

[nrnhines]

I added the NRN_ENABLE_ARCH_INDEP_EXP_POW feature to a temporary branch off release/8.2. It is remarkable. that the mere presence of that code (even without activating the mpfr calculation of pow(x, y)) produces identical gout results in comparison with the master for the above models, 126467 113997 256388 149100 144520 3800.

[nrnhines]

In this particular case, it seems isolated to the change (starting from 8.2) src/nmodl/nocpout.cpp

 \n#define exp hoc_Exp\nextern double hoc_Exp(double);\
+\n#undef pow\
+\n#define pow hoc_pow\
 \n#endif\n\

which is used in the translated hh.c file in the line

_lq10 = pow( 3.0 , ( ( celsius - 6.3 ) / 10.0 ) ) ;

[nrnhines]

So, when src/nmodl/nocpout.cpp is modified so that:

+++ b/src/nmodl/nocpout.cpp
@@ -280,8 +280,8 @@ void parout() {
 \n#if !NRNGPU\
 \n#undef exp\
 \n#define exp hoc_Exp\nextern double hoc_Exp(double);\
-\n#undef pow\
-\n#define pow hoc_pow\
+\n static double __attribute__((noinline)) foopow(double x, double y) { return pow(x, y); }\
+\n#define pow foopow\
 \n#endif\n\
 ");

the result are the same except for 1 as the master. But, I guess, the following is inlined

$ for i in `(cd 8.2.6-pow ; find . -name gout)`; do cmp 8.2.6-pow/$i 8.2.6-g078a34a9d/$i ; done
8.2.6-pow/./126467/NegroniLascano/gout 8.2.6-g078a34a9d/./126467/NegroniLascano/gout differ: byte 37, line 4
8.2.6-pow/./149100/HayEtAl2013/gout 8.2.6-g078a34a9d/./149100/HayEtAl2013/gout differ: byte 34, line 4
8.2.6-pow/./256388/PousinhaMouskaBianchiEtAl2019/gout 8.2.6-g078a34a9d/./256388/PousinhaMouskaBianchiEtAl2019/gout differ: byte 36, line 4
8.2.6-pow/./3800/cardiac1998/gout 8.2.6-g078a34a9d/./3800/cardiac1998/gout differ: byte 37, line 4

$ for i in `(cd 8.2.6-pow ; find . -name gout)`; do cmp 8.2.6-pow/$i 9.0a-390-g68b193d8d/$i ; done
8.2.6-pow/./144520/DiFrancescoNoble1985/gout 9.0a-390-g68b193d8d/./144520/DiFrancescoNoble1985/gout differ: byte 36, line 4

but when the __attribute__((noinline)) is commented out then all files are identical to the original 8.2.6-g078a34a9d

[nrnhines]

Chatgpt says:
Reasons for Different Results:
Floating-Point Optimization: When functions are inlined, compilers may apply different optimizations that affect the precision and intermediate calculations of floating-point operations. This can lead to small differences in results due to reordering or additional intermediate calculations.
Register Use and Rounding: Inlined functions may use registers for intermediate results more efficiently than when the function is called separately. This affects rounding behavior because floating-point registers might have extended precision compared to variables stored in memory.
Compiler Optimization Levels: Depending on your optimization flags (e.g., -O2, -O3), the compiler might perform aggressive optimizations, including more extensive inlining and rearrangement of instructions that affect numerical stability.
What to Consider:
Floating-Point Consistency: If precise consistency in floating-point results is critical, you can use compiler flags like -ffloat-store or adjust optimization flags to control floating-point behavior.
Compiler Options: Experiment with different compiler flags (e.g., -O0, -O2, -O3, -march=native) to understand how they affect inlining and floating-point handling.
Function Attributes: Using attribute((noinline)) ensures that the function call is never inlined, which may help maintain consistency when you need exact behavior for debugging or reproducibility.

[nrnhines]

It has been a bit frustrating since none of -ffloat-store or -fexcess-precision=standard or -fno-inline have changed either the master or 8.2 results. Only the __attribute__((noinline)) in 8.2 has had the effect of making (all but one of) the results the same as the master.

It seems that -O2 is a problem. 8.2 produces same results as master when 8.2 is configured with

cmake .. -G Ninja -DCMAKE_INSTALL_PREFIX=install -DCMAKE_BUILD_TYPE=Debug

[pramodk]

@JCGoran : Apologies for not able to provide this earlier. As mentioend during the standup, I scanned the commits and here is my summary. I might have missed few things or there might be some inconsistencies. Feel free to fix those!

NOTE: https://nrn.readthedocs.io/en/latest/guide/porting_mechanisms_to_cpp.html#porting-mechanisms-to-cpp should be relative and not the absolute like this one.

Edit: @JCGoran : by the way, it would be nice to turn #XY into links to corresponding PRs / issues.

NEURON 9.0 marks a major milestone with significant under-the-hood updates.
This release features a redesign of internal data structures using the
Structure-Of-Arrays (SoA) memory layout, migration of the codebase and translation
of MOD files to C++, and the introduction of random number generator construct natively
in the NMODL language. Additionally, many legacy NMODL constructs have been removed,
and legacy code and libraries have been replaced with modern alternatives
like Eigen, ensuring improved performance and maintainability.

### What’s New

- Use of SoA memory layout for internal data structures (#2027, #2381, #2712)  
- Support for random number generator language construct (RANDOM) in NMODL (#2627)  
- Replace pthread-based threading implementation with std::thread (#1859)  
- Introduce meaningful Python types: HOC classes associated with Python types (#1858)  
- Formalization of C API for NEURON (first version) (#2357)  
- Introduce nanobind to gradually replace C Python API usage (#2545)  
- Requires C++17 compiler and flex >= 2.6 (#1893)  
- CoreNEURON repository is merged into NEURON (#2055)  
- Support for `numpy>=2` added (#3040)  
- Replace legacy Meschach source copy with Eigen library (#2470)  

### Breaking Changes

- MOD files are compiled as C++ instead of C (see [migration guide](https://nrn.readthedocs.io/en/latest/guide/porting_mechanisms_to_cpp.html#porting-mechanisms-to-cpp) for VERBATIM blocks)  
- API changes in the functions related to random numbers, see #2618  
- Restored usage of TABLE statement in hh.mod, results are now same as 8.0.2 (see #1764)  
- NMODL: Disallow use of ion variable in the CONSTANT block (#1955)  
- NMODL: Disallow declaring variables and functions with the same name (#1992)  
- Usage of Eigen library could introduce floating point differences, but they are very small, validated and can be ignored  


### Removal / Deprecation

- Removed usage of legacy OS X carbon libraries (#1869)  
- Removed legacy LINDA code and Java bindings (#1919, #1937)  
- Removed unused NMODL methods: adams, heun, clsoda, seidel, simplex, gear (#2032)  
- Removed support for Python 3.7 and 3.8 (#2194, #3108)  
- Removed support for legacy units and dynamic units selection (#2539)  
- Removed unused NEMO simulator compatibility (#3035)  
- Removed checkpoint feature (`h.checkpoint()`), superseded by SaveState (#3092)  
- Removed legacy Random.MLCG and Random.ACG random number generators (#3189, #3190)  
- Removed support for mod2c transpiler for CoreNEURON (#2247)  
- Removed unused NMODL constructs: PUTQ, GETQ, RESET, MATCH, TERMINAL, SECTION,  
  IFERROR, MODEL_LEVEL, PLOT, SENS, etc. (#1956, #1974, #1975, #2001, #2004, #2005)  
- Removed `__MWERKS__` (CodeWarrior compiler) related code (#2655)  
- Removed obsolete uxnrnbbs code (#2203)  
- Removed old tqueue implementations (#2225, #2740)  


### Bug Fixes

- Fix segfault from unref hoc_obj, see #1857 (#1917)  
- Fix bug in rxd reactions involving rxd parameters (#1933)  
- Fix CoreNEURON bug when checkpointing VecPlay instances (#2148)  
- Fix nocmodl bug with unused STATE + COMPARTMENT variable in KINETIC block (#2210)  
- Fix for dynamic ECS diffusion characteristics (#2229)  
- Various fixes for compatibility with newer NVHPC compiler releases (#2239, #2591)  
- Fix Windows and Mac crash on multiline HOC statements input from terminal (#2258)  
- Fix for Vector.max_ind (#2251)  
- Fix for detecting ION channel type in LoadBalance (#2310)  
- Fix for RangeVarPlot adding extra point at section ends when using plotly (#2410)  
- Fixed mod file array variable assignment (seg.mech.array[i] = x) (#2504)  
- Various fixes in RxD features (#2593, #2588)  
- Fix CoreNEURON bug causing in-memory model copy with file mode (#2767)  
- Fix CoreNEURON bug when using array variables in MOD files (#2779)  
- Fix lexer for ONTOLOGY parsing (#3091)  
- Fix issue while using Anaconda Python on MacOS (#3088)

### Other Improvements and Changes

- Allow `segment.mechanism.func(args)` to call a FUNCTION or PROCEDURE (#2475)  
- Simplify INDEPENDENT statement as only `t` variable is accepted (#2013)  
- Fix various memory leaks and related improvements (#3255, #3257, #3243, #2456)  
- Support for reading reporting related information with in-memory mode (#2555)  
- NMODL language documentation update (#2152, #2771, #2885)  
- Documentation improvements including transfer from nrn.yale.edu to nrn.readthedocs.io (#1901, #2922, #2971, #3106, #3187)  
- Various test, CI, and build infrastructure improvements (#1991, #2260, #2474)  
- Migrate many C code parts to C++ (#2083, #2305)  
- Support online LFP calculation in CoreNEURON (via SONATA reports for BBP use cases) (#2118, #2360)  
- Adopt CoreNEURON cell permute functionality into NEURON (#2598)  
- Allow legacy `a.b(i)` syntax for 1d arrays (extend to `PointProcess.var[i]` and `ob.dblarray[i]`) (#2256)  
- Add an ability to run NEURON ModelDB CI using label (#2108)  
- Remove unused code parts under various preprocessor variables (#2007)  
- Return error/exit codes properly in `nrniv -c` and nrnivmodl (#1871)    
- No limit on number of ion mechanisms that can be used (#3089)  
- Added Jupyter support for ModelView (#1907)  
- Make notebooks work with `bokeh>=3` (#3061)  
- Added new mmethod `ParallelContext.get_partition()` (#2351)  

[nrnhines]

@pramodk I'm delighted with your changelog. Great job. Thanks!

[JCGoran]

I've made the necessary changes on the jelic/changelog_release9 branch for the changelog, have a look at the deployment here and let me know if something needs fixing (release date is provisional of course):
https://nrn.readthedocs.io/en/jelic-changelog_release9/changelog.html

[nrnhines]

I would like to make a few changes to the changelog. Where is the best place to do so. (The https://github.com/neuronsimulator/nrn/tree/jelic/changelog_release9 ?)

[JCGoran]

@nrnhines I don't think we have a central location, so that branch is one option.