To Do for 4.0.10

[scarlehoff]

If everything goes according to plan, 4.0.9 will bring a lot of changes to the structure of the code -without changing any of the results- and I would like to collect the list of items that we are planning to do (and that needs to be done) to minimize the chance of getting them lost in a sea of smaller PRs.
At the same time, whenever we add some big PR that we should keep track of, it will be added here.

NNPDF 4.0.10

This is the first release towards NNPDF 4.1 and as such this can be considered "NNPDF 4.1 beta". For now we are going to keep 4.0.X in order to keep the tags in the code synchronised with the papers, but it might be changed to 4.1 at a certain point.

Fits for the 4.0 "era" can be reproduced with the tag 4.0.9

Data and theory implementation #1709

• All commondata implemented in NNPDF4.0 has been ported to the new format
• The new commondata format has been partially documented
• The new commondata reader has been merged to master and substituted the old one
• Polish legacy jet datasets #2056
• The old commondata and the parsing of old commondata has been removed.
• The theory database is now in the form of .yaml files (Make the sqlite theorydb into a folder of yaml files #1997 Provide an unique truth for all theory cards #2028)
• Execute new commondata generation in CI #2099

Performance and logistics

• Refactoring to ease multireplica PDF, progress tracked in PR Multi Replica PDF #1880
• Training/Validation masks are no longer set equal across multireplica GPU runs (Parallel replicas with varying tr-vl masks #1788 )
• Refactored FK convolution for a slight speed up in CPU and a massive speed up in GPU (Fk refactor #1936)
• All examples in the validphys/examples list are now tested in the CI (Run example runcards in tests #1449)
• All theory covariance technology not tested and not used for hep-ph/xxxxx has been removed (Add a new plot_fancy_dataspecs and chi2 distributions with scale variations  #1830, New plotting functions for MHOU #1832, Remove the fivetheories and seventheories key and use point_prescription instead. #1847)
• LHAPDF .info files are read directly with .yaml Use yaml for .info files. #1552
• Test code in the documentation is now tested Run documentation examples using doctest #1426
• Rules are now immutable objects (Hashable Rules #2076)

Build system and installation

• The usage of cmake is now optional and only used to install evolven3fit_legacy and buildmaster (Improve the installation of nnpdf packages #1773, Simplify development environment installation instructions #1690)
• evolven3fit has become the new standard evolution tool
• Code updated to be compatible with python3 / tensorflow >2.16 / keras 3 (Fix multidense compatibility with tf 2.16 and keras 3 #1975, Changes needed to enable Python 3.12 #1970)
• Commondata and the theorydb is installed as part of the python package (Implement a python-only installation #1861)
• ${XDG_CONFIG_HOME}/NNPDF/nnprofile.yaml is now the default location for the NNPDF profile (Implement a python-only installation #1861)
• Added the flag redo-regression in github to autogenerate regression tests (Automate regression updates #1944)
• Data can now be installed as a separate package (NNPDF data package #2009)
• pre-commit has been added to the repository (add .pre-commit-config.yaml #2026)
• Data can be now regenerated automatically in the CI (Execute new commondata generation in CI #2099)
• All documentation has been updated to reflect these changes

New features

• Cuts can be now applied to positivity and integrability datasets (Positivity cuts #2059, replace POS_F2 process with more inclusive POS_DIS #2116)
• It is now possible to accept negative pseudodata in the replica generation (add resample_pseudodata flag #2088)
• The fitting now is compatible with pytorch as well Make the code compatible with other keras backends #2137

Breaking changes

• Replicas are now generated using directly the covariance matrix instead of the error breakdown Set separate_multiplicative as False by default #2107
• Convert validphys.core objects to dataclasses #408
• Paramfits has been removed remove paramfits scripts #2190

evolven3fit

• When an eko is not compatible with the current version, a helpful message will be displayed
• The theory and operator cards are added to the log file (Dump theory and operator cards to the evolution log file #2063)
• FastKernel Tables and EKO are separated in the server and donwloaded separately (Separate eko.tar and fastkernel tables in the server #2081, Make EKO into a separate resource #2087)

Bugfixes

• fix std_error function in MCStats class #2036
• make pseudodata_table correctly deal with multiple replicas #2034
• Fixing bug in the MUL / ADD treatment when writing down closure data #2045
• add nnpdf4.1 baseline theorycard #2048
• Use results to create th uncertainties with pdf errors. #2051
• add validphys.commondata to vp-setupfits providers list #2066
• fix table chi2 grouped by #2070
• replace scipy.integrate.simps which has been removed in favor of simpson #2119
• take n3lo_ad_variations from the corresponding key in theorycard instead of 'Comments' #2115
• make sure tempfile is alive when it's used #2114
• Revert the default for parallel fits to a single model per replica #2127

Please, don't use this issue to discuss these points (I'm going to lock it indeed, although most of you could still write if you really want :P). If you want to tackle some specific issue, please comment in the relevant PR/issue (or open it if it doesn't exist) or raise it during the code meetings.