v0.14.0

Release v0.14.0 - April 2022

Overall changes

• |feature| |api| Complete overhaul of the DeerLab modelling and fitting interface. Check the new documentation for help and details. (#218, #223, #228, #237, #225, #243).

  • A new modelling system has been introduced. DeerLab main interface runs on a new Model object class. Models implement and provide the distinction between linear and non-linear parameters.
  • Model parameters are no longer (solely) identified by their indexing inside a parameter vector, but are referenced by name. This avoids the need for a user to recall the ordering of the parameters. This is now all handled internally. For example, before paramA = parameters[idxA] is now model.paramA.
  • Any model parameter is accessible from the model object and its boundaries, start values and other properties can be easily modified. For example, to change the lower boundary of a parameter: model.paramA.lb = 0.
  • A new general fit function that fits arbitrary Model objects to single or multiple datasets has been implemented. The function automatically handles the selection of solvers to optimally fit the data to the model.
  • Implemented a new function link to link model parameters (setting equality constraints in models).
  • Implemented a new function merge to create a model returning more than one response (allowing the creation of global models).
  • Implemented a new function relate to define functional relationships between model parameters.
  • Implemented a new function lincombine to create a model whose response is a linear combination of the inputs' model responses.
  • Model parameters can now be frozen (set to a constant value and ignored during optimization) in the Model object and on the back-end snlls solver. For examples, to fix a parameter to a certain value: model.paramA.freeze(0.5).
  • Arbitrary normalization conditions can be imposed to the linear parameters.
  • Bootstrapping can now be requested directly from the fit function via the bootstrap keyword argument. The function will then return the bootstrap uncertainty quantification of all model parameters and of the model's response instead of the covariance-based uncertainty.
  • Implemented a new function dipolarmodel, which generates models based on the dipolar EPR multi-pathway theoretical model.
  • Added new examples, adapted existing ones, and removed unneeded examples.
  • Add many new tests and removed tests related to deprecated functionality.
  • All the built-in parametric models are now pre-compiled Model objects instead of just functions.
  • The function fitmodel has been deprecated and removed. The original has been substituted (and greatly expanded) by the new dipolarmodel and fit functions.
  • The function fitmultimodel has been deprecated and removed. The original functionality can be easily scripted with the new modelling system. An example of has been added, describing how to script the same functionality.

• |feature| Introduced the profile-likelihood methodology both for uncertainty quantification based on likelihood-confidence intervals, and for identifiability analysis (#222).

  • Added a new function profile_analysis to calculate the objective function profiles from model object parameters.
  • Implemented a new uncertainty quantification UQResult object type 'profile' for results obtained from profile_analysis.

• |feature| Implemented a system to specify arbitrary penalty functions to be included in the non-linear part of the objective function during optimization. The penalties can be custom-defined and constructed into a Penalty object that can be passed to the fit function. Outer optimization of the penalty weights can also be included based on certain information-based criteria (#197, #218, #225).

  • Implemented a new object Penalty that includes the penalty function, weight parameter (and its boundaries), and the selection functional for optimization.
  • Adds new outer optimization options for the penalty weights, based on hard-coded model selection functionals. For now, the ICC, AIC, AICc, and BIC functionals are available.
  • Implemented a new function dipolarpenalty that generates dipolar-EPR-specific penalties, e.g. to induce compactness and/or smoothness.

• |feature| Implemented masking of datasets during optimization (#250).

• |feature| Added a verbose option to display progress of the fit routines (#250).

• |feature| Added support for analyzing and fitting complex-valued models and data (#127, #218).

• |feature| Orientation selection in dipolar signals can now be simulated for arbitrary orientation weights distributions via the orisel keyword argument in the new dipolarmodel or the dipolarkernel functions (#183, #203).

• |feature| Re-purposed the ex_ models. Each of these function represents a specific dipolar EPR experiment. These now take the experimental time delays as input, and return a new ExperimentInfo object. This can be passed to dipolarmodel via the optional keyword argument experiment to refine the boundaries and start values of the dipolar pathway refocusing times and amplitudes based on the experimental setup (#225).

• |feature| Implemented masking of datasets during optimization (#250).

• |enhancement| Overhaul of the DeerLab documentation website (#235).

  • Full HTML/CSS overhaul. The new web design based on the PyData theme has a clearer design, with more readable pages and code blocks.
  • Deprecates the use of the RTD theme. This removes the hard constraint of using Sphinx 1.8.0. Now the documentation builds with the latest Sphinx release.
  • Add a user-guide for the new modelling and fitting system.
  • Re-organize all of the website content.
  • Improved the dipolar EPR starting guide, and adapted it to the new system.
  • Fixed some minor errors in the examples.
  • Redesigned all examples towards the use of actual experimental data files in BES3T format. Examples can now be taken and easily adapted by users to their experiment data files (#304).

• |enhancement| Added the functionality to print a FitResult object to display a summary of the fit goodness-of-fit statistics and a table of all estimated parameters and their uncertainties (#234).

• |enhancement| Added a new keyword argument regparamrange to snlls and fit to specify the search range of the optimal regularization parameter (#225).

• |enhancement| Noise levels of the datasets can be optionally specified in all functions taking datasets (#213).

• |enhancement| Added the option to include or exclude the edges of vector in regoperator via a new keyword argument includeedges (#204). The default for all functions in DeerLab has been set to includeedges=False (#205, #230).

• |enhancement| Generalized the regularization operator. Related functions no longer take regorder (regularization operator order) as an argument. Instead they now take regop (the full regularization operator) as an argument (#216).

• |enhancement| Generalized the regularized linear least-squares functionality. Now it can handle arbitrary bounds on linear parameters and adapts the linear LSQ solver back end accordingly (#216).

• |efficiency| Improved performance of post-optimization model evaluation/propagation for large datasets (#200, #238).

• |efficiency| Implemented (adaptable) memory limits for potentially memory-intense functions (#201, #239).

• |api| The function correctscale has been deprecated (#293). Its limited functionality is included in the now broader functionality provided by the new modelling and fitting system.

• |api| The functions fitregmodel and fitparamodel have been deprecated and their core functionality merged into snlls. The snlls function now handles any kind of least-squares problem and automatically employs optimal combinations of solvers to find the solution to the problems (#218).

• |api| Renamed the function bootan to bootstrap_analysis (#227).

• |api| Deprecated TV and Huber regularization. Accordingly the keyword arguments regtype, huberparameter have been removed throughout (#216).

• |api| Deprecated the nnlsbpp NNLS solver (#231).

• |api| Deprecated the regparamrange function (#232). It depended on home-written code for the GSVD, which (as shown in previous issues) was prone to LAPACK backend problems and other bugs. This function was still a derelict from DeerAnalysis methodology.

• |api| The function time2dist has been renamed to distancerange (#273, #274).- |api| The function time2dist has been renamed to distancerange (#273, #274).

• |api| The convergence control arguments of the fit functions have now been renamed for consistency with the least_squares function of the SciPy package (#296).

• |api| Changed the name of the parameter width to std in the dd_gauss, dd_gauss2, dd_gauss3, and dd_skewgaussmodels (#278, #280).

• |fix| When using the multistart keyword argument, no longer includes the parameter boundaries in the set of multiple start values (#218).

• |fix| Fixed error (manifesting as nan values in the confidence intervals) caused by a division-by-zero in the covariance matrix estimation (#218).

• |fix| Fix encoding error during installation (#252). This error could disrupt the installation in OS with default encoding different from CP-1252.

• |fix| Implement a new function to ensure that estimated Jacobians are positive semi-definite matrices. This fixes the appearance of warnings and bugs when calculating confidence intervals (#216).

• |fix| Corrected the scale invariance behavior of the covariance-based uncertainty estimates (#218).

• |fix| Fixed multiple numpy.VisibleDeprecationWarning and RunTime warnings (#207, #212).

• |fix| Corrected the math in the documentation of some distance distribution models (#215).

• |fix| Corrected the behavior of dataset weights. These are no longer normalized at runtime and kept as specified by the users (#248, #250).

• |fix| While testing, now skips a unit test if an error with the Tk backend of Matplotlib occurs (#211).

• |fix| Fix multiple bugs and errors related to the new modelling and fitting system (#226, #233, #235, #241, #242, #244, #245, #246, #249).

• |fix| Correct behavior of multistart optimization for one-sided parameter boundaries (#252).

• |fix| Fix bug when globally fitting multiple datasets. The global weights were not being manipulated correctly in the estimation of the linear parameters leading to incorrect results (#302)

bootstrap_analysis

• |efficiency| Added a new keyword argument memorylimit to specify the maximal memory used by the bootstrap analysis (by default 8GB). If the total analysis is expected to exceed the memory limit, the function will abort the execution (#200, #238).

dipolarkernel

• |feature| Added a new option complex to request the complex-valued dipolar kernel to simulate the out-of-phase contributions to the dipolar signals (#258).
• |efficiency| Added a new keyword argument memorylimit to specify the maximal memory used by the dipolar kernel (by default 8GB). If the dipolar kernel is expected to exceed the memory limit, the function will abort the execution (#200, #238).
• |fix| Prompts error if wrong method is selected when specifying a limited excitation bandwidth (#181, #183).

bg_models

• |feature| Implemented the time-dependent phase shifts for all the built-in physical background models, namely bg_hon3d_phase, bg_hom3dex_phase, and bg_homfractal_phase (#258).
• |enhancement| Changed the implementation of bg_hom3dex (#258). This avoids the use of tabulated pre-calculated values. Accordingly the utility functions calculate_exvolume_redfactor and load_exvolume_redfactor have been removed.
• |fix| Improved the implementation and behavior of the bg_homfractal model (#258).

diststats

• |fix| Fixed the behavior when dealing with distributions with arbitrary integral values

selregparam

• |enhancement| Implemented a general LSQ solver as backend to adapt to different regularized optimization problem structures.
• |enhancement| Generalized the linear least-squares solver. (#216).
• |enhancement| In the brent mode, the search range is no longer selected from the min/max of regparamrange output, but from a new keyword argument searchrange set by default to [1e-8,1e2]. The default values were chosen as the statistical means of Monte-Carlo simulations of the min/max values of regparamrange's output for typical 4-pulse DEER kernels (#232).
• |enhancement| In the grid mode, the grid-values are passed by the pre-existing keyword argument candidates. By default, if not specified, a grid will be generated from the searchrange argument (#232).

UQResult

• |fix| Ensures non-negativity of estimated parameter uncertainty probability density functions.
• |enhancement| Improve the behavior of UQresult.propagate() for bootstrapped uncertainty results. Now, instead of propagating bootstrapped uncertainty via the estimated covariance matrix, the uncertainty is propagated by bootstrapping from the bootstrapped uncertainty distributions (#218).
• |fix| Fix behavior of the bootstrap median (#254).
• |fix| Suppress multiple DeprecationWarning warnings during uncertainty calculations (#255).
• |fix| Fix error prompt when requesting private methods such as __deepcopy__ (#301, #303).

correctphase

• |fix| Implement a fully vectorized analytical solution, resulting in a 30-150x speedup (#256, #279).
• |api| Eliminate the phase='posrealint' and phase='negrealint' options (#279).

deerload

• |fix| Raise warning instead of exception when parsing lines without key-value pairs (#256). This avoid errors when trying to load BES3T files with PulseSPEL scripts edited in different OS systems.

whitegaussnoise

• |api| Renamed the argument level to std for clarity (#276).
• |api| Make the argument std a required positional argument and no longer provide a default value (#276).