Release patch v0.14.2 (#347)

* bumb version to v0.15.0-dev (#307) * examples: fix identifiablity analysis grid values (#314) * remove line that should be substituted by the subsequent GHA (#313) * Use Scipy's `eigh` instead of Numpy's to avoid convergence error (#311) * Use Scipy's eigh instead of Numpy's to avoid convergence error * fix error * Enable the pickling of DeerLab's objects (#312) * fix pickling of fit result objects * add save and load utility functions for pickling, add test * improve test * fix pickling of model objects * fix pickling of merged and linearly combined models * fix pickling of model objects with linked parameters * fix some tests * fix pickling of model objects with functionalized parameters - introduces the dill package as a dependency * fix error * improve docstrings and add to API reference * rename save to store_pickle and load to load_pickle add safety warning about reading pickles * rename imports * remove unused import * fix another import error * `docs`: add important note to user guide on experimental deadtime correction (#320) * Fix dependency of DeerLab's documentation logo on system fonts (#328) * add specific web font of DeerLab's title font * docs: make web logo independent of system fonts * docs: minor CSS fixes * fix minor bug when printing fit results with most parameters frozen (#329) * Fix bug when propagating bootstrapped uncertainty in presence of round-off errors (#325) * UQresult: expand join() method to work with bootstrapped results * fix bug when propagating bootstrapped results with round-off errors * docs: add docstring for evaluate and propagate methods (#321) * docs: minor edits, CSS fixes, and one 404 fix (#331) * docs: minor edits, CSS fixes, and one 404 fix * add missing functions to reference * minor edit and fix * Remove hard dependency of the DeerLab's installation routine and scripts on MKL (#330) * installation: remove hard dependency of installer on MKL - simplify setup.py - update workflows - add new script for linking against MKL in Windows * fixr random numerical error and warning during tests * check failing tests * CI workflow, do not use caching * change Ci job names * Avoid use of pre-release Numpy versions during installation (#336) * setup.py: avoid installation of numpy pre-releases * Fix randomly failing GHA WinOS workflow (#337) * refactor CI workflow * reintroduce MKL for GHA WinOS runners * fix MKL upgrade script * answer yes to uninstall * Prevent potential future bugs when normalizing distributions throughout (#335) * `bootstrap_analysis`: better noise estimation and add option to specify noise levels (#334) * bootstrap_analysis: allow specifying noise levels - otherwise estimate noise level from experimental dataset not from the fit residuals * refactor bootstrap_analysis for more general and faster tests * Fix automated upload to PyPI via token (#338) * Correct behavior of L-curve selection methods (#340) * selregparam: fix behavior of L-curve (LC method) selection criterion * selregparam: prevent `np.log(0)=np.inf ` cases in L-curve methods * selregparam: remove useless line from a debugging stage * Remove unused lines in `dipolarmodel` (#341) * Fix apparent typo * Remove unused lines Co-authored-by: Luis Fabregas <luis.fabregas@phys.chem.ethz.ch> * `dipolarmodel`: Fix error when specifying a limited excitation bandwidth (#342) * dipolarmodel: fix bug when specifying the excitation bandwidth * small fix * fix logic error * fit: properly pass the user-supplied noise level to the bootstrapping (#343) * remove unused lines in example (#345) * Fix documentation navigation menu on mobile phones and collapsed windows and other edits (#346) * fix navigation menu on mobile phone and collapsed windows * remove unnecessary navbar items * remove non-existent references * minor CSS fix * bump version to v0.14.2 * update changelog to v0.14.2 * setup.py: fix merge error Co-authored-by: edmundxcvi <59712516+edmundxcvi@users.noreply.github.com>
JeschkeLab · Jun 29, 2022 · 384deb6 · 384deb6
1 parent 2767550
commit 384deb6
Show file tree

Hide file tree

Showing 22 changed files with 216 additions and 290 deletions.
diff --git a/.github/workflows/ci_PR.yml b/.github/workflows/ci_PR.yml
@@ -5,55 +5,32 @@ on:
     - "**"
 
 jobs:
-  build:
+  tests:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: [3.8]
-    steps:
-      - uses: actions/checkout@v2
+        python-version: [3.9]
 
+    steps:
+      - uses: actions/checkout@v3
+
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v3
         with:
           python-version: ${{ matrix.python-version }}
 
-      - uses: actions/cache@v2
-        if: startsWith(runner.os, 'Windows')
-        with:
-          path: |
-            ~\pipwin
-            ~\AppData\Local\pip\Cache
-          key: ${{ runner.os }}-${{ hashFiles('**/setup.py') }}
-          restore-keys: |
-            {{ runner.os }}-pip-
-
-      - uses: actions/cache@v2
-        if: startsWith(runner.os, 'macOS')
-        with:
-          path: |
-            ~/Library/Caches/pip
-          key: ${{ runner.os }}-${{ hashFiles('**/setup.py') }}
-          restore-keys: |
-            {{ runner.os }}-pip-
-
-      - uses: actions/cache@v2
-        if: startsWith(runner.os, 'Linux')
-        with:
-          path: |
-            ~/.cache/pip
-          key: ${{ runner.os }}-${{ hashFiles('**/setup.py') }}
-          restore-keys: |
-            {{ runner.os }}-pip-
-
       - name: Install dependencies
-        if: steps.cache.outputs.cache-hit != 'false'
         run: |
           python -m pip install --upgrade pip
-          python setup.py install
-          python -m pip install pytest
-  
+          pip install .
+      
+      - name: Windows MKL linking 
+        if: matrix.os == 'windows-latest'
+        run: |
+          python upgrade_mkl.py
+
       - name: Test with pytest
-        run: pytest
+        run: |
+          pytest
diff --git a/.github/workflows/ci_scheduled.yml b/.github/workflows/ci_scheduled.yml
@@ -6,7 +6,7 @@ on:
     - cron:  '30 6 * * 1'
 
 jobs:
-  build:
+  matrix_test:
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
@@ -25,7 +25,6 @@ jobs:
         if: startsWith(runner.os, 'Windows')
         with:
           path: |
-            ~\pipwin
             ~\AppData\Local\pip\Cache
           key: ${{ runner.os }}-${{ hashFiles('**/setup.py') }}
           restore-keys: |
@@ -54,7 +53,6 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           python setup.py install
-          python -m pip install pytest
   
       - name: Test with pytest
         run: pytest
diff --git a/.github/workflows/package_upload.yml b/.github/workflows/package_upload.yml
@@ -23,10 +23,12 @@ jobs:
     - name: Build distribution
       run: |
           python setup.py sdist
+          ./setup.py bdist_wheel
     - name: Publish distribution to PyPI
-      uses: pypa/gh-action-pypi-publish@master
+      uses: pypa/gh-action-pypi-publish@release/v1
       with:
-        password: ${{ secrets.PYPI_API_TOKEN  }}
+        user: __token__
+        password: ${{ secrets.PYPI_API_TOKEN }}
 
   conda-build-n-publish:
     runs-on: ubuntu-latest

diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-v0.14.1
+v0.14.2
diff --git a/conda.recipe/meta.yaml b/conda.recipe/meta.yaml
@@ -20,17 +20,17 @@ requirements:
     - numpy
     - scipy
     - cvxopt
+    - dill
+    - pytest
 
 about:
   home: https://jeschkelab.github.io/DeerLab/index.html
   license: MIT
   summary: 'Comprehensive package for data analysis of dipolar EPR spectroscopy'
   description: |
-    DeerLab is a Python package for the analysis of dipolar EPR (electron paramagnetic resonance)
-    spectroscopy data. Dipolar EPR spectroscopy techniques include DEER (double electron-electron resonance),
-    RIDME (relaxation-induced dipolar modulation enhancement), and others.
-    DeerLab consists of a collection of functions for modelling, data processing, and least-squares fitting.
-    They can be combined in scripts to build custom data analysis workflows. DeerLab supports both classes of 
-    distance distribution models: non-parametric (Tikhonov regularization and related) and parametric 
-    (multi-Gaussians etc). It also provides a selection of background and experiment models.
+    DeerLab is a comprehensive free scientific software package for Python focused on modelling,
+    penalized least-squares regression, and uncertainty quantification. It provides highly 
+    specialized on the analysis of dipolar EPR (electron paramagnetic resonance) spectroscopy data. 
+    Dipolar EPR spectroscopy techniques include DEER (double electron-electron resonance), 
+    RIDME (relaxation-induced dipolar modulation enhancement), and others. 
   dev_url: https://jeschkelab.github.io/DeerLab
diff --git a/deerlab/bootstrap_analysis.py b/deerlab/bootstrap_analysis.py
@@ -9,8 +9,9 @@
 from joblib import Parallel, delayed
 from deerlab.classes import UQResult
 from deerlab.utils import isnumeric
+from deerlab.noiselevel import noiselevel
 
-def bootstrap_analysis(fcn,Vexp,Vfit, samples=1000, resampling='gaussian', verbose = False, cores=1, memorylimit=8):
+def bootstrap_analysis(fcn,Vexp,Vfit, samples=1000, noiselvl=None, resampling='gaussian', verbose = False, cores=1, memorylimit=8):
     r""" 
     Bootstrap analysis for uncertainty quantification
 
@@ -33,6 +34,11 @@ def bootstrap_analysis(fcn,Vexp,Vfit, samples=1000, resampling='gaussian', verbo
         results improve with the number of boostrap samples evaluated, the 
         default is 1000.
 
+    noiselvl : scalar or list thereof
+        Noise level of the input dataset(s), specified as standard deviations.
+        If not specified, these are automatically estimated from the experimental
+        dataset(s). 
+
     resampling : string, optional
         Specifies the method employed for re-sampling new bootstrap samples.
 
@@ -94,10 +100,11 @@ def myfcn(V):
         raise KeyError('The 1st argument must be a callable function accepting dataset(s) as input.')
 
     # Get residuals and estimate standard deviation
-    residuals,sigma = ([],[])
     for i in range(nSignals):
-        residuals.append(Vfit[i] - Vexp[i])
-        sigma.append(np.std(residuals[i]))
+        residuals = [Vfit[i] - Vexp[i] for i  in range(nSignals)]
+    if noiselvl is None:
+        noiselvl = [noiselevel(Vexp[i]) for i in range(nSignals)]
+    noiselvl = np.atleast_1d(noiselvl)
 
     # Prepare bootstrap sampler (reduced I/O-stream when parallelized) 
     def sample():
@@ -106,7 +113,7 @@ def sample():
             #Determine requested re-sampling method
             if resampling == 'gaussian':
                 # Resample from a Gaussian distribution with variance estimated from the residuals
-                Vsample[i] = Vfit[i] + np.random.normal(0, sigma[i], len(Vfit[i]))
+                Vsample[i] = Vfit[i] + np.random.normal(0, noiselvl[i], len(Vfit[i]))
             elif resampling == 'residual':
                 # Resample from the residual directly
                 Vsample[i] =  Vfit[i] + residuals[i][np.random.permutation(len(Vfit[i]))]

diff --git a/deerlab/dd_models.py b/deerlab/dd_models.py
@@ -140,7 +140,7 @@ def _nonparametric():
     dd_nonparametric = Model(_nonparametric,constants='r')
     dd_nonparametric.description = 'Non-parametric distribution model'
     # Parameters
-    dd_nonparametric.addlinear('P',vec=len(r),lb=0,par0=0,description='Non-parametric distance distribution',unit="nm⁻¹", normalization=lambda P: P/np.trapz(P,r))
+    dd_nonparametric.addlinear('P',vec=len(r),lb=0,par0=0,description='Non-parametric distance distribution',unit="nm⁻¹", normalization=lambda P: _normalize(r,P))
     return dd_nonparametric
 
 

diff --git a/deerlab/dipolarmodel.py b/deerlab/dipolarmodel.py
@@ -178,7 +178,7 @@ def dipolarpathways(*param):
                 if param in model._parameter_list(order='vector'):
                     subset[getattr(model,param).idx] = idx
 
-    kernelmethod = 'fresnel' if orisel is None else 'grid' 
+    kernelmethod = 'fresnel' if orisel is None and np.isinf(excbandwidth) else 'grid' 
 
     #------------------------------------------------------------------------
     def Vnonlinear_fcn(*nonlin):
@@ -243,9 +243,6 @@ def Vnonlinear_fcn(*nonlin):
 
     # Set other dipolar model specific attributes
     DipolarSignal.description = 'Dipolar signal model'
-    DipolarSignal.Pmodel = Pmodel
-    DipolarSignal.Bmodel = Pmodel
-    DipolarSignal.Npathways = npathways
 
     return DipolarSignal
 #===============================================================================
@@ -293,7 +290,8 @@ def dipolarpenalty(Pmodel, r, type, selection=None):
         # Define the compactness penalty function
         def compactness_penalty(*args): 
             P = Pmodel(*[r]*Nconstants,*args)
-            P = P/np.trapz(P,r)
+            if not np.all(P==0):
+                P = P/np.trapz(P,r)
             return np.sqrt(P*(r - np.trapz(P*r,r))**2*np.mean(np.diff(r)))
         # Add the penalty to the Pmodel
         penalty = Penalty(compactness_penalty,selection,
@@ -690,4 +688,4 @@ def ex_ridme(tau1, tau2, pathways=None):
         harmonics = [harmonics[pathway-1] for pathway in pathways] 
 
     return ExperimentInfo('RIDME',reftimes,harmonics)
-#===============================================================================
+#===============================================================================
diff --git a/deerlab/diststats.py b/deerlab/diststats.py
@@ -95,7 +95,7 @@ def analyze_rmode(V):
 
     # Auxiliary functions
     # -------------------
-    int = np.trapz(P,r)
+    int = np.trapz(P,r) if not np.all(P==0) else 1
     def normalize(P): 
         return P/int
     # Percentile function

diff --git a/deerlab/model.py b/deerlab/model.py
@@ -1135,7 +1135,7 @@ def bootstrap_fcn(ysim):
             if not isinstance(fit.model,list): fit.model = [fit.model]
             return (fit.param,*fit.model)
         # Bootstrapped uncertainty quantification
-        param_uq = bootstrap_analysis(bootstrap_fcn,ysplit,fitresults.model,samples=bootstrap)
+        param_uq = bootstrap_analysis(bootstrap_fcn,ysplit,fitresults.model,samples=bootstrap,noiselvl=noiselvl)
         # Include information on the boundaries for better uncertainty estimates
         paramlb = model._vecsort(model._getvector('lb'))[np.concatenate(param_idx)] 
         paramub = model._vecsort(model._getvector('ub'))[np.concatenate(param_idx)] 

diff --git a/deerlab/selregparam.py b/deerlab/selregparam.py
@@ -151,22 +151,22 @@ def register_ouputs(optout):
 
         # L-curve minimum-radius method (LR)
         if method == 'lr':
-            Eta = np.log(penalties)
-            Rho = np.log(residuals)
+            Eta = np.log(np.asarray(penalties)+1e-20)
+            Rho = np.log(np.asarray(residuals)+1e-20)
             dd = lambda x: (x-np.min(x))/(np.max(x)-np.min(x))
             functional = dd(Rho)**2 + dd(Eta)**2         
-
+            
         # L-curve maximum-curvature method (LC)
         elif method == 'lc': 
-            d1Residual = np.gradient(np.log(residuals))
+            d1Residual = np.gradient(np.log(np.asarray(residuals)+1e-20))
             d2Residual = np.gradient(d1Residual)
-            d1Penalty = np.gradient(np.log(penalties))
+            d1Penalty = np.gradient(np.log(np.asarray(penalties)+1e-20))
             d2Penalty = np.gradient(d1Penalty)
             functional = (d1Residual*d2Penalty - d2Residual*d1Penalty)/(d1Residual**2 + d1Penalty**2)**(3/2)
+            functional = -functional # Maximize instead of minimize 
 
         # Find minimum of the selection functional              
         alphaOpt = alphaCandidates[np.argmin(functional)]
-
     else: 
         raise KeyError("Search method not found. Must be either 'brent' or 'grid'.")
 

diff --git a/docsrc/source/_templates/nav.html b/docsrc/source/_templates/nav.html
@@ -1,11 +1,11 @@
+
 {%- set userguide_dropdown_navigation = [
     ('Basics', pathto('basics')),
     ('Getting Started', pathto('getting_started')),
     ('Modelling', pathto('dipolar_guide_modelling')),
     ('Fitting', pathto('dipolar_guide_fitting')),
     ('Theory', pathto('theory')),]
   -%}
-
 
 {%- set advancedguide_dropdown_navigation = [
 ('Modelling Guide', pathto('modelling_guide')),
@@ -23,18 +23,18 @@
 -%}
 
   <div class="container-fluid {{ top_container_cls }} px-0">
-
-    <div class="sk-navbar-collapse collapse navbar-collapse" id="navbarSupportedContent">
+  
+    <div class="navbar-collapse collapse navbar-collapse" id="navbar-collapsible">
       <ul class="navbar-nav mr-auto">
         <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('installation') }}">Installation</a>
+          <a class="nav-link nav-link" href="{{ pathto('installation') }}">Installation</a>
         </li>
 
         <li class="dropdown">
           <a class="dropbtn" href="{{ pathto('user_guide') }}" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">User Guide</a>
           <div class="dropdown-content" aria-labelledby="navbarDropdown">
             {%- for title, link in userguide_dropdown_navigation %}
-              <a class="sk-nav-dropdown-item dropdown-item" href="{{ link }}">{{ title}}</a>
+              <a class="nav-dropdown-item dropdown-item" href="{{ link }}">{{ title}}</a>
             {%- endfor %}
           </div>
         </li>
@@ -43,36 +43,36 @@
             <a class="dropbtn" href="{{ pathto('advanced_guide') }}" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Advanced Guides</a>
             <div class="dropdown-content" aria-labelledby="navbarDropdown">
               {%- for title, link in advancedguide_dropdown_navigation %}
-                <a class="sk-nav-dropdown-item dropdown-item" href="{{ link }}">{{ title}}</a>
+                <a class="nav-dropdown-item dropdown-item" href="{{ link }}">{{ title}}</a>
               {%- endfor %}
             </div>
           </li>
 
         <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('examples') }}">Examples</a>
+          <a class="nav-link nav-link" href="{{ pathto('examples') }}">Examples</a>
         </li>
 
         <li class="nav-item">
-          <a class="sk-nav-link nav-link" href="{{ pathto('modelsref') }}">Models</a>
+          <a class="nav-link nav-link" href="{{ pathto('modelsref') }}">Models</a>
         </li>
 
 
         <li class="nav-item">
-            <a class="sk-nav-link nav-link" href="{{ pathto('reference') }}">Reference</a>
+            <a class="nav-link nav-link" href="{{ pathto('reference') }}">Reference</a>
           </li>
 
         {%- for title, link in drop_down_navigation %}
         <li class="nav-item">
-          <a class="sk-nav-link nav-link nav-more-item-mobile-items" href="{{ link }}">{{ title }}</a>
+          <a class="nav-link nav-link nav-more-item-mobile-items" href="{{ link }}">{{ title }}</a>
         </li>
 
         {%- endfor %}
 
         <li class="dropdown">
-          <a class="dropbtn" href="{{ pathto('more') }}" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">More</a>
+          <a class="dropbtn nav-link dropdown-toggle" href="{{ pathto('more') }}" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">More</a>
           <div class="dropdown-content" aria-labelledby="navbarDropdown">
             {%- for title, link in more_dropdown_navigation %}
-              <a class="sk-nav-dropdown-item dropdown-item" href="{{ link }}">{{ title}}</a>
+              <a class="nav-dropdown-item dropdown-item" href="{{ link }}">{{ title}}</a>
             {%- endfor %}
           </div>
         </li>

diff --git a/docsrc/source/changelog.rst b/docsrc/source/changelog.rst
@@ -14,16 +14,31 @@ Release Notes
 - |fix| : Something which was not working as expected or leading to errors has been fixed.
 - |api| : This will require changes in your scripts or code.
 
-Release v0.14.1 - June 2022
+Release v0.14.2 - June 2022
 ---------------------------------
 
-.. rubric:: Overall changes
+- |feature| |efficiency| (Windows-systems only) Removed the unorthodox default installation procedure of DeerLab based on the installation of Numpy and related packages linked against MKL via the Gohlke repository (:issue:`322`, :pr:`330`).
+
+  * As a result the default performance of DeerLab can be affected in some Windows systems. To link the Numpy and related packages against MKL as in previous versions, an automated script ``upgrade_mkl.py`` is provided with the package.
+  * Fixes the error appearing during installation if the ``git`` command was not installed or available in the system (:issue:`326`). 
+  * Allows the distribution of DeerLab as wheels. 
+
+- |feature| Implemented better options for automated and user-supplied noise estimates to improve bootstrapping approaches (:pr:`334`, :pr:`343`).
+- |fix| Avoid the installation of (potentially unstable) pre-release versions of Numpy in systems with fresh Python installations (:pr:`336`).
+- |fix| Improved the robustness of several function against non-numerical values due to division-by-zero errors (:pr:`335`).
+- |fix| Corrected the behavior of regularization parameter selection with L-curve methods (:pr:`340`). Fixes the ``lc`` method in ``selgregparam`` which was seeking the optimal regularization parameter by minimizing curvature rather than by maximizing it. Prevents failure of the L-curve methods due to the appearance of non-numeric values when evaluating too large regularization parameter values.
+- |fix| Fixes the error when specifying a limited excitation bandwidth in ``dipolarmodel`` via the ``excbandwidth`` argument (:pr:`342`). 
+- |fix| Fixes the navigation menu on the documentation that appeared empty on mobile phones or for partially minimized windows on computers, impeding navigation through the documentation (:pr:`346`).
+- |fix| Minor corrections to the documentation and examples.    
+
+Release v0.14.1 - June 2022
+---------------------------------
 
-- |fix| Use Scipy's `eigh` instead of Numpy's to avoid convergence error ``numpy.linalg.LinAlgError: Eigenvalues did not converge`` during model uncertainty propagation (:issue:`310`, :pr:`311`).
+- |fix| Use Scipy's ``eigh`` instead of Numpy's to avoid convergence error ``numpy.linalg.LinAlgError: Eigenvalues did not converge`` during model uncertainty propagation (:issue:`310`, :pr:`311`).
 - |fix| Refactored the code to avoid the use of ``lambda`` and nested functions. This enables pickling DeerLab's objects with Python's ``pickle`` module without errors (:pr:`312`).
 - |feature| Added two new utility functions ``store_pickle`` and ``read_pickle`` that implement pickling with the ``dill`` package to be more robust against potential ``lambda`` functions defined by the users in scripts (:pr:`312`).
 - |fix| Fixed minor bug when printing fit results with many model parameters being frozen. The print command would return an error message (:pr:`329`).
-- |fix| Fixed bug when propagating bootstrapped uncertainty in presence of round-off errors (:pr:`325`).
+- |fix| Fixed bug when propagating bootstrapped uncertainty in presence of round-off errors (:pr:`325`). 
 - |fix| |enhancement| Multiple minor improvements and corrections in the documentation.
 
 Release v0.14.0 - April 2022