V2.0.4 (#258)

* added a new method to blobcorrector.correctorclass to do distorsion correction using a pixel LUT

* Attempt to manage openmp with multiprocessing

* Attempt to manage openmp with multiprocessing

* Attempt to manage openmp with multiprocessing

* drop 3.7 for 3.8 (but keep 2.7 for now)

* Update blobcorrector.py

Added new property self.dim with detector dimension to blobcorrector.correctorclass, inferred from the splinefile (xmin, xmax, ymin, ymax)

* Update blobcorrector.py

* TODO: fix fscan2d to be easier to use

* Thanks to Andrew Reader's youtube video

* added 3.11 (3.x is busted due to numba)

* auto reshape for fscan2d

* setting threads only for multiprocessing

* fixup fscan2d

* fixup fscan2d

* Update build_flake_pytest_ubuntu2004.yml

* Update build_flake_pytest_ubuntu2004.yml

* Update setup.py

* Update sigma_3_matrices.py (py2, print)

* Update properties.py (py2: print)

* Update test_datasets.py (py2: fstring/print)

* Update cImageD11.py : use OMP_NUM_THREADS and py27

* Enable manually specifying detector and motor names

* matrix vs dcm with old scipy

* fix py27 windows

* binary wheels for win32/py27

* 'fixup'

* 'fixup'

* 'fixup'

* Update appveyor.yml

* Update appveyor.yml

* Update appveyor.yml

* Update appveyor.yml

* Update appveyor.yml

* Update appveyor.yml

* Update appveyor.yml

* Add build wheels and tarball

* Update README.md

* Update __init__.py

* Create eps_sig_solver_NEW.py

Create a new solver to compute strain and stress from ImageD11 ubis

* Bugfix assemble_label.py

set rather than list randomly permuted the order of scans. Oups.

* compatible for python2.7

* fix file path

* corrected mistakes in eps_sig_solve_NEW.py + updated invariant properties function

- corrected a few mistakes and typos in function description
- corrected a mistake in convert_tensors_to_vect
- changed invariant_prop function to return von Mises stress / strain instead of octahedral shear stress / strain

* Update eps_sig_solver_NEW.py

* corrected typo in form_stiffness_tensor

* updated Cij_symmetry

* Update eps_sig_solver_NEW.py

* Rename eps_sig_solver_NEW.py to stress.py

* Rename ImageD11/eps_sig_solver.py to ImageD11/depreciated/eps_sig_solver.py

* Update stress.py

* Update stress.py

* Update stress.py

* Update stress.py

* Update stress.py

* Add scanning 3DXRD example notebooks

* Remove f-string for 2.7 compat.

* Exclude Python 3.12 in setup.py

Until Numba updates to Python 3.12
This should fix the CI for Python 3.x

* Exclude 3.12 from GitHub Actions CI

* Fix GitHub actions yml typo

* Third time lucky

* Use GitHub Actions badges in README.md

* Correct typo

* Update README.md

* fix the log strain - needs testing better

* helper function from several notebooks

* mlem and centers code

* function from notebooks

* function from notebooks

* Change int32 and float32 to 64

* remove broken test

* put stress on hold while updating it

* Update __init__.py v2.0.2

* Update README.md v2.0.2

* Update release.yml : no 3.12 version

* Allow a fuller mask, to mask powder rings for example

* fix mask for powder removal and reformat (black)

* Resurrect tests for columnfile.py, correct np.int

In the Github Actions CI, the command "python -m pytest" is run to test ImageD11.
By default, this only runs test files with the naming convention `test_*.py`
Renamed `testcolumnfile.py` to `test_columnfile.py` to fix this.
Also, replaced `np.int` references `int` to fix numpy depreciation errors.

* Fix np.int64

* Add colfile tests to run_tests.py

* Significant notebook layout changes

* Parallelised iradon

* Ensure 0 fill value for np.interp

* Add mask to MLEM

* First attempt at enforcing file paths

* A few bugfixes

* update cibuildwheel and upload|download-artifacts

* eps_sig_solver was deprecated

* eps_sig_solver was deprecated + black formatted

* Python 3.12 support

* Add col3d file to dataset paths

* Try to get rid of the fork problem

* point by point - needs some more docs to expain pars

* fixup 2.7

* Add outsize support to MLEM

* Clean up comment

* convert nnz to ip function

* error on typo in group names

* fix forking problem/error

* get the warnings to print, but only when needed

* for CI only having 1 thread?

* CI fails. Not sure why.

* fixup

* set forkserver here, todo: windows?

* and for windows that does not fork anyway

* and for windows that does not fork anyway

* and for windows that does not fork anyway

* percent, not fraction

* tested on SS316

* was not tested yet. pythonpath

* tested once: should not be forking

* seen to run once

* with a threshold image option

* Notebook for par file building from poni

* A few small sinogram bugfixes

* self.shape is undefined

* Significant scanning notebook changes

General changes:
- Moved quite a few common functions into utils.py. These need properly incorporating into ImageD11 in the future
- All notebooks updated to obey DataSet filestructures for peaks and grains files

0_S3DXRD_segment:
- Added widget written by Pierre-Olivier for adjusting segmenter parameters
- Added slurm bash "listener" that waits for slurm job to complete before next cell completes
- Moves individual sparse files to their own subfolder after assembling

1_S3DXRD_index:
- More sensible indexer that prioritises low-multiplicity rings
- Grains now saved in HDF5 format
- Added another example notebook for indexing a minor phase

2_S3DXRD_sinograms:
- Added whole-sample sinogram reconstruction to use as a mask for individual grain reconstructions.
- Added MLEM refinement stage (uses SLURM) to grain reconstructions using run_mlem_recon.py
- Added another example notebook for mapping a minor phase (uses grain recons for centre-of-mass)

3_S3DXRD_strain_maps_pbp:
- Performs a point-by-point refinement of the grain-averaged UBI using 2D peak selection
- Uses grain labels array from sinograms for grain shapes
- Should give "relative" Type-III strains

4_S3DXRD_plot_both_phases:
- Small example notebook for plotting both phases in the same map
- Some initial work on orientation relationship identification using Orix (not finished yet)

* Add box-beam example notebooks

- Similar to scanning notebooks but for box-beam 3DXRD
- Currently only working for Frelon data
- Uses Frelon segmenter from ma4750 (in sandbox) for peaksearching
- Global parameter refinement notebook
- Uses makemap.py to refine grain positions
- Some extra notebooks for importing multiple slices in a z series and looking at peak shapes (still under development)

* Build parameter file from CIF and pyFAI PONI

* Clear notebook outputs

* Correct pyplot typo

* Allow flake 2.7 linting to pass

* Python 2 compat

* Remove duplicate segmentation notebook

* Common utils between scanning and box-beam notebooks

- Moved a few more functions out of notebooks into utils
- This significantly simplifies the notebooks
- MLEM recon: y0 now gets written to/read from the grain HDF5 file rather than command-line option

* Replace FileNotFoundError with IOError for py2.7

* Minor phase scanning notebooks now use nb_utils, needs testing

* Correct ImageD11_gui bugs - broken log buttons and tth_histogram

* Rename and add batch scripting

* Add batch cells to scanning notebooks

* All notebooks now use nb_utils.py

* Remove WIP notebook from commit

* Add RGB colour output to sinogram notebooks

* Add widget with sliders for Frelon peaksearch data

* Add RGB IPF X,Y,Z outputs to HDF, allows viewing with H5Web

* py2 compat

* Small bugfixes and tidying up

* Small changes and bugfixes

* Clear output

* secret spacegroup number hidden option

* secret spacegroup number hidden option

* secret spacegroup number hidden option

* secret spacegroup number hidden option

* Correct very dodgy default cosine tolerance

* Correct grain translation sign problems

* Correct dodgy do_index scan

* Handle missing samples in skips_dict, dset lists are now sorted too

* More accurately find user home directory path

* Just use default arguments for assemble_label and properties

* Force only 2 threads for indexer (confirmed faster)

* Improve comments about peak filtration

* Correct origin for recon plots

* Much faster spatial correction using lut

* Lima segmenter now used in interactive notebook

* Add howmany to Eiger interactive segmenter

* Segment everything in parallel on SLURM

* Cleaned-up local PBP indexer

* Correct PBP results filename

* Fix dodgy pbp name

* Fix PBP output name for real this time

* fixes #236

* fixes #236

* Correct home directory searching

* added options to score_all_pairs for ring selection

* added options to score_all_pairs for ring selection

* Optionally scale sinogram rows by ring current

* Forgot to also add to batch cells

* Correct dodgy minor phase grain positioning

* nb_utils function for ring current correction per scan

* H5py in nb_utils

* Make ring current correction optional and separate

* More sensible default dstol

* tag maintenance release

---------

Co-authored-by: Jean-Baptiste Jacob <137695414+jbjacob94@users.noreply.github.com>
Co-authored-by: jadball <jadball@gmail.com>
Co-authored-by: Thomas VINCENT <thomas.vincent@esrf.fr>
Co-authored-by: James Ball <42615415+jadball@users.noreply.github.com>
Co-authored-by: Jerome Kieffer <github@terre-adelie.org>
Co-authored-by: Jonathan Wright <jonathan.wright@gmail.com>

.circleci/config.yml

@@ -0,0 +1,56 @@
+# Python CircleCI 2.0 configuration file
+#
+# Check https://circleci.com/docs/2.0/language-python/ for more details
+#
+version: 2
+jobs:
+ build:
+ docker:
+ # specify the version you desire here
+ # use `-browsers` prefix for selenium tests, e.g. `3.6.1-browsers`
+ - image: circleci/python:3.8.0
+
+ # Specify service dependencies here if necessary
+ # CircleCI maintains a library of pre-built images
+ # documented at https://circleci.com/docs/2.0/circleci-images/
+ # - image: circleci/postgres:9.4
+
+ working_directory: ~/repo
+
+ steps:
+ - checkout
+
+ # Download and cache dependencies
+ - restore_cache:
+ keys:
+ # fallback to using the latest cache if no exact match is found
+ - v1-dependencies-
+
+ - run:
+ name: install dependencies
+ command: |
+ python3 -m venv venv
+ . venv/bin/activate
+ python3 -m pip install pip setuptools --upgrade
+ python3 -m pip install pytest numpy
+ python3 -m pip install --editable .
+
+ - save_cache:
+ key: ImageD11-{{arch}}-{{ checksum "setup.py" }}
+ paths:
+ - ./venv
+
+ # run tests
+ # this example uses Django's built-in test-runner
+ # other common Python testing frameworks include pytest and nose
+ # https://pytest.org
+ # https://nose.readthedocs.io
+ - run:
+ name: run tests
+ command: |
+ . venv/bin/activate
+ python3 -m pytest
+
+ - store_artifacts:
+ path: test-reports
+ destination: test-reports

.gitattributes

@@ -1,4 +1,12 @@
+*.out_merge_t200 linguist-generated
+*.out linguist-generated
 *.flt linguist-generated
+*.gve linguist-generated
+*.ubi linguist-generated
+*.par linguist-generated
+*.prm linguist-generated
+*.pars linguist-generated
 *.html linguist-documentation
 *.pdf linguist-generated
 *.odt linguist-documentation
+*.pks linguist-generated

.github/workflows/build_flake_pytest_ubuntu2004.yml

@@ -0,0 +1,51 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: ImageD11_pytest
+on:
+ push:
+ branches: [ master ]
+ pull_request:
+ branches: [ master ]
+
+jobs:
+ build:
+ # because ESRF forces 20.04 for now.
+ runs-on: ubuntu-20.04
+ strategy:
+ fail-fast: false
+ matrix:
+ python-version: ["2.7", "3.8", "3.9", "3.10", "3.11", "3.12"]
+ steps:
+ - uses: actions/checkout@v3
+ - name: Set up Python ${{ matrix.python-version }}
+ if: matrix.python-version != '2.7'
+ uses: actions/setup-python@v4
+ with:
+ python-version: ${{ matrix.python-version }}
+ - name: Install Ubuntu python 2.7
+ if: matrix.python-version == '2.7'
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y --no-install-recommends python2 python2-dev python3-virtualenv
+ virtualenv -p python2 ${HOME}/cp27
+ source ${HOME}/cp27/bin/activate
+ ${HOME}/cp27/bin/python -m pip install -U pip
+ ${HOME}/cp27/bin/python -m pip install -U setuptools wheel
+ echo "${HOME}/cp27/bin" >> $GITHUB_PATH
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ python -m pip install flake8 pytest
+ if [ -f requirements.txt ]; then python -m pip install -r requirements.txt; fi
+ python -m pip install .
+ - name: Lint with flake8
+ run: |
+ # stop the build if there are Python syntax errors or undefined names
+ python -m flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics --exclude="*depreciated*"
+ # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+ # flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
+ - name: Test with pytest
+ run: |
+ cd test
+ python -m pytest

.github/workflows/release.yml

@@ -0,0 +1,115 @@
+name: Build and deploy
+
+on:
+ workflow_dispatch:
+ release:
+ types:
+ - published
+
+jobs:
+ build_wheels:
+ name: Build wheels on ${{ matrix.os }}-${{ matrix.cibw_archs }}
+ runs-on: ${{ matrix.os }}
+ strategy:
+ # Ensure that a wheel builder finishes even if another fails
+ fail-fast: false
+ matrix:
+ include:
+ - os: ubuntu-20.04
+ cibw_archs: "auto64"
+ #- os: ubuntu-20.04
+ # cibw_archs: "auto32"
+ - os: ubuntu-20.04
+ cibw_archs: "aarch64"
+ - os: ubuntu-20.04
+ cibw_archs: "ppc64le"
+ - os: windows-2019
+ cibw_archs: "auto64"
+ #- os: windows-2019
+ # cibw_archs: "auto32"
+ - os: macos-11
+ cibw_archs: "universal2"
+
+ steps:
+ - uses: actions/checkout@v4
+
+ - name: Set up QEMU
+ if: runner.os == 'Linux'
+ uses: docker/setup-qemu-action@v3
+ with:
+ platforms: all
+
+ - name: Build wheels
+ uses: pypa/cibuildwheel@v2.16.5
+ env:
+ CIBW_PROJECT_REQUIRES_PYTHON: ">=3.7"
+ CIBW_BUILD: cp37-* cp38-* cp39-* cp310-* cp311-* cp312-*
+ # Do not build for pypy, muslinux and python3.12 on ppc64le
+ CIBW_SKIP: pp* *-musllinux_* cp312-*linux_ppc64le
+ CIBW_ARCHS: ${{ matrix.cibw_archs }}
+
+ # Use silx wheelhouse: needed for ppc64le
+ CIBW_ENVIRONMENT_LINUX: "PIP_FIND_LINKS=https://www.silx.org/pub/wheelhouse/ PIP_TRUSTED_HOST=www.silx.org"
+
+ CIBW_TEST_REQUIRES: pytest
+ CIBW_TEST_COMMAND: pytest {project}/test
+ # Skip tests for 32bits and emulated architectures, arm64 macos and on Windows
+ CIBW_TEST_SKIP: "*-*linux_i686 *-*linux_{aarch64,ppc64le,s390x} *-macosx_arm64 *-macosx_universal2:arm64 *-win32 *-win_amd64"
+
+ - uses: actions/upload-artifact@v4
+ with:
+ name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
+ path: ./wheelhouse/*.whl
+
+ build_sdist:
+ name: Build source distribution
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+
+ - uses: actions/setup-python@v5
+ with:
+ python-version: '3.11'
+ cache: 'pip'
+
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ pip install build twine
+
+ - name: Build sdist
+ run: python -m build --sdist
+
+ - name: Check the package
+ run: |
+ python -m twine check dist/*
+
+ - name: Install and test sdist
+ run: |
+ pip install --pre dist/ImageD11*.tar.gz
+ pytest test
+
+ - uses: actions/upload-artifact@v4
+ with:
+ name: cibw-sdist
+ path: dist/*.tar.gz
+
+ pypi-publish:
+ needs: [build_wheels, build_sdist]
+ name: Upload release to PyPI
+ runs-on: ubuntu-latest
+ environment:
+ name: pypi
+ permissions:
+ id-token: write
+ if: github.event_name == 'release' && github.event.action == 'published'
+ # or, alternatively, upload to PyPI on every tag starting with 'v' (remove on: release above to use this)
+ # if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+ steps:
+ - uses: actions/download-artifact@v4
+ with:
+ pattern: cibw-*
+ path: dist
+ merge-multiple: true
+
+ - uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -18,6 +18,7 @@ __pycache__/
 \#*#
 
 # Distribution / packaging
+.vscode
 .Python
 env/
 build/

.travis.yml

@@ -0,0 +1,18 @@
+language: python # this works for Linux but is an error on macOS or Windows
+jobs:
+ include:
+ - name: "Python 3.8.0 on Xenial Linux"
+ python: 3.8 # this works for Linux but is ignored on macOS or Windows
+ - name: "Python 3.7.4 on macOS"
+ os: osx
+ osx_image: xcode11.2 # Python 3.7.4 running on macOS 10.14.4
+ language: shell # 'language: python' is an error on Travis CI macOS, could be generic
+install: 
+ - python3 -m pip install --upgrade pip setuptools 
+# fabio could not be installed as a dependency on mac automatically
+ - python3 -m pip install cython
+ - python3 -m pip install fabio
+ - python3 -m pip install --no-use-pep517 --editable . 
+# 'python3' is a 'command not found' error on Windows but 'py' works on Windows only
+script: 
+ - OMP_NUM_THREADS=2 python3 -m pytest