Need to update analysis tools to use anndata 0.10

[CameronFRWatson]

Breaking change between anndata 0.10 and prior versions means that some of our tools running anndata 0.7-0.8 are incompatible with tool suites that run anndata 0.10, resulting in the following error when trying to read h5ad files from the newer version:

KeyError: 'dict'
...
anndata._io.utils.AnnDataReadError: Above error raised while reading key 
'/layers' of type <class 'h5py._hl.group.Group'> from /.

This impacts tools in the following suites:

• Scimap (Requires version upgrade to 1.3.3)
• Squidpy (Requires version upgrade to >=1.3)
• Vitessce (May require version upgrade, but maybe be able to just re-pin anndata version in Dockerfile and conda build files)

Short term fix:

• Update packages to necessary versions, and make sure existing tools still function

Longer term:

• Wrap new features incorporated into recent versions of scimap, squidpy, and vitessce (This will become its own issue)

[CameronFRWatson]

Scimap

Working branch

https://github.com/goeckslab/tools-mti/tree/scimap-update-1.3.10

Goals

• Upgrade to scimap 1.3.10 which will bring anndata up to 0.10.5
• Apply param changes to, and check functionality of, existing wrapped tools

Current status

• Rebuilt Docker image (quay.io/goeckslab/scimap:1.3.10), pinned scimap and anndata versions, pushed to quay.io
• Existing tool changes
  • mcmicro_to_anndata.xml
    • Changed test data tutorial_data.h5ad to accommodate file size of h5ad files output by anndata >0.10 (file size differs substantially from previous anndata versions due to different default float type being used and addition of layers attribute)
  • scimap_spatial.xml
    • scimap update got rid of the PARC option for clustering, so removed related params from the Galaxy wrapper xml
  • scimap_phenotyping.xml
    • Dropped rescale_plots from XML to reflect tool change
    • Exposed random_state param to Galaxy
    • Made it optional to log transform the data by adding log boolean param to wrapper
    • Added a note to the help section explaining how scimap handles raw and X representations, as there is currently no controlling this for the user
• All changes pass planemo testing

To-do

• Rebuild conda build (build.sh and meta.yml) to reflect changes in the Docker image
• Test tools with conda build
• Try tools on some real data to replicate past results to check for any missed changes to scimap's internal functions

[CameronFRWatson]

Vitessce

Working branch

No working branch yet

Goals

• Update anndata to at least 0.8.0, maybe need to be higher?
• Change vitessce as minimally as possible for now

Current status

• Testing new Docker images on Galaxydev and getting a Linux-specific error despite --platform linux/amd64 in build command

ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

To-do

• Build a Docker image that works for vitessce at its current version, and anndata upgraded (otherwise, may need to upgrade everything)

[CameronFRWatson]

Squidpy

Working branch

No working branch yet

Goals

• Update Squidpy to 1.4.1 and bring anndata to 0.10

Current status

• Haven't yet started working on update, but have reviewed squidpy changelog from version 1.1.2 to 1.4.1 and have prioritized changes that may need to be addressed

To-do

• Update Docker and conda
• Likely must update for GalaxyME revision (copied from Squidpy changelog)
  • An optional ax keyword can now be passed to squidpy.pl.nhood_enrichment function
  • Add library_key in squidpy.gr.spatial_neighbors to support building graphs across multiple slides
  • Add percentile option to squidpy.gr.spatial_neighbors to filter neighbor graph using percentile of distances threshold.
  • Add attr option to squidpy.gr.spatial_autocorr to select values from anndata.AnnData.obs or anndata.AnnData.obsm.
• Need to check if changes impact existing wrapped tools, may not require action (copied from squidpy changelog)
  • Require numba>=0.52.0
  • Fix source/target being None in squidpy.gr.ligrec
  • Fix use_raw in squidpy.gr.spatial_autocorr
  • Fix a typo when saving a figure caused a strange directory name to be created
  • Fix ligrec from pandas update
  • Fix pandas inf squidpy.pl.ligrec
  • Remove column assignment to improve compatibility with new cell metadata
  • Update cell metadata index conversion
  • Fix previously updated cell metadata index conversion
  • Update pre-commits and unpin numba and numpy
  • Fix building graph in knn and delaunay mode
  • Correct shuffling of annotations in sq.gr.nhood_enrichment