v0.7

[ivirshup]

I'd like to have some organization around the upcoming release. I'd like to have a list of all planned and included changes for v0.7 Please edit this for anything I forgot!

New features

• DataFrames everywhere
• Views of views Views take around as much memory as object #62
• obsp, varp for pairwise measures Backwards breaking change: Sparse observation and variables annotation #136
• Allow sparse data and dataframes in .obsm, .varm, .layers, .uns adata.uns dataframe gets converted into numpy.ndarray when saving and loading h5ad #134
• Updated h5ad spec (with backwards compatable reading)
• Lazy views for faster subsetting
• Better test coverage

Breaking changes

• Modifying sparse .X inplace via a view, even if value was set with dense
• Indexing into AnnData object will not reduce dimensionality of contained arrays
  • Fixes inconsistent slicing with one element.  #74 Error slicing when there is only one observation #60 inconsistent slicing of .X vs .layer #145

TODO:

• Unbreak zarr implemntation (broken in [WIP] IO refactor #167)
• Restructure views, to allow views of view (merge Views of views #164)
• Standardize results of indexing into AnnData (merge Views of views #164) (always 2d) inconsistent slicing of .X vs .layer #145
• Add in obsp, varp attributes (uncomment)
• Fix dense as sparse reading for h5ad
• Appropriate warnings for changed behaviour
• Make sure scanpy and others work with new behaviour
• deprecate rename_categories

[flying-sheep]

Do we want “2d only” (the second breaking change) in something called 0.7?

I think to signal a breaking change and closeness to 1.0, we could name this version 0.90 or so!

(not 0.99, because then we’ll end up having 0.999, 0.9999, …)

[flying-sheep]

Another change we could think about: unifying .layers and .X

Maybe .X should just be a property returning the first layer? or the layer called X?

[ivirshup]

Do we want “2d only” (the second breaking change) in something called 0.7?

I think to signal a breaking change and closeness to 1.0, we could name this version 0.90 or so!

I would understand the change from v0.6.x to v0.7 to involve breaking changes. I don't like the idea of going to v0.90 since it breaks from semantic versioning. Also, who knows, we may want v0.8-v0.20. For example, I'd say the possibility of having .X be one of .layers sounds like a great v0.8 target to me, since it would require us to allow layers to be backed, and that'll take some time.

I would have to know that this was a dramatic change that broke a lot of stuff before thinking any out of the ordinary versioning semantics were used. The change to always 2d doesn't break many tests here or for scanpy. Additionally scvelo mostly runs with it.

I do think we should do a v0.6.22 release of AnnData and v1.4.4 of scanpy, so there's a version with working scipy/ statsmodels dependencies without this change. This would also give us space to transition users to {obs,var}_vector, since that will return a 1d array now, and once "always 2d" is in effect.

[flying-sheep]

Well, if the fallout isn’t that bad, I agree, we can just call it 0.7

That being said, skipping from 0.6.x to 0.90 is completely compatible with semantic versioning. Why do you think it wasn’t? (And instead of 0.8, the next version would just be called 0.91)

[falexwolf]

@ivirshup: Agreed with everything! Terrific! Agreed in particular with 0.6.22 and 1.4.4 before moving to 1.5 and 0.7. In 0.6.22, where the flattening is still present, can you add a warning future versions will not flatten .X anymore when it's called?

[ivirshup]

@flying-sheep, I suppose v0.90 would be compatible with establishing the order of versions, but I think of "incrementing the version" to imply incrementing on a fixed scale. I think my reaction to seeing a version change from v0.6.22 to v0.7.0 would be "there are breaking changes in this release", while v0.6.22 to v0.90.0 would be "what does this mean? What happened to v0.7-v0.89"?

[ivirshup]

I've added that warning, what else has to be done before we can release 0.6.22?

[falexwolf]

Nothing, I think. I'm happy to make the release if you don't want to do it. 🙂

[ivirshup]

Just realized there are probably a couple bugs to fix prior to that:

• n_counts not found? scanpy#728 - Just fixing backwards compat for _get_obs_array (Fixed by Allow asking for a key of obs when use_raw=True in deprecated _get_{obs,var}_array #176)
• obs_vector probably shouldn't throw a warning telling you to use obs_vector 😜 This doesn't actually happen, something else in sc.pl.scatter was causing this I think.
• "IndexError: boolean index did not match indexed array along dimension" when working with subsetted anndata object scanpy#699 - This is probably on the scanpy side. Should sc.pp.scale make a copy if it's called on a view? It currently does not if called on dense data, but does on sparse data. I think it should always make a copy if its on a view.
• Export Raw from anndata What do we export? #174

[falexwolf]

Great that you're addressing these!

I think it should always make a copy if its on a view.

Agreed!

[ivirshup]

I think we're good for 0.6.22. Scanpy will need a pass to remove warnings being thrown for v1.4.4 though. Would you mind making this release @falexwolf?

[falexwolf]

Made release 0.6.22 for the current state of master: https://github.com/theislab/anndata/commit/34e8616fd92b893bdd8e4b6d4036830270961d19/, https://github.com/theislab/anndata/releases, https://pypi.org/project/anndata/#history. 🙂

[flying-sheep]

It would be great if you could start publishing wheels too, it’s almost no additional work:

python3 setup.py sdist bdist_wheel
twine upload dist/anndata-0.7{.tar.gz,-py3-none-any.whl}

I always have to go back and retroactively make wheels.

[falexwolf]

Oh sorry, I thought we stopped publishing wheels 1.5 years after going from cython to numba. 😆Had you told me that we wanted to continue (what for, btw?, there is no compilation involved), I'd continued publishing wheels of Mac.

[flying-sheep]

There’s not that much of an advantage anymore but wheels are faster to install, as setup.py doesn’t need to be run on the target machines – the wheels will simply be unpacked.

[ivirshup]

Could the release process get documented, potentially with scripts or snippets? I'd like to be able to help out with that, but I'd like to know I was doing it right.

[flying-sheep]

It’s not more than this, but we can document it, yes:

$ git tag x.y.z
$ python3 setup.py sdist bdist_wheel
$ twine upload scanpy-x.y.z-py3-none-any.whl scanpy-x.y.z.tar.gz
$ git push --tags

[ivirshup]

Just push a big update which cleans up the reading and writing code a lot, as well as improving performance. It does make some changes to the on disk structure, but you should be able to read older files. Added features mean you can now have data frames and sparse matrices stored in any of the obsm, varm, layers etc.

It would be great if people could try out current master and let me know if they find any bugs!

[flying-sheep]

OK, so should we get this out before the X-layers merge (#244) and the modes (#237) land?

[ivirshup]

Yes! I need to take another look at fixing scverse/scanpy#832 before this can go though. I'm going to check if there's any chance this could get sorted upstream first.

[flying-sheep]

Hmm, we should deprecate rename_categories, as it’s no longer necessary with the new categorical storage. I’ll do this before releasing the rc.

[LuckyMD]

What is the alternative to rename_categories? I'm using it quite a lot. It works both on the categoricals for the obs or var column, but also on any rank_genes_groups() results that are generated, no?

[flying-sheep]

For .obs/.var it’s superfluous, just do:

adata.obs['foo'].cat.categories = new_categories
# or
adata.obs['foo'].cat.rename_categories(..., inplace=True)

For the .uns stuff, we need to find another way, this is a scanpy convention and doesn’t belong into AnnData. Either move it to scanpy or store it in a way that allows to do .uns[...].rename_categories(...)

[LuckyMD]

The nice thing about rename_categories as it is at the moment is that it renames the .uns and .obs/.var stuff in a single command.

[flying-sheep]

We just (pre-)released AnnData 0.7rc1 with it deprecated. We should move the code to scanpy as a documented utility I think.

[flying-sheep]

Alex reverted stuff on master because there’s no alternatives, so my idea of our path forwards is now:

1. Scanpy needs to grow utils.rename_categories (or so) and update_anndata (or so)
2. obsp/varp warning and rename_categories deprecations needs to be reinstated before 0.7
3. people should be encouraged at the right places to migrate to obsp/varp (using update_anndata) and scanpy.utils.rename_categories.

[falexwolf]

Could we try to have a discussion instead of you making absolute statements?

1. semi-agreed, because of unclarity of alternatives, see our Slack discussion
2. disagreed on obsp/varp, it gives the user a horrible experience, and there are no alternatives, I worked with these warnings for a week now and got very annoyed, so I muted them
3. agreed

[flying-sheep]

Sorry for not outlining this clearly enough, I was busy celebrating Christmas and my birthday!

OK, so we have two changes that need to happen (at least in the long, if not short run). We should only have the deprecations in the final anndata 0.7 release if we manage to get the correct behavior into anndata and scanpy quickly enough.

1. Better categorical handling: You think instead of deprecating rename_categories, we should allow storing arrays with a categorical dtype in .uns, right? You said in Slack that this is a difficult change. Can you elaborate why?
2. obsp/varp instead of square matrices in .uns: I don’t think this is a difficult change: We just need to figure out all places where scanpy currently stores such a matrix and change both the code and add that storage position to a list of “to be modernized” AnnData parts that can be used implicitly or via update_anndata.

I’d like 2 to happen before the next release because obsp/varp exists already and we need to keep people from relying on the old behavior.

[flying-sheep]

OK, seems like we got it. Now to all the exciting changes enabled by this!