Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-module KeyData interface #337

Merged
merged 16 commits into from
Feb 21, 2023
Merged

Multi-module KeyData interface #337

merged 16 commits into from
Feb 21, 2023

Conversation

takluyver
Copy link
Member

This brings an API similar to KeyData to multi-module detector data, so instead of agipd.get_array('image.data') you can do something like agipd['image.data'].xarray(). I hope this will make using EXtra-data more consistent, at least once people are used to using these newer APIs.

This also adds the ability to get an unlabelled numpy array for multi-module detector data; previously you had to get an xarray, even if you didn't want the labels. I still think dimension labels are a good idea, but xarray is certainly an extra level of complexity, and many people are more familiar with numpy arrays.

Finally, this changes how Dask arrays are created for multi-module data - rather than making chunks directly based on the files, so each frame is split across multiple chunks, it now uses the .split_trains() method to break data into chunks to load, so each Dask chunk spans all modules. This simplifies the task graph for the common case where you want to process data frame-wise with all modules together. I had hoped that this would solve all our issues with Dask and make #333 obsolete, but unfortunately that doesn't seem to be the case - we might want some variant of that PR as well.

@takluyver takluyver added the enhancement New feature or request label Aug 18, 2022
@lgtm-com
Copy link

lgtm-com bot commented Aug 18, 2022

This pull request introduces 1 alert when merging d750db7 into 1ba78dd - view on LGTM.com

new alerts:

  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Aug 18, 2022

This pull request introduces 1 alert when merging a138efa into 1ba78dd - view on LGTM.com

new alerts:

  • 1 for Module is imported with 'import' and 'import from'

@lgtm-com
Copy link

lgtm-com bot commented Aug 18, 2022

This pull request introduces 1 alert when merging fd4ad9a into 1ba78dd - view on LGTM.com

new alerts:

  • 1 for Module is imported with 'import' and 'import from'

@takluyver takluyver marked this pull request as ready for review September 22, 2022 12:58
Copy link
Contributor

@philsmt philsmt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this nice PR. Indeed the components API did feel slightly out of touch, especially when using them the first time after KeyData got introduced. This will nicely fill the gap.

Given the fairly complex axes-shifting and index-juggling machinery in here, I don't feel able to properly review logic as such. But I tried to give some constructive comments or ideas here or there.

extra_data/tests/test_components.py Outdated Show resolved Hide resolved
extra_data/components.py Show resolved Hide resolved
extra_data/components.py Show resolved Hide resolved
extra_data/components.py Show resolved Hide resolved
extra_data/components.py Show resolved Hide resolved
extra_data/components.py Show resolved Hide resolved
extra_data/components.py Show resolved Hide resolved
@lgtm-com
Copy link

lgtm-com bot commented Oct 28, 2022

This pull request introduces 1 alert when merging 8c1b922 into 9b36ce4 - view on LGTM.com

new alerts:

  • 1 for Module is imported with 'import' and 'import from'

extra_data/components.py Fixed Show fixed Hide fixed
@philsmt
Copy link
Contributor

philsmt commented Feb 20, 2023

Thank you, LGTM!

@takluyver takluyver added this to the 1.13 milestone Feb 21, 2023
@takluyver takluyver merged commit 7f80524 into master Feb 21, 2023
@takluyver takluyver deleted the multimod-keydata branch February 21, 2023 08:58
@takluyver
Copy link
Member Author

Thanks for the review 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants