Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move the buffer copy logic outside operators #3514

Merged
merged 6 commits into from
Mar 2, 2023

Conversation

anagainaru
Copy link
Contributor

@anagainaru anagainaru commented Feb 28, 2023

If an operator (direct or inverse) is not applied (due to a threshold or some other reason) the Operate or Decompress functions are now returning 0 and are not responsible with copying the data to the ADIOS2 internal buffers.

This allows ADIOS2 to use the appropriate function for copying the data (taking into consideration the memory space).

Operators that use this: Blosc, MGARD, MGARDPlus

Testing Adding test for MGARD with small GPU buffers.

Note: the original code inside operators was using memcpy so I am simply moving the logic outside (I am not checking for reverse endian)

Adding to #3465 to skip copy for blocks with 0 elements

@anagainaru anagainaru marked this pull request as ready for review March 1, 2023 14:26
@anagainaru
Copy link
Contributor Author

@JieyangChen7 I use 1024 as the threshold for when to apply MGARD based on my testing on Summit but you might have a better idea of what value to use.

@anagainaru anagainaru changed the title [WIP] Move the buffer copy logic outside operators Move the buffer copy logic outside operators Mar 1, 2023
@anagainaru anagainaru requested a review from lizdulac March 1, 2023 15:29
@JieyangChen7
Copy link
Contributor

@anagainaru I think 1e6 might be a safer choice based on my experience.

pnorbert
pnorbert previously approved these changes Mar 1, 2023
williamfgc
williamfgc previously approved these changes Mar 1, 2023
@anagainaru
Copy link
Contributor Author

anagainaru commented Mar 2, 2023

@pnorbert I will merge this after I rebase it over release_29 and if it doesn't solve Axel's issue I will update his PR (#3465) with the fix. I think it make more sense like that compared to changing this PR further.

@anagainaru anagainaru changed the base branch from master to release_29 March 2, 2023 13:08
@anagainaru anagainaru dismissed stale reviews from williamfgc and pnorbert March 2, 2023 13:08

The base branch was changed.

@anagainaru anagainaru added this to the v2.9.1 milestone Mar 2, 2023
@anagainaru anagainaru modified the milestones: v2.9.1, v2.9.0 Mar 2, 2023
@pnorbert pnorbert merged commit 2151935 into ornladios:release_29 Mar 2, 2023
@anagainaru anagainaru deleted the operators-on-gpu-buffers branch March 2, 2023 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants