Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Netcdf a100 #101

Draft
wants to merge 10 commits into
base: development
Choose a base branch
from
Draft

Netcdf a100 #101

wants to merge 10 commits into from

Conversation

CyprienBosserelle
Copy link
Owner

@CyprienBosserelle CyprienBosserelle commented Jan 2, 2024

Save to file takes too long for many blocks

The problem

The current dev branch save each block independantly to the netcdf file. The bottleneck is the netcdf write function overhead which make writing large model (20000 blocks and more) quite slow.

The solution

To alleviate this needs a new function that combines all blocks in a single array (for each level/variable) and write the whole array at once to the netcdf file.

Hold on

While this will be faster, it is likely inefficient when there are few arrays in a level. Typically when the highest level has only a few block we would still be writing a full array.

Memory is cheap?

This will also be impossible if going to very high level where we won't be able to allocate memory to hold the highest level array.

The real solution

Something that combine both option above and decide what is the best compromise.

Collapse array

This may be a good time to introduce collapsing level to a single array and saving it to disk.

Other things

This branch also includes:

  • fix on AOI polygon
  • Fix for Tsunami initial condition (deform)

@CyprienBosserelle CyprienBosserelle marked this pull request as draft January 2, 2024 07:38
@CyprienBosserelle CyprienBosserelle marked this pull request as ready for review February 16, 2024 00:16
@CyprienBosserelle
Copy link
Owner Author

Added a new variable savebyblk input parameter. default is on if set to off it will save per level which is a lot faster but that require enough memory to allocate the full high res level.

@CyprienBosserelle
Copy link
Owner Author

No testing has been done for zoneoutput

@CyprienBosserelle
Copy link
Owner Author

This is on-hold until testing with new bnd complete

@CyprienBosserelle CyprienBosserelle marked this pull request as draft June 17, 2024 01:32
@CyprienBosserelle CyprienBosserelle mentioned this pull request Jun 18, 2024
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant