Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replicate input structure in output files #1

Closed
angus-g opened this issue Jul 29, 2019 · 3 comments
Closed

Replicate input structure in output files #1

angus-g opened this issue Jul 29, 2019 · 3 comments
Assignees
Labels
enhancement New feature or request

Comments

@angus-g
Copy link
Owner

angus-g commented Jul 29, 2019

The output file is simply a concatenation of the filtered, flattened arrays for each timestep:

HDF5 "test.h5" {
GROUP "/" {
   DATASET "var_U" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 2, 70000 ) / ( 2, 70000 ) }
   }
}
}

It would be good to copy the structure of the input file (spatial and temporal dimensions) to the output file.

@angus-g angus-g added the enhancement New feature or request label Jul 29, 2019
@angus-g
Copy link
Owner Author

angus-g commented Aug 6, 2019

The most straightforward approach would probably be to open the xarray DataArray for each sampled variable, and combine them into a single Dataset. However, this interface doesn't support incremental writes. If we can set the Dataset's data to a lazy dask array, we could get around this?

@angus-g
Copy link
Owner Author

angus-g commented Aug 7, 2019

pydata/xarray#1672 pertains to the feature in xarray that would make this easier.

@angus-g
Copy link
Owner Author

angus-g commented Nov 12, 2019

Closed via #14

@angus-g angus-g closed this as completed Nov 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant