WIP: Allow users to set data alignment within netcdf4 files #2178
Conversation
|
A couple of immediate questions.
|
|
The main usecase for this is to memorymap large arrays in an alligned fashion. This enables efficient transfer of data between SSDs, network interfaces, and RAM. I'll look to see if i can do this later after opening. |
|
I honestly tried to look through the documentation, it isn't clear that you can change the FAPL after creation. From the examples, they seem to open a file, copy the FAPL, then close it, then reopen it with a different FAPL. I also posted the question on the forum, but I need to get moderated. It should appear |
|
Sorry, looking at the source, it is clear that it is a "copy" of the FAPL. |
hmaarrfk
force-pushed
the
set_alignment
branch
from
3828eda
to
55af948
Compare
|
No you cannot change the FAPL in HDF5 after file open... |
|
I will have to think about this. |
|
As we discussed in the other thread, alignment has to be set before the file open. When netcdf-c opens a file, it will use the alignment setting to call the HDF5 alignment function when it creates the FAPL... |
|
Closing this thread until we decide how the API will be designed. It seems a different approach is the stronger contender. |
|
On the HDF5 forum, it is confirmed that it must be specified at open time: |
re: Unidata#2177 re: Unidata#2178 Provide get/set functions to store global data alignment information and apply it when a file is created. The api is as follows: ```` int nc_set_alignment(int threshold, int alignment); int nc_get_alignment(int* thresholdp, int* alignmentp); ```` If defined, then for every file created opened after the call to nc_set_alignment, for every new variable added to the file, the most recently set threshold and alignment values will be applied to that variable. The nc_get_alignment function return the last values set by nc_set_alignment. If nc_set_alignment has not been called, then it returns the value 0 for both threshold and alignment. The alignment parameters are stored in the NCglobalstate object (see below) for use as needed. Repeated calls to nc_set_alignment will overwrite any existing values in NCglobalstate. The alignment parameters are applied in libhdf5/hdf5create.c and libhdf5/hdf5open.c The set/get alignment functions are defined in libsrc4/nc4internal.c. A test program was added as nc_test4/tst_alignment.c. ## Misc. Changes Unrelated to Alignment * The NCRCglobalstate type was renamed to NCglobalstate to indicate that it represented more general global state than just .rc data. It was also moved to nc4internal.h. This led to a large number of small changes: mostly renaming. The global state management functions were moved to nc4internal.c. * The global chunk cache variables have been moved into NCglobalstate. As warranted, other global state will be moved as well. * Some misc. problems with the nczarr performance tests were corrected.
Closes #2177
I'm really not sure how to make this "backward" compatible with older versions.