Add support for setting HDF5 alignment property when creating a file

[DennisHeimbigner]

• Fixes Allow users to specify data alignment #2177
• Fixes WIP: Allow users to set data alignment within netcdf4 files #2178

re: #2177
re: #2178

Provide get/set functions to store global data alignment
information and apply it when a file is created.

The api is as follows:

int nc_set_alignment(int threshold, int alignment);
int nc_get_alignment(int* thresholdp, int* alignmentp);

If defined, then for every file created opened after the call to
nc_set_alignment, for every new variable added to the file, the
most recently set threshold and alignment values will be applied
to that variable.

The nc_get_alignment function return the last values set by
nc_set_alignment. If nc_set_alignment has not been called, then
it returns the value 0 for both threshold and alignment.

The alignment parameters are stored in the NCglobalstate object
(see below) for use as needed. Repeated calls to nc_set_alignment
will overwrite any existing values in NCglobalstate.

The alignment parameters are applied in libhdf5/hdf5create.c
and libhdf5/hdf5open.c

The set/get alignment functions are defined in libsrc4/nc4internal.c.

A test program was added as nc_test4/tst_alignment.c.

Misc. Changes Unrelated to Alignment

• The NCRCglobalstate type was renamed to NCglobalstate to
  indicate that it represented more general global state than
  just .rc data. It was also moved to nc4internal.h. This led
  to a large number of small changes: mostly renaming. The
  global state management functions were moved to nc4internal.c.

• The global chunk cache variables have been moved into
  NCglobalstate. As warranted, other global state will be moved
  as well.

• Some misc. problems with the nczarr performance tests were corrected.

[hmaarrfk]

Is it guaranteed that the functions are "atomic" and threadsafe?

[DennisHeimbigner]

No, but no part of the netcdf-c library has those properties.

[hmaarrfk]

Understood. Thank you for the clarification.

[Dave-Allured]

nc_set_alignment
nc_get_alignment

Are these to be public facing functions in the public netcdf-c API?

If yes, then I suggest add a standard documentation block for each function. The above description block on this PR is a good start.

[edwardhartnett]

Are these new functions going in the dispatch table?

[DennisHeimbigner]

They are not part of the dispatch table since they are global. In this, they are like
the cache functions.

[Dave-Allured]

They are not part of the dispatch table since they are global. In this, they are like
the cache functions.

That seems reasonable to me. For file types other than netcdf-4, these nc_*_alignment functions will behave as no-ops.

[Dave-Allured]

@DennisHeimbigner, thank you for adding the function docs.

Initially I found HDF5 disk alignment to be confusing in several ways. I took the liberty of drafting some detail documentation to address the nuances. Please consider these extra suggestions for nc_set_alignment. Feel free to accept any parts and reword as you see fit.
.........................
nc_set_alignment causes new data written to a netCDF-4 file to be aligned on disk to a specified block size. To be effective, alignment should be the system disk block size, or a multiple of it. This setting is effective with MPI I/O and other parallel systems, not with single process file access.

This is a trade-off of write speed versus file size. Alignment leaves holes between file objects. The default of no alignment writes file objects contiguously, without holes. Alignment has no impact on file readability.

Alignment settings apply only indirectly, through the file open functions. Call nc_set_alignment first, then nc_create or nc_open for one or more files. Current alignment settings are locked in when each file is opened, then forgotten when the same file is closed. For illustration, it is possible to write different files at the same time with different alignments, by interleaving nc_set_alignment and nc_open calls.

Alignment applies to all newly written low-level file objects at or above the threshold size, including chunks of variables, attributes, and internal infrastructure. Alignment is not locked in to a data variable. It can change between data chunks of the same variable, based on a file's history.

Refer to H5Pset_alignment in HDF5 documentation for more specific details, interactions, and additional rules.

[Dave-Allured]

I should probably have done this as a pull request to your branch. Let me know if you prefer that.

[hmaarrfk]

I would like to suggest removing the phrase

not with single process file access.

In fact, my initial request was centered around optimizing single process access time to files of mine.

I'm not adapt enough to provide hard examples with C++, but with the standard strategies, I reach 1GB/s read/write throughput.

With data alignment, and the HDF5 Direct driver, I can reach 3GB/s (NVMe limited speeds) on read and write. Thus, I would disagree that data alignment does not help with "single process" access.

Generally, it allows lower level hardware/firmware/software to organize data transfers in a highly effective way.

[hmaarrfk]

I had an other writeup of some experiments of mine: pydata/xarray#6309

Eventually, xarray uses netcdf4-python as a backend.

[Dave-Allured]

@hmaarrfk, it is good to work out these issues up front. Thanks for commenting.

Those notes about single and parallel performance were taken straight from the HDF5 function documentation for H5Pset_alignment:

Generally the default values will result in the best performance for single-process access to the file. For MPI IO and other parallel systems, choose an alignment which is a multiple of the disk block size.

Is there a way to explain the difference between your experience and this statement from HDF5?

I have no direct experience with this. I am just trying to consolidate existing documentation, and make this functionality as clear as possible for netCDF users. Perhaps the way I rephrased it was not right.

If the issue is complicated and there are doubts about performance behavior, I have no problem completely omitting the performance notes from this function doc. They could be restored later after the behaviors are better understood. Your advice?

[hmaarrfk]

Is there a way to explain the difference between your experience and this statement from HDF5?

I have noticed:

1. for non-aligned writes, DIRECT performed terribly. I think I probably had some cache settings set poorly. So badly in fact that I've resorted to opening the file twice. Once for the small bits, the other for the large chunks.
2. I think HDF5 was written at a time where spinning disk drives were the norm. Today, modern SSDs are effectively full processors of their own with their own RAM. The limitations are simply different. The truth is, that up until this year, I wasn't able to measure any difference on a single SSD. 1GB/s was faster than any drive I could write to in a sustained way.

There are also softer advantages to the direct driver.

First, it bypasses the disk cache. For example, if you open and analyze in a serial fashion 100, 1GB files, with the standard SEC2 driver, the Operating system will in effect destroy all its cache by trying to "cache" the 100 files and likely evict more important files. In my experiments, the direct driver avoided the operating system cache alltogether.

This API (and the other I have requested) would allow me to more fully explore the performance axis.

TLDR, I think the old wisdom is centered around HDD. Today, HDDs got faster, and are no-longer the primary drive. (Fast!) SSDs are a paradigm shift that still hasn't made its way to documentation.

[Dave-Allured]

1. for non-aligned writes, DIRECT performed terribly ...

Okay. Are you saying that you made a reasonable comparison between aligned vs. non-aligned writes with single process and the DIRECT driver, and aligned performed better?

If so, this contradicts the performance statement in the HDF5 doc. That is not unreasonable. Performance tuning in general is complicated and a moving target, as you said.

[hmaarrfk]

I guess eventually I was comparing:

1. Small bits of unaligned data (my metadata) ~ 200 MB -- Much faster using the default driver (SEC2)
2. Large chunks of aligned data (the main images I was storing). ~40 GB -- Much faster using the Direct Driver

[Dave-Allured]

Okay. Can you easily make a side by side time measurement of large chunks, 5 to 40 Gb with Direct Driver, single process, one run with nc_set_alignment or equivalent, and one without?

My goal for today is only to decide what should, and should not, go into the netCDF function description.

[hmaarrfk]

Data size & alignment (RAM)	HDF5 No Alignment	HDF5 With Alignment
1x 4GB Unaligned	2.19 s	2.1 s
1x 4GB Aligned	1.87 s	1.89 s
32x 128 MB Unaligned	1.67 s	1.67 s
32x 128 MB Aligned	1.46 s	1.47 s
32x 128 MB Aligned + DIRECT (#2236)	5.06 s	1.01 s

Each test is run 4 times taking the average

I then repeated the benchmark running:

sudo su -c "echo 1 > /proc/sys/vm/drop_caches"
sudo swapoff -a
sudo swapon -a

between each test

Data size & alignment (RAM)	HDF5 No Alignment	HDF5 With Alignment
1x 4GB Unaligned	2.21 s	2.09 s
1x 4GB Aligned	1.92 s	1.87 s
32x 128 MB Unaligned	2.17 s	2.15 s
32x 128 MB Aligned	1.87 s	1.88 s
32x 128 MB Not Aligned + DIRECT	11.8 s	11.8 s
32x 128 MB Aligned + DIRECT	6.48 s	1.29 s

Benchmarking jupyter notebook
20220328_hdf5_benchmark.zip

Conclusion: It seems true that there is little benefit for the standard file to enforce alignment with the SEC2 driver.

[hmaarrfk]

Hopefully the benchmarks I ran were representative of what you wanted. It is a little hard to run these benchmarks in a repeatable way.

The SSDs get hot, and do their own internal shuffling between different levels of TLC. I've tested this particular SSD up to 2TB of write, then weird stuff happens my speeds started to slow down (was it heat???).

The operating system cache also comes into play with the SEC2 driver (default). Since the SEC2 driver will write to a temporary kernel buffer.

The SSD I got should be fine Sabrent Rocket 4 Plus (4TB) (according to https://www.tomshardware.com/reviews/sabrent-rocket-4-plus-m2-nvme-ssd-review/3) but you know, I could be using it poorly too.

[hmaarrfk]

For clarity the intial comparison I was making was between 2.2 s (1.8GB/s) and 1.29 s (3.1GB/s).

[hmaarrfk]

An other detail. I found that if i overwrote the same file in my benchmark it would give me strange results.

Therefore, each new write writes to a completely new file. I leave the old files in place.

This is the entry in the fstab

/dev/disk/by-uuid/xxxxx /data auto nosuid,nodev,nofail,x-gvfs-show 0 0

[hmaarrfk]

Output of relevant mamba list

h5py                      3.6.0           nompi_py39hf407c45_600    ramonaoptics   <---- Includes my patches to expose the direct driver.
hdf5                      1.12.1          nompi_h2750804_103    conda-forge

python version 3.9

[Dave-Allured]

@hmaarrfk, yes, that is what I was looking for. Thank you for running those comparisons. Obviously there is a large performance improvement when using disk alignment with the direct file driver.

HDF5 documentation for the direct file driver specifies the need for disk alignment to improve performance. It is interesting that the H5Pset_alignment description has no mention at all of this purpose.

With that in mind, is this more explicit wording for nc_set_alignment okay with you?
.............................
Generally the default alignment values result in best performance for single-process file access. When using the direct file driver, or with MPI I/O and other parallel systems, choose an alignment which is a multiple of the disk block size.

[hmaarrfk]

Really excited that this got through! Thanks you all!