Add support for _Float16 16-bit floating point type #4065

jhendersonHDF · 2024-03-05T01:05:16Z

This PR adds native support for a 16-bit floating point datatype through the H5T_NATIVE_FLOAT16, H5T_IEEE_F16LE and H5T_IEEE_F16BE macros. It is marked as a draft PR for early review while I'm still resolving issues with compilers that generate calls to functions like __extendhfxf2 and then they end up not being properly linked in.

Things not added in this PR:

No Fortran added yet; left for Scot though Fortran has no Float16 type. Should
just make the identifiers available.
No C++ added yet
No H5LTmake_dataset_float16 high-level method added yet since it would
be a public API and whether or not _Float16 is available is not controllable
by user. API would need to accept a const _Float16 *buffer parameter, where
the type might not be available.
Constants for types added to Java, but Java has no matching type yet
No support for data transforms on _Float16 data added yet, because
macro system is not setup to nicely support optional types
No option to enable AVX512 FP16 instructions - user could add flag if desired

Fixed some conversion issues with Clang due to problematic undefined behavior when casting a negative floating-point value to an integer Fixed a bug in the library's software integer to floating-point conversion function where a user's conversion exception function returning H5T_CONV_UNHANDLED in the case of overflows would result in incorrect data after conversion Added configure checks for functions and macros related to _Float16 usage since some compilers expose the datatype but not the functions or macros Fixed a dt_arith test failure when H5_WANT_DCONV_EXCEPTION isn't defined Fixed a few warnings from not explicitly casting some _Float16 variables upwards

Add CMake status message about _Float16 test-compile program

jhendersonHDF · 2024-03-05T01:07:23Z

src/H5Tinit_float.c

-    else if (perm[last] > perm[last - 1] && perm[last - 1] > perm[last - 2]) {
+    else if (perm[last] > perm[last - 1] &&
+             /* Only check perm[last - 2] if we have more than 2 points to consider */
+             ((last < 2) || (perm[last - 1] > perm[last - 2]))) {


The changes here fix an issue where this function was assuming it could access perm[last - 2] even though that was invalid and would also cause the _Float16 type to be detected as VAX endian order

jhendersonHDF · 2024-03-05T01:09:24Z

test/dt_arith.c

    int            endian; /*endianness            */
    unsigned int   fails_this_test = 0;
    size_t         j;
+#ifdef H5_WANT_DCONV_EXCEPTION


The changes in this function add _Float16 testing and also skip running this part of the test when H5_WANT_DCONV_EXCEPTION is not defined, as there was otherwise a testing failure in that case since the library wouldn't raise the expected floating-point conversion exception.

jhendersonHDF · 2024-03-05T01:10:55Z

test/dt_arith.c


        /* The hardware conversion */
-        /* Check for underflow when src is a "larger" float than dst.*/
-        if (FLT_FLOAT == src_type) {


At this point, it seemed reasonable to rewrite this part of dt_arith by creating separate functions for each type we're converting from. Otherwise, this started to become a visual mess and future support for additional types like complex numbers will only make it worse.

jhendersonHDF · 2024-03-05T01:16:15Z

src/H5Tnative.c

@@ -715,7 +717,14 @@ H5T__get_native_float(size_t size, H5T_direction_t direction, size_t *struct_ali
    assert(size > 0);

    if (direction == H5T_DIR_DEFAULT || direction == H5T_DIR_ASCEND) {
-        if (size <= sizeof(float)) {
+#ifdef H5_HAVE__FLOAT16
+        if (size <= sizeof(H5__Float16)) {


It seems dangerous for the code here to match smaller datatypes with larger datatypes, since, before this change, the test code we currently have in ntypes.c was matching _Float16 to float and causing an H5Dread call using the type returned from H5Tget_native_type to overrun the buffer. Seems to me that we should only match a type if the sizes match. That said, I'm just keeping in line with how the code behaved previously here.

I think that's a bug in the test code. H5Tget_native_type doesn't guarantee the size will match, it kind of fuzzily selects the closest native match to the datatype passed in. The app code should inspect the datatype returned to determine how to allocate the buffer and interpret the data. (If a real app knew the size a priori, it probably wouldn't need to call H5Tget_native_type anyways).

jhendersonHDF · 2024-03-05T01:20:55Z

src/H5Tconv.c

+            if (dst.prec / 8 > src_p->shared->size)
+                buf_size = (dst.prec + 7) / 8;
+            else
+                buf_size = src_p->shared->size;


This function was previously assuming ~32-bit floats. 16-bit and smaller floats would cause an invalid memory access further on when assuming it could copy more than 16 bits out of the temporary source buffer into the destination. In order to fix this bug, I had to rewrite parts of this function, which also led to a significant speedup when converting floats to ints in software.

The old way that this function used to work is that it would allocate a relatively large buffer to hold the temporary converted integer. The buffer would be (2^exponent_num_bits / 8) + 1 bytes to try and have enough space so that the mantissa value can be shifted by any possible shift value from the exponent without losing bits from the buffer so that the MSB can be checked after the shift. However, for something like an IEEE 128-bit float with an exponent of 15 bits, this would allocate a 4097-byte buffer and then perform multiple different bit operations on all 4097 * 8 = 32776 bits of the buffer, which was very slow. Now, the function just allocates a buffer as big as the size of the destination integer and checks the MSB before shifting, then later on calculates the new MSB position after the shift to determine if overflow or underflow occurred. A small test program that converts 10,000,000 64-bit doubles to 32-bit ints went from 7.6 seconds to 1.05 seconds. Converting 10,000,000 IEEE 128-bit floats to 32-bit ints goes from 97.15 seconds to 1.17 seconds.

Clarify a few things in the RELEASE.txt entry

jhendersonHDF · 2024-03-09T01:29:06Z

The build errors we saw before on stria and perlmutter are gone now.

Great, thanks for testing!

qkoziol · 2024-03-10T05:57:06Z

tools/test/h5ls/expected/tfloat16.ls

+Opened "tfloat16.h5" with sec2 driver.
+DS16BITS                 Dataset {8/8, 16/16}
+    Attribute: DS16BITS {128}
+        Type:      native _Float16


Will this be different on a big-endian machine?

Ah that's a good point. I should test this on a big-endian machine, but I suspect that yes it will probably trip over this test.

Added a BE test file for h5dump, h5dump xml and h5ls tests and verified testing on big-endian machine, but only in the case where _Float16 isn't supported since we don't have a BE machine where it is. However, I setup the tests to only check the appropriate BE or LE test file, depending on the target architecture, for native _Float16, and verified the correct file is picked between BE and LE machines with both CMake and Autotools, so this should work on BE machines where _Float16 is supported.

qkoziol · 2024-03-10T06:03:30Z

tools/test/h5dump/testh5dump.sh.in

I think the tools/test/h5dump/testh5dumpxml.sh.in test script needs a test(s) for float16 also.

Seems reasonable, I'll add that.

Added an XML test. Also added one for the new BE file while I was at it.

qkoziol · 2024-03-10T06:05:03Z

I don't know where the DDL definition file for h5dump output has gone, but it probably needs updating.

qkoziol · 2024-03-10T06:05:56Z

Changes to the library and utilities (except some small things in h5dump) look good so far.

jhendersonHDF · 2024-03-10T06:13:05Z

I don't know where the DDL definition file for h5dump output has gone, but it probably needs updating.

Looks like the old DDL files are at https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF110.dox and https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF112.dox, but we don't seem to have one for the current release branch. I updated the part in src/H5Tmodule.h that references the datatypes portion of these though.

qkoziol · 2024-03-10T15:04:10Z

I don't know where the DDL definition file for h5dump output has gone, but it probably needs updating.

Looks like the old DDL files are at https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF110.dox and https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF112.dox, but we don't seem to have one for the current release branch. I updated the part in src/H5Tmodule.h that references the datatypes portion of these though.

Good catch, that looks like it might hvae been missed in the doxygenation process. :-)

byrnHDF · 2024-03-10T18:56:10Z

Noone probably thought to update since 1.12 added the references? I was likely the last to work on the DDL specs.

…

________________________________________ From: Quincey Koziol ***@***.***> Sent: Sunday, March 10, 2024 10:04 AM To: HDFGroup/hdf5 Cc: Allen Byrne; Review requested Subject: Re: [HDFGroup/hdf5] Add support for _Float16 16-bit floating point type (PR #4065) I don't know where the DDL definition file for h5dump output has gone, but it probably needs updating. Looks like the old DDL files are at https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF110.dox and https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF112.dox, but we don't seem to have one for the current release branch. I updated the part in src/H5Tmodule.h that references the datatypes portion of these though. Good catch, that looks like it might hvae been missed in the doxygenation process. :-) — Reply to this email directly, view it on GitHub<#4065 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AL77KBS6AANFEK55XMKWACDYXSABDAVCNFSM6AAAAABEGGUTKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGI3DENBVGI>. You are receiving this because your review was requested.Message ID: ***@***.***>

byrnHDF · 2024-03-10T19:00:09Z

If there was no changes for DDL there would not be a need for a 1.14 version. It's more likely that the 1.8 version is the one missing. Allen

…

________________________________________ From: Allen Byrne ***@***.***> Sent: Sunday, March 10, 2024 1:56 PM To: HDFGroup/hdf5; HDFGroup/hdf5 Cc: Review requested Subject: Re: [HDFGroup/hdf5] Add support for _Float16 16-bit floating point type (PR #4065) Noone probably thought to update since 1.12 added the references? I was likely the last to work on the DDL specs.

________________________________________ From: Quincey Koziol ***@***.***> Sent: Sunday, March 10, 2024 10:04 AM To: HDFGroup/hdf5 Cc: Allen Byrne; Review requested Subject: Re: [HDFGroup/hdf5] Add support for _Float16 16-bit floating point type (PR #4065) I don't know where the DDL definition file for h5dump output has gone, but it probably needs updating. Looks like the old DDL files are at https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF110.dox and https://github.com/HDFGroup/hdf5/blob/develop/doxygen/dox/DDLBNF112.dox, but we don't seem to have one for the current release branch. I updated the part in src/H5Tmodule.h that references the datatypes portion of these though. Good catch, that looks like it might hvae been missed in the doxygenation process. :-) — Reply to this email directly, view it on GitHub<#4065 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AL77KBS6AANFEK55XMKWACDYXSABDAVCNFSM6AAAAABEGGUTKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGI3DENBVGI>. You are receiving this because your review was requested.Message ID: ***@***.***>

jhendersonHDF · 2024-03-10T20:22:23Z

I can just go ahead and add a 1.14 DDL file since it will have these new types and eventually complex numbers.

Add 1.14 DDL file

Fix a few warnings in dt_arith.c test when _Float16 isn't available

jhendersonHDF · 2024-03-13T02:45:40Z

I don't know where the DDL definition file for h5dump output has gone, but it probably needs updating.

Added a DDLBNF114.dox file and updated references in the library to point to the new file.

fortnern

Library changes look good to me

Fixed some conversion issues with Clang due to problematic undefined behavior when casting a negative floating-point value to an integer Fixed a bug in the library's software integer to floating-point conversion function where a user's conversion exception function returning H5T_CONV_UNHANDLED in the case of overflows would result in incorrect data after conversion Added configure checks for functions and macros related to _Float16 usage since some compilers expose the datatype but not the functions or macros Fixed a dt_arith test failure when H5_WANT_DCONV_EXCEPTION isn't defined Fixed a few warnings from not explicitly casting some _Float16 variables upwards

* Added new H5E with tests. (#4049) Added Fortran H5E APIs: h5eregister_class_f, h5eunregister_class_f, h5ecreate_msg_f, h5eclose_msg_f h5eget_msg_f, h5epush_f, h5eget_num_f, h5ewalk_f, h5eget_class_name_f, h5eappend_stack_f, h5eget_current_stack_f, h5eset_current_stack_f, h5ecreate_stack_f, h5eclose_stack_f, h5epop_f, h5eprint_f (C h5eprint v2 signature) Addresses Issue #3987 * Don't load toolchain file in CMake workflows (#4077) * Add support for the new MSVC preprocessor (#4078) Microsoft has added a new, standards-conformant preprocessor to MSVC, which can be enabled with /Zc:preprocessor. This preprocessor trips over our HDopen() function-like variadic macro since it uses a hack that only works with the legacy MSVC preprocessor. This fix adds ifdefs to use the correct HDopen() macro depending on the MSVC preprocessor selected. Fixes #2515 * Increased H5FD_ROS3_MAX_SECRET_TOK_LEN to 4096 to accommodate long AWS secret tokens (#4064) ros3: increased H5FD_ROS3_MAX_SECRET_TOK_LEN to 4096; stratified the debugging statements so there is more control over the output * Close and reopen file during dset vlen IO API tests (#4050) - Close/reopen file and file objects to prevent cache from being used instead of actual I/O. - Moved vlen io test datasets under the dset container group instead of the root group - Moved the PASSED() invocation to after individual test cleanup in case an error occurs during H5Treclaim * New option for building with static CRT in Windows (#4062) * addressed compilation errors with gfortran 4.8 (#4082) * Fix bin/trace script w/ out params (#4074) The bin/trace script adds TRACE macros to public API calls in the main C library. This script had a parsing bug that caused functions that were annotated with /*out*/, etc. to be labeled as void pointers instead of typed pointers. This is mainly a developer feature and not visible to consumers of the public API. The bin/trace script now annotates public API calls properly. Fixes GH #3733 * Use H5T_STD_I32LE to create datatype in vds examples (#4070) Fixes issues when VDS examples are tested on BE systems * Remove printf debugging in H5HL code (#4086) * Fixed asserts due to H5Pset_est_link_info() values (#4081) * Fixed asserts due to H5Pset_est_link_info() values If large values for est_num_entries and/or est_name_len were passed to H5Pset_est_link_info(), the library would attempt to create an object header NIL message to reserve enough space to hold the links in compact form (i.e., concatenated), which could exceed allowable object header message size limits and trip asserts in the library. This bug only occurred when using the HDF5 1.8 file format or later and required the product of the two values to be ~64k more than the size of any links written to the group, which would cause the library to write out a too-large NIL spacer message to reserve the space for the unwritten links. The library now inspects the phase change values to see if the dataset is likely to be compact and checks the size to ensure any NIL spacer messages won't be larger than the library allows. Fixes GitHub #1632 * Fix copy-paste comments * update macOS support statement (#4084) * fixes compilation failures due to H5E additions (#4090) * Remove extra whitespaces from nvhpc-cmake action. (#4091) * Remove printf debugging in H5I package (#4088) * Add subfiling for h5dump filedriver option help message (#3878) * Merge HDF4 release workflow changes to hdf5 (#4093) * Update long double test with correct values (#4060) Update long double test with correct values * virtual creates must use the same datatype as the main file (#4092) * Fixed -Wdeprecated-copy-dtor warnings by implementing a copy assignment operator (#3306) Example warning was: warning: definition of implicit copy assignment operator for 'Group' is deprecated because it has a user-declared destructor [-Wdeprecated-copy-dtor] * Expand check for variable-length or reference types when clearing datatype conversion paths (#4085) When clearing out datatype conversion paths involving variable-length or reference datatypes on file close, also check for these datatypes inside compound or array datatypes * Remove H5B debug checks (#4089) The H5B (version 1 B-tree) package would add some computationally expensive integrity checks when H5B_DEBUG was defined. Due to their negative effects on performance, this option was rarely turned on, making the H5B__assert() check function stale, if not dead, code. This change: * Builds H5B__assert() when NDEBUG is not defined (the function relies on assert()) so it gets compiled more often. * Removes some printf debugging statements in the B-tree code * Removes all H5B "extra debug" checks that are leftover from past debugging sessions. Maintainers can add H5B__assert() selectively to perform integrity checks when debugging. * Removes the HDF5_ENABLE_DEBUG_H5B CMake option H5B_DEBUG now has no effect * Fix the last C++ stack size warning (#4099) * Clean up off_t usage (#4095) * Add comments to C++ and Fortran API calls that use off_t * Remove noise casts for small integers * Correct missing change of source path for S3 build (#4100) * Remove leading / from relative path. (#4101) * msvc: don't declare `HAVE_TIMEZONE` for older MSVC (#3956) It was introduced in MSVC 15 (Visual Studio 2017). * Remove a few H5O printf debugging statements (#4096) These were in H5Oint.c, were protected by H5O_DEBUG, and only dumped to stdout if the HDF5_DEBUG environment variable were set to do so. * Remove H5DEBUG() calls from H5Dmpio.c (#4103) Just use stdout when a stream is needed. * Remove printf debugging from H5Smpio.c (#4098) * Change how stats are printed in H5Z (#4097) H5Z used the soon-to-be-removed HDEBUG macro to decide if stats would be dumped and to what stream. This is now handled by a DUMP_DEBUG_STATS_g variable and the output is always sent to stdout. This is an internal change, not normally visible to users. * Refactor datatype conversion code to use pointers rather than IDs (#4104) The datatype conversion code previously used IDs for the source and destination datatypes rather than pointers to the internal structures for those datatypes. This was mostly due to the need for an ID for these datatypes that can be passed to an application-registered datatype conversion function or datatype conversion exception function. However, using IDs internally caused a lot of unnecessary ID lookups and hurt performance of datatype conversions in general. This was especially problematic for compound datatype conversions, where the ID lookups were occuring on every member of every compound element of a dataset. The code has now been refactored to use pointers internally and only create IDs for datatypes when necessary. Fixed a test issue in dt_arith where a library datatype conversion function was being cast to an application conversion function. Since the two have different prototypes, this started failing after the parameters for a library conversion function changed from hid_t to H5T_t * and an extra parameter was added. This appears to have worked coincidentally in the past since the only different between a library conversion function and application conversion function was an extra DXPL parameter at the end of an application conversion function Fixed an issue where memory wasn't being freed in the h5fc_chk_idx test program. Even though the program exits quickly after allocating the memory, it still causes failures when testing with -fsanitize=address * Minimize use of abort() (#4110) The abort() call is used at several places where it probably shouldn't. * Clean up a few things in H5T.c (#4105) * remove (size_t) noise casts * replace (hid_t)FAIL with H5I_INVALID_HID * Convert H5B__assert to use error checks (#4109) Switches assert() calls to HGOTO_ERROR in H5B__assert() so it can be used in production mode. Also renames it to H5B__verify_structure() to better reflect what it checks. * Move common variables out of cache test blocks (#4108) Fixes a stack size warning w/ XCode * Remove lint comments (#4107) * Change compression tests reference files to use masking for compression ratios (#4083) Rework TEST_FILTER tests to handle slightly different compression ratios * Add Doxygen for HDFS VFD (#4106) * Add Doxygen for HDFS VFD * Fix Doxygen warning * Update H5FDhdfs.h * long double tests has problems setting precision with offset (#4102) * long double tests has problems setting precision with offset * Disable long double until more development fixes issues * Fix up dsets test for some platforms with different long double format (#4114) * Adjust aocc workflow to fit the autotools/cmake pattern of other callable workflows. (#4115) * Implement ID creation optimization for container datatype conversions (#4113) Makes the datatype conversion context object available during both the initialization and conversion processes for a datatype conversion function, allowing the compound, variable-length and array datatype conversion functions to avoid creating IDs for the datatypes when they aren't necessary Adds internal H5CX_pushed routine to determine if an API context is available to retrieve values from Also adds error checking to several places in H5T.c and H5Tconv.c where the code had previously assumed object close operations would succeed * Handle IBM long double issues in dsets.c test_floattypes test (#4116) * Handle IBM long double issues in dsets.c test_floattypes test * Remove old cmake check (#4117) * Use AC_SYS_LARGEFILE on Autotools (#4119) We previously used a hack introduced in 1.8.5 to paper over differences in off_t and off64_t when determining the type sizes. We no longer explicitly support off64_t in the library and AC_SYS_LARGEFILE works fine. * Initialize selection type in chunk struct (#4087) * Overhaul CMake LFS support (#4122) Externally visible: * The HDF_ENABLE_LARGE_FILE option (advanced) has been removed * We no longer run a test program to determine if LFS works, which will help with cross-compiling * On Linux we now unilaterally set -D_LARGEFILE_SOURCE and -D_FILE_OFFSET_BITS=64, regardless of 32/64 bit system. CMake doesn't offer a nice equivalent to AC_SYS_LARGEFILE and since those options do nothing on 64-bit systems, this seems safe and covers all our bases. We don't set -D_LARGEFILE64_SOURCE since we don't use any of the POSIX 64-bit specific API calls like ftello64, as noted above. * We didn't test for LFS support on non-Linux platforms. We've added comments for how LFS should probably be supported on AIX and Solaris, which seem to be alive, though uncommon. PRs would be appreciated if anyone wishes to test this. Internal: * Drops off64_t size checks since this is unused (as in Autotools) * Remove HDF_EXTRA_FLAGS, which is now unused * Remove hack around deprecated LINUX_LFS Fixes #2395 * Update CMake comment about _POSIX_C_SOURCE (#4124) Was missng the 2008 pread/write info * Deprecate bin/cmakehdf5 (#4127) * Deprecate bin/cmakehdf5 * Add reference text * Don't set the rpath when linking statically (#4125) * Remove invalid compile flag (#4129) * Fix segfault in vlen io API test (#4130) * Update URLs in RELEASE.txt (#4132) * Add cygwin CI and update yaml files for consistency and accuracy (#4131) * Add cygwin CI * add cygwin packages * Correct option names * Cleanup yaml file and synch look and feel * Synch CI look and feel and correct path issues * Upgrade oneapi version * pwsh needs env: for vars * No continuation char for pwsh * restore correct pwsh step * Run subset of tests for cygwin workflow * Remove space chars in regex * restore full tests * Remove ros3 and hdfs VFDs from Autotools VFD list (#4142) These will never pass `make check` and would require a custom test suite for more comprehensive testing. * Skip part of dsets.c test for IBM long double type (#4136) * Capitalize option message for consistency. (#4141) * Fixed misc. H5E fortran failures due to previous PR (#4138) * fixed promotion of integers and reals tests and check-passthrough-vol failure * fixed cygwin issue * Fix Autotools -Werror cleanup (#4144) The Autotools temporarily scrub -Werror(=whatever) from CFLAGS, etc. so configure checks don't trip over warnings generated by configure check programs. The sed line originally only scrubbed -Werror but not -Werror=something, which would cause errors when the '=something' was left behind in CFLAGS. The sed line has been updated to handle -Werror=something lines. Fixes one issue raised in #3872 * Fix doxygen link to example function usage (#4133) * Remove useless headers (#4145) Removes unnecessary headers from C library source files. * Clean up some hbool_t/TRUE/FALSE stragglers (#4143) It looks like most of these snuck in via selection I/O work * Fix error when overwriting an indirectly nested vlen with a shorter sequence (#4140) * defined CMAKE_H5_HAVE_DARWIN (#4146) * Make the newsletter scheme work like HDF4 (#4149) * Remove at the end of list item. (#4151) * Fix buffer size calculation in the deflate filter (#4147) * Remove H5O header and friend status from H5A.c (#4150) * Remove HDF from Fortran 2003 configuration check message. (#4157) * Suppress H5Dmpio debugging output unless HDF5_DEBUG=d is set (#4155) * Header cleanup in C library (#4154) * Ensure H5FL header is included everywhere * Ensure H5SL header is included everywhere * Ensure H5MM header is included everywhere * Add Doxygen to H5FDmirror.h (#4158) * Remove lseek64 and stat64 symbols from CMake (#4163) We don't use these in the library. * Remove HAVE_IOEO checks from CMake (#4160) This was intended to check for thread-safety functionality on Windows. The required functionality has been standard since Windows Vista, so these checks can be removed. * Fix some minor warnings (#4165) * Bump the size of the mirror VFD IP field (#4167) The IP address string isn't big enought to hold an IPv4-mapped IPv6 address. * Fix mirror VFD script (#4170) This had directory problems when running locally. * Fix an issue where the Subfiling VFD's context cache grows too large (#4159) * Address code page issues w/ Windows file paths (#4172) On Windows, HDF5 attempted to convert file paths passed to open() and remove() to UTF-16 in order to handle Unicode file paths. This scheme does not work when the system uses code pages to handle non-ASCII file names. As suggested in the forum post below, we now also try to see if we can open the file with open(), which should handle systems where non-ASCII code pages are in use. https://forum.hdfgroup.org/t/open-create-hdf5-files-with-non-utf8-chars-such-as-shift-jis/11785 * Add Doxygen to API calls in H5VLnative.h (#4173) * Allow H5Soffset_simple to accept NULL offsets (#4152) The reference manual states that the offset parameter of H5Soffset_simple() can be set to NULL to reset the offset of a simple dataspace to 0. This has never been true, and passing NULL was regarded as an error. The library will now accept NULL for the offset parameter and will correctly set the offset to zero. Fixes HDFFV-9299 * Add filter plugin user guide text. Fix registered URL in docs (#4169) * Add support for _Float16 16-bit floating point type (#4065) Fixed some conversion issues with Clang due to problematic undefined behavior when casting a negative floating-point value to an integer Fixed a bug in the library's software integer to floating-point conversion function where a user's conversion exception function returning H5T_CONV_UNHANDLED in the case of overflows would result in incorrect data after conversion Added configure checks for functions and macros related to _Float16 usage since some compilers expose the datatype but not the functions or macros Fixed a dt_arith test failure when H5_WANT_DCONV_EXCEPTION isn't defined Fixed a few warnings from not explicitly casting some _Float16 variables upwards * Remove some H5T_copy calls that are now unnecessary (#4164) Removes some datatype copying calls that are now unnecessary after refactoring the datatype conversion code to use pointers internally rather than IDs Rewrites the enum conversion function so that it uses cached copies of the source and destination datatypes in order to avoid modifying the datatypes passed in Adds a 'recursive' field to the datatype conversion context which allows the conversion functions for members of a container datatype to skip unnecessary repetitive conversion setup code Changes internal datatype conversion callback functions so that the source and destination datatype structure pointers are const Removes some unused and unnecessary internal IDs registered with H5I_register * Add RELEASE.txt note for cmpd segfault fix (#4175) RELEASE notice for the fix in #3842 * Clean up CMake direct VFD handling (#4161) There's no need to build and run programs, or even check the operating system. We just need to check for O_DIRECT and posix_memalign(). * Capitalize linux for consistency (#4178) * Reworked H5Epush_f (#4153) * Add const to new _Float16 conversion routine parameters (#4181) * Update Release Specific Information link. (#4179) * Filter plugins updates for registration URL (#4180) * Update filter plugin URL to new location * Adjust test array size * Add daily VFD CI workflow (#4176) Adds testing of Subfiling VFD * Exclude shell tests from sanitizers (#4186) * Add a missing period at the end of sentence. (#4184) * last-file.txt should not be created for release workflow (#4185) * Skip part of dtypes.c _Float16 file size check for certain VFDs (#4182) * Fixes several MinGW + Autotools issues (#4190) * Fixes detection of various Windows libraries, etc. * Corrects alarm(2) configure checks * Uses Win32 threads by default w/ Pthreads override, if desired * Set _WIN32_WINNT correctly for MinGW * Fix setenv(3) wrapper for MinGW, which does not have getenv_s() MinGW Autotools support is still not Amazing, but this at least allows the library and tools build and is better about thread-safety * Add semicolons to the end of HSYS_GOTO_ERROR (#4193) Looks like we forgot these when we did the other macros. * Remove broken links (#4187) * Skip vlen IO API test for cache VOL (#4135) * Fix cache VOL segfault in vlen io test * Skip vlen IO API test * Handle certain empty subfiling environment variables (#4038) * h5diff compares attribute data like dataset data (#4191) Updates tools docs to indicate that dataset and attribute data are compared in the same way * A path component may include a dot with other characters (#4192) * Add RELEASE.txt note for recent datatype conversion improvements (#4195) * Add NEWSLETTER item about _Float16 support (#4197) * Correct download link for develop doxygen (#4196) * Update version in new .yml files.

jhendersonHDF added 6 commits February 29, 2024 17:14

Add support for _Float16 16-bit floating point type

a793f30

Add test program to see if _Float16 really works

a209e23

Work around issues with MacOS 13

10c1c55

Add CMake status message about _Float16 test-compile program

Add RELEASE.txt note for _Float16 support

51e1cf5

Merge branch 'develop' into feature/new_datatypes

a846ae5

jhendersonHDF commented Mar 5, 2024

View reviewed changes

jhendersonHDF added this to the 1.14.4 milestone Mar 5, 2024

Update configure checks to account for problematic compilers

6295627

Clarify a few things in the RELEASE.txt entry

derobins marked this pull request as ready for review March 8, 2024 22:05

derobins requested review from byrnHDF, derobins, lrknox, fortnern, qkoziol and vchoi-hdfgroup as code owners March 8, 2024 22:05

derobins added Merge - To 1.14 and removed Merge Use this label when a PR is for a downstream merge labels Mar 9, 2024

qkoziol reviewed Mar 10, 2024

View reviewed changes

hyoklee assigned ajelenak Mar 11, 2024

jhendersonHDF added 5 commits March 12, 2024 18:44

Merge branch 'develop' into feature/new_datatypes

953951b

Address comments from review

982b7ea

Add 1.14 DDL file

Post-merge fixes

15a1fb8

Add _Float16 h5dump XML test

16826a8

Fix a few warnings in dt_arith.c test when _Float16 isn't available

Add big-endian _Float16 tools tests and files

4320d92

derobins and others added 3 commits March 15, 2024 20:42

Merge branch 'develop' into feature/new_datatypes

844172a

Merge branch 'develop' into feature/new_datatypes

36cc25b

Remove duplicate occurence of WORDS_BIGENDIAN in testh5ls.sh.in

78c9d01

fortnern approved these changes Mar 18, 2024

View reviewed changes

derobins approved these changes Mar 19, 2024

View reviewed changes

derobins merged commit 330b80a into develop Mar 19, 2024
102 checks passed

jhendersonHDF mentioned this pull request Mar 19, 2024

Add const to new _Float16 conversion routine parameters #4181

Merged

jhendersonHDF deleted the feature/new_datatypes branch March 19, 2024 22:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for _Float16 16-bit floating point type #4065

Add support for _Float16 16-bit floating point type #4065

jhendersonHDF commented Mar 5, 2024

jhendersonHDF Mar 5, 2024

jhendersonHDF Mar 5, 2024

jhendersonHDF Mar 5, 2024 •

edited

Loading

jhendersonHDF Mar 5, 2024

fortnern Mar 18, 2024

jhendersonHDF Mar 5, 2024 •

edited

Loading

jhendersonHDF commented Mar 9, 2024

qkoziol Mar 10, 2024

jhendersonHDF Mar 10, 2024

jhendersonHDF Mar 13, 2024 •

edited

Loading

qkoziol Mar 10, 2024

jhendersonHDF Mar 10, 2024

jhendersonHDF Mar 13, 2024

qkoziol commented Mar 10, 2024

qkoziol commented Mar 10, 2024

jhendersonHDF commented Mar 10, 2024 •

edited

Loading

qkoziol commented Mar 10, 2024

byrnHDF commented Mar 10, 2024 via email

byrnHDF commented Mar 10, 2024 via email

jhendersonHDF commented Mar 10, 2024

jhendersonHDF commented Mar 13, 2024

fortnern left a comment

Add support for _Float16 16-bit floating point type #4065

Add support for _Float16 16-bit floating point type #4065

Conversation

jhendersonHDF commented Mar 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhendersonHDF Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhendersonHDF Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

jhendersonHDF commented Mar 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhendersonHDF Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qkoziol commented Mar 10, 2024

qkoziol commented Mar 10, 2024

jhendersonHDF commented Mar 10, 2024 • edited Loading

qkoziol commented Mar 10, 2024

byrnHDF commented Mar 10, 2024 via email

byrnHDF commented Mar 10, 2024 via email

jhendersonHDF commented Mar 10, 2024

jhendersonHDF commented Mar 13, 2024

fortnern left a comment

Choose a reason for hiding this comment

jhendersonHDF Mar 5, 2024 •

edited

Loading

jhendersonHDF Mar 5, 2024 •

edited

Loading

jhendersonHDF Mar 13, 2024 •

edited

Loading

jhendersonHDF commented Mar 10, 2024 •

edited

Loading