Skip to content

Commit

Permalink
Hdf5 merge issue 488 v112 (#534)
Browse files Browse the repository at this point in the history
* Snapshot version 1.12 release 1-3.  Update  version to 1.12.1-4.

* First cut of the H5 public API documentation. (#80)

* First cut of the H5 public API documentation.

* Added H5Z "bonus track."

* Applied Quincey's patch.

* Added the missing patches from Quincey's original patch.

* H5PL (complete) and basic H5VL API documentation.

* Added H5I API docs.

* Added H5L API docs.

* First installment from Elena's H5T batch.

* Second installment of Elena's H5T batch.

* Final installment of Elena's H5T batch.

* Full set of current H5F documentation. (#105)

* First cut of the H5 public API documentation.

* Added H5Z "bonus track."

* Applied Quincey's patch.

* Added the missing patches from Quincey's original patch.

* H5PL (complete) and basic H5VL API documentation.

* Added H5I API docs.

* Added H5L API docs.

* First installment from Elena's H5T batch.

* Second installment of Elena's H5T batch.

* Final installment of Elena's H5T batch.

* Migrated documentation for SWMR functions.

* Catching up on MDC functions.

* Integrated the H5F MDC function documentation.

* Added MDC and parallel H5F functions.

* Slightly updated main page.

* Added doxygen/dox/H5AC_cache_config_t.dox to MANIFEST.

* Doxygen - added (mostly) beginner functions (#112)

* Doxygen - added (mostly) beginner functions

* Removed duplicate H5Pset_szip function

* Add src/H5module.h to MANIFEST.

* close #195. (#196)

* Update HDF5PluginMacros.cmake

* Update HDF5PluginMacros.cmake

* Avoid aligned access for references by decoding into temporary buffer and then copying the result into the actual buffer.   Update test to be more thorough with using compound datatype fields everywhere. (#206)

* Modify temporary rpath for testing in java example scripts. (#230)

* Fix undefined left shifting of negative numbers (#338)

Undefined Bahavior Sanitizer errored here about left shifting negative numbers.

* Fixes various warnings noticed on Windows (#425)

* Fixes various warnings noticed on Windows

- Adds a prototype for our implementation of vasprintf
- Return type of H5_get_utf16_str() is now non-const
- Fixes possible uninitialized return type in Wremove_utf8
- Better isolation of fork() code in accum.c:test_swmr_write_big()
- Better isolation of non-zlib code in dsets.c:test_filter_delete()
- Removed unused variable in trefer.c:test_reference_cmpnd_obj()

* Fixes clang-format issues

* Applied clang-tidy readability-non-const-parameter warning fixes auto… (#429)

* Automatically applied clang-tidy readability-avoid-const-params-in-decls fixes

Removes useless const declarations.

* Fixed most readability-non-const-parameter warnings

These changes were made automatically by clang-tidy, but I manually reverted the changes related to the H5Z_func_t signature.

* Reformat source with clang v10.0.1.

Co-authored-by: Larry Knox <lrknox@hdfgroup.org>

* Added C++11 override keyword where appropriate (#433)

Added H5_OVERRIDE macro for compatibility with both C++11 and older.

* Various clang tidy warning fixes (#448)

* Fixed clang-tidy bugprone-reserved-identifier warnings

* Fixed clang-tidy bugprone-assert-side-effect warnings

* Fixed clang-tidy bugprone-copy-constructor-init warning

* Fixed clang-tidy readability-redundant-preprocessor warning

For error_test.c the removed code was already dead, because it was in the else of an `#if H5_USE_16_API` block.

Based on H5Location.h, I think p_get_ref_obj_type was meant to be in `#ifndef DOXYGEN_SHOULD_SKIP_THIS` and an `#endif` was missing.  Similarly, in the header, getObjTypeByIdx is only in H5_NO_DEPRECATED_SYMBOLS, not DOXYGEN_SHOULD_SKIP_THIS.

* Fixed clang-tidy readability-redundant-string-init warnings

* Fixed some clang-tidy performance-type-promotion-in-math-fn warnings

* Fixed clang-tidy performance-unnecessary-value-param warnings

* Reformat source with clang v10.0.1.

Co-authored-by: Larry Knox <lrknox@hdfgroup.org>

* Removed checks/workarounds for pre-C++89 compatibility (#449)

After 30+ years, just assume that the following exist:
- extension-less includes
- namespaces
- std::
- static_cast
- bool

* Fixed all clang-tidy bugprone-suspicious-string-compare warnings (#451)

* Fixed all clang-tidy bugprone-suspicious-string-compare warnings

This change was generated entirely by clang-tidy itself.

* Reformat code with clang v10.0.1.

Co-authored-by: Larry Knox <lrknox@hdfgroup.org>

* Remove 2 functions incorrectly merged from develop in a cherry-pick merge of PR #451.

* Purge the buffer used in type conversion. (#263)

Some of the uniniitialized bits in the buffer may get carried through
all the way to disk, creating a risk for leaks.

We observed an msan error during the floating point output conversion.
Due to the encoding certain bits could remain untouched during the conversion.

In this draft we zero initialize the dbuf used by every convertor.

* Fixed HDFFV-10480 (CVE-2018-11206) and HDFFV-11159 (CVE-2018-14033) (#405)

* Fixed HDFFV-10480 (CVE-2018-11206) and HDFFV-11159 (CVE-2018-14033)
Description
    Checked against buffer size to prevent segfault, in case of data corruption.

    + HDFFV-11159 CVE-2018-14033 Buffer over-read in H5O_layout_decode
    + HDFFV-10480 CVE-2018-11206 Buffer over-read in H5O_fill_new[/old]_decode
Platforms tested:
    Linux/64 (jelly)

* Accidentally left in another occurrence of the previous patch from user
   after a more correct fix was applied, that is the check now accounted
   for the previous advance of the buffer pointer.  Removed it.

* Typo

* Fixed format issues.

* Added test.

* Changed arguments to ADD_H5_TEST

* Fixing arguments to ADD_H5_TEST again.

* Fixing arguments again.

* Took out the CMake changes until Allen can help.

* Added files:

tCVE_2018_11206_fill_old.h5
tCVE_2018_11206_fill_new.h5

* Revert "Took out the CMake changes until Allen can help."

This reverts commit c21324d.

* Revert "Fixing arguments again."

This reverts commit 5832a70.

* Revert "Fixing arguments to ADD_H5_TEST again."

This reverts commit b45de82.

* Revert "Changed arguments to ADD_H5_TEST"

This reverts commit 1671982.

* Added first argument to ADD_H5_TEST for HDFFV-10480 fix.

* Changed argument 0 to 1

* Revert "Changed argument 0 to 1"

This reverts commit b343d66.

* Revert "Added first argument to ADD_H5_TEST for HDFFV-10480 fix."

This reverts commit b8a0f9a.

* Added first argument and corrected the second.

* Updated fixes for HDFFV-10480 and HDFFV-11159/HDFFV-11049

* Improved error messages.

* Added description of the current HDF5 branches; added a draft of contribution policy. (#445)

* Added description of the current HDF5 branches.

* Removed capitalization in from Develop, Release, Feature to reflect the real naming schema

* Added a draft of contributions guidance document.

* Fixed typos.

* Fixed section title.

* Fixed typo.

* Fixed typos and formatting.

* Fixed many typos and simplified the text (e.g., removed testing instructions and left pointers to where to find them, etc.)

* Fixed a typo.
 Please enter the commit message for your changes. Lines starting

* Added contributing.md file and rearranged doc entries in alphabetical order.

* ddressed Gerd's review comments; found and fixed more typos.

* Addressed comments from Larry and Scot.

* Fix CMake error message location. (#478)

Print error message if Perl is not found.

* Committing clang-format changes

Co-authored-by: Gerd Heber <gheber@hdfgroup.org>
Co-authored-by: bljhdf <58825073+bljhdf@users.noreply.github.com>
Co-authored-by: H. Joe Lee <hyoklee@hdfgroup.org>
Co-authored-by: Quincey Koziol <quincey@koziol.cc>
Co-authored-by: Sean McBride <sean@rogue-research.com>
Co-authored-by: Dana Robinson <43805+derobins@users.noreply.github.com>
Co-authored-by: Yu Feng <rainwoodman@gmail.com>
Co-authored-by: bmribler <39579120+bmribler@users.noreply.github.com>
Co-authored-by: epourmal <epourmal@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
  • Loading branch information
11 people authored Apr 1, 2021
1 parent c38dbaf commit 5cba1c8
Show file tree
Hide file tree
Showing 12 changed files with 241 additions and 32 deletions.
4 changes: 4 additions & 0 deletions MANIFEST
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,9 @@

./config/site-specific/BlankForm

./doc/branches-explained.md
./doc/code-conventions.md
./doc/contributing.md

./doxygen/aliases
./doxygen/Doxyfile.in
Expand Down Expand Up @@ -2120,6 +2122,8 @@
./tools/testfiles/twithddl.exp
./tools/testfiles/twithddlfile.ddl
./tools/testfiles/twithddlfile.exp
./tools/testfiles/tCVE_2018_11206_fill_old.h5
./tools/testfiles/tCVE_2018_11206_fill_new.h5

# h5dump test error files
./tools/test/h5dump/errfiles/filter_fail.err
Expand Down
41 changes: 41 additions & 0 deletions doc/branches-explained.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# HDF5 Git Branching Model Explained

This document describes current HDF5 branches.

Branches are tested nightly and testing results are available at https://cdash-internal.hdfgroup.org/ and https://cdash.hdfgroup.org/.
Commits that break daily testing should be fixed by 3:00 pm Central time or reverted.
We encourage code contributors to check the status of their commits. If you have any questions, please contact help@hdfgroup.org.

## `develop`
Develop is the main branch whose source code always reflects a state with the latest delivered development changes for the next major release of HDF5.
This is also considered the integration branch, as **all** new features are integrated into this branch from respective feature branches.

## `Maintenance branches`

Each currently supported release-line of HDF5 (e.g. 1.8.x, 1.10.x, 1.12.x) has a support branch with the name 1_8, 1_10, 1_12.
Maintenance branches are similar to the develop branch, except the source code in a maintenance branch always reflects a state
with the latest delivered development changes for the next **maintenance** release of that particular supported release-line of HDF5.
**Some** new features will be integrated into a release maintenance branch, depending on whether or not those features can be
introduced in minor releases. Maintenance branches are removed when a release-line is retired from support.

## `feature/*`
Feature branches are temporary branches used to develop new features in HDF5.
Feature branches branch off of develop and exist as long as the feature is under development.
When the feature is complete, the branch is merged back into develop, as well as into any support branches in which the change will be included, and then the feature branch is removed.

## `release/*`
Release branches are used to prepare a new production release. They are primarily used to allow for last minute dotting of i's and crossing of t's
(things like setting the release version, finalizing release notes, et cetera) and do not include new development.
They are created from the maintenance branch at the time of the maintenance release and have
names 1_8_N, 1_10_N, 1_12_N, where N is the minor release number. Once the release is done it is tagged.
Patches can be applied to the release branch for patch releases that are treated as "scaled down" maintenance releases as defined by Release coordinator.

## `1.X/master/*` where X is 8, 10 or 12
These branches are used to tag 1.X.* maintenance releases.

## `inactive/<name>/*`
These branches are for experimental features that were developed in the past and have not been merged to develop, and are not under active development. The features
can be out of sync with the develop branch.

This document was last updated on March 16, 2021

87 changes: 87 additions & 0 deletions doc/contributing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# How to contribute to HDF5 (Draft)

The HDF Group encourages community members to contribute to the HDF5 project. We accept and are very grateful for any type of contributions
from small typos and bug fixes to new features. The HDF Group is committed to work with the code contributors and make contribution process simple and enjoyable.

This document describes guiding principles for the HDF5 code contributors and does not pretend to address any possible
contribution. If in doubt, please do not hesitate to ask us for guidance.
***Note that no contribution may be accepted unless the donor agrees with the HDF Group software license terms
found in the COPYING file in the top source directory of every branch.***


> We will assume that you are familiar with `git` and `GitHub`. If not, you may go through the GitHub tutorial found at [https://guides.github.com/activities/hello-world/](https://guides.github.com/activities/hello-world/). This tutorial should only take around 10 minutes.
## Table of Contents

* [Workflow](#workflow)
* [Acceptance criteria for pull request](#criteria)
* [Check List](#checklist)

# Workflow <A NAME="workflow"></A>

The process for contributing code to HDF5 is as follows:

* Open an issue on [HDF5 GitHub](https://github.com/HDFGroup/hdf5/issues).

> This step is ***required*** unless the change is minor (e.g., typo fix).
* Fork the [HDF5](https://github.com/HDFGroup/hdf5) repository.
* Make the desired changes to the HDF5 software.
* New features should always go to develop branch first and later should be merged to the appropriate maintenance branches.
* Bug fixes should go to all appropriate branches (develop and maintenance).
* Build and test your changes. Detailed instructions on how to build and test HDF5 can be found in the `INSTALL*` files in the `release_docs` directory.
* Push your changes to GitHub.
* Issue a pull request and address any code formatting and testing issues reported.

Once a pull request is correctly formatted and passes **ALL** CI tests, it will be reviewed and evaluated by The HDF Group developers and HDF5 community members who can approve pull requests..
The HDF Group developers will work with you to assure that the pull request satisfies acceptance criteria described in the next section.

# Acceptance criteria for pull request <A NAME="criteria"></A>

We appreciate every contribution we receive, but we may not accept them all. Those that we *do* accept satisfy the following criteria:

* **The pull request has a clear purpose** - What does the pull request address? How does it benefit the HDF5 community?
If the pull request does not have a clear purpose and benefits it will not be accepted.

* **The pull request is documented** - The HDF5 developers must understand not only *what* a change is doing, but *how* it is doing it.
Documenting the code makes it easier for us to understand your patch and will help to maintaine the code in the future.

* **The pull request passes HDF5 regression testing** - Any issue fixed or functionality added should be accompanied by the corresponding
tests and pass HDF5 regression testing run by The HDF Group. We do not expect you to perform comprehensive testing across multiple platforms
before we accept the pull request. If the pull request does not pass regression testing after the merge, The HDF Group developers will i
work with you on the fixes.

* **The pull request does not compromise the principles behind HDF5** - HDF5 has a 100% commitment to backward compatibility.
* Any file ever created with HDF5 must be readable by any future version of HDF5.
If the purpose of your patch is to modify HDF5 data model or file format,
**please** discuss this with us first. File format changes and features required by those changes can be introduced only in a new major release.
* HDF5 has a commitment to remaining *machine-independent*; data created on one platform/environment/architecture **must** remain readable by HDF5 on any other.
* For binary compatibility no changes are allowed to public APIs and data structures in the maintenance releases; new APIs can be added.

* **New features are documented** - Any new features should have proper documentation; talk to us if you have any questions.


# Checklist <A NAME="checklist"></A>

Please make sure that you check the items applicable to your pull request:

* Code
* [ ] Does the pull request have a corresponding GitHub issue and clear purpose?
* [ ] Does the pull request follow HDF5 best practices (naming conventions, code portability, code structure, etc.)? <<TODO: link to the document>>
* [ ] If changes were done to autotools build were they added to CMake and vice versa?
* [ ] Is the pull request applicable to any other branches? If yes, which ones? Please document it in the GitHub issue.
* [ ] Is the new code sufficiently documented for future maintenance?
* [ ] Does the new feature require a change to an existing API? See "API Compatibility Macros" document (https://portal.hdfgroup.org/display/HDF5/API+Compatibility+Macros)
* Documentation
* [ ] Was the change described in the release_docs/RELEASE.txt file?
* [ ] Was MANIFEST updated if new files had been added to the source?
* [ ] Was the new function documented in the corresponding public header file using Doxygen? <<TODO: link to Doxygen instructions>>
* [ ] Was new functionality documented for the HDF5 community (the level of documentation depends on the feature; ask us what would be appropriate)
* Testing
* [ ] Does the pull request have tests?
* [ ] Does the pull request affect HDF5 library perfromance?

We want as many contributions as we can get, and we are here to help. Feel free to reach out to us if you have any questions

Thank you for your contribution!

20 changes: 20 additions & 0 deletions release_docs/RELEASE.txt
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,26 @@ Bug Fixes since HDF5-1.12.0 release
===================================
Library
-------
- Fixed CVE-2018-17435

The tool h5dump produced a segfault when the size of a fill value
message was corrupted and caused a buffer overflow.

The problem was fixed by verifying the fill value's size
against the buffer size before attempting to access the buffer.

(BMR - 2021/03/15, HDFFV-10480)

- Fixed CVE-2018-14033 (same issue as CVE-2020-10811)

The tool h5dump produced a segfault when the storage size message
was corrupted and caused a buffer overflow.

The problem was fixed by verifying the storage size against the
buffer size before attempting to access the buffer.

(BMR - 2021/03/15, HDFFV-11159/HDFFV-11049)

- Remove underscores on header file guards

Header file guards used a variety of underscores at the beginning of the define.
Expand Down
27 changes: 17 additions & 10 deletions src/H5Ofill.c
Original file line number Diff line number Diff line change
Expand Up @@ -195,8 +195,9 @@ H5O__fill_new_decode(H5F_t H5_ATTR_UNUSED *f, H5O_t H5_ATTR_UNUSED *open_oh,
unsigned H5_ATTR_UNUSED mesg_flags, unsigned H5_ATTR_UNUSED *ioflags, size_t p_size,
const uint8_t *p)
{
H5O_fill_t *fill = NULL;
void * ret_value = NULL; /* Return value */
H5O_fill_t * fill = NULL;
const uint8_t *p_end = p + p_size - 1; /* End of the p buffer */
void * ret_value = NULL; /* Return value */

FUNC_ENTER_STATIC

Expand Down Expand Up @@ -227,8 +228,11 @@ H5O__fill_new_decode(H5F_t H5_ATTR_UNUSED *f, H5O_t H5_ATTR_UNUSED *open_oh,
INT32DECODE(p, fill->size);
if (fill->size > 0) {
H5_CHECK_OVERFLOW(fill->size, ssize_t, size_t);
if ((size_t)fill->size > p_size)
HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "destination buffer too small")

/* Ensure that fill size doesn't exceed buffer size, due to possible data corruption */
if (p + fill->size - 1 > p_end)
HGOTO_ERROR(H5E_OHDR, H5E_OVERFLOW, NULL, "fill size exceeds buffer size")

if (NULL == (fill->buf = H5MM_malloc((size_t)fill->size)))
HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "memory allocation failed for fill value")
H5MM_memcpy(fill->buf, p, (size_t)fill->size);
Expand Down Expand Up @@ -310,10 +314,11 @@ static void *
H5O__fill_old_decode(H5F_t *f, H5O_t *open_oh, unsigned H5_ATTR_UNUSED mesg_flags,
unsigned H5_ATTR_UNUSED *ioflags, size_t p_size, const uint8_t *p)
{
H5O_fill_t *fill = NULL; /* Decoded fill value message */
htri_t exists = FALSE;
H5T_t * dt = NULL;
void * ret_value = NULL; /* Return value */
H5O_fill_t * fill = NULL; /* Decoded fill value message */
htri_t exists = FALSE;
H5T_t * dt = NULL;
const uint8_t *p_end = p + p_size - 1; /* End of the p buffer */
void * ret_value = NULL; /* Return value */

FUNC_ENTER_STATIC

Expand All @@ -334,8 +339,10 @@ H5O__fill_old_decode(H5F_t *f, H5O_t *open_oh, unsigned H5_ATTR_UNUSED mesg_flag
/* Only decode the fill value itself if there is one */
if (fill->size > 0) {
H5_CHECK_OVERFLOW(fill->size, ssize_t, size_t);
if ((size_t)fill->size > p_size)
HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL, "destination buffer too small")

/* Ensure that fill size doesn't exceed buffer size, due to possible data corruption */
if (p + fill->size - 1 > p_end)
HGOTO_ERROR(H5E_OHDR, H5E_OVERFLOW, NULL, "fill size exceeds buffer size")

/* Get the datatype message */
if ((exists = H5O_msg_exists_oh(open_oh, H5O_DTYPE_ID)) < 0)
Expand Down
29 changes: 19 additions & 10 deletions src/H5Olayout.c
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
* Purpose: Messages related to data layout.
*/

#define H5D_FRIEND /*suppress error about including H5Dpkg */
#define H5D_FRIEND /*suppress error about including H5Dpkg */
#include "H5Omodule.h" /* This source code file is part of the H5O module */

#include "H5private.h" /* Generic Functions */
Expand Down Expand Up @@ -90,12 +90,13 @@ H5FL_DEFINE(H5O_layout_t);
*/
static void *
H5O__layout_decode(H5F_t *f, H5O_t H5_ATTR_UNUSED *open_oh, unsigned H5_ATTR_UNUSED mesg_flags,
unsigned H5_ATTR_UNUSED *ioflags, size_t H5_ATTR_UNUSED p_size, const uint8_t *p)
unsigned H5_ATTR_UNUSED *ioflags, size_t p_size, const uint8_t *p)
{
H5O_layout_t *mesg = NULL;
uint8_t * heap_block = NULL;
unsigned u;
void * ret_value = NULL; /* Return value */
H5O_layout_t * mesg = NULL;
uint8_t * heap_block = NULL;
unsigned u;
const uint8_t *p_end = p + p_size - 1; /* End of the p buffer */
void * ret_value = NULL; /* Return value */

FUNC_ENTER_STATIC

Expand Down Expand Up @@ -179,6 +180,10 @@ H5O__layout_decode(H5F_t *f, H5O_t H5_ATTR_UNUSED *open_oh, unsigned H5_ATTR_UNU
if (mesg->type == H5D_COMPACT) {
UINT32DECODE(p, mesg->storage.u.compact.size);
if (mesg->storage.u.compact.size > 0) {
/* Ensure that size doesn't exceed buffer size, due to possible data corruption */
if (p + mesg->storage.u.compact.size - 1 > p_end)
HGOTO_ERROR(H5E_OHDR, H5E_OVERFLOW, NULL, "storage size exceeds buffer size")

if (NULL == (mesg->storage.u.compact.buf = H5MM_malloc(mesg->storage.u.compact.size)))
HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, NULL,
"memory allocation failed for compact data buffer")
Expand All @@ -198,6 +203,10 @@ H5O__layout_decode(H5F_t *f, H5O_t H5_ATTR_UNUSED *open_oh, unsigned H5_ATTR_UNU
UINT16DECODE(p, mesg->storage.u.compact.size);

if (mesg->storage.u.compact.size > 0) {
/* Ensure that size doesn't exceed buffer size, due to possible data corruption */
if (p + mesg->storage.u.compact.size - 1 > p_end)
HGOTO_ERROR(H5E_OHDR, H5E_OVERFLOW, NULL, "storage size exceeds buffer size")

/* Allocate space for compact data */
if (NULL == (mesg->storage.u.compact.buf = H5MM_malloc(mesg->storage.u.compact.size)))
HGOTO_ERROR(H5E_OHDR, H5E_CANTALLOC, NULL,
Expand Down Expand Up @@ -887,13 +896,13 @@ H5O__layout_reset(void *_mesg)
} /* end H5O__layout_reset() */

/*-------------------------------------------------------------------------
* Function: H5O__layout_free
* Function: H5O__layout_free
*
* Purpose: Free's the message
* Purpose: Free's the message
*
* Return: Non-negative on success/Negative on failure
* Return: Non-negative on success/Negative on failure
*
* Programmer: Quincey Koziol
* Programmer: Quincey Koziol
* Saturday, March 11, 2000
*
*-------------------------------------------------------------------------
Expand Down
Loading

0 comments on commit 5cba1c8

Please sign in to comment.