Merge dev into main for release 2.0 #806
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Add a new broadcast RPC that requests all servers to return metadata for the files they own. This is just the initial commit and RPC is not fully functional yet.
Continuing the work on the metaget_all broadcast RPC. It's now functional, but there's still a handful of "ToDo" comments that need to be addressed.
Fix up the error handling and remove TODO comments from process_metaget_bcast_rpc().
Implement proper error handling for some of the functions related to the metaget_all_bcast RPC. Also fix a few bugs found during testing.
We weren't deleting the child_attr_list array after we were done with it.
Move the tree-walk code out of the process_metaget_bcast_rpc function and into its own function in unifyfs_inode.c Two reasons: 1) we want to keep all the inode manipulation code in one place for convenience and 2) it'll make it easier to re-use that code.
We decided "Exit" was too generic for a label, so now it's called "Exit_Invoke_BMA".
Change unifyfs_invoke_broadcast_metaget_all() to also walk the local tree and include file metadata with the results returned from the RPC call.
Fix a case where we could incorrectly return ENOMEM even though the malloc() succeeded. Also cleaned up several comments.
Signed-off-by: Chen Wang <wangvsa@gmail.com>
…pose Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Fix a warning about a write() call having too small a buffer for the number of bytes it was supposed to write. Also replaced a pair of literal numbers that were used repeatedly with #defines. Resolves issue LLNL#782
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Also: * fix premature heartbeat rpc attempt
This adds batching of client extent sync requests at each server to try to limit the number of RPCs to the owner to one-per-server, rather than one-per-client. It also avoids doing metaget RPCs for the mountpoint. Also: * use ABT_rwlock instead of pthread_rwlock_t * add comparison functions for int/float types to common * fix misused return code * fix free() of uninitialized margo bulk buf pointers * remove last vestiges of MPI in server * remove last vestiges of MDHIM in server
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Newer versions of the OpenSSL library have deprecated the MD5() API function in favor of the EVP_* functions. This commit replaces MD5() with EVP_Digest(). Note: The new function generates the same hash values as the old function.
Add a check in the configure script to make sure the OpenSSL library does have the EVP_Digest() function that we now need.
Newer versions of the OpenSSL library have deprecated the MD5 API functions in favor of the EVP_* functions. This commit is essentially the same as c9516ac, but for the unifyfs-stage util.
Read the Docs has moved to a yaml file format that is needed to continue building our docs in the future. This adds that file. This enables fail_on_warning to check for rst syntax errors when building. Builds for pull requests has also been enabled to add a status check to open pull requests. PR builds will be skipped if there are no changes to the docs. These features can easily be disabled if they don't work as desired.
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Signed-off-by: Chen Wang <wangvsa@gmail.com>
Add OPENSSL_CFLAGS and OPENSSL_LIBS to the UNIFYFS_COMMON_BASE_FLAGS and UNIFYFS_COMMON_BASE_LIBS variables, respectively. Fixes issue LLNL#792
* replace server pids pthread mutex/cond with ABT versions * add margo_state_dump() on client-server or server-server failures (currently commented out) * add a 'bootstrap complete' broadcast rpc after rank 0 sees all servers have reported * fix function declaration for unifyfs_invoke_broadcast_extents()
Update sphinx version and fix resulting conf.py build errors. (language and extra css files) Add sphinx todo extension Add requirements.in file to list top-level docs dependencies. Have rtd install pip-tools and use pip-compile to always generate an updated requirements file in the rtd build output. Add the requirements.txt generated by pip-compile to pin this version of the docs as a reproducible build. Note: When wanting to pin a new docs build, copy the results of pip-compile from the build logs and overwrite the existing requirements.txt
Add UnifyFS videos to Overview page. For ECP Tutorial, also include the slides to download and a section on changes since the tutorial. Add small section on option to use LD_PRELOAD when an application does not need to be rebuilt. Minor fixes
The tutorial download image for the slides did not come through in the original PR. Trying again.
Update Gitlab CI environments, dependencies, and jobs to reflect what UnifyFS now uses, as well as what is now available on the testing systems.
Update conf.py versions. Pin a new reproducible build by updating versions of doc dependencies
|
@adammoody, this is ready to merge. |
adammoody
approved these changes
Dec 22, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Merge for release tag v2.0
After merged, I'll tag the release and upload a tar and pdf of the docs.
Reminder: For future reference, this needs to be merged as a rebase+merge or a regular vanilla merge with a commit and NOT a squash+merge.