v3.0.0

Summary of changes:

• In this release, we have made several significant updates to improve write performance and user experience.
  • We have revised our implementation of key-value store that translates key-value pairs in the request to block layer data written to device.
  • Specifically, we have replaced RocksDB with an internal implementation of a hash-based key-value translator and a hint-based logical block manager sensitive to optimal NVMe drive IO boundary to improve write performance.
• For impacted users, these updates should provide a smoother, more intuitive experience with a noticeable improvement to write operation.
  • We have deprecated compaction and the requirement for written data to be compacted before reading to achieve maximum read performance is no longer needed.
  • We have introduced a new format tool called dss-formatter to format devices; similar to existing mkfs-blobfs tool which has now been deprecated.

Limitations

• Persistent key listing is not yet implemented. This will result in MinIO failing to return object listing after a reboot.
• Target startup time is presently longer than expected - and will be addressed in a future release.

What's Changed

• Adding patch for hanging monitor status poller on behalf of SRCX by @nsarras in #47
• Separating system and unit tests to enable smooth consumption for code coverage in build pipelines by @nsarras in #61
• 56 pipeline integrate pytest code coverage in commitpr pipeline by @velomatt in #60
• Refactor for code coverage by @velomatt in #72
• Adding MINIO REST endpoint collector, refactoring main metrics.py to be more readable for multiple collectors by @nsarras in #69
• Add mocks and unittests for Client/Server socket communication, add m… by @Chenliang-Eric-Li in #74
• Add mid-layer system tests for datamover put with large/empty file, n… by @Chenliang-Eric-Li in #76
• Integrating MINIO cluster UUID and capacity metrics by @nsarras in #80
• Adding gen2 s3 benchmark changes by @nsarras in #78
• Removing s3 benchmark binary by @nsarras in #85
• Use latest sonar-scanner and build-wrapper on runtime by @velomatt in #86
• Modifying logic to provide metrics per MINIO endpoint by @nsarras in #83
• Adding support for badnwidth and iops collection in MINIO REST collector by @nsarras in #87
• Modifying regex pattern to accurately retrieve MINIO endpoint based on pid by @nsarras in #89
• Adding handling for if MINIO(s) go down by @nsarras in #92

New Contributors

• @Chenliang-Eric-Li made their first contribution in #74

Full Changelog: v2.0.0...v3.0.0

v2.0.0

Gen2 Overview

• DSS can now be deployed with gen2 mode, which utilizes RDD (RDMA Data Direct).
  • This feature enables clients to transfer object data directly from DSS targets without the need to go through the MinIO servers.
  • This allows better performance, as well as improved network and memory resource utilization in the DSS Stack.
  • This feature is integrated with DSS Client APIs and allows for seamless integration with applications using the thin DSS client library.
• To deploy DSS with gen2 mode enabled, it is required that both front-end (in tcp_ip_list) and back-end (in rocev2_ip_list) IP addresses be from RDMA-enabled network interfaces. See dss-ansible README for details.
• dss-ansible will deploy DSS in "gen2" mode simply by setting gen2=true in your Ansible inventory file. The default value is false, which will deploy DSS in gen1 mode (not using the RDD feature, does not require RDMA front-end network).

What's Changed

• Setting socket option to reuse address for socket connection to resolve TCP TIME_WAIT blocker from recent runs of Data Mover by @nsarras in #11
• Fixing payload misalignment/corruption issue by @nsarras in #14
• Added extra None message when the logger is stopped. This will break … by @grandsuri in #15
• Adding user configurable socket options by @nsarras in #17
• Added support to run DM with IPv4 or IPv6 or both. by @grandsuri in #21
• Adding Pytest Framework and initial Pytest unittests to DataMover by @nsarras in #23
• Changing logic to return payload as json in the case of any exception depending on received bytes by @nsarras in #20
• Added simple test tool to perform object upload, download, verify by @grandsuri in #12
• Fix for MIN-1312: Graceful exit of DataMover for invalid prefix by @sohomb91 in #25
• MIN-1540: Fixed the logger to use QueueHandler and RotatingFileHandler by @grandsuri in #26
• Adding NFS CLI arg options and NFS port by @nsarras in #31
• Modifying compaction option to be yes/no from either config file or CLI by @nsarras in #32
• Fixing bug in not specifying NFS CLI options by @nsarras in #41
• Fix for hanging application, Submitting on behalf of Eric by @nsarras in #39
• Add dynamic reusable workflows by @velomatt in #42
• Add stagemergeartifacts by @velomatt in #49
• Gen2 based changes for DSS Client, benchmarks etc by @grandsuri in #28
• Added the flags to link librdd_cl.so and libdss.so properly from the artifacts/repos by @grandsuri in #51
• 37 datamover add flexibility to prefix structure to work without nfs server name by @nsarras in #43
• Adding & saving progress on dss_metrics module by @nsarras in #45
• Add tests to artifact by @velomatt in #52
• Fixing metrics agent bug for parsing and populating metrics objects for all subsystems by @nsarras in #54
• Updated the s3-benchmark binary compiled with librdd_cl and libdss_client by @grandsuri in #55

New Contributors

• @grandsuri made their first contribution in #15
• @sohomb91 made their first contribution in #25

Full Changelog: v1.0.0...v2.0.0

Known Issues

• datamover - target compaction may be unintentially triggered in dry-run mode
• datamover - cache files are generated in dry-run mode causing subsequent PUTs to be skipped
  • workaround: delete the /var/log/dss/prefix_index_data.json file after dry-run

dss-ecosystem v1.0.0

Initial public release

What's Changed

• Add GitHub Actions by @velomatt in #2
• Documenting need to resolve hanging client process by @nsarras in #3
• Add CODEOWNERS by @velomatt in #10

New Contributors

• @velomatt made their first contribution in #2

Full Changelog: https://github.com/OpenMPDK/dss-ecosystem/commits/v1.0.0

Artifact Downloads:

dss_client-v1.0.0.tgz
nkv-datamover-v1.0.0.tgz