v3.1 Multipart Upload

Overview

TODO

Limitations

1. The storage API supporting the MPU is used. These are Amazon S3 and Openstack Swift currently.
2. In the distributed mode, all object parts are processed by the single storage driver.
3. "Create" load type is used to split the large objects into the parts uploaded separately.
4. No guarantees about the time of each upload completion request is issued and finished. This means that user should use count limit for a multipart upload load job.

Approach

Mongoose has the so called I/O task abstraction. I/O tasks are executed by the specific storage drivers. The storage driver may be able to detect the "multipart" I/O tasks and execute the corresponding sequence of the "sub-tasks":

1. Initiate the object MPU.
2. Upload the object parts.
3. Commit the object MPU.

Configuration

The "item-data-ranges-threshold" configuration parameter controls the MPU behavior. The value is the size in bytes. Any new generated object is treated as "large" if its size is more or equal than the configured threshold.

Reporting

TODO

Parts List Output

TODO

Future Enhancements

• Support Read for the segmented objects
• Support Update for the segmented objects
• Support Copy for the segmented objects