How to handle multi-storage scenarios?

[MikhailGordienk]

Hello!
The InlineDataFlowController can initiate two types of data transfer: streaming and inline. As I see the former one is for the case when both sides use the same storage platform (ex. both have S3 buckets).
At the same time, only the destinationType is used in S3toS3DataStreamer.canHandle(). What about the multi-storage case? Is it possible to have two storage systems with different assets in them, both capable of streaming to the same destination?

[paullatzelsperger]

Data transfers always have one origin and one destination. If you either want to transfer multiple assets into the same destination, or a single asset into multiple destinations, those would have to be modeled as separate data transfers.

Having two two data transfers that transfer data to the same destination is certainly possible, but be aware that the managedResources flag should be set to false then, otherwise the provisioning may fail.

[MikhailGordienk]

Still not clear, let's assume that I have StorageA and StorageB, both capable of streaming to S3. So I have StorageAtoS3Streamer and StorageBtoS3Streamer. In this case, the following algorithm will not suffice

@Override
    public boolean canHandle(DataRequest dataRequest) {
        return "dataspaceconnector:s3".equals(dataRequest.getDestinationType());
    }

Because the requested artifact might reside in StorageA but DataOperatorRegistry will pick up StorageBtoS3Streamer.
I think canHandle() implementation in all Streamers must consider both destination and source.

[paullatzelsperger]

First of all, a storage implementation does not stream anywhere. We have DataReaders, that read data from somewhere, and then we have DataWriters, that, well, write data somewhere.

So in your example you'd end up with a StorageAReader, a StorageBReader and an S3BucketWriter (which already exists BTW). The InlineDataFlowControllers job is simply to invoke them.

Be aware though that we have the DataPlane feature (issue #463 and PR #496) incoming around end of January 2022.