Proposal: Support COPY use case

[dmitrizagidulin]

(This is a more detailed proposal continuation of issue solid/solid#49.)

Motivation / Problem Statement

We need a bandwidth-efficient method to copy data to and from Solid pods.

Imagine you're building a 'Save to Solid' widget / app / browser extension. The idea is - the user is browsing some Web resource (a PDF file, or an image, or a video, etc), and would like to save it to their pod (to be able to tag it and do other sorts of CMS stuff on it).

Currently, the only way to perform this operation would be multi-step:

1. GET resource and somehow store it temporarily in the client (in browser memory?)
2. PUT the temp resource on the pod (upload it)

Putting aside the implementation details, this presents two problems: temporary storage space (if the file is being held in a Javascript variable in a web app, or in LocalStorage, this quickly presents challenges when the resource is large), and bandwidth.

This is especially problematic on resource-constrained clients (like mobile apps or browsers) -- the user first has to use their mobile data to download a file temporarily, and then use mobile data to upload that resource to their pod.

Proposed Solution 1: COPY Method

Note: As @RubenVerborgh points out, we should separate the problem / use case from the proposed solution.

Add a new Solid-specific LDP method, COPY, inspired by the existing WebDAV COPY method.

This proposed solution lets pods play to their strength, to act as always-connected high(er) bandwidth servers. Specifically, it allows a client to issue a single COPY command, and the server would perform the necessary data transfer (using its own bandwidth, not the client's).

This would be just a single step:

1. issue a command to COPY resource from source URL to destination URL (server would proceed with the operation).

COPY Example

To copy FROM an external URL, https://example.com/example.pdf, TO a user's POD, alice.inrupt.net:

COPY /papers/example.pdf HTTP/1.1
Host: alice.inrupt.net
Source: https://example.com/example.pdf

HTTP/1.1 201 Created
Date: Mon, 23 May 2019 22:38:34 GMT
Content-Type: application/pdf
Location: https://alice.inrupt.net/papers/example.pdf

COPY Method Specs

This method is idempotent, but not safe (see Section 9.1 of RFC2616). Responses to this method must not be cached.

Copying non-Container resources

(Note, this is what's currently implemented on node-solid-server; copying of containers is not implemented.)

When the source resource is not a collection, the result of the COPY method is the creation of a new resource at the destination whose state and behavior match that of the source resource as closely as possible.

To copy a resource FROM an external URL TO a Solid pod:

COPY <destination url>
Source: <source url>

(Note that this is backwards from the current WebDAV semantics, which uses COPY <source url>, Destination: <destination url>, see below.)

Copying Container resources

(Question: Should this be supported?)

See the WebDAV COPY for Collections spec for discussion of what's involved, including the handling of recursion via the Depth: header.

Difference from WebDAV COPY

WebDAV's COPY is intended primarily for transfering data out of the WebDAV server and into an external destination. Its syntax is: COPY <source url> with the Destination: <destination url> header.

It does not, however, support the common use case where the source URL resides on an external server, and you want to copy it to yours. (In other words, it does not support the Source: header.)

Since the motivation for Solid's COPY method is the latter (transfering from an external resource to a Solid pod), the Solid COPY method should support the Source: header (in addition or instead of the Destination: header).

ACL Interactions

Copying from a non-LDP public Web resource to a container:
A COPY operation requires that the authenticated user has Write access to the destination container.

Copying FROM an LDP container to an LDP container:
A COPY operation requires that the authenticated user has Read access on the source container, and Write access on the destination container.

Interfacing with actual WebDAV servers:
Out of scope for the moment. We just need this as a convenience method to move to and from Solid pods.

Design Questions

1. This Solid/LDP Copy method uses a Source: request header, to handle the use case of transfering resources from an external source to a Solid pod. Question: Should the WebDAV style Destination: header operation be supported as well? (For transfering resources from a pod to another external pod).
2. Should COPY be supported on LDP Containers? (What about recursive copy?)

COPY Implementation Notes

• The proposed COPY method is currently implemented on node-solid-server, for experimental purposes.
• The current implementation only supports the most common authentication use case (the external resource is public).
• Developers using this method should be aware of .acl files when copying resources (for example, if a file has its own .acl, first copy the .acl, and then the resource itself).

Proposed Solution 2: ?

?? (Discuss whether the use case can be solved using existing LDP methods).

[RubenVerborgh]

I'd propose perhaps to keep the problem being addressed separate from the solution; then we can distinguish between a) who supports the use case (and maybe which additional requirements they have), and b) who supports the solution.

I +1 the use case, but unsure about the solution (need to think if we can have an alternative without a custom HTTP method).

[dmitrizagidulin]

@RubenVerborgh sure, good point. (I'll edit the proposal text to separate problem from solution).

[dmitrizagidulin]

@RubenVerborgh Are you thinking like, that a potential solution would be to use POST or PUT with the Source: header?

Minor comment on COPY being a custom header: fetch() supports the COPY method, as does Express.js, so, I figure, not so bad?

[RubenVerborgh]

@RubenVerborgh Are you thinking like, that a potential solution would be to use POST or PUT with the Source: header?

Drawback about POST is the lack of idempotence; PUT has idempotence but requests the exact representation to be saved (so the copy logic would probably not be "allowed").

So there is a case for a custom method, I think. Just would want to consider alternatives as well.

[acoburn]

It is worth noting that the Fedora specification created a feature called "External Content". That feature is specific to binary content but there are many similarities between that and what is proposed here. In addition, the use case described here follows the copy (rather than redirect) handling mechanism of Fedora's external content feature.

This feature is tremendously appealing, but it also opens the door to various security holes, so one would want to tread very carefully. For example, if an agent is able construct a COPY (or similar) request that causes the server to behave as a generic HTTP client, there are several categories of exploits that suddenly become possible.

First, if the Solid pod is running inside a firewall, this feature can be used as a type of port/resource scanner and it may be possible to gain access to resources that a user shouldn't typically have access to.

Second, DoS attacks become really easy. While it is common to limit the amount of data being transferred into a server via POST or PUT, in this scenario, the COPY request would be tiny, but it could trigger the download of an arbitrarily large resource into the server.

Third, authorization becomes more complicated if the remote resource requires special access that differs from the credentials required for writing to the destination resource.

As an alternative, it may be easier and (arguably) more secure to build this feature as a stand-alone sort of application that sits along side of a resource server, rather than being a feature of the resource server itself. The distinction here is small but architecturally important because one could treat all data as ephemeral and isolated from the resource server itself, which insulates the resource server from the kinds of exploits described above.

[RubenVerborgh]

As an alternative, it may be easier and (arguably) more secure to build this feature as a stand-alone sort of application that sits along side of a resource server, rather than being a feature of the resource server itself.

+1

[dmitrizagidulin]

@acoburn all excellent points! (And yeah, I was wondering if Trellis etc had a way to do something similar).

Fedora "External Content" feature

Yep, that would work too! (uses a PUT or POST with a Link: header)

And good point, we should add a section on Security Considerations.

if the Solid pod is running inside a firewall, this feature can be used as a type of port/resource scanner

Agreed. It'd be worth being able to enable/disable this feature on the config level.

Second, DoS attacks become really easy.

Not sure I fully agree there. I assume the COPY implementation would use the same quota limits as PUT or POST. (And of so, it's as resistant to DDOS as those verbs.)

Third, authorization becomes more complicated if the remote resource requires special access

Sure, yeah. The primary use case is for public web-accessible resources.

As an alternative, it may be easier and (arguably) more secure to build this feature as a stand-alone sort of application that sits along side of a resource server, rather than being a feature of the resource server itself.

Interesting!
I'm not sure it would be more secure. (It would be exactly as secure as implementing it in the server, given the caveats above (operators can disable it, and quota limits apply).)
Architecturally.. So that means pod operators need to set up yet another server, alongside?

[RubenVerborgh]

Second, DoS attacks become really easy.

Not sure I fully agree there. I assume the COPY implementation would use the same quota limits as PUT or POST. (And of so, it's as resistant to DDOS as those verbs.)

Yeah, but suddenly I can ask 100 servers to perform 10 COPY requests each, for the same resource on a server A, which finds itself now bombarded with 1000 requests coming from 100 sources.

So that means pod operators need to set up yet another server, alongside?

Generic copy server/app/agent.

[dmitrizagidulin]

@acoburn Oh, I just remembered (re 'port scanner' / firewall security consideration) -- our existing /proxy endpoint also has this issue. (And I believe the current mitigation is - you can turn it off on the config level).