Streaming multipart support

[tomchristie]

Closes #2249

Something we're not currently handling here, that'd also need updating is how we handle non-rewindable cases.

If the file instance is a non-seekable IOBase then we ought to raise a StreamConsumed error if we attempt to read from the file more than once. This occurs if we attempt to upload a streaming multipart file, but then get a redirect. Everything's fine if we've got a rewindable file instance, but if we've got a non-rewindable case then we need to error out, since we can't repeat the upload at that point.

---

Edit: My description of this ticket wasn't really sufficient, so...

If we have a multipart file upload but cannot determine it's length, then used chunked transfer encoding, instead of reading the entirety of the file into memory in order to determine the Content-Length.

[adriangb]

Wouldn't we get the redirect before we start reading files? I would have thought we'd assemble the multipart stuff into an AsyncIterable[bytes] that lazily does work -> if we get redirected we never start pulling from it.

[tomchristie]

Wouldn't we get the redirect before we start reading files?

Nope. You send the complete request, then you get the complete response.

If the response is a redirect, then you need to send the complete request again. (Which is fine if you're sending a file since you can just rewind to the start, but isn't recoverable if you're eg. streaming some data from a generator)

[ghost]

Any update when this will be added?

[tomchristie]

@justinjeffery-ipf This pull request is currently awaiting a review. If you'd like I can send you an invite to the GitHub organisation, which would mean you'd have permission to make approving reviews?

[zanieb]

LGTM

[adriangb]

Awesome thanks Tom and Michael!