Add advanced usage sections to README.rst

[ddelange]

Title

Add advanced usage sections to README.rst (ACL, StorageClass, Metadata etc)

Motivation

worth updating the smart_open docs somehow?

Because if it tripped you up, it might trip up others too. So I wonder if there's a better way to record / present that knowledge.

Originally posted by @piskvorky in #738 (comment)

Tests

Docs only

Work in progress

--

Checklist

Before you create the PR, please make sure you have:

• Picked a concise, informative and complete title
• Clearly explained the motivation behind the PR
• Linked to any existing issues that your PR will be solving
• Included tests for any new functionality
• Checked that all unit tests pass

Workflow

Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress.
Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code.

[mpenkov]

Maybe put these in https://github.com/RaRe-Technologies/smart_open/blob/develop/howto.md instead?

Stuff like this feels a bit too specific for the general readme (the readme links to the howto anyway).

[ddelange]

What to do about the doctests there? No easy way to mock s3, azure, and gcs writes I guess?

[mpenkov]

For S3 we can use our own bucket (maybe later...).

For Azure and GCS there isn't much we can do, as a team we don't use those cloud providers.

[ddelange]

we don't use those cloud providers

Same goes for me...

Can moto be enabled for doctests by running them via pytest? That would cover the s3 part at least.

To test azure and gcs, in pypicloud we spin up (fake) servers, configure the client objects accordingly which are then propagated to the final smart_open calls. Is it an option for smart_open to spin those up before running unit tests and doctests? Side-effect would be you can drop (maintenance of) the fake classes for the tests, and run the functional assertions against the fake servers instead.

[mpenkov]

I don't see an optimal solution here.

1. Do nothing (the status quo): we risk parts of the documentation going stale or suggesting something that doesn't actually work, which is bad
2. We invest time and effort, and test under moto (and whatever the equivalent is for Azure and GCS). Not sure if this is possible, though. Plus, makes the documentation harder to read.
3. As above, but with localstack. Should be possible by e.g. sneaking the endpoint URL via environment variables. Makes the documentation harder to read.
4. We invest time, effort and money, and sign up for each supported backend (AWS, Azure, GCS, and whatever gets added next).

[ddelange]

Thanks for the reply! And fair enough, let's go with the status quo

[piskvorky]

I don't think the ticket was related to Azure or GCS, was it?

Any way to help the original user / fix the discovery issue?

[ddelange]

for S3 at least, howto covers it I think: it explains the mechanic and suggests to study the boto3 api reference

[ddelange]

Hey @piskvorky @mpenkov 👋

These last months, I've had to point quite a few people to this PR for reference.

Can you reconsider this PR against README.rst?

• It is very brief (+35 LOC only)
• It showcases the slightly different transport_params semantics for the big three
• It is a unified structure, basically a 'comparison'
• Not subject to the doctest blocker mentioned above

I think this addition will save many devs a journey!

[ddelange]

Is <provider> Advanced Usage maybe a misleading subheader for README?

Does something like <provider> Custom Uploads make more sense?

[mpenkov]

Merged, thank you