validate base_path for s3 repository

[MarcinGinszt]

Description

Documentation states, that base_path for s3_repository should not start or end with /.
However, this is not obvious and not validated anyhow. We had a repository with base_path starting from /, that had status verified and it contained completed snapshots- however, the snapshots were actually not synchronized with S3. I find that dangerously misleading.
In addition to that, I wonder if the documentation is actually right on this topic- for us (Elasticsearch v7.9.3) repository doesn't work for base path without trailing /.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

I feel similarly, but this is a pretty tricky area and unfortunately not one we can address without making changes that would be breaking for at least some users. See #46869 for some related discussion of the problems.

Yet, we haven't seen this as an actual problem when using the real S3, only when using something that claims (incorrectly) to be S3-compatible. Are you using something other than the real S3 @MarcinGinszt?

[MarcinGinszt]

@DaveCTurner We are using S3.
I'm sorry, I know there is a lot to unwrap in my message and maybe I should rather create a bug issue.
Let me know if I could help with any information.

[DaveCTurner]

Interesting.

More detail would be helpful, yes. In particular, could you share a recursive listing of the bucket contents that seems not to be working for you? There shouldn't be any sensitive data in such a listing, the blob names are all random, so ideally use something like https://gist.github.com. If you'd rather not share it publically, let me know and I can arrange a private channel too.

[MarcinGinszt]

Ok, I'm ashamed of myself. Looks like the Elasticsearch works correctly, and just S3 path handling is making this complicated.

The snapshot with base_path containing trailing / were not visible in the S3, because AWS can't properly display this directory if it is at bucket root level- since the directory name is interpreted as directory path. Recursive listing found those files.
If the base_path contains the / suffix, removal suffix prefix will cause errors, because meta files will be in the same directory, while indices path will differ by mentioned '/'- making Elasticsearch unable to find expected indices.

Having a warning/ documentation about '/' prefix or suffix while creating the repository could save a lot of confusion. Currently, base_path is only described as The bucket path to the repository data..

[DaveCTurner]

😁 I did say this was a tricky area...

My understanding of your last message is that there's no problem to fix here, it's just that when looking at the bucket contents in the AWS UI you didn't see something you expected to see. I think that's really a problem with your expectations rather than because there's any action needed on the Elasticsearch side. We don't think we should push users towards a config that makes the repository contents appear in the AWS UI as you expect, so we won't be adjusting any docs or warnings as you suggest. Instead, I recommend you only interact with the repository contents via the Elasticsearch APIs. In particular:

• the get snapshot(s) API will list the snapshots in the repository
• the verify snapshot API does some basic checks that the repository works as we need
• the repository analysis API does some more in-depth checks too
• the (not yet released) integrity check API will do a very in-depth analysis of the integrity of data in the repository

As there's no action needed here I'm closing this. If you have any more questions, please open a thread on the Elasticsearch forum instead.