add endpoint url param for s3 dal

[dantengsky]

I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

• add endpoint_url to S3StorageConfig
• using custom Region if endpoint_url is enabled
• use minio in statless-standalone-test(ubuntu-latest only)
• minor code gc

@ZhiHanZ thanks for the github action code snippets , it works really well! but

• I failed to use it in config macos-latest, no docker there. thus minio is enabled only for ubuntu-latest. if there are better solutions, please let me know.
• workflows/stateless-tests-standalone.yml has been tweaked to enable minio. there are three extra steps doing nothing in macos-latest... not as elegant as before. is there a better way to do this? or shall we improve it later?

@drmingdrmer thanks for suggesting using moto as the mock service, but @wubx kindly reminds me that there might be real cases of using minio later, and mocking is only used in integration tests in this PR, thus minio is taken.

part of

• issue fuse table implementations #1780 : Mock S3 Service by using MinIO
• issue ontime test suite insert and select with fuse table engine #2561

Changelog

• New Feature
• Not for changelog (changelog entry is not required)

Related Issues

part of issue #1780 #2561

Test Plan

Unit Tests

Stateless Tests

[databend-bot]

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

[codecov-commenter]

Codecov Report

Merging #2555 (5daa9a1) into main (a398ed6) will decrease coverage by 0%.
The diff coverage is 32%.

@@          Coverage Diff          @@
##            main   #2555   +/-   ##
=====================================
- Coverage     72%     72%   -1%     
=====================================
  Files        646     644    -2     
  Lines      36383   36378    -5     
=====================================
- Hits       26310   26299   -11     
- Misses     10073   10079    +6     

Impacted Files	Coverage Δ
common/dal/src/data_accessor.rs	91% <ø> (ø)
common/dal/src/impls/aws_s3/s3.rs	0% <0%> (ø)
...ommon/dal/src/impls/aws_s3/s3_input_stream_test.rs	0% <0%> (ø)
...on/dal/src/impls/azure_blob/azure_blob_accessor.rs	0% <ø> (ø)
common/dal/src/impls/local.rs	75% <ø> (+3%)	⬆️
query/src/datasources/common/dal_builder.rs	42% <0%> (-2%)	⬇️
query/src/datasources/table/fuse/io/mod.rs	100% <ø> (ø)
.../src/datasources/table/fuse/meta/table_snapshot.rs	56% <ø> (ø)
...y/src/datasources/table/fuse/index/min_max_test.rs	91% <71%> (-2%)	⬇️
query/src/configs/config_storage.rs	87% <100%> (+<1%)	⬆️
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a398ed6...5daa9a1. Read the comment docs.

[BohuTANG]

Cool~
What happens if we run make stateless-test locally?

[BohuTANG]

Cool~ What happens if we run make stateless-test locally?

Got it, the github stateless test sets the s3 as the default storage :)

[BohuTANG]

but @wubx kindly reminds me that there might be real cases of using minio later, and mocking is only used in integration tests in this PR, thus minio is taken.

This is stateful test(issue #2516) not stateless test, cc @wubx

[PsiACE]

LGTM, although stateless-test doesn't necessarily require s3 , it can help us verify that it works.

Perhaps we also need to provide such support for azure blob or other later?

[drmingdrmer]

Does it mean we mac users can only test s3 backend in a CI?

[dantengsky]

Perhaps we also need to provide such support for azure blob or other later?

yes, I think we need them

[dantengsky]

Does it mean we mac users can only test s3 backend in a CI?

for "auto" tests, unfortunately, yes. we can improve this eventually I think.

but by exporting the env vars before cargo test make stateless-test..., something like this:

          export STORAGE_TYPE=s3
          export S3_STORAGE_BUCKET=testbucket
          export S3_STORAGE_REGION=us-east-1
          export S3_STORAGE_ENDPOINT_URL=http://127.0.0.1:9000
          export S3_STORAGE_ACCESS_KEY_ID=minioadmin
          export S3_STORAGE_SECRET_ACCESS_KEY=minioadmin
          bash ./scripts/ci/ci-run-stateless-tests-standalone.sh

we can test s3 backend locally (but manually setup)

[drmingdrmer]

Does it mean we mac users can only test s3 backend in a CI?

for "auto" tests, unfortunately, yes. we can improve this eventually I think.

but by exporting the env vars before cargo test make stateless-test..., something like this:

          export STORAGE_TYPE=s3
          export S3_STORAGE_BUCKET=testbucket
          export S3_STORAGE_REGION=us-east-1
          export S3_STORAGE_ENDPOINT_URL=http://127.0.0.1:9000
          export S3_STORAGE_ACCESS_KEY_ID=minioadmin
          export S3_STORAGE_SECRET_ACCESS_KEY=minioadmin
          bash ./scripts/ci/ci-run-stateless-tests-standalone.sh

we can test s3 backend locally (but manually setup)

What about adding these config to a script like stateless-test-s3 and let user run it by just one click?

[dantengsky]

What about adding these config to a script like stateless-test-s3 and let user run it by just one click?

good idea! like ci-run-stateless-tests-cluster-tls.sh.

[drmingdrmer]

/lgtm

[databend-bot]

Approved! Thank you for the PR @dantengsky

[databend-bot]

CI Passed
Reviewer Approved
Let's Merge