add S3 tests, LocalStack, MinIO #10044
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
pdurbin
changed the title
pdurbin
added
the
Component: Containers
pdurbin
added
the
Size: 30
pdurbin
force-pushed
the
6783-s3-tests
branch
from
2f3d836 to
a087c0b
Compare
This comment has been minimized.
This comment has been minimized.
pdurbin
force-pushed
the
6783-s3-tests
branch
from
a087c0b to
b624206
Compare
This comment has been minimized.
This comment has been minimized.
pdurbin
force-pushed
the
6783-s3-tests
branch
from
b624206 to
0d1691a
Compare
This comment has been minimized.
This comment has been minimized.
2 similar comments
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
pdurbin
force-pushed
the
6783-s3-tests
branch
from
5763b93 to
4f00cda
Compare
This comment has been minimized.
This comment has been minimized.
This was referenced Nov 16, 2023
Developers can now test S3 locally by using the Dockerized development environment, which now includes both LocalStack and MinIO. See S3AccessIT which executes API (end to end) tests. In addition, a new integration test test class (not an API test, the new kind launched with `mvn verify`) has been added at S3AccessIOLocalstackIT. It uses Testcontainers to spin up Localstack for S3 testing and does not require Dataverse to be running. Note that the format of docker-compose-dev.yml had to change to allow for JVM options to be added. Finally, docs were improved for listing and setting stores via API.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
|
This comment has been minimized.
This comment has been minimized.
This should have been part of 811d79a
|
📦 Pushed preview images as 🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name. |
pdurbin
added
Size: 3
I just tried [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0 |
|
I approved the PR on the Ansible side: Thank you, @donsizemore for all the hard work! 🚀 ❤️ 🎉 Once it's merged, we'll kick off another build to see if Jenkins starts passing. |
@pdurbin the Dataverse-Ansible branch is merged, so if you're comfortable with your own testing, Jenkins is ready. |
|
Thanks a lot @donsizemore !! |
|
@donsizemore thanks! I just kicked off a build: https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-10044/17/ |
jp-tosca
approved these changes
Dec 7, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Developers can now test S3 locally by using the Dockerized development environment, which now includes both LocalStack and MinIO. See S3AccessIT which executes API (end to end) tests.
In addition, a new integration test test class
(not an API test, the new kind launched with
mvn verify) has been added at S3AccessIOLocalstackIT. It uses Testcontainers to spin up Localstack for S3 testing and does not require Dataverse to be running.TODO: Exercise direct upload, possibly borrowing code from DVUploader.Which issue(s) this PR closes:
Special notes for your reviewer:
Note that the format of docker-compose-dev.yml had to change to allow for JVM options to be added.
Suggestions on how to test this:
mvn -Pct clean package docker:run
mvn test -Dtest=S3AccessIT
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No.
Is there a release notes update needed for this change?:
Yes, included.
Additional documentation:
Tweaked various docs: