Add S3 backend to Pelican #419
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
turetske
reviewed
Nov 29, 2023
|
This seems like the correct approach to me, but I'm assuming @bbockelm will have more comments. |
jhiemstrawisc
commented
Nov 30, 2023
jhiemstrawisc
force-pushed
the
issue-169
branch
from
983d0b7 to
8abbfc1
Compare
jhiemstrawisc
added a commit
to jhiemstrawisc/pelican
that referenced
this pull request
Dec 4, 2023
Need these in dev.Dockerfile for the tests to work.
|
Tests are currently failing because the test CI doesn't pull in the updates I've made to the dev container we use to run tests. Those updates are reflected in PR #450. I will re-run once that's merged, built, and released. |
turetske
added a commit
that referenced
this pull request
Dec 4, 2023
…-docker Add test dependencies to dev.Dockerfile for PR #419
jhiemstrawisc
force-pushed
the
issue-169
branch
2 times, most recently
from
112e7ab to
d83be05
Compare
turetske
requested changes
Dec 6, 2023
turetske
requested changes
Dec 6, 2023
…oints This is the first step in giving Pelican the ability to use an S3 backend. For now, it only covers unauthenticated buckets. Importantly, this doesn't do anything to actually make the plugin available to xrootd unless its installed by hand. That will come in a later commit. Add plugin to containers Right now, the RPM for the S3/HTTP plugin isn't functional, and we don't have that repo's GH set up to publish anything automatically. Until those can be sorted out, we'll clone from source to install. Concurrent to this commit, I'm opening a few issues at the plugin repo to handle getting the various GHA set up to take care of this.
This adds a supervisor script for serving origins, as well as updates some of the dependencies needed to install the s3/http plugin and get an origin running with all of its other plugins.
This adds what we need to set an access/secret key for the S3 plugin. HOWEVER, upon trying to test this locally with MinIO, I found some bugs in the actual plugin that prevent authenticated buckets from working in the first place. That being said, that's a problem with the plugin and not with Pelican. All Pelican needs to do is to run xrootd with the right `s3` directives, and that appears to be the case with these changes.
This adds a very basic test for the S3 origin -- basically it only checks that we can start an xrootd S3 server with the provided configuration. The reason for this punt on the E2E testing is that I'm waiting for things like `originServe` to be unwrapped from the cobra cli, because otherwise there's an awful lot of setup to do. Since I don't actually test a file download, there's technically no need to set up the MinIO server in the test, but that's there and ready when those other pieces are figured out. The last bit is that for these tests to run, the minio server needs to be installed. With changes to `dev.Dockerfile`, the tests should be runnable.
jhiemstrawisc
force-pushed
the
issue-169
branch
from
fa598eb to
4f9cdad
Compare
turetske
requested changes
Dec 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.