-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement client direct reads #1166
Implement client direct reads #1166
Conversation
43aed20
to
128519e
Compare
8ee552b
to
a5410b6
Compare
3362b67
to
7a4aba4
Compare
utils/utils.go
Outdated
if !isPlugin && hasRecursive { | ||
return fmt.Errorf("Cannot use the recursive query parameter when not utilizing the pelican plugin") | ||
return errors.New("cannot use the recursive query parameter when not utilizing the pelican plugin") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On second thought, let's say "recursive query parameter not yet supported in URLs outside of plugin" or something to that effect. That tells the user it's coming, but not yet ready.
My second comment is that we need an issue to keep track of this. Can you open an issue and link it to this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is the issue, I added it to it #1165
The issue includes this PR and I have notes to modify this function as well as the tests. Anything else I need to do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope, thanks for linking it :)
This implements direct read functionality within the client. Adds a query parameter to add to the transfer url ?directread. When this is set, the client will avoid using a cache and go directly to the origin for its downloads. This PR also adds some unit tests for this functionality with a few bug fixes along the way. This should work with both the client and the plugin as well as with the ?pack and ?recursive query.
Moved CheckQueryParam to utils instead of in two different packages as well as moved the test for it. Added more test cases for directread functionality.
changes: - Changed tests to instead of look at cache location for if the file does not exist, look at the endpoint returned from the transfer - Modified the checkValidQuery function, fixing up errors returned as well as ways to check the query - Fixed typo for embedding test configs, modified one of the configs to work with test by adding another namespace - Modified comments in test yaml configs
7a4aba4
to
31338f2
Compare
80b36a1
to
2763403
Compare
This implements direct read functionality within the client. Adds a query parameter to add to the transfer url ?directread. When this is set, the client will avoid using a cache and go directly to the origin for its downloads. This PR also adds some unit tests for this functionality with a few bug fixes along the way. This should work with both the client and the plugin as well as with the ?pack and ?recursive query.
Closes #516