-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modify the Reader classes to support getting files from somewhere other than a local disk #135
Comments
I checked out your links, but I'm afraid I still don't understand your use case. Why do you need to read the bag-info.txt from S3? |
That is exactly what our cloud storage service is doing - especially during replication for multiple copies "Lots Of Copies Keeps Stuff Safe - LOCKSS". To store the things, you have to move them into storage - and to move the things between replication locations between object stores you have to move them in large batches. It might be useful for us to communicate about this not via GitHub issues - can we invite you to our Slack perhaps? |
The classes in
reader
take a path to a file on disk, read that file and then parse the contents. For example:For the Wellcome storage service (https://github.com/wellcometrust/storage-service), we aren’t keeping bags on the local disk, but in S3. If we want to read a file, we make a GetObject call to the S3 SDK, which returns an
InputStream
.We could download the bag files to disk, and read them from there, but that seems a bit icky – would you be open to some pull requests that add allow parsing files even if they aren’t local files? Something like:
So the existing API is preserved, and calls into the new method that takes any BufferedReader – and now we can call that rather than round-tripping to the filesystem first.
Thoughts?
The text was updated successfully, but these errors were encountered: