-
Notifications
You must be signed in to change notification settings - Fork 443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make hfile_s3 refresh AWS credentials on expiry #1462
Conversation
This is to make HTSlib work better with AWS IAM credentials, which have a limited lifespan, and so may need to be refreshed. To allow this, hfile_s3 is made to look for an unofficial 'expiry_time' entry in the AWS_SHARED_CREDENTIALS_FILE. If present, the file will be re-read if the current time is within one minute of the given expiry (new credentails are available five minutes before expiry, according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). Currently no effort is made to understand the JSON format emitted by the AWS security-credentials endpoint. It's up to the end user to reformat this into the style of the normal '.aws/credentials' file. An example of how this can be done for one source of credentials on AWS is added to the manual page. Fixes bug where parse_ini would append to rather than replace existing values. Moves x-amz-security-token to the set of headers updated via callback, as it can now change when the credentials are updated. Includes an implementation of the timegm() function, which is not portable (e.g. mingw doesn't have it) but needed to convert the expiry time to a time_t. This is put in a separate header so that it can be more easily reused elsewhere if we want. Includes tests to check that details like leap years and normalisation work properly.
Notes on how to test this.
The credentials should refresh after three hours (or up to six if AWS is slow about replacing them). A realistic test needs to open a file, and keep on using through the refresh. To do this, I wrote a simple program to open a file, then in a loop, |
@@ -720,6 +726,7 @@ test/test-parse-reg.o: test/test-parse-reg.c config.h $(htslib_hts_h) $(htslib_s | |||
test/test_realn.o: test/test_realn.c config.h $(htslib_hts_h) $(htslib_sam_h) $(htslib_faidx_h) | |||
test/test-regidx.o: test/test-regidx.c config.h $(htslib_kstring_h) $(htslib_regidx_h) $(htslib_hts_defs_h) $(textutils_internal_h) | |||
test/test_str2int.o: test/test_str2int.c config.h $(textutils_internal_h) | |||
test/test_time_funcs.o: test/test_time_funcs.c $(htslib_time_funcs_h) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
$(hts_time_funcs_h)
.
(See also autoconf on config.h, para 2.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, yes. Will fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See #1474
This is to make HTSlib work better with AWS IAM credentials, which have a limited lifespan, and so may need to be refreshed. To allow this, hfile_s3 is made to look for an unofficial
expiry_time
entry in theAWS_SHARED_CREDENTIALS_FILE
. If present, the file will be re-read if the current time is within one minute of the given expiry (new credentails are available five minutes before expiry, according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html, and in practice much earlier).Currently no effort is made to understand the JSON format emitted by the AWS security-credentials endpoint - mainly because there are several ways to get credentials, which all have subtle differences. Rather than try to support them all, it's left up to the end user to reformat the credentials into the style of the normal '.aws/credentials' file. An example of how this can be done for one source of credentials on AWS is added to the manual page.
Fixes a bug where parse_ini would append to rather than replace existing values.
Moves x-amz-security-token to the set of headers updated via callback, as it can now change when the credentials are updated.
Includes an implementation of the timegm() function, which is not portable (e.g. mingw doesn't have it) but needed to convert the expiry time to a time_t. This is put in a separate header so that it can be more easily reused elsewhere if we want. Includes tests to check that details like leap years and normalisation work properly.