Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage/swift: better retries and bug fixes #552

Merged
merged 1 commit into from
May 21, 2018

Conversation

buchanae
Copy link
Contributor

This PR is not ready, and depends on #551

This upgrades a number of features of our Swift storage retrier:

  • logging messages about when retries are happening
  • fixing a bugs in error matching in ShouldRetry
  • getting rid of MaxRetries in favor of max elapsed time (closes storage/swift: tweak retry config #475)

@buchanae
Copy link
Contributor Author

Also closes #468

Message is now:


local                upload failed
attempt              0
error                swift: opening host file for URL "swift://buchanan/funnel-test/object-corrupted": open /home/ubuntu/funnel-work-dir/bc087t6nu5c1770h17t0/outputs/buchanan/funnel-test/object-corrupted: no such file or directory
index                0
taskID               bc087t6nu5c1770h17t0
time                 2018-05-19T20:07:19Z
timestamp            2018-05-19T20:07:19.438671954Z
url                  swift://buchanan/funnel-test/object-corrupted

@buchanae
Copy link
Contributor Author

buchanae commented May 19, 2018

I've added an environment variable for testing swift with an alternative storage service URL. This allows me to set up a swift proxy that deliberately injects failures into the HTTP request/response, for testing retries and failure handling in Funnel.

example:

export FUNNEL_SWIFT_STORAGE_URL=http://10.50.50.130:8080/v1/AUTH_abced<auth token here>/

TODO:

  • figure out how best to get the /AUTH_<token goes here> part of the URL and document that.

@adamstruck
Copy link
Contributor

I strongly prefer MaxRetries to MaxRetryDuration. Making retries duration based makes it pretty much impossible to know how many times Funnel will try to download/upload a particular file.

@buchanae
Copy link
Contributor Author

I strongly prefer MaxRetries to MaxRetryDuration. Making retries duration based makes it pretty much impossible to know how many times Funnel will try to download/upload a particular file.

You're right. I've reverted that and updated the default Swift config instead.

figure out how best to get the /AUTH_ part of the URL and document that.

The Swift storage URL can be found by going to the OpenStack dashboard > Access & Security page > API Access tab. I've noted this in the code comments.

@buchanae
Copy link
Contributor Author

I might need to separate the Swift storage URL override. Calling Authenticate() in the constructor seems to have downstream effects on our storage code.

@buchanae buchanae force-pushed the swift-debug branch 3 times, most recently from 238e213 to da49dfc Compare May 21, 2018 19:56
@buchanae
Copy link
Contributor Author

I've removed the CLI conflict with #551 and the Swift storage URL override code; it's not critical here. I'll revisit it in the future.

@adamstruck adamstruck merged commit 40fe85f into ohsu-comp-bio:master May 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

storage/swift: tweak retry config
2 participants