Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

Log in agent retries for debugging #263

Closed
ranweiler opened this issue Nov 4, 2020 · 1 comment · Fixed by #623
Closed

Log in agent retries for debugging #263

ranweiler opened this issue Nov 4, 2020 · 1 comment · Fixed by #623
Labels
enhancement New feature or request

Comments

@ranweiler
Copy link
Member

When making agent HTTP requests with retries and backoff, log what's happening to provide insight and debuggability.
@elee
Copy link

elee commented Nov 5, 2020

FWIW onefuzz respects -vvv which does show the underlying HTTP requests and retries, maybe documenting this in the usage would be helpful to some users?

The output from trying to connect to the playground the other day when it was 404-ing for me:

$ onefuzz versions check -vvv
DEBUG:onefuzz:getting info
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): login.microsoftonline.com:443
[...]
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): onefuzz-playground.azurewebsites.net:443
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None
DEBUG:urllib3.connectionpool:https://onefuzz-playground.azurewebsites.net:443 "GET /api/info HTTP/1.1" 404 0
INFO:nsv-backend:request bad status code: 404
DEBUG:nsv-backend:request GET https://onefuzz-playground.azurewebsites.net/api/info None

@bmc-msft bmc-msft added the enhancement New feature or request label Nov 11, 2020
@bmc-msft bmc-msft mentioned this issue Mar 2, 2021
Merged
@ghost ghost closed this as completed in #623 Mar 2, 2021
ghost pushed a commit that referenced this issue Mar 2, 2021
@bmc-msft
fix agent retry on connection level failures (#623)
b970937 
In debugging the connection retry issues, I dug into this more.  

Apparently, some of hyper's connection errors are not mapped to std::io::Error, rendering the existing downcast impl ineffective.

As such, this PR makes the following updates:
1. Any request that fails for what `reqwest` calls a `connection` error is considered transient.
2. Updates the retry notify code to use our `warn` macro such that the events show up in application insights.
3. Updates the unit test to demonstrate that failures by trying to connect to `http://localhost:81/`, which shouldn't be listening on any system.
4. Adds a simple unit test to verify with send_retry_default, connections to https://www.microsoft.com work

Fixes #263
@ghost ghost locked as resolved and limited conversation to collaborators Apr 1, 2021
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix agent retry on connection level failures bmc-msft/onefuzz
3 participants
@elee @ranweiler @bmc-msft