Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api_request should accept a list of api endpoints for HA #122

Open
cwjohnston opened this issue Jun 2, 2016 · 7 comments
Open

api_request should accept a list of api endpoints for HA #122

cwjohnston opened this issue Jun 2, 2016 · 7 comments
Labels

Comments

@cwjohnston
Copy link
Contributor

api_request currently constructs the URL for the API end point using the api scope host and port values. As described in sensu/sensu#1299 and sensu-plugins/sensu-plugins-sensu#6, this is problematic when we want handlers to connect to the API on a load balancer whose port differs from the one the API itself is listening on.

Load balancer-specific concerns aside, Sensu Server currently has no way of knowing if the API is unavailable, which means that pipe handlers using this library may fail to execute as expected if the one and only API server they know about (via api config scope) is unavailable for some reason.

I think we can improve this by enhancing sensu-plugin api_request method to support iterating over an array of API endpoints described under a configuration scope that won't require changes to the API server itself, e.g. pointing at a single load balancer:

{ "api": { "endpoints": [ { "host": "192.168.10.11", "port": 45670 } ] } }

or directly at one or more APIs:

{ 
  "api": {
    "endpoints": [ 
      { 
        "host": "192.168.10.9",
        "port": 4567
      },
      { 
        "host": "192.168.10.10",
        "port": 4567
      }
    ] 
  } 
}
@cwjohnston
Copy link
Contributor Author

cwjohnston commented Jun 2, 2016

We can also try to address #90 and/or #95 in the same change.

@cwjohnston
Copy link
Contributor Author

In sensu/sensu#1299 @jaykumar-jnpr asks:

what is the expected behavior when one of the API dies in the Array, do we have any health check logic to ignore dead API endpoint? Or just basic timeout with exponential back-off.

I believe that because sensu-plugin invocations are short-lived, i.e. the library is invoked when Sensu forks a process to run a plugin, we'd probably need a short timeout on each request so as to handle issues as quickly as possible. Exponential back-off would likely have major performance effects.

I imagine we would iterate over the array of end points until we found one that handles our request successfully, or we hit the execution timeout for the handler.

@cwjohnston
Copy link
Contributor Author

cwjohnston commented Jun 10, 2016

I'm still open to feedback on this, and not entirely sure its necessary in light of #134. :)

@cwjohnston
Copy link
Contributor Author

This api.endpoints spec should probably be adopted/canonized in Sensu Core before we try to create another defacto standard here...

@cwjohnston
Copy link
Contributor Author

@portertech what do you think about implementing defaults for api.endpoints in sensu-settings for use by extensions and plugins?

@cwjohnston
Copy link
Contributor Author

API endpoint configuration validation was added to sensu-settings 9.9.0 in sensu/sensu-settings#65, shipped in Sensu Core 0.28

@jaboc
Copy link

jaboc commented Oct 19, 2017

Why wouldn't you use something like HA proxy infront of your pool of api servers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants