Agent service registry slowdown

[spheromak]

Per the conversation on the mailing list here: https://groups.google.com/forum/#!searchin/consul-tool/thousands$20of$20services/consul-tool/vPAhS1EhABM/_SPAIXuMFDQJ

There is a slowdown due to how the gent loops over all services and the catalog returned from the server that completes in O(N^2). After talking with @slackpad at HashiConf we think that we can improve on this fairly simply with an index on the server result.

[spheromak]

@slackpad did you do anything with this since hashiconf ?

[armon]

@spheromak I think #1296 is a partial fix at least. Do those services have checks associated, or are there just 10K services without checks?

[slackpad]

Hi @spheromak sorry for the delay on this, and thanks @armon. Your change in #1296 looks good but iirc there were no health checks for these services. Looking at the code I'm surprised there's not an n^2 loop over services so I'm not sure this alone will fix it.

[spheromak]

No agent checks. We were planning on doing http callbacks for service
status.

On Mon, Oct 12, 2015 at 9:15 PM James Phillips notifications@github.com
wrote:

Hi @spheromak https://github.com/spheromak sorry for the delay on this,
and thanks @armon https://github.com/armon. Your change in #1296
#1296 looks good but iirc there
were no health checks for these services. Looking at the code I'm surprised
there's not an n^2 loop over services so I'm not sure this alone will fix
it.

—
Reply to this email directly or view it on GitHub
#1265 (comment).

[spheromak]

This may have been causing the slowdown. I am traveling this week but will
try to get a test run with the patch
On Tue, Oct 13, 2015 at 5:29 AM Jesse Nelson spheromak@gmail.com wrote:

No agent checks. We were planning on doing http callbacks for service
status.

On Mon, Oct 12, 2015 at 9:15 PM James Phillips notifications@github.com
wrote:

Hi @spheromak https://github.com/spheromak sorry for the delay on
this, and thanks @armon https://github.com/armon. Your change in #1296
#1296 looks good but iirc
there were no health checks for these services. Looking at the code I'm
surprised there's not an n^2 loop over services so I'm not sure this alone
will fix it.

—
Reply to this email directly or view it on GitHub
#1265 (comment).

[slackpad]

#1296 has been working well - closing this out.