Limit total global Hystrix threads

[mikeycohen]

Systems can sprial out of control with hystrix threads as the number of hystrix commands increase. During periods of latency, multiple pools may be affected, causing rapid runnable thread expansion. We should be able to cap total hystrix threads.

[benjchristensen]

Agreed, thanks for posting this. Hopefully @KoltonAndrus and I can get to this in Hystrix 1.4 in the near future.

[mattrjacobs]

I see a couple of options for addressing this problem.

1. Just like each individual Hystrix thread pool has a size, and exceeding that number of active threads results in a thread-pool rejection, we could create a synthetic limit on the sum of all active threads, and exceeding that would result in a synthetic thread-pool rejection.

2. Hystrix already exposes the hooks onThreadStart and onThreadComplete. You can write a custom hook that keeps track of the number of outstanding threads and then consults a policy on how to handle different levels of saturation. For instance, you could check this value upon each receipt of an HTTP request, and then 503 it if the value was too high, hopefully shedding enough load to keep your system up.

My preference is Option 2, as not all applications will want to opt-in to this behavior, and it doesn't add any cognitive load to Hystrix-core.

Thoughts? /cc @mikeycohen @benjchristensen @KoltonAndrus ?

[benjchristensen]

I agree with option 2.

[mattrjacobs]

Punting to 1.4.x

[mattrjacobs]

I just implemented a middle-ground between options 1+2 above. There is now a static method (HystrixCounters.getGlobalConcurrentThreadsExecuting()) which directly returns the number of Hystrix threads executing across the JVM.

This doesn't affect any core Hystrix behavior, but makes it easier to build a custom strategy in any application which consumes Hystrix.