fix(bindings): ConfigPool should always yield associated connections

[jmayclin]

Description of changes:

always reset the config on connections yielded from ConfigPool to account for Callbacks that might be swapping the config on connections.

This also adds a doc comment for the ConfigPool confirming that it is a connection pool, and adds a benchmark to get real numbers about the performance savings of connection reuse.

This change degrades ConfigPool performance by about 4%
(772 ns -> 804 ns).

Connection Creation/connection reuse
                        time:   [803.89 ns 804.03 ns 804.17 ns]

Connection Creation/connection allocation
                        time:   [2.6046 µs 2.6057 µs 2.6068 µs]

Testing:

Added a unit test to prevent regression on this behavior.

And added a benchmark to understand the performance of the ConfigPool.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[Mark-Simulacrum]

If you didn't already, it's probably a good idea to re-run this with a more recent allocator (e.g., jemalloc) and see how much that changes the results. Still, ~2 microsecond of difference feels to me like there is probably no reason to use a connection pool -- maybe we have plans that suggest we can get more out of pooling than we do today in this benchmark?

[jmayclin]

Rerunning the benchmarks with jemalloc I got the following results.

Connection Creation/connection reuse
                        time:   [751.49 ns 751.64 ns 751.78 ns]
                        change: [-3.3691% -3.3445% -3.3186%] (p = 0.00 < 0.05)
                        Performance has improved.

Connection Creation/connection allocation
                        time:   [2.2163 µs 2.2170 µs 2.2178 µs]
                        change: [-23.654% -23.616% -23.581%] (p = 0.00 < 0.05)
                        Performance has improved.

Essentially jemalloc improved the connection allocation/free performance by ~23%.

I agree that ~2 microseconds is a relatively small improvement.

maybe we have plans that suggest we can get more out of pooling than we do today in this benchmark?

I'm not aware of any, although we're certainly open to suggestions! My goal with this benchmark was to have some empirical data to help highlight the benefit or lack thereof.

My understanding is that connection reuse was one of the older features of s2n-tls, so perhaps the performance benefit was more significant with older platforms/allocators?

[lrstewart]

Connection reuse is specifically intended for situations where allocation is expensive. There would be no reason to reuse connections with a modern allocator like jemalloc. Unfortunately most of our customers don't use jemalloc ;)