Review Certificate Reconciliation Logic

[charith-elastic]

As noted in #1832, the certificates.Reconcile method has a large footprint in terms of memory usage. This is probably due to parsing of certificates over and over. We should review the logic to see if we can avoid some of this work.

Alloc Space:

[anyasabo]

Some specific examples called out in this PR: #2541

[charith-elastic]

In a single iteration of the Elasticsearch reconciliation loop, the certificates are parsed and encoded multiple times.

• HTTP CA
  • Check that parsing succeeds
  • Check expiry time to create a tentative requeue request
  • Return parsed CA certificates in case the HTTP certificate needs to be regenerated
• HTTP certificate
  • If user-provided, parse to ensure that the secret is valid
  • If self-signed, parse to check whether the certificate should be updated
  • Return either the user-provided or self-signed certificate as a secret which then gets parsed again to check expiry time and create a new tentative requeue request
  • Encode the certificate again to create the public secret
• Transport CA
  • Check that parsing succeeds
  • Check expiry time to create a tentative requeue request
  • Return parsed CA certificates to be used to generated the xpack configuration
  • Encode the certificate again to create the public secret
• Transport certificates
  • For each node in the Elasticsearch cluster:
    • Parse the transport certificate entry to ensure it is valid (and regenerate if necessary)
    • Parse the certificate again to check for expiry and create tentative requeue request
• HTTP certificate chain
  • Parse the certificate chain to pass to the Elasticsearch client

A synthetic benchmark (using the fake client) of certificates.Reconcile produced the following:

goos: linux
goarch: amd64
pkg: github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/certificates
BenchmarkReconcile          500  11719931 ns/op 1246609 B/op      19199 allocs/op
BenchmarkReconcile-2        500  13337516 ns/op 1268568 B/op      19227 allocs/op
BenchmarkReconcile-4        500  10793754 ns/op 1275032 B/op      19219 allocs/op
BenchmarkReconcile-8        500  11304709 ns/op 1279162 B/op      19223 allocs/op

CPU and memory profiles are attached.

As this requires a careful design and a large refactoring to get rid of the amount of unnecessary work done in each iteration, I'll come back to this later after finishing the more urgent tasks on hand.

profile_dumps.tar.gz