New certificate layout for tiproxy (#5825)

[ti-chi-bot]

This is an automated cherry-pick of #5825

What problem does this PR solve?

This PR introduces a new spec field for tiproxy: certLayout. It determines how tidb-operator mounts certificate related k8s secrets and how to configure TLS related configuration for tiproxy.
For compatibility, the old certLayout is kept as legacy (the default value), the newly added one is the v1. So that upgrading tidb-operator to new version is safe and change nothing.

What is changed and how does it work?

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

• After deploy the new tidb-operator, the existing TCs having tiproxy enabled shouldn't have any pod restarts.
• Set certLayout to `v1 for this tc, the tiproxy pods should be restarted

  tiproxy:
    certLayout: v1

** the tiproxy pods have been restarted

db-tiproxy-0                                          0/1     Terminating   0          6m6s
db-tiproxy-0                                          0/1     Pending       0          0s
db-tiproxy-0                                          0/1     Pending       0          0s
db-tiproxy-0                                          0/1     ContainerCreating   0          0s
db-tiproxy-0                                          1/1     Running             0          1s

• the tiproxy pods should have new cert mount layout

volumeMounts:
    - mountPath: /var/lib/cluster-client-tls
      name: cluster-client-tls
      readOnly: true
    - mountPath: /var/lib/tiproxy-tls
      name: tiproxy-tls
      readOnly: true
    - mountPath: /var/lib/tiproxy-server-tls
      name: tidb-server-tls
      readOnly: true
    - mountPath: /var/lib/tidb-client-tls
      name: tidb-client-tls
      readOnly: true

  volumes:
  - name: cluster-client-tls
    secret:
      defaultMode: 420
      secretName: db-cluster-client-secret
  - name: tiproxy-tls
    secret:
      defaultMode: 420
      secretName: db-tiproxy-cluster-secret
  - name: tidb-server-tls
    secret:
      defaultMode: 420
      secretName: db-tidb-server-secret
  - name: tidb-client-tls
    secret:
      defaultMode: 420
      secretName: db-tidb-client-secret

• The configuration of tiproxy should have been changed to adapt the new cert layout

   [security]
     [security.cluster-tls]
       ca = "/var/lib/tiproxy-tls/ca.crt"
       cert = "/var/lib/tiproxy-tls/tls.crt"
       key = "/var/lib/tiproxy-tls/tls.key"
     [security.server-http-tls]
       ca = "/var/lib/tiproxy-tls/ca.crt"
       cert = "/var/lib/tiproxy-tls/tls.crt"
       key = "/var/lib/tiproxy-tls/tls.key"
       skip-ca = true
     [security.server-tls]
       ca = "/var/lib/tidb-client-tls/ca.crt"  # mount the client CA cert from the `tidb-client-tls` cert
       cert = "/var/lib/tiproxy-server-tls/tls.crt"  # use the same server-side cert/key as the tidb-server
       key = "/var/lib/tiproxy-server-tls/tls.key"
       skip-ca = true
     [security.sql-tls]
       ca = "/var/lib/tiproxy-server-tls/ca.crt"  # mount the tidb server CA cert to verify tidb-server's server-side cert

• Connect to tidb without client certificates should be success

mycli --ssl -htidb-server -uroot -P4000 --password='xxxx' -e 'show variables like "%Ssl%"'
Variable_name	Value
have_openssl	YES
have_ssl	YES
ssl_ca	/var/lib/tidb-server-tls/ca.crt
ssl_cert	/var/lib/tidb-server-tls/tls.crt
ssl_cipher
ssl_key	/var/lib/tidb-server-tls/tls.key

• Connect to tidb with incorrect client cert/key should be failed

 mycli --ssl --ssl-key /tmp/tls.key --ssl-cert /tmp/tls.crt -htidb-server -uroot -P4000 --password='xxxx' -e 'show variables like "%Ssl%"'
(2013, 'Lost connection to MySQL server during query ([SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:2580))')

• Connect to tidb with correct client cert/key should be successfull

mycli --ssl --ssl-key /tmp/tls1.key --ssl-cert /tmp/tls1.crt -htidb-server -uroot -P4000 --password='xxxx' -e 'show variables like "%Ssl%"'

Variable_name	Value
have_openssl	YES
have_ssl	YES
ssl_ca	/var/lib/tidb-server-tls/ca.crt
ssl_cert	/var/lib/tidb-server-tls/tls.crt
ssl_cipher
ssl_key	/var/lib/tidb-server-tls/tls.key

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mikechengwei for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[csuzhangxc]

/run-all-tests

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0% with 162 lines in your changes missing coverage. Please review.

Project coverage is 46.23%. Comparing base (c23a794) to head (d832dcc).
Report is 7 commits behind head on release-1.6.

❗ There is a different number of reports uploaded between BASE (c23a794) and HEAD (d832dcc). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (c23a794)	HEAD (d832dcc)
unittest	1	0

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.6    #5933       +/-   ##
================================================
- Coverage        57.20%   46.23%   -10.97%     
================================================
  Files              259      219       -40     
  Lines            33233    31061     -2172     
================================================
- Hits             19010    14362     -4648     
- Misses           12290    14915     +2625     
+ Partials          1933     1784      -149     

Flag	Coverage Δ
e2e	46.23% <0.00%> (?)
unittest	?

[csuzhangxc]

/run-pull-e2e-kind

[csuzhangxc]

/run-pull-e2e-kind-across-kubernetes

[csuzhangxc]

/run-pull-e2e-kind-serial