Switch to Centos 8 Stream base image

[elfosardo]

Images taken from https://www.centos.org/centos-stream/

[elfosardo]

/test-integration

[fmuyassarov]

/lgtm

[metal3-io-bot]

@fmuyassarov: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dtantsur]

/approve

[elfosardo]

related to metal3-io/metal3-dev-env#671

[namnx228]

/test-integration
/test-centos-integration

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dtantsur, namnx228

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dtantsur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[furkatgofurov7]

That is weird, both tests have run exactly for 2h52min, whereas timeout set for integration tests is 1h30min. Re-triggering to see what went wrong.

/test-integration
/test-centos-integration

[elfosardo]

/test-centos-integration

[elfosardo]

giving it another try, locally seems working
/test-centos-integration

[elfosardo]

it seems we're moving back to centos 8.3 for the time being metal3-io/metal3-dev-env#671
/hold

[furkatgofurov7]

@elfosardo I think we can move forward with this change, since we took back CentOS Stream again into use in metal3-dev-env?
metal3-dev-env#707

[elfosardo]

/unhold

[elfosardo]

/test-centos-integration

[elfosardo]

/test-centos-integration

[furkatgofurov7]

@elfosardo hi! This seems to be failing even though we switched to Stream in m3-dev-env, also a question, should we update vbmc Dockerfile too?

[elfosardo]

@elfosardo hi! This seems to be failing even though we switched to Stream in m3-dev-env, also a question, should we update vbmc Dockerfile too?

I can't reproduce the failure locally :/
yes, we should change the vbmc dockerfile too, I thought to do that in a different patch though

[elfosardo]

the issue seems related to the connection to mariadb
this is from the ironic-api logs
[Wed Jul 07 10:44:53.011685 2021] [wsgi:error] [pid 112:tid 140611176912640] [remote 172.17.0.16:55960] 2021-07-07 10:44:53.009 112 WARNING oslo_db.sqlalchemy.engines [req-aa2c23c9-eee8-42bc-ac2a-42a7b3fbf51a - - - - -] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '127.0.0.1' ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897))")\x1b[00m

[elfosardo]

seems like we're using different versions of oslo-db between ubuntu and centos integration tests
in the centos integration test we use a more recent one (9.1.0) which probably introduces the issue with the db connection

[elfosardo]

/test-integration

[elfosardo]

/test-integration

[elfosardo]

/test-centos-integration

[elfosardo]

/test-integration

[namnx228]

/test-centos-integration

[elfosardo]

After some troubleshooting (thanks @fmuyassarov for setting up the environment) it looks like the issue is related to the certificate used to connect to mariadb.
When connecting to mariadb we use ssl, but the cert used to connect, which is /certs/ca/mariadb/tls.crt, doesn't have any Common Name specified.
The Issuer and the Subject fields are empty.
Not sure where we get those certs from.

[root@minikube mariadb]# openssl x509 -in tls.crt -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 3f:78:00:f4:43:83:85:70:83:00:74:42:fb:21:11:13 Signature Algorithm: sha256WithRSAEncryption Issuer: Validity Not Before: Aug 5 15:10:25 2021 GMT Not After : Nov 3 15:10:25 2021 GMT Subject: Subject Public Key Info:

Even trying with the normal mysql client we get the same error:
[root@minikube mariadb]# mysql -uironic -pchangeme -hlocalhost --ssl_ca=/certs/ca/mariadb/tls.crt ERROR 2026 (HY000): SSL connection error: self signed certificate

Although the certs generated in metal3-dev-env look fine:
[centos@feruz-dev-ci certs]$ pwd /opt/metal3-dev-env/certs [centos@feruz-dev-ci certs]$ openssl x509 -in mariadb.crt -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 6a:aa:75:25:bf:32:cd:5f:cd:51:74:da:21:b4:5d:78:7a:8a:ff:63 Signature Algorithm: sha256WithRSAEncryption Issuer: CN = ironic CA Validity Not Before: Aug 4 14:02:01 2021 GMT Not After : Nov 7 14:02:01 2023 GMT Subject: CN = mariaDB

[elfosardo]

/test-integration

[namnx228]

@elfosardo /certs/ca/mariadb/tls.crt is a self-signed certificate, and it is used as a CA certificate.
The other cert that was generated in @fmuyassarov dev-env is the certificate issued by the CA, and it should looks different from the CA certificate. So, mariadb uses the second cert for the TLS handshake, and ironic or other clients use the first cert to verify if they are talking to the right mariadb. In my opinion, the self-signed certificate should be fine without CN as long as the client can trust and use its public key for verification.

[elfosardo]

@namnx228 it doesn't look like that cert is used as CA according to mariadb configuration:
--ssl=on --ssl_cert=/certs/mariadb/tls.crt --ssl_key=/certs/mariadb/tls.key
the correct option for the CA cert should be ssl_ca

[namnx228]

@elfosardo The one used for --ssl_cert is /certs/mariadb/tls.crt (without ca) while the self-signed one is /certs/ca/mariadb/tls.crt (with ca). The one without ca should have CN.

[elfosardo]

@elfosardo The one used for --ssl_cert is /certs/mariadb/tls.crt (without ca) while the self-signed one is /certs/ca/mariadb/tls.crt (with ca). The one without ca should have CN.

@namnx228 what I mean is that the CA cert is not defined anywhere in mariadb, or at least I can't see it
the problem here is that ironic is using /certs/ca/mariadb/tls.crt to try to establish an ssl connection to mariadb and it's failing because the cert is self-signed and because it doesn't have any Common Name
the ERROR 2026 (HY000): SSL connection error: self signed certificate message looks clear enough

[namnx228]

@elfosardo Yes, I agree that the culprit here is that there is a issue with the CA certificate of Mariadb. I just wonder why we didn't see it in Centos before.
Actually, I also saw a problem with TLS CA certificate before in the CI when we switch from Centos to Centos stream in metal3-dev-env, and the solution was to disable SELinux which is enforced by default. Maybe it can also solve this problem?

[namnx228]

CommonName has been added to Ironic and Mariadb certs in this PR: metal3-io/baremetal-operator#951
Let's see if it can solve the issue of this PR.
/test-integration
/test-centos-integration

[namnx228]

/test-integration
/test-centos-integration

[namnx228]

/test-integration

[fmuyassarov]

/test-integration

[fmuyassarov]

/test-integration

[elfosardo]

/test-integration

[elfosardo]

/test-centos-integration

[elfosardo]

@namnx228 thanks for taking care of that! Let's see if it fixes the issue
I haven't checked but it could be due to an upgrade in mariadb packaged version in centos stream

[elfosardo]

/test-integration

[elfosardo]

/test-centos-integration

[namnx228]

/lgtm