rucio-server container does not allow x509_proxy authentication

[maxnoe]

When trying to authenticate using an x509_proxy, the httpd responds with proxy certificates not allowed:

[Fri Jun 14 08:05:15.966306 2024] [ssl:info] [pid 34:tid 56] [client 172.21.0.8:34670] AH02276: Certificate Verification: Error (40): proxy certificates not allowed, please set the appropriate flag [subject: CN=1209913245,CN=DPPS User / issuer: CN=DPPS User / serial: 481DCF9D / notbefore: Jun 14 08:04:12 2024 GMT / notafter: Aug  4 23:04:12 2025 GMT]

[bari12]

Did you enable RUCIO_HTTPD_GRID_SITE_ENABLED ?

[maxnoe]

No... even when checking the env variables the docker image uses I didn't make that connection...

Is that documented somewhere? What else does this option enable / change?

[bari12]

I don't think it's documented, but we should perhaps be mentioning it here https://github.com/rucio/containers/tree/master/server

I am not 100% sure if this fixes the issue, but in principle you need to run mod_gridsite in apache to handle x509 proxies. So it is at least a prerequisite.

[maxnoe]

Trying it now....

[maxnoe]

Ok, one step further, I now get a different error:

[user@d242836a00af dpps-docker-compose]$ rm -rf /tmp/user/.rucio_user/
[user@d242836a00af dpps-docker-compose]$ RUCIO_ACCOUNT=root RUCIO_AUTH_TYPE=x509_proxy RUCIO_X509_CLIENT_PROXY=/tmp/x509up_u1000 rucio -vvv whoami
2024-06-14 14:59:02,018	DEBUG	baseclient.py	No trace_host passed. Using rucio_host instead
2024-06-14 14:59:02,019	DEBUG	baseclient.py	No creds passed. Trying to get it from the config file.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	HTTPS is required, but no ca_cert was passed. Trying to get it from X509_CERT_DIR.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	HTTPS is required, but no ca_cert was passed and X509_CERT_DIR is not defined. Trying to get it from the config file.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	No account passed. Trying to get it from the RUCIO_ACCOUNT environment variable or the config file.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	No VO passed. Trying to get it from environment variable RUCIO_VO.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	No VO found. Trying to get it from the config file.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	No VO found. Using default VO.
2024-06-14 14:59:02,019	DEBUG	baseclient.py	get a new token
2024-06-14 14:59:02,019	DEBUG	baseclient.py	HTTP request: GET https://rucio-server/auth/x509_proxy
2024-06-14 14:59:02,019	DEBUG	baseclient.py	HTTP header:  X-Rucio-Auth-Token: [hidden]
2024-06-14 14:59:02,019	DEBUG	baseclient.py	HTTP header:  X-Rucio-VO: def
2024-06-14 14:59:02,019	DEBUG	baseclient.py	HTTP header:  Connection: Keep-Alive
2024-06-14 14:59:02,020	DEBUG	baseclient.py	HTTP header:  User-Agent: rucio-clients/34.4.3
2024-06-14 14:59:02,020	DEBUG	baseclient.py	HTTP header:  X-Rucio-Script: rucio::-vvv
2024-06-14 14:59:02,020	DEBUG	baseclient.py	HTTP header:  X-Rucio-Account: root
2024-06-14 14:59:02,023	ERROR	ConnectionError: ('Connection aborted.', IsADirectoryError(21, 'Is a directory'))
2024-06-14 14:59:02,025	ERROR	ConnectionError: ('Connection aborted.', IsADirectoryError(21, 'Is a directory'))
2024-06-14 14:59:02,027	ERROR	ConnectionError: ('Connection aborted.', IsADirectoryError(21, 'Is a directory'))
2024-06-14 14:59:02,027	ERROR	Cannot connect to the Rucio server.
Completed in 0.0092 sec.

[maxnoe]

In case this might help narrow down the issue: it seems that via curl, the request goes through:

[user@5607e50d4eec dpps-docker-compose]$ curl -v -f \
    --cert /tmp/x509up_u1000 \
    --cacert /etc/grid-security/certificates/74df993b.0 \
    -H "X-Rucio-Account: root" \
    -H "X-Rucio-VO: def" https://rucio-server/auth/x509_proxy 
*   Trying 192.168.64.7:443...
* Connected to rucio-server (192.168.64.7) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/grid-security/certificates/74df993b.0
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=rucio-server
*  start date: Jun 13 16:49:55 2024 GMT
*  expire date: Feb  2 16:49:55 2049 GMT
*  subjectAltName: host "rucio-server" matched cert's "rucio-server"
*  issuer: CN=DPPS Development CA
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /auth/x509_proxy HTTP/1.1
> Host: rucio-server
> User-Agent: curl/7.76.1
> Accept: */*
> X-Rucio-Account: root
> X-Rucio-VO: def
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 25 Jun 2024 08:47:10 GMT
< Server: Apache/2.4.57 (AlmaLinux) OpenSSL/3.0.7 mod_wsgi/4.7.1 Python/3.9 mod_gridsite/3.0.0
< Content-Length: 0
< Access-Control-Allow-Origin: None
< Access-Control-Allow-Headers: None
< Access-Control-Allow-Methods: *
< Access-Control-Allow-Credentials: true
< Access-Control-Expose-Headers: X-Rucio-Auth-Token, X-Rucio-Auth-Token-Expires, X-Rucio-Auth-Account, X-Rucio-Auth-Accounts
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< X-Rucio-Auth-Token: root-CN=DPPS User-unknown-9134466d1a09444b87e66e28c69167ab
< X-Rucio-Auth-Token-Expires: Tue, 25 Jun 2024 09:47:10 UTC
< X-Rucio-Auth-Account: root
< X-Rucio-Host: rucio-server
< Content-Type: application/octet-stream
< 
* Connection #0 to host rucio-server left intact

[maxnoe]

Debugging with changing log messages to show the full exceptions and adding some addtional logs, I found that for some reason, the client tries to use my current working directory as location for the client certificate:

2024-06-25 09:04:32,204	INFO	Using client certificate from path '/home/user'
2024-06-25 09:04:32,207	ERROR	ConnectionError: ('Connection aborted.', IsADirectoryError(21, 'Is a directory'))
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 418, in ssl_wrap_socket
    context.load_cert_chain(certfile, keyfile)
IsADirectoryError: [Errno 21] Is a directory

[maxnoe]

The reason for this is that my config, due to how the docker system is setup, contained an empty entry for the client proxy.

Which due to this code:

https://github.com/rucio/rucio/blob/0bbc069faf17f06eff1891aa90e634adea67b216/lib/rucio/client/baseclient.py#L292-L309

leads to the current working directory being inserted into the cert option

[bari12]

Would you suggest to change something in this case? I am not sure how much this is a Rucio issue or how much this stems from your docker setup? :-)

[bari12]

Ahh, should have read my next eMail first. I guess #6845 is the followup from this. Can we close this containers issue then?

[maxnoe]

A fix here could be to also not have the empty sections in the default config that gets merged