Error connecting to Jira sever with self-signed certificate - SSLCertVerificationError - certificate verify failed: unable to get local issuer certificate

[mh182]

We have an on-premise Jira installation which uses a self-signed certificate. I integrated the self-signed certificate int the OS infrastructure (I can connect with curl). `

The tap-jira extractor fails to handle self-signed certificates. Using REQUESTS_CA_BUNDLE or SSL_CERT_FILE had no effect. Both environment variables worked for me when I created a simple HTTP client using urllib3 (which is used by the plugin). tap-jira uses certifi to validate SSL certificates, ignoring the environment variables.

Is there a way to provide the self-signed root certificate to the extractor?

Here is a setup to reproduce the error - precondition:

• Jira server with self-signed certificate
• Extract the certificate and integrate it on your client's root certificate database (for example, with update-ca-certificate on Debian)

➜ meltano add extractor tap-jira --variant singer-io
Added extractor 'tap-jira' to your project
Variant:	singer-io
Repository:	https://github.com/singer-io/tap-jira
Documentation:	https://hub.meltano.com/extractors/tap-jira--singer-io

2024-06-29T07:54:23.010831Z [info     ] Installing extractor 'tap-jira'
2024-06-29T07:54:33.398365Z [info     ] Installed extractor 'tap-jira'

To learn more about extractor 'tap-jira', visit https://hub.meltano.com/extractors/tap-jira--singer-io

# Configure the plugin - replaced our domain name with example.com
➜ meltano config tap-jira set base_url "https://jira.example.com"
➜ meltano config tap-jira set start_date 2024-06-28
➜ echo TAP_JIRA_USERNAME=your_jira_user >> .env
➜ echo TAP_JIRA_PASSWORD=your_jira_password >> .env

➜ meltano config tap-jira test
2024-06-29T07:54:48.547509Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/home/max/dev/qxvp/mx-dev-analytics/.meltano/extractors/tap-jira/venv/bin/tap-jira', '--config', '/home/max/dev/qxvp/mx-dev-analytics/.meltano/run/tap-jira/tap.74b43b5c-493a-4b2a-9dcb-bf01818f8959.config.json', '--discover'] returned 1 with stderr:
 INFO Using Basic Auth API authentication
INFO Backing off send(...) for 1.0s (requests.exceptions.SSLError: HTTPSConnectionPool(host='jira.example.com', port=443): Max retries exceeded with url: /rest/api/2/serverInfo (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))))
...

I found a hacky work-around by injecting pip-system-certs into the extractors virtual environment:

➜ . .meltano/extractors/tap-jira/venv/bin/activate
(venv) ➜ python -m pip install pip-system-certs
(venv) ➜ deactivate
➜ meltano config tap-jira test
2024-06-29T08:12:21.188098Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Plugin configuration is valid