TLS SSL Certificates

Note: This document mostly applies to the BSP environment. For AWS Sandbox environments, public certificates are automatically provisioned through AWS and host certificates are self-signed.

For Data.gov, we use GSA provisioned certificates in the BSP environment. Site certificates cover "public" network facing components like load balancers or CDNs. Host certificates are installed on each host to provide transport encryption between hosts within the network, which is required for security compliance.

Production site certificates are external GSA certificates while non-production/staging site certificates are internal GSA certificates. All host certificates are internal GSA certificates.

The process for getting (or updating) a certificate is as follows:

1. Create a new key and CSR
2. Open a ticket in Service Now and attach the CSR
3. Install the certificate and key
4. Expiration reminder

Each environment will need a site certificate and a host certificate. Wildcard certificates are not allowed by GSA, so we include all hosts/domains via Subject Alternate Names (SANs).

1. Generate a key and CSR

datagov-deploy contains a few scripts to automate the process and reduce the chance for mistakes or missing a host.

gen-csr.sh reads Subject Alternative Names (SANs) from stdin to create a single certificate. Files are generated in gen-csr-out, including the key and CSR.

SANs for site certificates

There should be a text file next to gen-csr.sh for production and staging.

$ cd ansible
$ bin/gen-csr.sh data.gov < bin/production-sans.txt

ansible/gen-csr-out/data.gov.csr will be attached to the Service Now request. Upload the .key file to Google Drive and save it for later. It will be installed to the load balancers and CDNs once the certificate is provisioned by GSA. Use Google Drive's file versioning feature to update the existing file rather than creating a new one (right-click and select Manage Versions).

You can inspect the CSR with openssl.

$ openssl req -text -noout -in gen-csr-out/data.gov.csr

SANs for host certificates

Host certificates get the SANs from the Ansible inventory. Run the gen-csr.sh with the inventory name and feed the hostnames from ansible-inventory to stdin.

$ cd ansible
$ bin/gen-csr.sh $inventory < <(ansible-inventory -i "inventories/$inventory" --list | jq -r '.[].hosts[]?' | sort | uniq)

ansible/gen-csr-out/${inventory}.csr will be attached to the Service Now request. Save the .key file to the Ansible Vault.

You can inspect the CSR with openssl.

$ openssl req -text -noout -in gen-csr-out/$inventory.csr

2. Requesting the certificate

Open a Service Now ticket for GSA SSL certificate request. Fill out the required information.

For production site certificates, you want to request an external certificate. For all other certificates, you want to request an internal certificate.

You will be filling in a lot of redundant information that is included in the CSR. You can inspect the CSR for this information:

$ openssl req -text -noout -in gen-csr-out/data.gov.csr

Some additional fields not in the CSR:

Field	Value
Enhanced Key Usage	Server Authentication (1.3.6.1.5.5.7.3.1)
Responsible SSO	Federal Acquisition Services (Q)
FISMA System	Data.gov
Application Team	FCS Team Approval

Submit the form. Once the request is created you can attach the CSR. Note: On May 20th, 2020 it was not accepting .csr extensions, so I appended .txt before I could upload the CSR.

3. Installing the certificate and key

Provisioning the certificate may take a few days. Once provisioned, the certificate will be attached to the Service Now ticket. The GSA certificate is in PEM format and should be installed with the key generated in step 1.

For site certificates, upload the certificate and key to Google Drive.

Host certificate and keys should be added to the Ansible inventory as default_tls_host_certificate and default_tls_host_key, respectively.

For production site certificates:

Site certificate and keys are installed manually to load balancers, CDNs, and WAFs. You may need to open BSP tickets to get these installed.

For staging site certificates:

There are no CDN's or WAF's for staging. So, you will need to open a ticket with BSP to install the cert on Netscaler. Follow these steps:

Create a generic request using these values:

• Short Description: Install certificate on Netscaler
• Justification for request: [Current certificate is expired] OR [SANs list is modified]. Please install new certificate on Netscaler. Needed for data.gov. Urls are: insert all urls here -> You should be able to get this list from pervious service now ticket.
• Attachments: Attach both the key file you generated and certificate file you received from previous steps.
• Add comments: In addition to install the new certificate on Netscaler, please coordinate with GSA Secops to have it installed on PA due to recent decryption policy change on the Firewalls.

4. Set a reminder before expiration

We're currently manually tracking certificate expiration. Check when the new certificate is due to expire. Add a reminder to the data.gov calendar 60 days before the expiration date.