Add HSTS support to Federalist S3 setup

[dhcole]

S3 doesn't currently support HSTS headers or arbitrary custom headers. This means we can't launch Federalist sites with them. We should come up with a solution for this in the medium term. Options include investigating other CDNs that would allow for setting custom headers or a proxy layer between S3 and the CDN.

[dhcole]

Started work on a simple S3 proxy here: https://github.com/dhcole/s4, available for testing at https://federalist-origin.18f.gov/site/dhcole/developer-hub/

The proxy either serves content with the usual /site/owner/repo/ pattern or if its routed through a custom domain, /site/domain.gov/. We'd need to adjust Federalist's build engine to build sites with a custom domain to that path, but this would simply routing for custom domains.

[dhcole]

Just capturing some thoughts for next actions on this. I think a new CDN that supports custom response headers is the easiest way to go, because unlike the proxy, it avoids the need to create a new ELB for each custom domain. However, that step could be automated by some calls to the AWS API, so it may not be a blocker.

The proxy is now set up for the 18f.gsa.gov site, and I've documented that here: 18F/18f.gsa.gov#1430 (comment)

To make this work for all Federalist sites, we'd need to adjust the build script in the application's buildEngine as well as the build container to publish to [bucket]/site/custom-domain.example.gov/ instead of /site/owner/repo/. This would allow for the proxy to match the request's domain with a bucket prefix. As a simple first step, a line could be added to simply copy the site from its current location to [bucket]/site/custom-domain.example.gov/, for example something like:

if [ -n "$DOMAIN" ]; then
  aws s3 sync s3://${BUCKET}/${PREFIX}/ s3://${BUCKET}/site/${DOMAIN}/ --delete
fi

DOMAIN isn't currently passed to the build container, we'd need to set that up in the SQS service of the API and parse out just the domain from the full value of the custom domain attribute, which includes https://.

Once sites are available at that prefix on the bucket, a proxy could be set up with BASE_DOMAIN set to the domain of the proxy, such as federalist-site.18f.gov.

The final part would be making sure cloud.gov routes requests on custom domains to this app. That would require adding a cf route for each custom domain, or setting up a cf global fallback to this particular app (which I believe is possible and could also serve a custom 404 page from Federalist for unavailable routes, instead of the current plain-text message at something like https://no-site-here.18f.gov/).

[dhcole]

Also, for the above, we'd need a simple check to see if a site already exists at a custom domain and if so, show the user an error rather than saving that custom domain to another site.

[wslack]

Requested by 18f.gsa.gov

[konklone]

Requested by me too ;)

[wslack]

S3 doesn't send those. Options are a proxy or something else.

[jmhooper]

A few options:

1. Wait for cloud.gov to leverage Lambda@Edge to enable HSTS on all the CDN brokers. This does not require any changes on our part and the architecture can stay the same
2. Add a proxy between the CDN and S3. This solution has the most complicated end result with the simplest immediate implementation.
3. Replace CDN with a proxy that sets the headers. This solution has the simplest end result with the most complicated implementation. The implementation is difficult because removing the CDN means updating DNS records for every site using a custom domains. This option is only viable if the only value we get from the CDN is support for custom URLs and not the CDN itself.
4. Place a proxy after the CDN. This solution will work, but means creating a proxy for every site, and completely defeats the purpose of having a CDN.

[konklone]

Do 2 for now (simplest implementation), and then strip them out when 1 is accomplished?

[wslack]

Decision: Do #2 while we wait for #1 to be made available by AWS as a native part of cloudfront.

[jmhooper]

federalist-proxy is now deployed to https://federalist-proxy.app.cloud.gov. It proxies content from S3 and add the HSTS header. Now we'll need to update the documentation for custom URLs to instruct site owners to set the origin for CloudFront to federalist-proxy instead of the S3 bucket.

[wslack]

And we have some ATO work to do as well, but this is great!

[wslack]

Verified in retro! This is done; go team!

[brownpl]

It seems like this change may have been reverted recently. I ran an SSL labs test a few weeks ago and the HSTS header was present, and I just ran one today and it is giving me different results.

Any timeline for when it will be added back?

[jseppi]

Can you help us find where you're seeing HSTS missing?

It is present when I just ran a test on https://www.ssllabs.com/ssltest/analyze.html?d=federalist-proxy.app.cloud.gov&s=52.61.61.155

[brownpl]

I was running it against https://collegescorecard.ed.gov. It was present on 1/8 (I have a PDF of a test I ran that day).

[wslack]

@brownpl will be back by EOD tomorrow.

[brownpl]

Awesome. I also noticed a few others that used to be there (X-Frame-Options, X-XSS-Protection, etc.) are absent. Will those also come back with this change? I'd assume so as they are present on federalist-proxy.

[wslack]

@brownpl correct!

[brownpl]

They're back! Thanks for the quick work.

[wslack]

You're welcome! Did you notice the downtime that caused them to get removed last week? 😄

[brownpl]

Ha, I did notice a little blip last week. ;)

[wslack]

The headers were dropped when I worked with cloud.gov to cut the (down) part of the hosting process out of the loop, in order to get the site back up faster. In the future, we can do and undo this action much more quickly, though hopefully we’ll never need it again.