Apache default site

[ghost]

The Apache default site is what gets served when a request doesn't match a virtual host.
This happens when visitors go to places that you don't expect, for example: the server IP address, a DNS record that isn't used to host a site (ocp02.open-contracting.org), a site we've switched off or if someone points a domain we have no control over at the server.

The biggest risk with an absent default site is that requests to these endpoints returns one of our sites, this would lead to duplicate content which is bad for SEO.

Currently HTTP traffic receives the Apache default site which is OK, it addresses the risks of this issue but it does look ugly.

HTTPS however doesn't having a specific default site and needs addressing.

Best practice is to serve a custom holding page for these requests or to redirect (302 redirect) to an actual site.

[ghost]

Ideally, the holding page would also be served for all paths under the domain.
Especially if it is replacing a site that we've turned off.

Currently the apache default page is being returned for just the main FQDN, paths under it receive 404s.

[jpmckinney]

When a disallowed host reaches Django, it responds with 400 Bad Request: https://docs.djangoproject.com/en/3.1/ref/exceptions/#django.core.exceptions.SuspiciousOperation

Can you link to some best practice? If best practice is to serve/redirect content, I'm fine with just redirecting everything to https://www.open-contracting.org

[jpmckinney]

For HTTPS, I assume we don't need certs – but for any traffic that disregards the lack of certs, we can still respond.

[ghost]

I don't have any links regarding best practices for how to configure the default site I'm afraid.
It is common practice to disable the inbuilt Apache page.
It is bad to serve the same site on multiple domains as this is duplicate content.

For HTTPS we'll need a cert but we're not worried about it being valid because no legitimate traffic should end up here. We often use an SSL cert matching the server hostname or a self signed one.

[jpmckinney]

I was curious whether it's better to error with HTTP 400 (simplest), serve some holding page or redirect to another site.

[ghost]

I'm happy with returning an error code, probably 404 or 410 depending on the link.
In NGINX it has a "444" status that just drops connection and returns nothing to the client, I've set that up as the default site before. Not sure what the Apache equivalent is.

If we switch off a site and it falls back to this it might be nice to send visitors a "OCP" page rather than anything systematic.

[jpmckinney]

I think 444 is nginx-specific. I'm happy with 404.

Update: We haven't yet switched off a site, but we can add a note here to consider different behavior if/when that happens: https://ocdsdeploy.readthedocs.io/en/latest/develop/update/delete.html#delete-a-virtual-host

Putting a simple way to check the behavior here:

for i in {1..9}; do echo $i; curl --connect-timeout 1 -I http://ocp`printf '%02d' $i`.open-contracting.org; done

where Content-Length: 10918 means it's the same Apache default page.

[jpmckinney]

This configuration would do the same as "444" in nginx (drop the connection): https://serverfault.com/a/859463/542637

Since this needs to work on IP addresses as well, I think self-signed certs are the only way to go.

[jpmckinney]

Okay, I made changes to #434 and deployed to ocp23. You can see:

• http://ocp23.open-contracting.org
• http://ocp23.open-contracting.org/non-existent
• https://ocp23.open-contracting.org
• https://ocp23.open-contracting.org/non-existent

The first 3 are the same. I haven't configured the default default-ssl.conf (copied below), so the last 404 page above is just the default Apache 404 page. I think that's fine.

Does this all look good, then?

<IfModule mod_ssl.c>
	<VirtualHost _default_:443>
		ServerAdmin webmaster@localhost

		DocumentRoot /var/www/html

		# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
		# error, crit, alert, emerg.
		# It is also possible to configure the loglevel for particular
		# modules, e.g.
		#LogLevel info ssl:warn

		ErrorLog ${APACHE_LOG_DIR}/error.log
		CustomLog ${APACHE_LOG_DIR}/access.log combined

		# For most configuration files from conf-available/, which are
		# enabled or disabled at a global level, it is possible to
		# include a line for only one particular virtual host. For example the
		# following line enables the CGI configuration for this host only
		# after it has been globally disabled with "a2disconf".
		#Include conf-available/serve-cgi-bin.conf

		#   SSL Engine Switch:
		#   Enable/Disable SSL for this virtual host.
		SSLEngine on

		#   A self-signed (snakeoil) certificate can be created by installing
		#   the ssl-cert package. See
		#   /usr/share/doc/apache2/README.Debian.gz for more info.
		#   If both key and certificate are stored in the same file, only the
		#   SSLCertificateFile directive is needed.
		SSLCertificateFile	/etc/ssl/certs/ssl-cert-snakeoil.pem
		SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

		#   Server Certificate Chain:
		#   Point SSLCertificateChainFile at a file containing the
		#   concatenation of PEM encoded CA certificates which form the
		#   certificate chain for the server certificate. Alternatively
		#   the referenced file can be the same as SSLCertificateFile
		#   when the CA certificates are directly appended to the server
		#   certificate for convinience.
		#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt

		#   Certificate Authority (CA):
		#   Set the CA certificate verification path where to find CA
		#   certificates for client authentication or alternatively one
		#   huge file containing all of them (file must be PEM encoded)
		#   Note: Inside SSLCACertificatePath you need hash symlinks
		#		 to point to the certificate files. Use the provided
		#		 Makefile to update the hash symlinks after changes.
		#SSLCACertificatePath /etc/ssl/certs/
		#SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt

		#   Certificate Revocation Lists (CRL):
		#   Set the CA revocation path where to find CA CRLs for client
		#   authentication or alternatively one huge file containing all
		#   of them (file must be PEM encoded)
		#   Note: Inside SSLCARevocationPath you need hash symlinks
		#		 to point to the certificate files. Use the provided
		#		 Makefile to update the hash symlinks after changes.
		#SSLCARevocationPath /etc/apache2/ssl.crl/
		#SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl

		#   Client Authentication (Type):
		#   Client certificate verification type and depth.  Types are
		#   none, optional, require and optional_no_ca.  Depth is a
		#   number which specifies how deeply to verify the certificate
		#   issuer chain before deciding the certificate is not valid.
		#SSLVerifyClient require
		#SSLVerifyDepth  10

		#   SSL Engine Options:
		#   Set various options for the SSL engine.
		#   o FakeBasicAuth:
		#	 Translate the client X.509 into a Basic Authorisation.  This means that
		#	 the standard Auth/DBMAuth methods can be used for access control.  The
		#	 user name is the `one line' version of the client's X.509 certificate.
		#	 Note that no password is obtained from the user. Every entry in the user
		#	 file needs this password: `xxj31ZMTZzkVA'.
		#   o ExportCertData:
		#	 This exports two additional environment variables: SSL_CLIENT_CERT and
		#	 SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
		#	 server (always existing) and the client (only existing when client
		#	 authentication is used). This can be used to import the certificates
		#	 into CGI scripts.
		#   o StdEnvVars:
		#	 This exports the standard SSL/TLS related `SSL_*' environment variables.
		#	 Per default this exportation is switched off for performance reasons,
		#	 because the extraction step is an expensive operation and is usually
		#	 useless for serving static content. So one usually enables the
		#	 exportation for CGI and SSI requests only.
		#   o OptRenegotiate:
		#	 This enables optimized SSL connection renegotiation handling when SSL
		#	 directives are used in per-directory context.
		#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
		<FilesMatch "\.(cgi|shtml|phtml|php)$">
				SSLOptions +StdEnvVars
		</FilesMatch>
		<Directory /usr/lib/cgi-bin>
				SSLOptions +StdEnvVars
		</Directory>

		#   SSL Protocol Adjustments:
		#   The safe and default but still SSL/TLS standard compliant shutdown
		#   approach is that mod_ssl sends the close notify alert but doesn't wait for
		#   the close notify alert from client. When you need a different shutdown
		#   approach you can use one of the following variables:
		#   o ssl-unclean-shutdown:
		#	 This forces an unclean shutdown when the connection is closed, i.e. no
		#	 SSL close notify alert is send or allowed to received.  This violates
		#	 the SSL/TLS standard but is needed for some brain-dead browsers. Use
		#	 this when you receive I/O errors because of the standard approach where
		#	 mod_ssl sends the close notify alert.
		#   o ssl-accurate-shutdown:
		#	 This forces an accurate shutdown when the connection is closed, i.e. a
		#	 SSL close notify alert is send and mod_ssl waits for the close notify
		#	 alert of the client. This is 100% SSL/TLS standard compliant, but in
		#	 practice often causes hanging connections with brain-dead browsers. Use
		#	 this only for browsers where you know that their SSL implementation
		#	 works correctly.
		#   Notice: Most problems of broken clients are also related to the HTTP
		#   keep-alive facility, so you usually additionally want to disable
		#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
		#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
		#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
		#   "force-response-1.0" for this.
		# BrowserMatch "MSIE [2-6]" \
		#		nokeepalive ssl-unclean-shutdown \
		#		downgrade-1.0 force-response-1.0

	</VirtualHost>
</IfModule>

[jpmckinney]

Note that the 1st and 3rd links respond with status 200. I could move index.html to another (unreachable) path, but then I'll also need to set Options -Indexes to not display a directory listing when visiting the root – but Options -Indexes causes 403, so I'll need to also add ErrorDocument 403 – but now the error code is 403 when it should be 404... Anyway, maybe @RobHooper you know a better way.

[jpmckinney]

Aha, I can disable the autoindex module, so directory listings are never generated (which I think is fine – it produces a scary warning, but I don't know that anything depends on being able to generate listings, which seems insecure to allow...):

WARNING: The following essential module will be disabled.
This might result in unexpected behavior and should NOT be done
unless you know exactly what you are doing!
 autoindex

To continue type in the phrase 'Yes, do as I say!' or retry by passing '-f':

I've also now added the self-signed SSL certs to the default site. So, all links 404 without any complicated configuration.

I've now also tested on the IP:

• http://65.109.102.188
• http://65.109.102.188/non-existent
• https://65.109.102.188
• https://65.109.102.188/non-existent

[jpmckinney]

If we switch off a site and it falls back to this it might be nice to send visitors a "OCP" page rather than anything systematic.

In the end, I prefer this option, rather than just dropping the connection. So, closing as complete. The new configuration will spread to other servers as they get deployed.