Proxying Tomcat AJP dynamic content

Not applicable for profile-hub version 3.0 and above

Intro

When proxying between a hub frontend and a service backend, eg profiles publication downloads, there is some sort of incompatibility between Tomcat 7(.0.55?) with an Apache mod_proxy_ajp frontend and the Java 7 HttpUrlConnection HTTP client. The exact cause is not 100% certain but appears to be related to chunked transfer encoding.

Example.

Assuming an Apache -> AJP -> Tomcat server for both hub and service, running code like this on the hub:

def proxyGetRequest(HttpServletResponse response, String url) {

    HttpURLConnection conn = configureConnection(url, true)

    def headers = [HttpHeaders.CONTENT_DISPOSITION, HttpHeaders.TRANSFER_ENCODING]
    response.setContentType(conn.getContentType())
    response.setContentLength(conn.getContentLength())

    headers.each { header ->
        response.setHeader(header, conn.getHeaderField(header))
    }
    response.status = conn.responseCode
    response.outputStream << conn.inputStream
}

that connects to code like this on the service:

def getPublicationFile() {
    File file = profileService.getPublicationFile(params.publicationId)
    String contentType = Utils.getFileExtension(file.getName())
    if (!file) {
        notFound "The requested file could not be found"
    } else {
        response.setContentType("application/${contentType}")
        response.setHeader("Content-disposition", "attachment;filename=publication.${contentType}")
         response.outputStream << file.newInputStream()
    }
}

Then the hub's HttpUrlConnection inputstream will probably die with an exception when copying to the outputstream.

Configuring your dev environment for AJP

Configure Grails Embedded Tomcat

In your scripts directory (on the service app but you can do hub as well if you like) open or create the _Events.groovy script and add the following to it, adjusting the port number as necessary (I used 8010 for the service and 8009 for the hub):

import grails.util.Environment
import org.apache.catalina.connector.*
import org.apache.catalina.startup.Tomcat

eventConfigureTomcat = { Tomcat tomcat ->
    if (Environment.current == Environment.DEVELOPMENT) {
        println "### Enabling AJP/1.3 connector"

        def ajpConnector = new Connector("org.apache.coyote.ajp.AjpProtocol")
        ajpConnector.port = 8009
        ajpConnector.protocol = 'AJP/1.3'
        ajpConnector.redirectPort = 8443
        ajpConnector.enableLookups = false
        ajpConnector.setProperty('redirectPort', '8443')
        ajpConnector.setProperty('protocol', 'AJP/1.3')
        ajpConnector.setProperty('enableLookups', 'false')
        tomcat.service.addConnector ajpConnector

        println "### Ending enabling AJP connector"
    }
}

Setup the reverse proxy

For OS X Apache, add a .conf file in /etc/apache2/other, eg rp.conf with the following (pay attention to port numbers and adjust context paths as appropriate):

ProxyRequests Off
ProxyPreserveHost On
    
<LocationMatch "/profile-hub">
  ProxyPass ajp://localhost:8009/profile-hub
#  ProxyPass http://localhost:8080/profile-hub
</LocationMatch>

<LocationMatch "/profile-service">
  ProxyPass ajp://localhost:8010/profile-service
#  ProxyPass http://localhost:8081/profile-service
</LocationMatch>

Update the app's external config file

Ensure that both apps are configured to refer to themselves without a port in the myraid of places this is required, eg:

serverURL=http://devt.ala.org.au
grails.serverURL=http://devt.ala.org.au/profile-hub
serverName=http://devt.ala.org.au
security.cas.appServerName=http://devt.ala.org.au

# And also for the link between the hub and service
profile.service.url=http://devt.ala.org.au/profile-service

Potential fix 1

We can try to help by:

• Disabling HTTP Keep Alive, which possibly disables chunking (whether this actual disables chunking is not confirmed and may require the next item)
• giving the HttpUrlConnection more information in the form of a Content-Length header
• Prevent HttpUrlConnection from using caches

Disable Keep Alive and disable caches

On the hub side, add the following to the code:

HttpUrlConnection conn = ...
conn.useCaches = false
conn.setRequestProperty(HttpHeaders.CONNECTION, 'close') // disable Keep Alive

Content Length header

On the service side, add the following to the code (in the case of dynamically generated content this may not be possible)

response.contentLength = (int) file.length()

Other considerations

Always close resources (such as InputStreams and UrlConnections). For example, instead of this:

response.outputStream << file.newInputStream()

Do this:

file.withInputStream { response.outputStream << it }

And instead of this:

HttpURLConnection conn = configureConnection(url, true)
...
response.outputStream << conn.inputStream

Do this:

HttpURLConnection conn = configureConnection(url, true)
try {
    ...
    conn.inputStream.withStream { response.outputStream << it }
} finally {
    conn.disconnect()
}

Potential Fix 2

Use an alternative HTTP client such as Apache HTTP Client or (my choice) Square's OkHttp, which might look like this:

import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.Response
import okhttp3.ResponseBody

...

// TODO inject this
OkHttpClient client = new OkHttpClient();

void proxy(HttpServletResponse response, String url) throws IOException {
    Request request = new Request.Builder().url(url).build()
    Response proxiedResponse = client.newCall(request).execute()
    response.status = proxiedResponse.code()

    def headers = [CONTENT_DISPOSITION, TRANSFER_ENCODING]
    headers.each { header ->
        String headerValue = proxiedResponse.header(header)
        if (headerValue) {
            response.setHeader(header, headerValue)
        }
    }

    proxiedResponse.body().withCloseable { ResponseBody body ->
        response.contentType = body.contentType().toString()
        final contentLength = body.contentLength()
        if (contentLength != -1) response.contentLength = contentLength
        response.outputStream << body.byteStream()
    }
}