Environment variables not properly passed in 8.5.16

[jlaci]

Background

We run Spring Boot Web applications in Dockerized Tomcats, configured using Spring Boot's externalized configuration (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html).

When upgrading from 8.5.14 to 8.5.16 we noticed our configurations was no longer working. After some investigations we came to conclude that this is a problem with the docker image, not the SpringBoot itself. To demonstrate and reproduce the issues I created a small demo application setup, which can be found here:
https://github.com/jlaci/SpringBootTomcatTestEnv

Setup

The demo application tries to resolve the

test.variable

property, which is provided in to two places, the application.properties file (which is the default) and overridden in the container by an environmental variable. To illustrate this, here is the application.properties and the docker-compose file for both versions:

application.properties

test.variable=This comes from the default application.properties file

docker-compose.yml

  8516-tomcat:
    image: tomcat:8.5.16
    container_name: 8516-tomcat
    volumes:
      - ./8516webapps:/usr/local/tomcat/webapps/
    environment:
      - test.variable=This is set on the container as an ENV var
      
  8514-tomcat:
    image: tomcat:8.5.14
    container_name: 8514-tomcat
    volumes:
      - ./8514webapps:/usr/local/tomcat/webapps/
    environment:
      - test.variable=This is set on the container as an ENV var

The application prints out both the Spring Boot resolved value and the raw environmental variable visible by the JVM.

Expected outcome

The application then resolves the property and the expected outcome is for the environmental variable to override the default one. This was the case in 8.5.14 but not in 8.5.16:

Logs from 8.5.14

2017-07-26 13:08:13.600  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Spring: Test Variable is: This is set on the container as an ENV var
2017-07-26 13:08:13.600  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Java  : Test Variable is: This is set on the container as an ENV var

Logs from 8.5.16

2017-07-26 13:08:13.350  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Spring: Test Variable is: This comes from the default application.properties file
2017-07-26 13:08:13.350  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Java  : Test Variable is: null

Eliminated causes

Docker / Docker compose

A manually created container also displays the issue:

docker create --name 8516-manual-tomcat -v /Users/jlaci/Temp/SpringBootEnvTest/8516_manual_webapps:/usr/local/tomcat/webapps/ -e test.variable=This_is_set_on_the_container_as_an_ENV_var tomcat:8.5.16

When listing the environment variables for the containers, apparently each container has the required variable:

8.5.14

CATALINA_HOME=/usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION=20161107~bpo8+1
GPG_KEYS=05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME=/root
HOSTNAME=e35e5d4b67fe
JAVA_DEBIAN_VERSION=8u121-b13-1~bpo8+1
JAVA_HOME=/docker-java-home/jre
JAVA_VERSION=8u121
LANG=C.UTF-8
LD_LIBRARY_PATH=/usr/local/tomcat/native-jni-lib
OPENSSL_VERSION=1.1.0e-1
PATH=/usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TOMCAT_ASC_URL=https://www.apache.org/dist/tomcat/tomcat-8/v8.5.14/bin/apache-tomcat-8.5.14.tar.gz.asc
TOMCAT_MAJOR=8
TOMCAT_NATIVE_LIBDIR=/usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.14/bin/apache-tomcat-8.5.14.tar.gz
TOMCAT_VERSION=8.5.14
no_proxy=*.local, 169.254/16
test.variable=This is set on the container as an ENV var

8.5.16

CATALINA_HOME=/usr/local/tomcat
CA_CERTIFICATES_JAVA_VERSION=20170531+nmu1
GPG_KEYS=05AB33110949707C93A279E3D3EFE6B686867BA6 07E48665A34DCAFAE522E5E6266191C37C037D42 47309207D818FFD8DCD3F83F1931D684307A10A5 541FBE7D8F78B25E055DDEE13C370389288584E7 61B832AC2F1C5A90F0F9B00A1C506407564C17A3 713DA88BE50911535FE716F5208B0AB1D63011C7 79F7026C690BAA50B92CD8B66A3AD3F4F22C4FED 9BA44C2621385CB966EBA586F72C284D731FABEE A27677289986DB50844682F8ACB77FC2E86E29AC A9C5DF4D22E99998D9875A5110C01C5A2F6059E7 DCFD35E0BF8CA7344752DE8B6FB21E8933C60243 F3A04C595DB5B6A5F1ECA43E3B7BBB100D811BBE F7DA48BB64BCB84ECBA7EE6935CD23C10D498E23
HOME=/root
HOSTNAME=d2c60a8e8f13
JAVA_DEBIAN_VERSION=8u131-b11-2
JAVA_HOME=/docker-java-home/jre
JAVA_VERSION=8u131
LANG=C.UTF-8
LD_LIBRARY_PATH=/usr/local/tomcat/native-jni-lib
OPENSSL_VERSION=1.1.0f-3
PATH=/usr/local/tomcat/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TOMCAT_ASC_URL=https://www.apache.org/dist/tomcat/tomcat-8/v8.5.16/bin/apache-tomcat-8.5.16.tar.gz.asc
TOMCAT_MAJOR=8
TOMCAT_NATIVE_LIBDIR=/usr/local/tomcat/native-jni-lib
TOMCAT_TGZ_URL=https://www.apache.org/dyn/closer.cgi?action=download&filename=tomcat/tomcat-8/v8.5.16/bin/apache-tomcat-8.5.16.tar.gz
TOMCAT_VERSION=8.5.16
no_proxy=*.local, 169.254/16
test.variable=This is set on the container as an ENV var

Tomcat itself

When run natively on the host os with the following command:

env "test.variable=This is set on the container as an ENV var" ./catalina.sh run

both version works fine

Logs from 8.5.14

2017-07-26 13:29:46.429  INFO 2779 --- [ost-startStop-1] hu.jlaci.Application                     : Spring: Test Variable is: This is set on the container as an ENV var
2017-07-26 13:29:46.429  INFO 2779 --- [ost-startStop-1] hu.jlaci.Application                     : Java  : Test Variable is: This is set on the container as an ENV var

Logs from 8.5.16

2017-07-26 13:30:17.043  INFO 2791 --- [ost-startStop-1] hu.jlaci.Application                     : Spring: Test Variable is: This is set on the container as an ENV var
2017-07-26 13:30:17.043  INFO 2791 --- [ost-startStop-1] hu.jlaci.Application                     : Java  : Test Variable is: This is set on the container as an ENV var

Spring Boot

The System.getenv(); call doesn't work either and natively the application is fine.

Workaround

If we use the KEY_NAME=value syntax instead of the key.name=value 8.5.16 works fine:

Docker-Compose

  8516-tomcat-different-syntax:
    image: tomcat:8.5.16
    container_name: 8516-tomcat-different-syntax
    volumes:
      - ./8516webapps:/usr/local/tomcat/webapps/
    environment:
      - TEST_VARIABLE=This is set on the container as an ENV var with a different syntax

Logs from 8.5.16 with different syntax

2017-07-26 13:08:13.618  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Spring: Test Variable is: This is set on the container as an ENV var with a different syntax
2017-07-26 13:08:13.618  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Java  : Test Variable is: null

Conclusion

The support for environmental variables with uncapitalized keys seperated by dots has been lost between 8.5.14 and 8.5.16.

[yosifkit]

This is related to docker-library/openjdk#135. I will make a PR to update catalina.sh to use bash rather than sh (and will do the same for the alpine images) so that tomcat applications can use environment variables with periods in them.

[yosifkit]

Without rebuilding the tomcat image I used your test with just one minor tweak and it worked:

version: '2'
services:
  8516-tomcat:
    image: tomcat:8.5.16
    container_name: 8516-tomcat
    volumes:
      - ./8516webapps:/usr/local/tomcat/webapps/
    environment:
      - test.variable=This is set on the container as an ENV var
    command:
      - 'bash'
      - 'catalina.sh'
      - 'run'

8516-tomcat    | 2017-07-27 22:23:50.035  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Spring: Test Variable is: This is set on the container as an ENV var
8516-tomcat    | 2017-07-27 22:23:50.036  INFO 1 --- [ost-startStop-1] hu.jlaci.Application                     : Java  : Test Variable is: This is set on the container as an ENV var

And it still is just java running in the container (bash gets out of the way):

$ docker top 8516-tomcat
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                28014               27996               2                   15:18               ?                   00:00:14            /docker-java-home/jre/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start

[jlaci]

Thank you for your help, both with the pull request and the workaround! It works and its rather painless.

[JeroenKnoops]

The change to bash seems to break the locale. When using tomcat:jre8 the js files have encoding issues. Reverting it to tomcat:8.5.14 will resolve these issues.

[tomasdev]

I believe this fix is removing javac from /usr/bin

[yosifkit]

@tomasdev, the tomcat images only install the jre and so have never come with javac which is part of openjdk-*-jdk-headless packages.

[tomasdev]

True, but apt-get purge -y --auto-remove $nativeBuildDeps is removing all JDK deps when trying to extend this script (use case: tomcat + solr within same container)
Maybe I'm completely wrong.

[yosifkit]

This Dockerfile was written assuming that we need to install the jdk to build tomcat native; if you make your own Dockerfile to change to a base that already has the jdk installed, then you'll need to adjust any packages that are added or removed in the RUN lines.