newer tomcat rejects [:]

[gajowi]

Historically, thredds supports opendap subsetting urls including syntax with [:]. Recent versions of tomcat reject this by default.

See: https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
https://stackoverflow.com/questions/41053653/tomcat-8-is-not-able-to-handle-get-request-with-in-query-parameters
https://stackoverflow.com/questions/50361171/how-to-allow-character-in-urls-for-tomcat-8-5

For unidata I think there are two actions:

1. update server.xml in thredds-docker (to use/set at least relaxedQueryChars='[]:')
2. update general thredds deployment instructions

Other characters may also be in scope. I just hit the problem with opendap subsetting ascii response with 5.0beta5

[julienchastang]

Thanks for the heads up. I'll look at this shortly. (It's Thanksgiving week here in the US).

[julienchastang]

I see how this can give you problems, but is there anything specific to docker here? If you would like to adjust server.xml with any customizations, you can do that by mounting over the server.xml. I do just that here. BTW, this project does not contain a server.xml file. That comes from the parent tomcat-docker project which is meant to be generic, and not specific to the TDS.

It seems like this could be a point of clarification in the TDS documentation, if it is not already. Ping @lesserwhirls.

[lesserwhirls]

We discussed this issue over on the THREDDS issue tracker at Unidata/thredds#1144, although it's a bit lengthy and has some red herrings on our part.

Perhaps most relevant is this particular comment: Unidata/thredds#1144 (comment)

We decided that unless there is an explicit need for a server to allow unencoded urls, we suggest not using the relaxedQueryChars (which is why we haven't documented it as part of the setup/tutorial). We will point out the relaxedQueryChars option in our release announcement, however.

There were some issues with the opendap html page / javascript not encoding the url properly before submitting a request, and this has been fixed in beta6 of 5.0 and the soon-to-be-released 4.6.12.

Interestingly enough, when using an Apache server as a front-end to Tomcat, the illegal characters are handled without issue. This is the setup we use at Unidata, and so we had no idea there was an issue...I expect Apache will start disallowing these too at some point soon.

[julienchastang]

Unless, there are objections, I am going to close this issue as it relates more to the core TDS project than this docker container. In addition, it was discussed more recently in the 4.6.13 release notes.

[julienchastang]

Ugh. I just ran into this issue myself and as this docker container does not use Apache as the front end, I think, at the very least, it needs to be better documented and any example server.xml needs to have a relaxedQueryChars as part of it.

[kwilcox]

@julienchastang any chance you re-visit this and make a "working" server.xml part of this project? I maintain an ERDDAP docker container and recently did the same. The other option is that everyone maintains a custom server.xml files instead of the batteries being included here.

axiom-data-science/docker-erddap@32ba572#diff-4acbaad6af43f602bbb598fba00d43e0R72-R73

[tc33133]

Thanks for mentioning this Kyle. I agree it would be much easier since I don't like the idea of editing the server.xml inside the container. I thinking having users maintain their own server.xml might get messy, and really other than the relaxed characters setting, I never touch the server.xml

[srstsavage]

Running this inside the container (e.g. in $CATALINA_HOME/bin/setenv.sh, if you're already mounting a custom setenv.sh) will update server.xml to accept path characters []| and query characters []|{}^\`"<> without having to mount a custom server.xml.

if ! grep relaxedPathChars $CATALINA_HOME/conf/server.xml > /dev/null; then
  echo "Updating server.xml to allow for special URL characters (needed for DAP requests)"
  echo "See https://github.com/Unidata/thredds-docker/issues/209"
  echo "and https://github.com/Unidata/thredds/releases/tag/v4.6.13"
  sed -i 's~\(port="8080" protocol="HTTP/1.1"\)$~\1 relaxedPathChars="[]|" relaxedQueryChars="[]|{}^\\\`\"\<\>"~' \
    $CATALINA_HOME/conf/server.xml
fi

[julienchastang]

&#x5c is \

&#x60 is `

&quot is "

&lt is <

&gt is >

We will still need these, right?

[ [

] ]

: :

[julienchastang]

Is something like this what you have in mind Unidata/tomcat-docker#69?

Slight variation of axiom-data-science/docker-erddap@32ba572#diff-4acbaad6af43f602bbb598fba00d43e0R72-R73

with the :

[julienchastang]

Closing with Unidata/tomcat-docker#69. Let me know if there are any additional issues.