included LowerCaseQueryStringMiddleware middleware affects query strings

[jackharrhy]

Problem description

Currently, the LowerCaseQueryStringMiddleware included in middleware.py seems to affect the query_string property on request.scope such that it strips it from url encoded values when passed to GDAL.

Under some circumstances, this is fine and causes no problems, but I'm running into issues in which presigned URLs (with contain query a few extra query parameters) are being handed to /vsicurl/ within GDAL incorrectly:

"'/vsicurl/https://storage.googleapis.com/<...>?X-Goog-Algorithm=GOOG4-RSA-SHA256' does not exist in the file system, and is not recognized as a supported dataset name."

The above path should show much more than just X-Goog-Algorithm, the original request has many more query params after it, but this was the first one, hence the rest are being removed somehow.

I'm not sure what is trimming out the full path before being passed to GDAL, but after temporarily removing the LowerCaseQueryStringMiddleware, the problem goes away.

...
            DECODE_FORMAT = "latin-1"

            print("query params", request.query_params.multi_items())

            query_string = ""
            for k, v in request.query_params.multi_items():
                query_string += k.lower() + "=" + v + "&"

            print("query string before", request.scope["query_string"])

            query_string = query_string[:-1]
            request.scope["query_string"] = query_string.encode(DECODE_FORMAT)

            print("query string after", request.scope["query_string"])
...

I added the following code to middleware.py, and got the following output:

query params [('url', '[https://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif?abc=123')](https://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif?abc=123%27))]
query string before b'url=https%3A%2F%2Foin-hotosm.s3.amazonaws.com%2F59c66c5223c8440011d7b1e4%2F0%2F7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif%3Fabc%3D123'
query string after b'url=https://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif?abc=123'
INFO:     127.0.0.1:57572 - "GET /cog/tiles/WebMercatorQuad/5/10/14%401x?url=https://oin-hotosm.s3.amazonaws.com/59c66c5223c8440011d7b1e4/0/7ad397c0-bba2-4f98-a08a-931ec3a6e943.tif?abc=123 HTTP/1.1" 404 Not Found

It seems this isn't where the items are being trimmed, but the urlencoded string is being converted to a non

Expected Output

LowerCaseQueryStringMiddleware should not mutate the query_string parameter in such a way that later on GDAL fails to be passed all of the query parameters.

Environment Information

Ubuntu, Python 3.10.11, titiler-core ^0.12.0

---

An attempted fix is available in #677