Concerns with String.toLowerCase() in default Locale

[dimas-b]

Query engine

ALL

Question

This stemmed from a discussion thread on PR 8909:

I did a quick scan of the calls to String.toLowerCase() in Iceberg codebase, and I do a few places where it may be a concern:

1. VectorizedSupport seems like it may be a problem, but I do not really know whether the lower case data is exposed and how.

2. IcebergRecordObjectInspector converts field names to lower case, and that seems to be affected by the locale problem as various case names are accepted as input parameters to its methods.

3. jQuery code uses a lot of toLowerCase(), but I do not really know how it is supposed to be used.

To the best of my knowledge this kind of case conversion can be problematic only in German and Turkish locales. The German locale affects only proper German language words (so it is less of a problem), but the Turkish locale can cause English words to be converted in unexpected ways.

For example, this assertion fails: assertThat("VIEW".toLowerCase(new Locale("TR"))).isEqualTo("view");

Does Iceberg support using its libraries in user-defined locales?

[findepi]

Does Iceberg support using its libraries in user-defined locales?

Not for me to decide this, but i believe that we have basically these options

1. make the code independent of JVM locale (Palantir's error-prone check DefaultLocale can be helpful, but seems insufficient)
2. test the code on various locales including Turkish
3. add a runtime check enforcing JVM locale

Of these I think (3) is least awesome, because Iceberg is a library and should support being embedded in various contesxts. I think we should do (1).

[findepi]

I did a quick scan of the calls to String.toLowerCase() in Iceberg codebase

The toLowerCase & toUpperCase calls are being fixed in #10521.
There may be other Locale-dependent APIs though.