commit 304488d3... (2025-02-05) broke JOIN ... USING("UPPERCASE_FIELD_NAME")

[brunal]

Describe the bug

Commit 304488d updated sqlparser (0.53->0.54) and updated datafusion source for it.

It broke SQL queries such as

    SELECT test."UPPER" FROM "test"
    INNER JOIN (
        SELECT test."UPPER" FROM "test"
    ) AS selection USING ("UPPER")

This query fails with the message:
Schema error: No field named upper. Valid fields are test."UPPER".

You can find attached a patch that creates a test for this query. You can run it with
$ cargo test -p datafusion -- sql::uppercase_fields::test_uppercase_columns --exact --nocapture

using_uppercase_bug.patch

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[jfahne]

take

[alamb]

Thank you for the report @brunal and for taking this @jfahne

[jfahne]

Okay so I dug through it and found the error is coming from the following call chain:

• The LogicalPlanBuilder returned by the parse_join calls join_using on the builder (link)
• The left_keys calculation from join_using calls the LogicalPlanBuilder method, normalize (link)
• The normalize method calls the Column struct method, normalize_with_schemas_and_ambiguity_check (link)
• The normalize_with_schemas_and_ambiguity_check build a schema error for FieldNotFound because the passed using_columns argument is empty (link)

This appears to be due to something happening here in the sql unparser (link). I am still working out exactly what is going wrong but wanted to give an update.

[brunal]

The issue actually comes from the loss of the Column::from_name() call in SqlToRel::parse_join(). I have a fix for it (#16560).