Properly handle nested fields when computing stats / stats-schema

[roeap]

Bug

Right now our logic to compute stats, specifically using the `` property only considers root level fields, but does not traverse into fields. Also we need to parse fields as the field names may be escaped and contain special characters ..

https://github.com/delta-io/delta/blob/4b102d34a2ce881b2a851b4c6cfbf2ab3ab5534f/spark/src/main/scala/org/apache/spark/sql/delta/DeltaConfig.scala#L549-L561

What you expected to happen:

Properly parse field names when generating stats and stats schema

More details:

[ion-elgreco]

@roeap fyi, stats parsing seems also not entirely working for checkpoints
#2571

[alexwilcoxson-rel]

My team will need to address this to adopt 0.18.x.

My proposal is when looking up stats column in delta schema

• if stats column is formatted col.na$me then use schema.field("col.na$me")
• if stats column does not have `s, treat . as delimiter for nested columns and traverse the schema a la: https://github.com/delta-io/delta-rs/pull/2519/files this was reverted essentially when the kernel code was adopted

for example: top.middle.bottom would find stats for

top struct([
    middle struct([
        bottom primitive
    ])
])