[SPARK-28098][SQL] Supports reading hive tables when there are subdirectories under the table or partition location.

[zhengchenyu]

What changes were proposed in this pull request?

Add a new variable partitionDirToChildrenFiles to InMemoryFileIndex to distinguish the use of cachedLeafDirToChildrenFiles.

Why are the changes needed?

When table location or partition location have multi-level non partitioned paths. In particular, TEZ will generate the HIVE_UNION_SUBDIR_1 directory. Spark cannot read the contents.

The reason is that when there are subdirectories, the key stored in cachedLeafDirToChildrenFiles might be something like /xx/table/pt=1/sudir. Then when using cachedLeafDirToChildrenFiles.get('/xx/table/pt=1') to retrieve the file corresponding to the path, will get nothing, resulting in data loss.

The root cause lies in the use of cachedLeafDirToChildrenFiles. Three underlying code uses cachedLeafDirToChildrenFiles. The correct use of cachedLeafDirToChildrenFiles in inferPartitioning is because partitions are easier to infer from lower directory levels. However, the other two uses of cachedLeafDirToChildrenFiles are incorrect. The correct use is to retrieve files from the partition or table location, but not from the leaf directory. Here add new partitionDirToChildrenFiles to solve this problem.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit test, and real job.

[zhengchenyu]

I know some pr try to solve this problem.

• [SPARK-28098][SQL]Support read hive table while LeafDir had multi-level paths #32679 I found that here can not infer partition for non-catalog table.
• [SPARK-28098][SQL]Supporting non-partitioned Hive tables with subdirectories #32202 only consider non-partitioned table.

So I submit this new PR.

[zhengchenyu]

@cloud-fan @dongjoon-hyun @HyukjinKwon Can you please review this PR? I think it is more reasonable to read files under subdirectories.