[SPARK-28098][SQL]Support read hive table while LeafDir had multi-level paths #32679
Conversation
There was a problem hiding this comment.
What about this option? https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#recursive-file-lookup
There was a problem hiding this comment.
This action can't support read partitioned tables, and exception are thrown directly when handling partitioned table subdirectories.
chong0929
force-pushed
the
SPARK-28098
branch
2 times, most recently
from
5d282a1 to
a984c28
Compare
|
ok to test |
| if (rootPaths.contains(path)) { | ||
| path | ||
| } else { | ||
| getRootPathsLeafDir(path.getParent) |
There was a problem hiding this comment.
This fails with NPE if a parquet file is provided directly (instead of a directory).
The fix: lyft@4375b8a
| } | ||
| } | ||
|
|
||
| test("SPARK-28098 - supporting read partitioned Hive tables with subdirectories") { |
There was a problem hiding this comment.
not sure what this is testing for
this case also passes on 2.4 without any patched code
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
|
Any chance of this getting picked up again? I saw it was merged in a fork: lyft#40 but it would be great to have it upstream |
chong0929
changed the title
but, it's not on the official repo (apache/spark) ! |
|
I have the same problem: With the TEZ engine writing data in the presence of union all: part_date=xxxx/HIVE_UNION_SUBDIR_1/part_000 (parquet) when I run a query on this data Spark cannot read the subdir ! I have a solution biut is not recommmended |
| cachedLeafDirToChildrenFiles = files.toArray.groupBy(_.getPath.getParent) | ||
| cachedLeafDirToChildrenFiles = | ||
| if (readPartitionWithSubdirectoryEnabled) { | ||
| files.toArray.groupBy(file => getRootPathsLeafDir(file.getPath.getParent)) |
There was a problem hiding this comment.
I found that here can not infer partition for non-catalog table. For table location /dir/table/pt=1/file, here the key is /dir/table, the value is /dir/table/pt=1/file. So we can not infer partition from the key /dir/table.
8 participants
What changes were proposed in this pull request?
This support could read source files of partitioned hive table with subdirectories.
Why are the changes needed?
While use spark engine to read a partititioned hive table with subdirectories, The source files in subdirectories couldn't
get.
Does this PR introduce any user-facing change?
no
How was this patch tested?
new test