[SPARK-7570] [SQL] Ignores _temporary during partition discovery

[liancheng]

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[SparkQA]

Test build #32524 has started for PR 6091 at commit 8ff07e8.

[yhuai]

Also cc @marmbrus

[marmbrus]

Should we be ignoring anything that starts with _?

[SparkQA]

Test build #32524 has finished for PR 6091 at commit 8ff07e8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • s"FileOutputCommitter or its subclass is expected, but got a $
  • trait FSBasedRelationProvider
  • abstract class OutputWriter

[AmplabJenkins]

Merged build finished. Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32524/
Test PASSed.

[liancheng]

@marmbrus No, because there can be dynamic partition column names starting with _ or . (e.g. /path/to/_1=1/_2=2).

[liancheng]

For now, we can check whether there is an = in the directory name. If the directory name

1. doesn't contain =, and
2. starts with _ or .

then we can ignore them. But we may want to support customized partition discovery strategies later. For example, one possible strategy that some users had once asked for is not to encode partition column names in directory paths, (i.e., path/1/2 instead of path/a=1/b=2). In that case, we simply can't have any assumption over partition directory names, even using _temporary is not 100% safe.

[SparkQA]

Test build #819 has started for PR 6091 at commit 8ff07e8.

[yhuai]

LGTM

[SparkQA]

Test build #819 has finished for PR 6091 at commit 8ff07e8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.