[SUPPORT] sparksql query perfermance degrade in hudi 0.14-rc

[xuzifu666]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

details from ：https://issues.apache.org/jira/browse/HUDI-6941
@boneanxs @codope @danny0405 @stream2000 have a review plz

@codope #9273 seems cause it

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.

[jonvex]

looking into this

[wecharyu]

Eager listing all partition files in file index cause it. It seems not a good idea to get all files in initialization especially when the files are huge.

[codope]

@xuzifu666 @wecharyu Thanks for highlighting the issue. This is indeed a regression. The code block

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala

Lines 471 to 474 in e6be396

	if (partitionColumns.get().length > 1
	&& (listingModeOverride == null || DataSourceReadOptions.FILE_INDEX_LISTING_MODE_LAZY.equals(listingModeOverride))) {
	properties.setProperty(DataSourceReadOptions.FILE_INDEX_LISTING_MODE_OVERRIDE.key, DataSourceReadOptions.FILE_INDEX_LISTING_MODE_EAGER)
	}

is the main cause. It should be further special cased and not simply depend on number of partition columns. I am going to revert the eager listing first and then put in a proper fix, which should also take care of HUDI-6320. Additionally, we need some tests to assert spark DAGs for reads much like TestSparkDataSourceDAGExecution.

[xuzifu666]

@xuzifu666 @wecharyu Thanks for highlighting the issue. This is indeed a regression. The code block

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala

Lines 471 to 474 in e6be396

	if (partitionColumns.get().length > 1
	&& (listingModeOverride == null || DataSourceReadOptions.FILE_INDEX_LISTING_MODE_LAZY.equals(listingModeOverride))) {
	properties.setProperty(DataSourceReadOptions.FILE_INDEX_LISTING_MODE_OVERRIDE.key, DataSourceReadOptions.FILE_INDEX_LISTING_MODE_EAGER)
	}

is the main cause. It should be further special cased and not simply depend on number of partition columns. I am going to revert the eager listing first and then put in a proper fix, which should also take care of HUDI-6320. Additionally, we need some tests to assert spark DAGs for reads much like TestSparkDataSourceDAGExecution.

ok，thanks and I submit a quick fix， but can not confirm whether cause other risk，if you had fix it，can close #9862 anytime @codope @danny0405

[codope]

Thanks @xuzifu666. I have fixed some tests and other issues in #9863