Spark 3.5: Support recursive delegate unwrapping to find ExtendedParser in parser chains

[majian1998]

Related PR: #13625
Since support for Spark 3.4 will be removed in version 1.11, this fix is only applied to Spark 3.5.

[huaxingao]

@majian1998 Is this a clean back-port?

[majian1998]

@majian1998 Is this a clean back-port?

@huaxingao Yes, just a straight copy.

[huaxingao]

LGTM

[kamcheungting-db]

Suggested change

	ParserInterface next = getNextDelegateParser(current);
	if (next == null) {
	break;
	}
	current = next;
	current = getNextDelegateParser(current);

[kamcheungting-db]

shouldn't we at least log something?

[kamcheungting-db]

add log to help RCA if something goes wrong

[kamcheungting-db]

same as above

[kamcheungting-db]

Suggested change

	when(icebergParser.parseSortOrder("id ASC NULLS FIRST")).thenReturn(expected);
	ParserInterface wrapper = new WrapperParser(icebergParser);
	setSessionStateParser(spark.sessionState(), wrapper);
	List<ExtendedParser.RawOrderField> result =
	ExtendedParser.parseSortOrder(spark, "id ASC NULLS FIRST");
	assertThat(result).isSameAs(expected);
	verify(icebergParser).parseSortOrder("id ASC NULLS FIRST");
	when(icebergParser.parseSortOrder("id ASC NULLS FIRST")).thenReturn(expected);
	verify(icebergParser).parseSortOrder("id ASC NULLS FIRST");
	ParserInterface wrapper = new WrapperParser(icebergParser);
	setSessionStateParser(spark.sessionState(), wrapper);
	List<ExtendedParser.RawOrderField> result =
	ExtendedParser.parseSortOrder(spark, "id ASC NULLS FIRST");
	assertThat(result).isSameAs(expected);

[majian1998]

This doesn’t work as expected—verify needs to go after you actually call the method. Otherwise, there’s nothing to verify yet

[kamcheungting-db]

Suggested change

	when(icebergParser.parseSortOrder("id ASC NULLS FIRST")).thenReturn(expected);
	ParserInterface parser = new GrandChildParser(icebergParser);
	setSessionStateParser(spark.sessionState(), parser);
	List<ExtendedParser.RawOrderField> result =
	ExtendedParser.parseSortOrder(spark, "id ASC NULLS FIRST");
	assertThat(result).isSameAs(expected);
	verify(icebergParser).parseSortOrder("id ASC NULLS FIRST");
	when(icebergParser.parseSortOrder("id ASC NULLS FIRST")).thenReturn(expected);
	verify(icebergParser).parseSortOrder("id ASC NULLS FIRST");
	ParserInterface parser = new GrandChildParser(icebergParser);
	setSessionStateParser(spark.sessionState(), parser);
	List<ExtendedParser.RawOrderField> result =
	ExtendedParser.parseSortOrder(spark, "id ASC NULLS FIRST");
	assertThat(result).isSameAs(expected);

[huaxingao]

This is a backport PR; let’s merge it unchanged to keep it identical to the previous one.
@majian1998 Could you open a follow‑up PR (for both 3.5 and 4.0) to address @kamcheungting-db's comments?

[huaxingao]

Merged. Thanks @majian1998 for the PR! Thanks @singhpk234 @kamcheungting-db for the review!
@majian1998 Could you open a follow‑up PR to address @kamcheungting-db's comments?

[majian1998]

This is a backport PR; let’s merge it unchanged to keep it identical to the previous one. @majian1998 Could you open a follow‑up PR (for both 3.5 and 4.0) to address @kamcheungting-db's comments?

@huaxingao Got it, I’ll read through the comments and create a follow-up PR for 3.5 and 4.0.