Improve implementation of parse command

[joshuali925]

Overview

There are few problems with the initial approach of parse command implementation from #411, it brings some limitations described in #359.

One of the root cause is that PPL cannot determine whether to use NamedExpression or ParseExpression when resolving identifiers. For example:

source=index | parse message "(?<firstChar>\w).*" | fields firstChar

In this query when resolving firstChar, PPL should refer to ParseExpression firstChar and derive it from field message.

source=index | parse message "(?<firstChar>\w).*" | stats count() by firstChar | fields count(), firstChar

In this query behavior should be the same when resolving firstChar in stats, but in fields command, firstChar should come from NamedExpression because message no longer exists.

To solve this issue, initial implementation always prioritizes to use ParseExpression instead of NamedExpression if both exists. If the raw text field of parse expression is missing, then it will read from tuple values. This brings the restrictions of not able to rewrite parsed fields and etc. Related code:

sql/core/src/main/java/org/opensearch/sql/planner/physical/ProjectOperator.java

Lines 54 to 82 in 578c6b2

	@Override
	public ExprValue next() {
	ExprValue inputValue = input.next();
	ImmutableMap.Builder<String, ExprValue> mapBuilder = new Builder<>();
	for (NamedExpression expr : projectList) {
	ExprValue exprValue = expr.valueOf(inputValue.bindingTuples());
	if (namedParseExpressions.stream()
	.noneMatch(parsed -> parsed.getNameOrAlias().equals(expr.getNameOrAlias()))) {
	mapBuilder.put(expr.getNameOrAlias(), exprValue);
	}
	}
	// ParseExpression will always override NamedExpression when identifier conflicts
	for (NamedExpression expr : namedParseExpressions) {
	ExprValue value = inputValue.bindingTuples()
	.resolve(((ParseExpression) expr.getDelegated()).getExpression());
	if (value.isMissing()) {
	// value will be missing after stats command, read from inputValue if it exists
	// otherwise do nothing since it should not appear as a field
	ExprValue exprValue = ExprValueUtils.getTupleValue(inputValue).get(expr.getNameOrAlias());
	if (exprValue != null) {
	mapBuilder.put(expr.getNameOrAlias(), exprValue);
	}
	} else {
	ExprValue parsedValue = expr.valueOf(inputValue.bindingTuples());
	mapBuilder.put(expr.getNameOrAlias(), parsedValue);
	}
	}
	return ExprTupleValue.fromExprValueMap(mapBuilder.build());
	}

Ideas for a better implementation:

1. If OpenSearch supports creating fields at runtime, parse command can be pushed down to that.
2. Do not convert parse command as parse expressions in analyzer, do it here instead?
   

   sql/opensearch/src/main/java/org/opensearch/sql/opensearch/storage/OpenSearchIndexScan.java

   Lines 76 to 77 in fe20fb1

   	this.request = new OpenSearchQueryRequest(indexName,
   	settings.getSettingValue(Settings.Key.QUERY_SIZE_LIMIT), exprValueFactory);