diff --git a/release-notes/opensearch-sql.release-notes-3.4.0.0.md b/release-notes/opensearch-sql.release-notes-3.4.0.0.md new file mode 100644 index 00000000000..05a3a22d355 --- /dev/null +++ b/release-notes/opensearch-sql.release-notes-3.4.0.0.md @@ -0,0 +1,155 @@ + +## Version 3.4.0 Release Notes + +Compatible with OpenSearch and OpenSearch Dashboards version 3.4.0 + +### Features +* Support `chart` command in PPL ([#4579](https://github.com/opensearch-project/sql/pull/4579)) +* Support `Streamstats` command with calcite ([#4297](https://github.com/opensearch-project/sql/pull/4297)) +* Support `multisearch` command in calcite ([#4332](https://github.com/opensearch-project/sql/pull/4332)) +* Add replace command with Calcite ([#4451](https://github.com/opensearch-project/sql/pull/4451)) + +### Enhancements +* Add `bucket_nullable` argument for `Streamstats` command ([#4831](https://github.com/opensearch-project/sql/pull/4831)) +* Add `regexp_replace()` function as alias of `replace()` ([#4765](https://github.com/opensearch-project/sql/pull/4765)) +* Convert `dedup` pushdown to composite + top_hits ([#4844](https://github.com/opensearch-project/sql/pull/4844)) +* Merge group fields for aggregate if having dependent group fields ([#4703](https://github.com/opensearch-project/sql/pull/4703)) +* Merge the implementation of `timechart` and `chart` ([#4755](https://github.com/opensearch-project/sql/pull/4755)) +* PPL tostring() implementation issue #4492 ([#4497](https://github.com/opensearch-project/sql/pull/4497)) +* Perform RexNode expression standardization for script push down ([#4795](https://github.com/opensearch-project/sql/pull/4795)) +* Pushdown sort by complex expressions to scan ([#4750](https://github.com/opensearch-project/sql/pull/4750)) +* Pushdown the `top` `rare` commands to nested aggregation ([#4707](https://github.com/opensearch-project/sql/pull/4707)) +* Refactor alias type field by adding another project with alias ([#4881](https://github.com/opensearch-project/sql/pull/4881)) +* Remove count aggregation for sort on aggregate measure ([#4867](https://github.com/opensearch-project/sql/pull/4867)) +* Remove redundant push-down-filters derived for bucket-non-null agg ([#4843](https://github.com/opensearch-project/sql/pull/4843)) +* Remove unnecessary filter for DateHistogram aggregation ([#4877](https://github.com/opensearch-project/sql/pull/4877)) +* Specify timestamp field with `timefield` in timechart command ([#4784](https://github.com/opensearch-project/sql/pull/4784)) +* Support `appendpipe` command in PPL ([#4602](https://github.com/opensearch-project/sql/pull/4602)) +* Support `mvdedup` eval function ([#4828](https://github.com/opensearch-project/sql/pull/4828)) +* Support `mvindex` eval function ([#4794](https://github.com/opensearch-project/sql/pull/4794)) +* Support push down sort on aggregation measure for more than one agg call ([#4759](https://github.com/opensearch-project/sql/pull/4759)) +* Support wildcard for replace command ([#4698](https://github.com/opensearch-project/sql/pull/4698)) +* Add `bucket_nullable` argument for `Eventstats` ([#4817](https://github.com/opensearch-project/sql/pull/4817)) +* Bin command error message enhancement ([#4690](https://github.com/opensearch-project/sql/pull/4690)) +* Update clickbench queries with parameter bucket_nullable=false ([#4732](https://github.com/opensearch-project/sql/pull/4732)) +* Support 'usenull' option in PPL `top` and `rare` commands ([#4696](https://github.com/opensearch-project/sql/pull/4696)) +* Support millisecond span ([#4672](https://github.com/opensearch-project/sql/pull/4672)) +* Enhance dynamic source clause to support only metadata filters ([#4554](https://github.com/opensearch-project/sql/pull/4554)) +* Support push down sort after limit ([#4657](https://github.com/opensearch-project/sql/pull/4657)) +* Add `per_minute`, `per_hour`, `per_day` function support ([#4531](https://github.com/opensearch-project/sql/pull/4531)) +* Pushdown sort aggregate metrics ([#4603](https://github.com/opensearch-project/sql/pull/4603)) +* Allow renaming group-by fields to existing field names ([#4586](https://github.com/opensearch-project/sql/pull/4586)) +* Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions ([#4599](https://github.com/opensearch-project/sql/pull/4599)) +* Pushdown case function in aggregations as range queries ([#4400](https://github.com/opensearch-project/sql/pull/4400)) +* Update GEOIP function to support IP types as input ([#4613](https://github.com/opensearch-project/sql/pull/4613)) +* Pushdown distinct count approx ([#4614](https://github.com/opensearch-project/sql/pull/4614)) +* Optimize pushdown script size with necessary fields per expression ([#4615](https://github.com/opensearch-project/sql/pull/4615)) +* Support referring to implicit `@timestamp` field in span ([#4138](https://github.com/opensearch-project/sql/pull/4138)) +* Make composite bucket size configurable ([#4544](https://github.com/opensearch-project/sql/pull/4544)) +* Add internal MAP_REMOVE function for Calcite PPL ([#4511](https://github.com/opensearch-project/sql/pull/4511)) +* Add MAP_APPEND internal function to Calcite PPL ([#4515](https://github.com/opensearch-project/sql/pull/4515)) +* Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance ([#4569](https://github.com/opensearch-project/sql/pull/4569)) +* [Enhancement] Error handling for illegal character usage in java regex named capture group ([#4434](https://github.com/opensearch-project/sql/pull/4434)) +* Add JSON_EXTRACT_ALL internal function for Calcite PPL ([#4489](https://github.com/opensearch-project/sql/pull/4489)) +* Set 0 and negative value of subsearch.maxout as unlimited ([#4534](https://github.com/opensearch-project/sql/pull/4534)) +* Add configurable system limitations for `subsearch` and `join` command ([#4501](https://github.com/opensearch-project/sql/pull/4501)) +* Add `per_second` function support for `timechart` command ([#4464](https://github.com/opensearch-project/sql/pull/4464)) +* Add MAP_CONCAT internal function ([#4477](https://github.com/opensearch-project/sql/pull/4477)) +* Support Regex for replace eval function ([#4456](https://github.com/opensearch-project/sql/pull/4456)) +* Add data anonymizer for spath command ([#4479](https://github.com/opensearch-project/sql/pull/4479)) +* Support eval returns decimal division result instead of integer ([#4440](https://github.com/opensearch-project/sql/pull/4440)) +* Add mvappend function for Calcite PPL ([#4438](https://github.com/opensearch-project/sql/pull/4438)) +* PPL `fillnull` command enhancement ([#4421](https://github.com/opensearch-project/sql/pull/4421)) +* Support format=yaml in Explain API ([#4446](https://github.com/opensearch-project/sql/pull/4446)) +### Bug Fixes +* Add hashCode() and equals() to the value class of ExprJavaType ([#4885](https://github.com/opensearch-project/sql/pull/4885)) +* BucketAggretationParser should handle more non-composite bucket types ([#4706](https://github.com/opensearch-project/sql/pull/4706)) +* Do not remove nested fields in resolving AllFieldsExcludeMeta ([#4708](https://github.com/opensearch-project/sql/pull/4708)) +* Fix binning udf resolution / Add type coercion support for binning UDFs ([#4742](https://github.com/opensearch-project/sql/pull/4742)) +* Fix bug that `Streamstats` command incorrectly treats null as a valid group ([#4777](https://github.com/opensearch-project/sql/pull/4777)) +* Fix filter push down producing redundant filter queries ([#4744](https://github.com/opensearch-project/sql/pull/4744)) +* Fix function identify problem in converting to sql dialect ([#4793](https://github.com/opensearch-project/sql/pull/4793)) +* Fix search anonymizer only ([#4783](https://github.com/opensearch-project/sql/pull/4783)) +* Fix sub-fields accessing of generated structs ([#4683](https://github.com/opensearch-project/sql/pull/4683)) +* Fix wrong parameter and return result logic for LogPatternAggFunction ([#4868](https://github.com/opensearch-project/sql/pull/4868)) +* Grouping key field type can only be overwritten when the `ExprCoreType`s are different ([#4850](https://github.com/opensearch-project/sql/pull/4850)) +* Fix eval on grouped fields after timechart ([#4758](https://github.com/opensearch-project/sql/pull/4758)) +* Support access to nested field of struct after fields command ([#4719](https://github.com/opensearch-project/sql/pull/4719)) +* Support escaped field names in SPath parsing ([#4813](https://github.com/opensearch-project/sql/pull/4813)) +* Support script pushdown in sort-on-measure pushdown rewriting ([#4749](https://github.com/opensearch-project/sql/pull/4749)) +* Support serializing external OpenSearch UDFs at pushdown time ([#4618](https://github.com/opensearch-project/sql/pull/4618)) +* Support using decimal as span literals ([#4717](https://github.com/opensearch-project/sql/pull/4717)) +* Translate `SAFE_CAST` to `TRY_CAST` in Spark SQL ([#4788](https://github.com/opensearch-project/sql/pull/4788)) +* Update syntax: like(string, PATTERN[, case_sensitive]) ([#4837](https://github.com/opensearch-project/sql/pull/4837)) +* [BugFix] Fix Memory Exhaustion for Multiple Filtering Operations in PPL ([#4841](https://github.com/opensearch-project/sql/pull/4841)) +* Fix CVE-2025-48924 ([#4665](https://github.com/opensearch-project/sql/pull/4665)) +* [BugFix] Fix unexpected shift of extraction for `rex` with nested capture groups in named groups ([#4641](https://github.com/opensearch-project/sql/pull/4641)) +* Fix asc/desc keyword behavior for sort command ([#4651](https://github.com/opensearch-project/sql/pull/4651)) +* Fixes for `Multisearch` and `Append` command ([#4512](https://github.com/opensearch-project/sql/pull/4512)) +* Make nested alias type support referring to outer context ([#4673](https://github.com/opensearch-project/sql/pull/4673)) +* Use table scan rowType in filter pushdown to fix rename issue ([#4670](https://github.com/opensearch-project/sql/pull/4670)) +* Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations ([#4621](https://github.com/opensearch-project/sql/pull/4621)) +* Fix bin nested fields issue ([#4606](https://github.com/opensearch-project/sql/pull/4606)) +* Change ComparableLinkedHashMap to compare Key than Value ([#4648](https://github.com/opensearch-project/sql/pull/4648)) +* Replace all dots in fields of table scan's PhysType ([#4633](https://github.com/opensearch-project/sql/pull/4633)) +* Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue ([#4629](https://github.com/opensearch-project/sql/pull/4629)) +* Fix filter parsing failure on date fields with non-default format ([#4616](https://github.com/opensearch-project/sql/pull/4616)) +* Fix compile issue in main ([#4608](https://github.com/opensearch-project/sql/pull/4608)) +* Fix push down failure for min/max on derived field ([#4572](https://github.com/opensearch-project/sql/pull/4572)) +* Add value type hint for derived aggregate group by field ([#4583](https://github.com/opensearch-project/sql/pull/4583)) +* Fix sort push down into agg after project already pushed ([#4546](https://github.com/opensearch-project/sql/pull/4546)) +* Update request builder after pushdown sort into agg buckets ([#4541](https://github.com/opensearch-project/sql/pull/4541)) +* Including metadata fields type when doing agg/filter script push down ([#4522](https://github.com/opensearch-project/sql/pull/4522)) +* Fix percentile bug ([#4539](https://github.com/opensearch-project/sql/pull/4539)) +* Fix mapping after aggregation push down ([#4500](https://github.com/opensearch-project/sql/pull/4500)) +* Throw an error when the conditions of case are not boolean values ([#4520](https://github.com/opensearch-project/sql/pull/4520)) +* Fallback to sub-aggregation if composite aggregation doesn't support ([#4413](https://github.com/opensearch-project/sql/pull/4413)) +* Fix the bug of explicit makeNullLiteral for UDT fields ([#4475](https://github.com/opensearch-project/sql/pull/4475)) +* Fix missing keywordsCanBeId ([#4491](https://github.com/opensearch-project/sql/pull/4491)) +* Fix issue 4441 ([#4449](https://github.com/opensearch-project/sql/pull/4449)) +* Fix join type ambiguous issue when specify the join type with sql-like join criteria ([#4474](https://github.com/opensearch-project/sql/pull/4474)) +* Remove shared mutable optimizer field that caused race condition ([#4454](https://github.com/opensearch-project/sql/pull/4454)) +* Reverting to _doc + _id ([#4435](https://github.com/opensearch-project/sql/pull/4435)) + +### Infrastructure +* Add config for CodeRabbit review ([#4890](https://github.com/opensearch-project/sql/pull/4890)) +* Split bwc-tests to bwc-rolling-upgrade and bwc-full-restart ([#4716](https://github.com/opensearch-project/sql/pull/4716)) +* Update github workflows to move from macos-13 to 14 ([#4779](https://github.com/opensearch-project/sql/pull/4779)) +* Fix the flaky CalcitePPLTcphIT ([#4846](https://github.com/opensearch-project/sql/pull/4846)) +* Fix UT failure and Linkchecker failure ([#4809](https://github.com/opensearch-project/sql/pull/4809)) +* Adding IT suite for PPL-based dashboards in Neo for CloudWatch Lake ([#4695](https://github.com/opensearch-project/sql/pull/4695)) +* Add allowed_warnings in yaml restful tests ([#4731](https://github.com/opensearch-project/sql/pull/4731)) +* Mitigate the CI failure caused by 500 Internal Server Error ([#4646](https://github.com/opensearch-project/sql/pull/4646)) +* Publish internal modules separately for downstream reuse ([#4484](https://github.com/opensearch-project/sql/pull/4484)) +* Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT ([#4623](https://github.com/opensearch-project/sql/pull/4623)) +* Onboarding async query core and grammar files to maven snapshots ([#4598](https://github.com/opensearch-project/sql/pull/4598)) +* Onboarding new maven snapshots publishing to s3 (sql) ([#4588](https://github.com/opensearch-project/sql/pull/4588)) +* Fix JsonExtractAllFunctionIT failure ([#4556](https://github.com/opensearch-project/sql/pull/4556)) +* Check server status before starting Prometheus ([#4537](https://github.com/opensearch-project/sql/pull/4537)) +* Update stalled action ([#4485](https://github.com/opensearch-project/sql/pull/4485)) +* Switch to Guice#createInjector and add concurrent SQL/PPL regression ITs ([#4462](https://github.com/opensearch-project/sql/pull/4462)) +* Update delete_backport_branch workflow to include release-chores branches ([#4025](https://github.com/opensearch-project/sql/pull/4025)) +* Add ignorePrometheus Flag for integTest and docTest ([#4442](https://github.com/opensearch-project/sql/pull/4442)) + +### Documentation +* Update PPL Command Documentation ([#4562](https://github.com/opensearch-project/sql/pull/4562)) +* Doc update for `json_valid` ([#4803](https://github.com/opensearch-project/sql/pull/4803)) +* Enhance tests and doc for eval isnull/isnotnull functions ([#4724](https://github.com/opensearch-project/sql/pull/4724)) +* Update search.rst documentation ([#4686](https://github.com/opensearch-project/sql/pull/4686)) +* Add more examples to the `where` command doc ([#4457](https://github.com/opensearch-project/sql/pull/4457)) +* Update eventstats.rst ([#4447](https://github.com/opensearch-project/sql/pull/4447)) + +### Maintenance +* Bump Calcite to 1.41.0 ([#4714](https://github.com/opensearch-project/sql/pull/4714)) +* Execute yamlRestTest in integration job ([#4838](https://github.com/opensearch-project/sql/pull/4838)) +* Fix test failures due to version in mapping ([#4748](https://github.com/opensearch-project/sql/pull/4748)) +* Support timeouts for Calcite queries ([#4857](https://github.com/opensearch-project/sql/pull/4857)) +* [Maintenance] Enforce PR label of 'bugFix' instead of 'bug' ([#4773](https://github.com/opensearch-project/sql/pull/4773)) +* [3.4.0] Bump Gradle to 9.2.0 and GitHub Action JDK to 25 ([#4824](https://github.com/opensearch-project/sql/pull/4824)) +* Fix clickbench query 43 ([#4861](https://github.com/opensearch-project/sql/pull/4861)) +* Update big5 ppl queries and check plans ([#4668](https://github.com/opensearch-project/sql/pull/4668)) +* Revert "Update grammar files and developer guide (#4301)" ([#4643](https://github.com/opensearch-project/sql/pull/4643)) +* Increment version to 3.4.0-SNAPSHOT ([#4452](https://github.com/opensearch-project/sql/pull/4452)) +* Revert partial of #4401 ([#4503](https://github.com/opensearch-project/sql/pull/4503)) +* Implement one-batch lookahead for index enumerators ([#4345](https://github.com/opensearch-project/sql/pull/4345)) +* Refactor name resolution in Calcite PPL ([#4393](https://github.com/opensearch-project/sql/pull/4393))