This repository has been archived by the owner on Sep 4, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 24
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adds 1.5.0 release notes to the docs. (#254)
Signed-off-by: Michael Robinson <merobi@gmail.com>
- Loading branch information
1 parent
487ed8d
commit 789aebf
Showing
1 changed file
with
30 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
--- | ||
title: 1.5.0 | ||
sidebar_position: 9952 | ||
--- | ||
|
||
# 1.5.0 - 2023-11-2 | ||
|
||
### Added | ||
* **Flink: add Flink lineage for Cassandra Connectors** [`#2175`](https://github.com/OpenLineage/OpenLineage/pull/2175) [@HuangZhenQiu](https://github.com/HuangZhenQiu) | ||
*Adds Flink Cassandra source and sink visitors and Flink Cassandra Integration test.* | ||
* **Spark: support `rdd` and `toDF` operations available in Spark Scala API** [`#2188`](https://github.com/OpenLineage/OpenLineage/pull/2188) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski) | ||
*Includes the first Scala integration test, fixes `ExternalRddVisitor` and adds support for extracting inputs from `MapPartitionsRDD` and `ParallelCollectionRDD` plan nodes.* | ||
* **Spark: support Databricks Runtime 13.3** [`#2185`](https://github.com/OpenLineage/OpenLineage/pull/2185) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski) | ||
*Modifies the Spark integration to support the latest Databricks Runtime version.* | ||
|
||
### Changed | ||
* **Airflow: loosen attrs and requests versions** [`#2107`](https://github.com/OpenLineage/OpenLineage/pull/2107) [@JDarDagran](https://github.com/JDarDagran) | ||
*Lowers the version requirements for attrs and requests and removes an unnecessary dependency.* | ||
* **dbt: render yaml configs lazily** [`#2221`](https://github.com/OpenLineage/OpenLineage/pull/2221) [@JDarDagran](https://github.com/JDarDagran) | ||
*Don't render each entry in yaml files at start.* | ||
|
||
### Fixed | ||
* **Airflow/Athena: change dataset name to its location** [`#2167`](https://github.com/OpenLineage/OpenLineage/pull/2167) [@sophiely](https://github.com/sophiely) | ||
*Replaces the dataset and namespace with the data's physical location for more complete lineage across integrations.* | ||
* **Python client: skip redaction in column lineage facet** [`#2177`](https://github.com/OpenLineage/OpenLineage/pull/2177) [@JDarDagran](https://github.com/JDarDagran) | ||
*Redacted fields in `ColumnLineageDatasetFacetFieldsAdditionalInputFields` are now skipped.* | ||
* **Spark: unify dataset naming for RDD jobs and Spark SQL** [`#2181`](https://github.com/OpenLineage/OpenLineage/pull/2181) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski) | ||
*Use the same mechanism for RDD jobs to extract dataset identifier as used for Spark SQL.* | ||
* **Spark: ensure a single `START` and a single `COMPLETE` event are sent** [`#2103`](https://github.com/OpenLineage/OpenLineage/pull/2103) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski) | ||
*For Spark SQL at least four events are sent triggered by different SparkListener methods. Each of them is required and used to collect facets unavailable elsewhere. However, there should be only one `START` and `COMPLETE` events emitted. Other events should be sent as `RUNNING`. Please keep in mind that Spark integration remains stateless to limit the memory footprint, and it is the backend responsibility to merge several Openlineage events into a meaningful snapshot of metadata changes.* |