Skip to content
This repository has been archived by the owner on Sep 4, 2024. It is now read-only.

Commit

Permalink
Adds 1.5.0 release notes to the docs. (#254)
Browse files Browse the repository at this point in the history
Signed-off-by: Michael Robinson <merobi@gmail.com>
  • Loading branch information
merobi-hub authored Nov 2, 2023
1 parent 487ed8d commit 789aebf
Showing 1 changed file with 30 additions and 0 deletions.
30 changes: 30 additions & 0 deletions docs/releases/1_5_0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
title: 1.5.0
sidebar_position: 9952
---

# 1.5.0 - 2023-11-2

### Added
* **Flink: add Flink lineage for Cassandra Connectors** [`#2175`](https://github.com/OpenLineage/OpenLineage/pull/2175) [@HuangZhenQiu](https://github.com/HuangZhenQiu)
*Adds Flink Cassandra source and sink visitors and Flink Cassandra Integration test.*
* **Spark: support `rdd` and `toDF` operations available in Spark Scala API** [`#2188`](https://github.com/OpenLineage/OpenLineage/pull/2188) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
*Includes the first Scala integration test, fixes `ExternalRddVisitor` and adds support for extracting inputs from `MapPartitionsRDD` and `ParallelCollectionRDD` plan nodes.*
* **Spark: support Databricks Runtime 13.3** [`#2185`](https://github.com/OpenLineage/OpenLineage/pull/2185) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
*Modifies the Spark integration to support the latest Databricks Runtime version.*

### Changed
* **Airflow: loosen attrs and requests versions** [`#2107`](https://github.com/OpenLineage/OpenLineage/pull/2107) [@JDarDagran](https://github.com/JDarDagran)
*Lowers the version requirements for attrs and requests and removes an unnecessary dependency.*
* **dbt: render yaml configs lazily** [`#2221`](https://github.com/OpenLineage/OpenLineage/pull/2221) [@JDarDagran](https://github.com/JDarDagran)
*Don't render each entry in yaml files at start.*

### Fixed
* **Airflow/Athena: change dataset name to its location** [`#2167`](https://github.com/OpenLineage/OpenLineage/pull/2167) [@sophiely](https://github.com/sophiely)
*Replaces the dataset and namespace with the data's physical location for more complete lineage across integrations.*
* **Python client: skip redaction in column lineage facet** [`#2177`](https://github.com/OpenLineage/OpenLineage/pull/2177) [@JDarDagran](https://github.com/JDarDagran)
*Redacted fields in `ColumnLineageDatasetFacetFieldsAdditionalInputFields` are now skipped.*
* **Spark: unify dataset naming for RDD jobs and Spark SQL** [`#2181`](https://github.com/OpenLineage/OpenLineage/pull/2181) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
*Use the same mechanism for RDD jobs to extract dataset identifier as used for Spark SQL.*
* **Spark: ensure a single `START` and a single `COMPLETE` event are sent** [`#2103`](https://github.com/OpenLineage/OpenLineage/pull/2103) [@pawel-big-lebowski](https://github.com/pawel-big-lebowski)
*For Spark SQL at least four events are sent triggered by different SparkListener methods. Each of them is required and used to collect facets unavailable elsewhere. However, there should be only one `START` and `COMPLETE` events emitted. Other events should be sent as `RUNNING`. Please keep in mind that Spark integration remains stateless to limit the memory footprint, and it is the backend responsibility to merge several Openlineage events into a meaningful snapshot of metadata changes.*

0 comments on commit 789aebf

Please sign in to comment.