0.6.2 - DataSource V2, fixed Window on Databricks and more

Features

• #191, #256, #259, #266 - Data Source V2 support

Bugfixes

• #285 - Recover from ObjectStructureDumper reflection errors allowing for partial object structure dumps to be logged
• #272, #265 - Fix duplicated attribute IDs
• #264 - Place a workaround for AbsaOSS/spline#931
• #262 - Place a workaround for a closed source Window function from the Databricks runtime
• #226 - Fix: "Illegal cyclic reference" in the ExpressionConverter

Spark 3.1, AWS EMR, Databricks Notebook

Features

#227 - Spark 3.1 support

Bugfixes

#230 - fix IllegalAccessError on Databricks Notebook
#248 - fix java.lang.NoClassDefFoundError on AWS EMR

Other improvements

#232 - Add INFO logging for the dispatcher type in use
#242 - Improve dependency management for different Spark profiles

0.6.0 - Spark 3.0, Attribute-level lineage and more

Features

Support for Apache Spark 3.0

Spline agent can now be used with Apache Spark 3.0+

• #93 - Spark 3.0 support

Attribute-level lineage & new Spline Producer API (#61)

Spline Producer API has been refined and expanded to support attribute-level lineage tracking

• #61 Upgrade to Producer API ver.1.1
• AbsaOSS/spline#114 - Attribute lineage

More lineage dispatcher types (#156)

LineageDispatcher is an abstraction for the lineage meta-data transport layer. It allows to create you own lineage data sink, or use one of the pre-configured options available out of the box. Using multiple dispatchers is also supported with the help of CompositeLineageDispatcher.

• #166 - Kafka (experimental)
• #174 - HDFS (experimental)
• #177 - Console & Logging
• #172 - Composite dispatcher

Post processing & password hiding

Post processing filters allow you to intercept the capture lineage meta-data before it is passed to the transport layer.
There you can enriching the lineage with additional meta-data, or remove some sensitive info (like passwords) from the lineage meta-data for security or privacy reasons.

• #69 - Security issue: JDBC URI contains plaintext password & username feature

Bugfixes

• Fixed Window operation support (#60)
• Fixed Union operation attribute lineage (#65)

Improvements

Configuration

Flexible and easy-to-use configuration is required to support layered and pluggable agent design.
An improved configuration approach allows for a great level of flexibility and extensibility of the agent.

• #165 - Spline config to support more dispatcher types
• #223 - IWDStrategy to use the standard config pattern

Dockerization

• #187 - Docker container for Spark agent & examples

Bugfix release

release/0.5.6

POM: GPG: fix "gpg: signing failed: Inappropriate ioctl for device"

Bugfix release

Fixes a cyclic dependency in an attribute graph #125

Cobrix (COBOL) support + a new Plugin API

Among a number of bugfixes this release adds support for Cobrix (Cobol Spark connector)
and introduces a new Plugin API that makes adding support for other datasource types, frameworks and libraries easy.
See README

[#114] - Log unrecongized opearation structure
[#108] - Plugin API to support 3rd party connectors, relations etc.
[#107] - Support for SaveAsTable and SQL (Azure Databricks)
[#104] - Pluggable handling of arbitrary relations
[#100] - Possibility to add datasource support
[#92] - Make spark-xml library optional dependency
[#71] - Upgrade to Producer API v1.1
[#64] - Multiple Producer API versions support
[#38] - Cobol connector support (Cobrix)
[#31] - Cannot re-attempt Spline initialization after an error
[#30] - Sanity check for codeless initialization

release/0.5.3

[maven-release-plugin] copy for tag release/0.5.3

release/0.5.1

Bugfixes

[#62] Spark write mode "Ignore" support
[#40] support for databricks delta format
[#39] LocalRelation node should not capture the data
[#29] AVRO: a class name is captured instead of a format name
[#28] UnsupportedOperationException: dataType

release/0.5.0

[AbsaOSS/spline#212] Scala 2.12 support (for Spark 2.4)
[AbsaOSS/spline#606] ElasticSearch connector support
[AbsaOSS/spline#605] MongoDB connector support
[AbsaOSS/spline#604] Cassandra connector support