egeria & https://openlineage.io/

[jbadeau]

Hello,

Do you guys have any plans to integrate with

• openlineage a subproject of egeria ?

This would prob reduce or eliminate the need to maintain agents and spline public model.

It seems to be trying to do what opentelemetry is doing for observability.

[wajda]

Never heard about this. Thanks for the info. Will do some research.

[wajda]

I made a little research about Open Lineage and I was surprised how conceptually close the OpenLineage API and Spline Producer API models are. The key entities are semantically the same and even the way the model extension points are organized is similar (although the OpenLineage went a step further in their facets which I like).
I think it would definitely make sense to combine efforts of both projects, the question is how. We'll probably start with adopting OpenLineage standard for Spline Agent to Server communication, that will make both parties interoperable with the OpenLineage projects stack.

@jbadeau Do you have any other ideas?

[ganeshnikumbh]

@wajda just wanted to check if you have already done something with this topic. I am currently trying few things with openlineage and wanted to know if the spline server can accept and view the lineage events sent by openlineage agent?

[cerveada]

Not yet. I'm working on open lineage dispatcher for the spline agent, the server will be done as a second step, so we have both sides covered.