How to get lineage response using Consumer API

[mohini-tripathi]

Hi team,

Thank you for your support, I am new to Spline Agent and I request your help with the following issue:

I have integrated spline agent in a spark job and I can see lineage as well on Spline UI, I am trying to get Lineage json responses using Consumer API. I can successfully get API response for /data-sources, /execution-events endpoints but I am getting error when I am trying to hit curl -v 'http://localhost:8080/consumer/lineage-detailed/'
curl -v 'http://localhost:8080/consumer/lineage-overview/'
curl -v 'http://localhost:8080/consumer/attribute-lineage-and-impact/'
I am getting response as

*   Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> GET /consumer/attribute-lineage-and-impact HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/1.1 500 
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Tue, 25 Jun 2024 09:51:09 GMT
< Connection: close
< 
* Closing connection
{"errorId":"7e718411-60a4-40ba-8eb4-c4016effb268","timestamp":1719309069776,"message":null}%

I have already checked the connectivity for Spline UI, Producer API and Consumer API. Kindly help me with the same.

Best Regards,
Mohini

[wajda]

Well yes, it's a wrong response. The correct error should be 404 or 400, but definitely not 500. I've created an issue for that - #1341

Nevertheless, take a look at the REST API doc - http://localhost:8080/docs/consumer.html#tag/lineage
All URL you try are failing because they require additional GET parameters.

[mohini-tripathi]

Thank you so much for your quick response and help @wajda . I am trying as per the given endpoint.
for Lineage: curl -v 'http://localhost:8080/consumer/attribute-lineage-and-impact/' ~ it returned HTTP/1.1 500

whereas for http://localhost/execution-plans/{plan_id}/data-sources

curl -v "http://localhost:8080/consumer/execution-plans/54f85231-ca61-5f68-8af7-f85e30a2e0b9/data-sources" ~ it returned the correct json response with status 200.

I've added required parameters, please let me know if I am missing anything to update in the url to add GET parameter. Thanks.

[wajda]

/consumer/attribute-lineage-and-impact/ should be called as /consumer/attribute-lineage-and-impact/?attributeId=xxxx where xxxx is an internal ID (_key in the ArangoDB collection) of the attribute you are requesting. Also note that currently attribute-level lineage is limited by the boundaries of a single execution-plan. End-to-end attribute-level lineage query isn't implemented yet.