Management API dataplanes endpoints not found

[uwburn]

I've been experimenting with the local setup of the EDC connector, + extensions i've been able to properly add assets, policies, contracts and perform transfers both from the UI and the REST API.

As part of my experimenting i was looking into the differences of "provider push" and "consumer pull" data transfer flows, following the steps described at EDC - Transfer 02 Provider push and EDC - Transfer 02 Consumer Pull.

For the latter, if i've understood correctly the logic behind it, a dataplane should be registered before, so that it can be selected by the connectors at the beginning of transfer flow.

To do so, i've attempted to follow preliminary steps described at EDC - Transfer 00 Prerequisites, but performing GET or POST http://localhost:11002/api/management/v2/dataplanes resulted in 404 errors. Is this expected?

According to eclipse-edc-management-api.yaml those endpoints should be available. Are them exposed on another interface (similarly to observability endpoints, e.g. http://localhost:22001/api/check/health)? I've already attempted some different combinations but to no avail so far.

[tmberthold]

Hello and thank you for asking, we'll take a look at it and get back to you.

[tmberthold]

From a technical perspective, the clue to the used sovity Community-Edition is that the data-plane is already integrated into the control-plane and registration doesn't need to be done by you as done in the core-edc samples. You can see this by booting up our EDC CE DEV variant: for our EDC CE, only one image/container is booted up for our EDC itself, for our EDC-UI and for the Database. However, the running EDC CE itself is not divided into a control- and data-plane container.

If there are any technical problems when trying the consumer-pull with our EDC CE, I would ask you to let us know again.

[uwburn]

Hello again, thanks for answering. Sorry if my question sound silly, i'm trying to figure what are the notable differences between "pure" EDC and sovity "extended" connector, but i'm rather green on EDC based solutions in general.

For testing consumer pull i started from the Postman collection for testing API calls and went up to the point where i could execute a transfer with the provider push, having the following:

POST /api/management/v2/transferprocesses HTTP/1.1
Host: localhost:22002
X-Api-Key: ApiKeyDefaultValue
Content-Type: application/json
Content-Length: 937

{
    "@type": "https://w3id.org/edc/v0.0.1/ns/TransferRequest",
    "https://w3id.org/edc/v0.0.1/ns/assetId": "provider-test-data",
    "https://w3id.org/edc/v0.0.1/ns/contractId": "cHJvdmlkZXItdGVzdC1kYXRhLXVucmVzdHJpY3RlZA==:cHJvdmlkZXItdGVzdC1kYXRh:NTkxYmNhZDItYzBiOC00ZmUyLWEzNGUtOTY5ZGQxYzEyMjA4",
    "https://w3id.org/edc/v0.0.1/ns/connectorAddress": "http://edc-provider:11003/api/dsp",
    "https://w3id.org/edc/v0.0.1/ns/connectorId": "my-edc-provider",
    "https://w3id.org/edc/v0.0.1/ns/dataDestination": {
        "https://w3id.org/edc/v0.0.1/ns/type": "HttpData",
        "https://w3id.org/edc/v0.0.1/ns/baseUrl": "http://node-red-consumer:1880/sink"
    },
    "https://w3id.org/edc/v0.0.1/ns/properties": {},
    "https://w3id.org/edc/v0.0.1/ns/privateProperties": {},
    "https://w3id.org/edc/v0.0.1/ns/protocol": "dataspace-protocol-http",
    "https://w3id.org/edc/v0.0.1/ns/managedResources": false
}

with response:

{
  "@type": "edc:IdResponse",
  "@id": "a41eaaea-d3d2-4674-b1ad-26542386c88e",
  "edc:createdAt": 1706167272028,
  "@context": {
    "dct": "https://purl.org/dc/terms/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/"
  }
}

and data successfully transfered from the source to the targtet.

Then, i tried to perform the consumer pull request by changing the request body as follows:

POST /api/management/v2/transferprocesses HTTP/1.1
Host: localhost:22002
X-Api-Key: ApiKeyDefaultValue
Content-Type: application/json
Content-Length: 849

{
    "@type": "https://w3id.org/edc/v0.0.1/ns/TransferRequest",
    "https://w3id.org/edc/v0.0.1/ns/assetId": "provider-test-data",
    "https://w3id.org/edc/v0.0.1/ns/contractId": "cHJvdmlkZXItdGVzdC1kYXRhLXVucmVzdHJpY3RlZA==:cHJvdmlkZXItdGVzdC1kYXRh:NTkxYmNhZDItYzBiOC00ZmUyLWEzNGUtOTY5ZGQxYzEyMjA4",
    "https://w3id.org/edc/v0.0.1/ns/connectorAddress": "http://edc-provider:11003/api/dsp",
    "https://w3id.org/edc/v0.0.1/ns/connectorId": "my-edc-provider",
    "https://w3id.org/edc/v0.0.1/ns/dataDestination": {
        "https://w3id.org/edc/v0.0.1/ns/type": "HttpProxy"
    },
    "https://w3id.org/edc/v0.0.1/ns/properties": {},
    "https://w3id.org/edc/v0.0.1/ns/privateProperties": {},
    "https://w3id.org/edc/v0.0.1/ns/protocol": "dataspace-protocol-http",
    "https://w3id.org/edc/v0.0.1/ns/managedResources": false
}

so the data destination has changed, type going from HttpData to HttpProxy, baseUrl being removed, as per EDC examples.

The response is:

{
  "@type": "edc:IdResponse",
  "@id": "861c118b-a1d2-4bd7-adc7-abaabcfa37de",
  "edc:createdAt": 1706167472829,
  "@context": {
    "dct": "https://purl.org/dc/terms/",
    "edc": "https://w3id.org/edc/v0.0.1/ns/",
    "dcat": "https://www.w3.org/ns/dcat/",
    "odrl": "http://www.w3.org/ns/odrl/2/",
    "dspace": "https://w3id.org/dspace/v0.8/"
  }
}

This attempt results in the transfer staying in REQUESTED state:

{
    "@id": "861c118b-a1d2-4bd7-adc7-abaabcfa37de",
    "@type": "edc:TransferProcess",
    "edc:correlationId": "861c118b-a1d2-4bd7-adc7-abaabcfa37de",
    "edc:state": "REQUESTED",
    "edc:stateTimestamp": 1706167473182,
    "edc:type": "CONSUMER",
    "edc:assetId": "provider-test-data",
    "edc:contractId": "cHJvdmlkZXItdGVzdC1kYXRhLXVucmVzdHJpY3RlZA==:cHJvdmlkZXItdGVzdC1kYXRh:NTkxYmNhZDItYzBiOC00ZmUyLWEzNGUtOTY5ZGQxYzEyMjA4",
    "edc:callbackAddresses": [],
    "edc:dataDestination": {
      "@type": "edc:DataAddress",
      "edc:type": "HttpProxy"
    },
    "edc:connectorId": "my-edc-provider",
    "@context": {
      "dct": "https://purl.org/dc/terms/",
      "edc": "https://w3id.org/edc/v0.0.1/ns/",
      "dcat": "https://www.w3.org/ns/dcat/",
      "odrl": "http://www.w3.org/ns/odrl/2/",
      "dspace": "https://w3id.org/dspace/v0.8/"
    }
  }

while on the provider consumer logs some errors do appear:

2024-01-25 07:24:34 TransferProcess: ID 1d02ccf0-349b-4b84-bcf6-a34af7c26398. Fatal error while Initiate data flow. Error details: Failed to find DataPlaneInstance for source/destination: HttpData/HttpProxy 
2024-01-25 07:24:34 Failed to find DataPlaneInstance for source/destination: HttpData/HttpProxy 
2024-01-25 07:24:34 TransferProcess: ID 1d02ccf0-349b-4b84-bcf6-a34af7c26398. Fatal error while Send TransferTerminationMessage to http://edc-consumer:11003/api/dsp. Error details: {"@type":"dspace:TransferError","dspace:code":"404","dspace:reason":"TransferProcess with correlationId 1d02ccf0-349b-4b84-bcf6-a34af7c26398 not found","dspace:processId":"1d02ccf0-349b-4b84-bcf6-a34af7c26398","@context":{"dct":"https://purl.org/dc/terms/","edc":"https://w3id.org/edc/v0.0.1/ns/","dcat":"https://www.w3.org/ns/dcat/","odrl":"http://www.w3.org/ns/odrl/2/","dspace":"https://w3id.org/dspace/v0.8/"}} 
2024-01-25 07:24:34 {"@type":"dspace:TransferError","dspace:code":"404","dspace:reason":"TransferProcess with correlationId 1d02ccf0-349b-4b84-bcf6-a34af7c26398 not found","dspace:processId":"1d02ccf0-349b-4b84-bcf6-a34af7c26398","@context":{"dct":"https://purl.org/dc/terms/","edc":"https://w3id.org/edc/v0.0.1/ns/","dcat":"https://www.w3.org/ns/dcat/","odrl":"http://www.w3.org/ns/odrl/2/","dspace":"https://w3id.org/dspace/v0.8/"}}

[tmberthold]

Hello,

• we have open-source documentation of what the API-calls should look like, for example for transfer initiation. There I can see slight differences to the API-call used above for the consumer pull: LINK
• Unfortunately, it also says that consuming data via HttpProxy/HTTP Pull is only supported in our Enterprise Edition: LINK

[uwburn]

Thanks a lot for taking time to look into this.