Unable to parametrize HTTP data source

[panosprotopapas]

Hi everyone,

I've been trying for a couple of days now to consume a HTTP data source with parameterization but have been unable to do so.

I am using (i) image-versions:

EDC_IMAGE=ghcr.io/sovity/edc-dev:7.3.0
TEST_BACKEND_IMAGE=ghcr.io/sovity/test-backend:7.3.0
EDC_UI_IMAGE=ghcr.io/sovity/edc-ui:3.0.0

(ii) the latest (ver. 7.3.0) non-dev docker compose that can be found here, and (iii) purely the eclipse-management-api.

If it's not feasible to achieve parametrization through this combination I'm more than happy to switch to a different image version and/or the edc-api-wrapper.

The API requests I am making follow the postman collection and Swagger. Apologies for perhaps being over verbose.

1. [PRODUCER] Create an Asset by taking into account the parameterized HTTP data sources documentation, and POST to {{PROVIDER_EDC_MANAGEMENT_URL}}/v3/assets

{
  "@type": "https://w3id.org/edc/v0.0.1/ns/Asset",
  "https://w3id.org/edc/v0.0.1/ns/properties": {
    "https://w3id.org/edc/v0.0.1/ns/id": "{{ASSET_ID}}",
    "http://www.w3.org/ns/dcat#version": "1.0",
    "http://purl.org/dc/terms/language": "https://w3id.org/idsa/code/EN",
    "http://purl.org/dc/terms/title": "test-document",
    "http://purl.org/dc/terms/description": "my test document",
    "http://www.w3.org/ns/dcat#keyword": [
      "keyword1",
      "keyword2"
    ],
    "http://purl.org/dc/terms/creator": {
      "http://xmlns.com/foaf/0.1/name": "My Org"
    },
    "http://purl.org/dc/terms/license": "https://creativecommons.org/licenses/by/4.0/",
    "http://www.w3.org/ns/dcat#landingPage": "https://mydepartment.myorg.com/my-offer",
    "http://www.w3.org/ns/dcat#mediaType": "text/plain",
    "https://semantic.sovity.io/dcat-ext#httpDatasourceHintsProxyMethod": "true",
    "https://semantic.sovity.io/dcat-ext#httpDatasourceHintsProxyPath": "true",
    "https://semantic.sovity.io/dcat-ext#httpDatasourceHintsProxyQueryParams": "true",
    "https://semantic.sovity.io/dcat-ext#httpDatasourceHintsProxyBody": "true",
    "http://purl.org/dc/terms/publisher": {
      "http://xmlns.com/foaf/0.1/homepage": "https://myorg.com/"
    }
  },
  "https://w3id.org/edc/v0.0.1/ns/privateProperties": {},
  "https://w3id.org/edc/v0.0.1/ns/dataAddress": {
    "https://w3id.org/edc/v0.0.1/ns/type": "HttpData",
    "https://w3id.org/edc/v0.0.1/ns/baseUrl": "{{PROVIDER_EDC_SOURCE_URL}}"
  }
}

Note that I am using webhook.site as a PROVIDER_EDC_SOURCE_URL so I can investigate the actual POST calls sent out during the transfer process.

2. [PRODUCER] Create a Policy - POST to {{PROVIDER_EDC_MANAGEMENT_URL}}/v2/policydefinitions

{
    "@type": "https://w3id.org/edc/v0.0.1/ns/PolicyDefinition",
    "@id": "{{POLICY_ID}}",
    "https://w3id.org/edc/v0.0.1/ns/policy": {
        "@type": "http://www.w3.org/ns/odrl/2/Set",
        "http://www.w3.org/ns/odrl/2/permission": [
            {
                "http://www.w3.org/ns/odrl/2/action": {
                    "http://www.w3.org/ns/odrl/2/type": "USE"
                },
                "http://www.w3.org/ns/odrl/2/constraint": []
            }
        ]
    }
}

3. [PRODUCER] Define a (simple) policy - POST to {{PROVIDER_EDC_MANAGEMENT_URL}}/v2/contractdefinitions

{
    "@id": "{{CONTRACT_DEFINITION_ID}}",
    "@type": "https://w3id.org/edc/v0.0.1/ns/ContractDefinition",
    "https://w3id.org/edc/v0.0.1/ns/accessPolicyId": "{{POLICY_ID}}",
    "https://w3id.org/edc/v0.0.1/ns/contractPolicyId": "{{POLICY_ID}}",
    "https://w3id.org/edc/v0.0.1/ns/assetsSelector": [
        {
            "@type": "CriterionDto",
            "https://w3id.org/edc/v0.0.1/ns/operandLeft": "https://w3id.org/edc/v0.0.1/ns/id",
            "https://w3id.org/edc/v0.0.1/ns/operator": "=",
            "https://w3id.org/edc/v0.0.1/ns/operandRight": "{{ASSET_ID}}"
        }
    ]
}

4. [CONSUMER] Retrieve the odrl:hasPolicy.@id value of the offered asset - POST to {{CONSUMER_EDC_MANAGEMENT_URL}}/v2/catalog/dataset/request

{
  "@context": {
    "edc": "https://w3id.org/edc/v0.0.1/ns/"
  },
  "@type": "DatasetRequest",
  "@id": "{{ _.ASSET_ID }}",
  "counterPartyAddress": "{{ _.PROVIDER_EDC_PROTOCOL_URL }}",
  "protocol": "dataspace-protocol-http"
}

5. [CONSUMER] Start the negotiation - POST to {{CONSUMER_EDC_MANAGEMENT_URL}}/v2/contractnegotiations

{
  "@type": "https://w3id.org/edc/v0.0.1/ns/ContractRequest",
  "https://w3id.org/edc/v0.0.1/ns/consumerId": "{{CONSUMER_EDC_PARTICIPANT_ID}}",
  "https://w3id.org/edc/v0.0.1/ns/providerId": "{{PROVIDER_EDC_PARTICIPANT_ID}}",
  "https://w3id.org/edc/v0.0.1/ns/connectorAddress": "{{PROVIDER_EDC_PROTOCOL_URL}}",
  "https://w3id.org/edc/v0.0.1/ns/protocol": "dataspace-protocol-http",
  "https://w3id.org/edc/v0.0.1/ns/offer": {
    "https://w3id.org/edc/v0.0.1/ns/offerId": "<odrl:hasPolicy.@id>",
    "https://w3id.org/edc/v0.0.1/ns/assetId": "{{ASSET_ID}}",
    "https://w3id.org/edc/v0.0.1/ns/policy": {
            "@id": "<odrl:hasPolicy.@id>",
            "@type": "http://www.w3.org/ns/odrl/2/Set",
            "http://www.w3.org/ns/odrl/2/permission": {
                "http://www.w3.org/ns/odrl/2/target": "{{ASSET_ID}}",
                "http://www.w3.org/ns/odrl/2/action": {
                    "http://www.w3.org/ns/odrl/2/type": "USE"
                },
                "http://www.w3.org/ns/odrl/2/constraint": []
            },
            "http://www.w3.org/ns/odrl/2/prohibition": [],
            "http://www.w3.org/ns/odrl/2/obligation": [],
            "http://www.w3.org/ns/odrl/2/target": "{{ASSET_ID}}"
        }
  }
}

6. [CONSUMER] Using the @id from the response of (5), retrieve the contractAgreementId - GET to {{CONSUMER_EDC_MANAGEMENT_URL}}/v2/contractnegotiations/<@id>

7. [CONSUMER] Start Data Push - POST to {{CONSUMER_EDC_MANAGEMENT_URL}}/v2/transferprocesses

{
    "@type": "https://w3id.org/edc/v0.0.1/ns/TransferRequest",
    "https://w3id.org/edc/v0.0.1/ns/assetId": "{{ASSET_ID}}",
    "https://w3id.org/edc/v0.0.1/ns/contractId": "<contractAgreementId>",
    "https://w3id.org/edc/v0.0.1/ns/connectorAddress": "{{PROVIDER_EDC_PROTOCOL_URL}}",
    "https://w3id.org/edc/v0.0.1/ns/connectorId": "{{PROVIDER_EDC_PARTICIPANT_ID}}",
    "https://w3id.org/edc/v0.0.1/ns/dataDestination": {
        "https://w3id.org/edc/v0.0.1/ns/type": "HttpData",
        "https://w3id.org/edc/v0.0.1/ns/baseUrl": "{{CONSUMER_EDC_TRANSFER_TARGET_URL}}"
    },
    "https://w3id.org/edc/v0.0.1/ns/properties": {},
    "https://w3id.org/edc/v0.0.1/ns/privateProperties": {
    "https://w3id.org/edc/v0.0.1/ns/pathSegments": "my-endpoint",
    "https://w3id.org/edc/v0.0.1/ns/method": "POST",
    "https://w3id.org/edc/v0.0.1/ns/queryParams": "filter=abc&limit=10",
    "https://w3id.org/edc/v0.0.1/ns/contentType": "application/json",
    "https://w3id.org/edc/v0.0.1/ns/body": "{\"myBody\": \"myValue\"}"
  	},
    "https://w3id.org/edc/v0.0.1/ns/protocol": "dataspace-protocol-http",
    "https://w3id.org/edc/v0.0.1/ns/managedResources": false
}

Note that this final request takes into account the parameterized HTTP data sources documentation.

Following the above process, I am only able to make a GET request to baseUrl without any parametrization at all. None of the privateProperties set in the request are taken into account.

Any help on the above and/or any referral to documentation outside of this repo that would further clarify the API structure/usage would be immensely helpful.

Thanks!

[tmberthold]

Hi and thank you for reporting!

We are currently facing issues with the paramterization:

• Path- and Query-Parameterization are ignored Path- and Query-Parameterization are ignored edc-ui#707
• Transfer fails if method-parametrization is enabled Transfer fails if method-parametrization is enabled edc-ui#703

These issues also affect the backend, please wait until we fix these issues.

[panosprotopapas]

Thanks for your quick reply.

Would you have a very rough estimate on when this fix might be implemented? Are we talking weeks or months?

Much appreciated!

[tmberthold]

We are currently planning to fix this with a new Community Edition release this week, but I can't promise that.

[tmberthold]

Fixed in latest release.