Missing Asset Properties in Contract Negotiation Store (Consuming Side)

[richardtreier]

Hi,

the EDC DataDashboard's Contract Agreement currently shows very little information compared to what was available during the consumption of the data offer or that was produced during the negotiation.

We are currently implementing an API endpoint where we take the EDC's Contract Agreements, Contract Negotiations and other stores and try to piece together the information we need for a fully featured Contract Agreement List Page.

One of the important parts is that we want to show the asset that the contract agreement was negotiated for, meaning, we need the name, description, keywords etc, and preferably the entire amount of properties, meaning the metadata.

We noticed that the tables and data the EDC produces are not in the 3rd normal form. They contain embedded JSONs and the like. This, however, did not discourage us, and we are confident in handling aggregating the available information either in native DB queries or in Java Code.

However,

we noticed that the Contract Negotiation's List<ContractNegotiation> contractOffers JSON, which is the only source for the other connector's asset in the EDC, does not include all asset properties. At least not on the consuming side, where it would matter the most, as the asset is not locally resolvable by id.

A contract negotiation on the providing side contains all asset propeties.

Is this a bug? Would we need to hack the EDC with Events in hopes to retain the information that was available in the contract offer?

[ndr-brt]

hey, are you referring to the Asset properties or the DataAddress ones? Could you report a full example?

[richardtreier]

Sure!

Repro

1. Start Two EDCs

# clone edc-extensions at version 3.1.0 that uses MS8.5 / 0.0.1-20230220.patch1
git clone git@github.com:sovity/edc-extensions.git sovity-edc-extensions \
    && cd sovity-edc-extensions \
    && git checkout tags/v3.1.0

# Build an edc with DB so we can look into the data
docker build \
    -f "connector/Dockerfile" \
    -t "edc-ce-dev:local" \
    --build-arg BUILD_ARGS="-Pdmgmt-api-key -Ppostgres-flyway" \
    .;

# Start Dev Docker Compose (2 Connectors)
DEV_EDC_UI_IMAGE=ghcr.io/sovity/edc-ui:0.0.1-milestone-8-sovity4 DEV_EDC_IMAGE=edc-ce-dev:local EDC_UI_ACTIVE_PROFILE=sovity-open-source docker compose -f docker-compose-dev.yaml up

2. Negotiate a Contract

The connector at http://localhost:22000/contracts will be the producer.

The connector at http://localhost:11000/contracts will be the consumer.

# Create an asset
curl 'http://localhost:22002/api/v1/management/assets' -X POST -H 'Accept: application/json' -H 'X-Api-Key: ApiKeyDefaultValue' -H 'Content-Type: application/json' --data-raw '{"asset":{"properties":{"asset:prop:id":"urn:artifact:test","asset:prop:name":"test","asset:prop:originator":"http://edc2:11003/api/v1/ids/data","asset:prop:originatorOrganization":"Example GmbH","asset:prop:language":"ISO_639:EN"}},"dataAddress":{"properties":{"type":"HttpData","baseUrl":"https://jsonplaceholder.typicode.com/todos/1"}}}' && \
    # Create a contract definition with the pre-existing policy "always-true"
    curl 'http://localhost:22002/api/v1/management/contractdefinitions' -X POST -H 'Accept: application/json' -H 'X-Api-Key: ApiKeyDefaultValue' -H 'Content-Type: application/json' --data-raw '{"id":"test-cd","accessPolicyId":"always-true","contractPolicyId":"always-true","criteria":[{"operandLeft":"asset:prop:id","operator":"in","operandRight":["urn:artifact:test"]}]}' && \
    # Negotiate the Contract
    curl 'http://localhost:11002/api/v1/management/contractnegotiations' -X POST -H 'Accept: application/json' -H 'X-Api-Key: ApiKeyDefaultValue' -H 'Content-Type: application/json' --data-raw '{"connectorAddress":"http://edc2:11003/api/v1/ids/data","offer":{"offerId":"test-cd:521ac095-9af9-463f-b627-4c4b3a58ec8a","assetId":"urn:artifact:test","policy":{"permissions":[{"edctype":"dataspaceconnector:permission","uid":null,"target":"urn:artifact:test","action":{"type":"USE","includedIn":null,"constraint":null},"assignee":null,"assigner":null,"constraints":[{"edctype":"AtomicConstraint","leftExpression":{"edctype":"dataspaceconnector:literalexpression","value":"ALWAYS_TRUE"},"rightExpression":{"edctype":"dataspaceconnector:literalexpression","value":"true"},"operator":"EQ"}],"duties":[]}],"prohibitions":[],"obligations":[],"extensibleProperties":{},"inheritsFrom":null,"assigner":null,"assignee":null,"target":"urn:artifact:test","@type":{"@policytype":"set"}}},"connectorId":"my-connector","protocol":"ids-multipart"}'

Check the Consumer's DB

Connect to jdbc:postgresql://localhost:54321/edc user edc password edc with a PostgreSQL DB Client of your choice.

Query the following:

select
    id,
    agreement_id,
    created_at,
    counterparty_address,
    jsonb_array_elements(contract_offers::jsonb) -> 'asset' -> 'properties' as asset_properties
from edc_contract_negotiation;

The results should look something like this:

+------------------------------------+--------------------------------------------+-------------+---------------------------------+--------------------------------------+
|id                                  |agreement_id                                |created_at   |counterparty_address             |asset_properties                      |
+------------------------------------+--------------------------------------------+-------------+---------------------------------+--------------------------------------+
|9a049400-53c6-4fbd-badc-460a3737ed31|test-cd:b464e045-cc86-401d-a533-6926026f5f00|1683298016255|http://edc2:11003/api/v1/ids/data|{"asset:prop:id": "urn:artifact:test"}|
+------------------------------------+--------------------------------------------+-------------+---------------------------------+--------------------------------------+

The asset's missing all properties except for the ID.

This can also be observed when comparing the Contract Agreement Details:

• Consumer Contract Agreement Details (http://localhost:11000/contracts) contain only the Asset ID.
• Producer Contract Agreement Details (http://localhost:22000/contracts) contain all asset properties.

[jimmarino]

Assets should not be contained within the contract offer. Instead, they should only contain id pointers.

Contract offers on the producer stored those as a workaround for an issue in the old IDS specification that has now been fixed in the DSP specifications. See the issue #2889. Assets should be dereferenced from the AssetIndex or catalog cache.

[richardtreier]

Okay. Where is the consumed consuming contract agreements' asset information persisted then?

Is it persisted in the consumer's asset list? If so, is there a differentiation between owned assets and foreign assets?

Does one have to do another request to the producer, which might not have that asset anymore? The latter might also be highly inefficient considering a list of contract agreements consumed from many different producers.

Considering how the list of asset properties are the placeholder for most information about transferred payloads and categorization of payloads. One really needs these information to know what to initiate transfers for, on the consumer side, amongst other things.

That's considered, right?

[jimmarino]

Okay. Where is the consumed consuming contract agreements' asset information persisted then?

The contract only references an asset, it doesn't contain it. Asset information should be managed by the federated catalog or an analogous system on the consumer. Since the asset id is a GUID, the asset associated with a contract agreement can be dereferenced using its id.

Is it persisted in the consumer's asset list? If so, is there a differentiation between owned assets and foreign assets?

Does one have to do another request to the producer, which might not have that asset anymore? The latter might also be highly inefficient considering a list of contract agreements consumed from many different producers.

The design of the EDC is that the consumer should periodically query the federated catalogs of producers and cache the results. Deleting an asset is like removing a resource on the web. Also, if the provider deletes an asset, data transfers won't work, so the provider needs to be careful about deleting them (i.e., no active contract agreements or transfer processes).

Considering how the list of asset properties are the placeholder for most information about transferred payloads and categorization of payloads. One really needs these information to know what to initiate transfers for, on the consumer side, amongst other things.

That's considered, right?

Yes, that's what the federated catalog is for. Users can create their own catalog systems using technologies they choose as well - there is no requirement to use the EDC catalog.