0.5.x to 0.7.x migration

[ndr-brt]

Hey, some days ago version 0.7.0-rc1 was released, it brings different breaking changes.
In order to provide a comprehensive migration guide we ask some feedback to those who tried (and hopefully succeeded :) ) updating from 0.5.x to 0.7.x, about encountered difficulties, changes needed to make and so on.
Please add yours as comment in this discussion.

[florianrusch-zf]

Breaking change: EDR callback is gone

Mentioned here: https://matrix.to/#/!mYxOilDPMLCQhMoIVc:matrix.eclipse.org/$DSlotGD4aqsC9LpNnBg0TjwOPTjlwa_puwmwBH3cb9Y?via=matrix.eclipse.org&via=matrix.org&via=dev-null.rocks

[wolf4ood]

The setting backendService.httpProxyTokenReceiverUrl, which configured an EDR receiver backend using the upstream EDC extension is no longer available as well as the extension is not shipped in the tractusx 0.7.x distribution.

The main reason is that EDC/TractusX-Edc switched to data plane signaling for handling transfers and the EDR format specified in DPS spec was not handled properly but the receiver extension.

Now all the EDRs are automatically stored in the upstream implementation of the EDR store/cache and available in the new EDRs API. If users still need to receive the EDR on their backend and hand this manually, upstream EDC handle this natively without a custom extension.

TractusX ships those extensions to make it possible to receive an EDR associated with the transfer.process.start event using HTTP callbacks.

This can be done in two ways:

• dynamic callbacks (associated to each transfer process)
• static callbacks (configured at boot time)

Example of dynamic callback

When starting a new transfer process callbackAddresses can be attached to the transfer request:

"callbackAddresses": [
        {
            "events": [
                "transfer.process.started"
            ],
            "uri": "https://mybackend/edr"
        }
    ]

Example of static callbacks configuration

EDC_CALLBACK_CB1_URI="https://mybackend/edr"
EDC_CALLBACK_CB1_EVENTS= "transfer.process.started"

When configuring static callbacks like above, users will receive notifications about transfer process start events of both sides consumer/provider if the connector acts as both. This can be check by the type property in the event itself.

Users need to be aware that if they handle the receiver manually, they still to have to handle EDR refresh with the EDRs API.

More info about callbacks here

[paullatzelsperger]

Another Breaking Change: configuration MIW -> IATP. details to follow.

[wern]

Would love to see the "new" SAP wallet to be mentioned in the docs instead. ;-)

[istvan-nagy-epam]

Hi,
since the SAP wallet is not self-hostable as you mentioned @paullatzelsperger , what would you recommend for CI/CD test scenarios? Is there a locally hostable alternative we can rely on?

Also, looking forward to the configuration details of the whole BdrsClientImpl to make it work.

Thank you in advance!
CC: @thomas-henn @agg3fe @tunacicek @RazvanZmau

[paullatzelsperger]

@istvan-nagy-epam I cannot give you any recommendation, at least not an officially sanctioned and supported one.

If you wanna play around with BdrsClientImpl, you could look at its component test: https://github.com/eclipse-tractusx/tractusx-edc/blob/main/edc-extensions/bdrs-client/src/test/java/org/eclipse/tractusx/edc/identity/mapper/BdrsClientImplComponentTest.java

[istvan-nagy-epam]

Thanks @paullatzelsperger ! To be honest, I have no idea whether I need it, I am just following where the EDC error logs are leading me. It would be great to have some documentation about these changes and why we need them.

[paullatzelsperger]

absolutely. you'll need it to map BPNs to DIDs and vice versa. Unfortunately that is needed since Catena-X opted for BPNs (business partner numbers) as stable identifiers, instead of going with DIDs, so a mapping is needed, as IATP (the authentication protocol) requires DIDs.

[paullatzelsperger]

another one: change in environment variables for the Azure Vault variant

[jerzy-wachowiak]

Can you elaborate on this please?

[paullatzelsperger]

the names of the env vars have changed as outlined in this upstream PR, check "Further Notes":
eclipse-edc/Technology-Azure#82

[rafaelmag110]

From a DMA user perspective

• Changes in the policy definition properties. A policy definition now adheres to ODRL and the @type has changed to "Set".
• An Offer does not contain a target property anymore and it has to be injected in the ContractRequest message.
• The consumerId and providerId were replaced by the ODRL assignee and assigner, respectively. The latter also has to be injected in the ContractRequest Message.
• A transfer process has now a new transfer type property that should adhere to the following mapping: https://github.com/eclipse-edc/Connector/blob/main/docs/developer/data-plane-signaling/data-plane-signaling-mapping.md
• The EDR API changed and it's not longer possible to query for EDRs by contract agreement directly. A queryspec with a filter has to be used for this.

[wolf4ood]

Management API changes

Catalog

• counterPartyId is a needed property for the catalog request to be working in tx-edc. It's not mandatory (backward compatibility) but if not provided the IATP flow won't work and thus the catalog request will fail.

Transfer Process

• transferType is a needed property for the transfer request to be working in tx-edc. transferTypes are the supported transfer format in tx-edc. They return as part of the Catalog response as part of distribution format for an asset. In TX-EDC we currently supports HttpData-PULL, AzureStorage-PUSH and AmazonS3-PUSH. The transferType i's not mandatory (backward compatibility) but if not provided the transfer request will fail.

Edrs

The 0.5.4 EDRs API, where removed due the adoption of DPS (data plane signaling) and reborn under a different shape in /v2/edrs

The main difference between 0.5.4 and 0.7.0 is how the token refresh is handled. In 0.5.x when an EDR was about to expire or the users had to manually trigger another transfer process, or by using the EDR API which did that automatically. This approach had major drawbacks like proactively creating transfer process on EDR expiration. In 0.7.0 this changed radically by adopting DPS and overlaying on top of it a refresh mechanism for EDRs.

For a detailed usage I would suggest changing this section of the docs.

[matbmoser]

We don't know for what is this configuration:

iatp:
  # Decentralized IDentifier
  id: "did:web:changeme"
  sts:
    dim:
      url: "https://somewhere.dim.org"
    oauth:
      token_url: "https://changeme.org"
      client:
        id: "test-client-id"
        secret_alias: "test-alias"
controlplane:
  bdrs:
      server:
        url: "https://bdrs.test.org"

Where can we get that in the portal?

[sabarishrl]

@paullatzelsperger , We don't have much documentation about installing IATP (or any open source). Can you help to provide details on this?

[sabarishrl]

https://github.com/eclipse-tractusx/identity-trust, only we have the specs, no details about the implementation

[wolf4ood]

Trusted Issuer configuration

The configuration ssi.miw.authorityId for configuring a trusted issuer has been removed. Since we might support multiple trusted issuer in the future, an extension for configuring the trusted issuers is bundled in TractusX-EDC without exposing the configuration via chart.

Example of configuration with env vars:

env "EDC_IAM_TRUSTED-ISSUER_ISSUER1_ID=<issuer1-did>" 
env "EDC_IAM_TRUSTED-ISSUER_ISSUER2_ID=<issuer2-did>"

[wolf4ood]

This has been integrated here #1237

and it will be available in 0.7.1. Meanwhile the env vars config can be used.

[gerbigf]

The support for BPNLs in access policies has been removed with 0.7.x which was the main way to steer visibility of offers to partners for us. However, it's still in the docs, e.g.

here: https://github.com/eclipse-tractusx/tractusx-edc/blob/8c05e86c78b01fb4e0dea228dd681d8e498a1a4c/docs/usage/management-api-walkthrough/02_policies.md

[paullatzelsperger]

we only removed the deprecation warning and left the BPN policy eval function in place

[gerbigf]

Then it's maybe worth mentioning in the migration guide that they stay? My users are still planning that they need to upgrade and recreate all policies because the warning was very clear on that.

[paullatzelsperger]

yup, no problem. Although they can (and should!) still upgrade.

[b3n3d17]

I am sorry for my ignorance (@paullatzelsperger), but to what should I upgrade when I currently rely on using the BPN in my access policies? Is there another way to uniquely reference Sites or Business partners without the BPN?

[paullatzelsperger]

@b3n3d17 please refer to the documentation. TL;DR: you shouldn't reference individual BPNs, as that would cause a database migration of the policies if they ever change, but BPN Groups

[paullatzelsperger]

closing as resolved.

[sabarishrl]

Hi @paullatzelsperger ,
We're looking to upgrade the EDC; Is there a local version of DIM to test it against?

[paullatzelsperger]

there is not.

[sabarishrl]

ok, thanks