Provide Workflow-API with EDC Provisioning to manage Compute Jobs in Private Cloud (code2data, code2compute) #2405
-
In the latest Q&A-Session, I introduced our plans to use an edc connector as a Gatekeeper to our compute infrastructure and trigger Jobs and return the produced result to the Consumer. Our first attempt was to use a http-data-asset, which will be trigger the backend with an incoming transfer request, and process a specific execution workflow. For short jobs this workflow succeeded, but for longer jobs timeouts occur in the edc and produces errors in the backend. To face this problem, I got an introduction the In the Q&A-Session about the http-provision extension, which faces such long-running workflows to prepare data before the actual transfer. Because of the missing documentation of that extension (only system-tests used as resource), it required some time to implement a connector alongside a proper backend. I have here a small summary which indicate all points that must be considered. I would love to see that committed/added in the main repo :) I have now a running prototype which is able to deploy jobs into a kubernetes system and wait for that termination. With the callback, the edc can be informed of the finished process and provide a dataAddress from which the data can be loaded and transferred to the consumer. IMHO, the current implementation of the http-provisioning extension is very limited and very strict. Please correct me if I'm wrong with the following points.
|
Beta Was this translation helpful? Give feedback.
Replies: 8 comments 1 reply
-
I'm having difficulty understanding what the ask is. Can you please briefly outline what you are trying to do in a couple of sentences? As a rule of thumb, |
Beta Was this translation helpful? Give feedback.
-
In general, we want to connect our on-premise compute infrastructure (currently, it is just a Kubernetes cluster) with the edc to provide compute resources to a dataspace for different services to run them e.g. a tensorflow, PyTorch or something else but for now it is just Carla and ROS as a data-generator. At the moment, we're just using the edc as it is with the extensions you provide. With the http-provision extension and our basic middlelayer-api, I was able to implement a first prototype, which is able to schedule a static data-generator job in Kubernetes, and after it's execution the generated file will be transferred to the data-consumer. Now, what I want to do, is to provide services assets, where a service-consumer can or is required to add additional input. E.g. For a PyTorch service, a consumer wants to choose its own scripts or training data, or use that from other participants from the dataspace. This "Additional Information was also discussed in #1386 but not for |
Beta Was this translation helpful? Give feedback.
-
Based on the description you provided, I would consider revising the approach. It appears you are trying to include client "artifacts" (e.g., scripts, metadata, code, containers, etc.) as part of the connector message exchanges. This will scale poorly and will likely result in a very complicated implementation. Here's a brief outline of how I would look at this problem. Take it with a grain of salt since I don't have visibility into your specific requirements. As a design principle, client "artifacts" should never be associated with the control-plane flow; instead, they should be contributed out-of-band in the same way the control-plane/data-plane is architected. In fact, client artifacts are part of the data-plane flow. I would therefore create a custom data-plane:
This approach does not require specialized processing or tunneling artifacts in the control plane. It will also scale much better since the reliability and performance of object storage (or another system) can be leveraged to transfer very large artifacts. This also has the advantage that the data-plane could be provisioned on a trusted third party so that the provider never has access to the client's artifacts. In this case, the provision step could call the API of a third party. |
Beta Was this translation helpful? Give feedback.
-
This scenario is different than the one we discussed (or I misunderstood the requirements):
Importantly, the EDC data plane is not needed in this scenario. |
Beta Was this translation helpful? Give feedback.
-
Sorry for the confusion, I was thinking a little ahead in the scenario I created. The following picture reflects more the post above, and has only two participants.
|
Beta Was this translation helpful? Give feedback.
-
At a high level, that seems reasonable, but with the caveat that I don't know the requirement details. Also, line 2.2 should not exist as connectors only interact with one another (the credentials should be part of the transfer protocol). |
Beta Was this translation helpful? Give feedback.
-
Thank you for your positive opinion! Line 2.2 shows the flow of the created result by the provisioner (credentials). So after the provisioner has its jobs finished, it uses the callback with the information for the provider-connector where it can retrieve the result and send it to the DateDestination (Line 2.2). This is how I understand the provider-push behavior. The DataDestination was specified in the body of the This is the call:
|
Beta Was this translation helpful? Give feedback.
-
@reisman234 : I also created a related discussion thread for compute-to-data capabilities based on the functional requirements of the new IDSA rulebook v2. I would suggest to work together. |
Beta Was this translation helpful? Give feedback.
At a high level, that seems reasonable, but with the caveat that I don't know the requirement details. Also, line 2.2 should not exist as connectors only interact with one another (the credentials should be part of the transfer protocol).