[FEATURE] Add fine-grained provisioning

[joshpalis]

Is your feature request related to a problem?

Currently, a user has the ability to create a template and provision resources but does not have the ability to update the provisioned resources without first de-provisioning the entire workflow. To improve the user experience, we should add the ability to provision new or update existing resources for a workflow that has been previously executed.

What solution would you like?

To enable fine-grained provisioning, it will be necessary to support the following update APIs as WorkflowSteps in order to execute these on the resources already created.

• Update Search Pipeline (Documentation)
• Update Ingest Pipeline (Documentation)
• Update Model (Documentation)
• Update Connector (Documentation)
• Update Index Settings (Documentation)
• Update Index Mapping (Documentation)

The directional idea is to utilize the update API to handle updates to provisioned workflows with the status COMPLETED. Through the update API, the user will provide an updated template and workflow_id pointing to the already provisioned workflow. This will allow us to compare the diff, execute the related update API (if one exists), and update the resources of that provisioned workflow.

Update API :

// (Currently implemented) Updates a NOT_STARTED template, full document replacement. Fails if template is COMPLETED
PUT /_plugins/_flow_framework/workflow/<workflow_id>
{
    <updated template>
}

// (For fine-grained provisioning) Updates a COMPLETED template
PUT /_plugins/_flow_framework/workflow/<workflow_id>?provision=true
{
    <updated template>
}

Task Breakdown :

• Update WorkflowResources to include a related Update WorkflowStep for each of the existing WorkflowSteps
• Add support for the above Update WorkflowSteps
• Update the CreateWorkflowTransportAction to consider the case in which a workflow has a status of COMPLETED
• Topologically sort both the given updated template and existing template into separate lists of ProcessNode and compare, identifying new and updated nodes.
• Execute the necessary action for the new list of ProcessNode, either update or provision, updating the related state index document when necessary

Do you have any additional context?

This capability is intended to support the following use cases :

• fine-grained update to be able to update an existing create ingest pipeline / create search pipeline step
• fine grained provision to execute the update of ingest pipeline / search pipeline
• fine grained provision to add an ingest pipeline step / search pipeline step