[FEATURE] Support workflow grouping

[ohltyler]

The frontend plugin's mental model for a "workflow" is evolving into several individual "launches" (in other words, just separately created/provisioned workflows) all falling under the same end goal of creating a single, final, application. This allows for a simpler rapid prototyping experience, and lets users view and/or test out older launches to compare performance, review previous configurations, previously created resources, etc. This backend plugin is currently building towards a one-and-done model of building a workflow without taking into account any hierarchy or context that could group workflows together under a single "application" or "final workflow" umbrella.

In its simplest form, this just means having an extra id or tag associated to a workflow that allows them to be grouped together. It is most complex form, this means introducing a new data model that would contain x number of workflows. The frontend plugin could technically handle all of this as well by adding and managing metadata tags when creating and reading workflows. But, this could lead to some potential performance bottlenecks, tricky frontend-only edge cases, and leaves out backend-only users who may want simple mechanisms for grouping workflows together. Or perhaps this wouldn't be very useful from a backend-only perspective since it would less likely involve rapid prototyping user behavior.

Curious to hear everyone's thoughts.

[kamingleung]

I wonder if "Launches" can be a separate object rather than a variant of a provisioned workflow.

Here's what I am thinking in terms of mental model:
A workflow is a working file for users can create and edit. OpenSearch keeps track of the revisions of workflow as users are editing the workflow, so that users can recall and potentially restore to a certain point.

A launch involves handling provisioning the infrastructure, which tracks the status and reference of resources that are provisioning as part of the launch. It also tracks which workflow (and which workflow revision) the user trigger the launch from.

So, users can see all of launches in the cluster. Users can also go into a workflow and see a list of launches triggered from that particular workflow.

[ohltyler]

@dbwiddis / @owaiskazi19 / @joshpalis / @amitgalitz any thoughts on the above?

[ohltyler]

Additional use cases may involve iterating over a workflow by swapping out a model, for example, but still using created resources from a previous run, e.g., a knn index. This would mean having shared resources across launches.

[dbwiddis]

When I have experimented with setting up cloud-based workflows, I've always enjoyed the ability to create a "resource group" or equivalent that lets me easily isolate those experiments and enable a "delete all" (or at lease browse-all) functionality when cleaning things up at the end. I definitely think there's a ton of value for "tags" on the backend to filter resources created. Some applications:

• workflow-id as tag allows for quick undo of a failed run
• user-provided tag allows user to delete all from a series of experiments, or at least find the changes

[kamingleung]

I think grouping the different variations of a workflow makes sense, perhaps we can think of these variations as "Versions" of a workflow. Considering the different use cases:

Use Case 1: I want to perform A/B test between the different variations of a semantic search workflow.

1. I can deploy a version of the workflow, so that the resources are independent from other versions of workflow.
2. I can deploy or un-deploy a particular version, so that I can manage the resources of the cluster.
3. I can use the Test / Prototype tool to compare versions of the same or different workflow.

Use Case 2: I want to make small updates to my existing semantic search workflow, but I don't want to deploy a new set of resources.

1. I can make updates to an existing version of the workflow, the backend will handle updating the necessary changes, but it won't create a brand new set of resources.
2. If I want to compare the changes, I will need to save as and deploy the changes as a new version.

This may also aligns with the existing model versioning concept in OpenSearch's Model Registry. I also think grouping these "Versions" by tag keep things flexible. Love to hear everyone's thoughts on this.

[ohltyler]

Introducing a workflow grouping mechanism can be the simplest approach and most natural way to handle "versioning" and persisting of standalone resources / standalone template version / standalone state / etc. In other words, rather than making a single workflow more fine-grained such that we persist task IDs and add more logic and mapping changes to the state index / global context index, and have different optional or default query params in different CRUD APIs, we can take the opposite approach of having a hierarchical "workflow group" which simply persists a set of workflows and prevent any backend or mapping changes regarding a single workflow.

We can have different mechanisms of default workflow groups and other user-friendly interfaces similar to ML commons model groups, such that users who don't want/need to care about the grouping concept can ignore it altogether.

[ohltyler]

Closing in favor of #526