[Feature] Allow to register new flow dynamically

[dan2li]

[Feature] Support dynamically appending flows after the flow service started.
Currently, it is only possible to register a workflow before starting the flow service. If you need to register a new workflow after starting the flow service, a restart is required. We need to provide a more flexible solution in the future.

[s8sg]

I think by design it should not require a restart. If this is happening it can be a bug.

The idea is to define the workflow in each requests. Even for partial request

[dan2li]

I think by design it should not require a restart. If this is happening it can be a bug.

I'm not quite sure, but it seems like the task queue and consumer for each flow are initialized in the StartQueueWorker function, which is only called once when the flow service is started. This is why a restart of the flow service is needed to make new flows effective.

[dan2li]

The idea is to define the workflow in each requests. Even for partial request

Are you suggesting that after separating flow and workload, we can attach the complete flow information to each request? That's ok if there is also a method to dynamically register workloads.

[s8sg]

I’ll check this over the weekend

[dan2li]

Hi @s8sg ，how are things going? :)

[Lucifer8729]

hey @s8sg , this is very crucial for a workflow engine

[s8sg]

@Lucifer8729 I would love to know why it’s crucial for a workflow orchestration. For example what kind of business use cases needs this and which other platform provides such support

Goflow supposed not to be a traditional workflow engine. It supposed to solve common use cases such as orchestrating a set of task that need to be executed in async reliably. I wanna stress on the reliability part because that is where I think the primary focus for this library should be

In my experience I have hardly came across use cases where you need to change the orchestration on the fly very often.

For not so common use cases I'll suggest to contribute to the repo so that we can close the high priority items and make progress for the additional requirement such as this

[s8sg]

Duplicate of #66

[dan2li]

@s8sg To clarify, the goal is not to dynamically change the orchestration during runtime, but to register a new flow, which is a different scenario from #66 . When I use goflow to create a workflow platform similar to Airflow, which allows users to dynamically define and submit new workflows, this is a necessary function.

[s8sg]

@dan2li do you want to register the flow with a new name ?

[s8sg]

I might have misunderstood. I think as long as we are not updating the existing workflow I’m fine to dynamically register a new workflow on the same flow service. Reopening the same

[dan2li]

Awesome! @s8sg could you help to review #74 ? :)