[FEATURE] Allow `$.flow.suspend` to resume within the same step

[dylburger]

See this comment for details.

What I would like to achieve, if possible of course, would be a suspension which allows to be resumed within the calling component. Maybe a win-win solution, from a user experience standpoint, could invoke a new resume() function within the calling component itself, like the following:

export default {
  async run({ $ }) {
    const { cancel_url, resume_url } = $.flow.suspend();
    // ...
  },
  async resume({ resumeData }) {
     // ...
  }
}

[ecobisi]

Thank you, @dylburger and team.

I am authoring a long-running logical step and would need to choose whether to split it into two components (one to init the step and another to handle its resume) now or just wait for this feature to be released, so that I can deploy just one component, with the latter being a much more natural option to the end user's eyes, I believe.

I am sorry if I appear to be pushy - I really don't mean to be like that and just would need to know what would be the best option for me right now.

Thus, may I know if this feature is something you are currently working on and if you are going to release something shortly, please? If you have an approximate ETA to share, that would be really awesome.

Many thanks! 🖖

[dylburger]

@ecobisi We were chatting a lot about this feature internally, and @casret shipped an MVP I'd love for you to try and give feedback on.

You can now call $.flow.rerun from within a step:

import axios from 'axios'

export default defineComponent({
  async run({ steps, $ }) {
    const { run } = $.context
    // $.context.run.runs starts at 1 and increments when the step is rerun
    if (run.runs === 1) {
      // $.flow.rerun(timeout in ms if not resumed, context, max retries)
      const { cancel_url, resume_url } = $.flow.rerun(86400 * 1000, { hello: "world"}, 1)

      // Send resume_url to external service
      await axios({
        method: "POST",
        url: "your callback URL",
        data: {
          resume_url,
          cancel_url,
        }
      })
    }
    // When the external service calls back into the resume_url, I have access to 
    // the callback data within $.context.run.callback_request
    else {
      const { callback_request } = run
      return callback_request
    }
  },
})

If you need to keep track of additional state between runs, the context argument ({ hello: "world"} passed to $.flow.rerun) is also exposed in $.context.run between runs.

The final argument to $.flow.rerun — max_retries — defaults to 10 to avoid infinite loops, but can be set to any integer.

Let me know if that helps and if you think we can improve the DX!

[ecobisi]

Thank you, @dylburger and @casret - this looks promising!

I am trying it but can't have my component to resume correctly; there is a TEST_HANDLE thing in the resume_url which appears to be very suspicious to my eyes https://api.pipedream.com/v1/sources/p_brCQdW8/resume/TEST_HANDLE.

Am I missing something here or does the MVP need to be fully deployed somehow?

[ecobisi]

Okay - I have just noticed resume_url works only if the workflow is deployed. In test mode it doesn't and return the TEST_HANDLE mentioned above.

I will keep trying this new feature and share my feedback here. Many thanks, guys!

[ecobisi]

A great job overall, congrats @dylburger @casret and team!
Just a couple of things which I believe could be improved:

1. If $.flow.rerun() / delay() / suspend() is called there seems to be no way to abort the underlying suspension operation initiated by pause() and continue with the normal flow, in the event a suspension in no more needed. Apparently, deleting data.pause should do the trick but that object is hidden within the calling object's namespace: maybe you can add a function which does just that, please? Something like $.flow.resume() or similar.
2. After suspension, the subsequent resume appears to take several seconds (~20, in my test) to take place. Is that by design or could that be quickened a bit?

Thanks again.

[dylburger]

@ecobisi I hear you on the use case. I'm tracking #1 in a separate ticket at #3151 .

Are you seeing a total end-to-end time of 20 seconds between the HTTP request to the resume_url and the time the workflow resumes? Do you see this behavior on a new workflow? i.e. do you have a specific case where we can reproduce the behavior?

[ecobisi]

@dylburger I believe that was a transient issue which I can't replicate anymore, possibly on my own testing code. I have retried several times so far and everything appears to be super fast.

Thanks!

[dylburger]

Great, thanks for confirming! Let us know if you see it again.