Dynamic creation of scheduled tasks

[reflog]

Hi.
Maybe I'm missing something - but from the README it appears that all scheduled tasks have to be predefined, is that correct? I cannot add/remove tasks at runtime?

[Ehesp]

Hey, yeah that's correct. @Salakar will take a look at this, should be able to get something in today so manage them in realtime.

Cheers

[reflog]

That would be amazing! Looking forward to it. 👍

[Salakar]

@reflog yep I'll look at this today, was initially gonna move this kibda dynamic schedule functionality into jobs (delayed jobs), but guess it makes more sense here.

[reflog]

Excellent. In any case - this project is amazing, if I'm not mistaken, it's the first and only open source task scheduler that supports multi-server setup. Pretty impressive feat!

[Salakar]

@reflog I got quite far with this today, it required a re-write as the current setup required predefining schedules on boot and relying on node to handle timers - so it's taking longer than I thought because this logic is now up to redis to handle but here's what I've got working so far:

• Timestamp based scheduling via Lua scripts
• Enabling / disabling jobs - even if several occurrences have already been pushed to the queue.
• Cron now supported as an interval string, as well as the standard text parser.
• Schedule querying api - enable/disable, update, add and delete schedules.
• History of schedules.
• Schedule Stall prevention.
• Schedule uniqueness locks - to avoid conflicts - thinking of doing this by sha1sum of the schedule but excluding the interval as part of the sha1sum by default but also allow giving a custom id for the schedules.
• Allow schedules to be set to only run X number of times or infinitely
  • By default all schedules are set to infinitely repeat based on their scheduled times. i.e 'once tomorrow at 12.15 pm'

Lowest precision supported is 1 second.

@Ehesp @reflog can you think of anything else that might be good to have?

Little rundown of the approach I'm taking if interested:

• Sorted set with schedule occurrences pushed up by node js.
• Lua script that RPUSH's any occurrences in the set that are less than or equal to current timestamp into a 'ready' list. But only if not already in 'ready' or 'inprogress' lists. (prevents same schedule from running more than once at a time)
• Workers BRPOPLPUSH off the 'ready' list into a 'inprogress' list and then run the schedule function in node.
• 'inprogress' queue monitored for stalled schedules
• As soon as a worker finishes running the schedule occurrence it removes it from 'inprogress' and then pushes the next occurrence for that schedule into the sorted set.

Think the logic is valid hah =]

[reflog]

This most definitely covers all my usage scenarios! 💯

[cesar]

Greetings, I would like to help with this issue

[Salakar]

@reflog Another update just to let you know I am still working on this ;p

Schedules can now also define a 'starts' and 'ends' or the number of 'times' its should run.

Example:

const schedule = {
  name: 'My Schedule',
  runs: 'path.to.global.function',
  interval: 'at 5:00 pm every 2 days', // required - human interval string or cron string
  starts: 'on sunday', // optional - string, timestamp or date
  ends: 'in 28 days', //optional - string, timestamp or date
  data: {
    foo: 'bar',
  },
};

Above example internally outputs as below:

Progress can be tracked on the 'next' branch: https://github.com/redibox/schedule/commits/next

[Salakar]

Pretty much there now with it, gonna work on tests now before releasing.

Can see progress on the 'next' branch: https://github.com/redibox/schedule/commits/next - still cleaning up shop though =]

[Salakar]

@reflog v2 alpha code pushed to master, 93% test coverage, still need to work on the docs and some more public api methods. Feel free to give it a whirl - the tests in schedule.spec demonstrate how things are created.

Schedules provided in the options are default schedules, these persist - even if you flush redis they'll re-appear (added some flush detection logic), only way to get rid of them is a code change. Schedules that are created dynamically using .create() do not persist on flush, these are your throwaway / run once / a couple times schedules.

Alpha supports starting at, ending at, X number of times and interval/cron - can mix and match.

[jitesh121]

want to schedule some dynamic jobs like mail campaigns as a drip campaign... please suggest me the best options for node

[jitesh121]

currently i am using agenda library of node but i am unable to find the proper solution for this

[Ehesp]

@jitesh121 have you looked at https://github.com/OptimalBits/bull

[jitesh121]

@jitesh121 have you looked at https://github.com/OptimalBits/bull

yes i looked but ...don't know to do time based scheduling in it

[PricerFR]

I'm working on nodej + typescript application what do you think about this piece of code ?
the "let" operator give you each instance as you need .

for (let i = 1; i <= 5; i++) {

const cronvalue = i + " * * * * *";
console.log("Starting cron value of : " + cronvalue);
cron.schedule(cronvalue, () => {
    console.log("catched  : " + cronvalue);

});

}

here is the result :

[Node] Starting cron value of : 1 * * * * *
[Node] Starting cron value of : 2 * * * * *
[Node] Starting cron value of : 3 * * * * *
[Node] Starting cron value of : 4 * * * * *
[Node] Starting cron value of : 5 * * * * *

[Node] catched : 1 * * * * *
[Node] catched : 2 * * * * *
[Node] catched : 3 * * * * *
[Node] catched : 4 * * * * *
[Node] catched : 5 * * * * *