Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random hangs #141

Closed
coldlamper opened this issue Jan 29, 2015 · 20 comments
Closed

Random hangs #141

coldlamper opened this issue Jan 29, 2015 · 20 comments

Comments

@coldlamper
Copy link

After updating to 1.0.6, the cron will randomly hang and not fire. I have it set to run every second and at random times, hours in, it will just hang and the cron will not fire anymore. It's hard to reproduce except waiting for a long period of time. Maybe you could possibly see a scenario in which that could happen.

var cronJob = require('cron').CronJob;
moment = require('moment-timezone');
var count = 0;

var job = new cronJob('*/1 * * * * *', function () {
    count++;
    console.log('Count: ' + count);

        // read and write from redis 
});

job.start();

It will end up stuck on:
...
Count: 14565
Count: 14566
Count: 14567

or similiar

@ncb000gt
Copy link
Member

Interesting, I'll take a look when I get a chance. Thanks.

@goldfire
Copy link
Contributor

I can confirm that we are seeing the same issue. We have a cron that runs every 10 seconds that we started logging and since upgrading to 1.0.6 it will just randomly stop firing after a few hours. There are no error messages or anything else I can see to explain why it stops.

@alihalabyah
Copy link

Looks like that's true

@ncb000gt
Copy link
Member

ncb000gt commented Feb 3, 2015

Ok, thanks for the notes.

@ncb000gt
Copy link
Member

ncb000gt commented Feb 4, 2015

I'm still having issues reproducing this...what version of node are you on?

var CronJob = require('./lib/cron').CronJob;
var sinon = require('sinon');

var clock = sinon.useFakeTimers();
var previous = 0;
var counter = 0;
var cronSyncSchedule = '* * * * * *';

var scheduleJob = new CronJob(cronSyncSchedule, function() {
    counter++;
    checkCounter();
}, function() {
}, true, null);
clock.tick(24*60*60*1000);

function checkCounter() {
    //console.log(counter + ' - ' + previous);
    if (counter != (previous + 1))
            console.log('SKIPPED: ' + (counter - 1) + ", COUNTER: " + counter);
    previous = counter;
}

clock.restore();

Also, would you agree this is an acceptable test to try to reproduce what you're seeing (simulated onTicks that is). The reason for simulated is that if it's a problem with the module then the onTicks should not work as expected. If it's a problem with timers or something similar then that's probably a node/iojs issue...

@goldfire
Copy link
Contributor

goldfire commented Feb 4, 2015

We are using Node 0.10.36 and the only difference was going from 1.0.5 to 1.0.6 and then back to fix it. The cron job that was having the issues for us was this one:

new CronJob('*/10 * * * * *', function() {
  console.log('onTick');
  // ....
}, null, true);

This would run for several hours and then would just stop firing (I don't know if it was the same length of time each time).

@coldlamper
Copy link
Author

We are using Node 0.10.33. Reverting back to node-cron 1.0.5 fixes the issue.
I tried your test and could not reproduce the error like that.

@ncb000gt
Copy link
Member

ncb000gt commented Feb 4, 2015

Ok. I'll see if I can try to reproduce it here.

@coldlamper
Copy link
Author

I ran your test without sinon simulated clock ticks modified like:

var CronJob = require('cron').CronJob;

//var sinon = require('sinon');

//var clock = sinon.useFakeTimers();
var previous = 0;
var counter = 0;
var cronSyncSchedule = '*/1 * * * * *';

var scheduleJob = new CronJob(cronSyncSchedule, function() {
    counter++;
    checkCounter();
}, function() {
}, true, null);
//clock.tick(24*60*60*1000);

function checkCounter() {

    console.log(counter + ' - ' + previous);
    if (counter != (previous + 1))
        console.log('SKIPPED: ' + (counter - 1) + ", COUNTER: " + counter);
    previous = counter;
}

//clock.restore();

It ended up stuck on
...
18999 - 18998
18900 - 18999
18901 - 18900

It didn't exit, just stopped.

@jack887889
Copy link

I have the same problem. it will stop after run for about 16000 times.

@ncb000gt
Copy link
Member

Does the incomplete function fire for any of you? Also, are any of you
running the latest stable node?
On Feb 11, 2015 7:36 PM, "keqinzhao" notifications@github.com wrote:

I have the same problem. it will stop after run for about 16000 times.


Reply to this email directly or view it on GitHub
https://github.com/ncb000gt/node-cron/issues/141#issuecomment-73995693.

@ncb000gt
Copy link
Member

Oncomplete not incomplete...
On Feb 11, 2015 8:22 PM, "Nicholas Campbell" nicholas.j.campbell@gmail.com
wrote:

Does the incomplete function fire for any of you? Also, are any of you
running the latest stable node?
On Feb 11, 2015 7:36 PM, "keqinzhao" notifications@github.com wrote:

I have the same problem. it will stop after run for about 16000 times.


Reply to this email directly or view it on GitHub
https://github.com/ncb000gt/node-cron/issues/141#issuecomment-73995693.

@godspeedelbow
Copy link

+1 experienced this last night, we are on 1.0.6 and 0.10.33

@cyanly
Copy link

cyanly commented Feb 15, 2015

+1 , interestingly from our log, we had a rule for 00 * * * * *, it kept firing every minute at exact ##:##:00 for 3 days until the last event when it stoppped with log at timestamp ##:##:01
node-cron: 1.0.6
node: 0.10.35
Ubuntu Server 14.04 LTS 64bit

@jdarling
Copy link

I have been having a similar issue in my own code and was about to switch to Cron when I saw this defect. From what I can tell this is something to do with the latest versions of Node and the way setTimeout/nextTick work. Deep inside error reports you will get something about recursion. Apparently setTimeout isn't a tailed method anymore.

To work around try wrapping all setTimeout's in setImmediate:

return setImmediate(function(){
  setTimeout(whatever, whenever);
});

At least this seems to work.

@h0x91b
Copy link
Contributor

h0x91b commented Feb 24, 2015

Confirm, after several hours some of my crons stop working

@h0x91b
Copy link
Contributor

h0x91b commented Feb 24, 2015

Cron stops working in function start() if fallen in

if(timeout >= 0) { ... } else {
//there....
this.stop();
}

because sometime cronTime.getTimeout() return -1 for some reason
var timeout = this.cronTime.getTimeout();
is -1

  getTimeout: function() {
      var sendAt = this.sendAt();
      var m = moment();
      var delta = sendAt - m;
      var ret = Math.max(-1, delta)
      console.log('getTimeout', sendAt.valueOf(), m.valueOf(), delta, ret);
    return ret;
  },

output:

getTimeout 1424777178994 1424777177994 1000 1000
timeout 1000 remaining 0
cron called 5067 times
getTimeout 1424777178997 1424777177997 1000 1000
timeout 1000 remaining 0
cron called 5068 times
getTimeout 1424777177999 1424777178000 -1 -1
timeout -1 remaining 0
timeout < 0
cron called 5069 times

problem in sendAt()
in if

//add 1 second so next time isn't now (can cause timeout to be 0)
        if (date.seconds() == new Date().getSeconds()) {
            //console.log('add 1 second?');
            date = date.add(1, 's');
        }

@ncb000gt
Copy link
Member

ncb000gt commented Mar 1, 2015

I just pushed a new version of the module with GH-141, this should take care of the hang issues you're seeing. Please post back if not and I can reopen.

@coldlamper
Copy link
Author

Unfortunately this issue is occurring again using node 0.12.5 and node-cron 1.0.9. Running the crons from 1 to 2 sec intervals like */2 * * * * *
The cron just stops firing and the program just hangs until i ctrl-c out.
Seems to occur more when system is under a moderate to heavy load. I would estimate it's been happening 1-3 times a week for at least the last month.

@medisoft
Copy link

Is this fixed on 1.3? On 1.1 I'm still having the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants