Fix flakey tests

[megawac]

Several of the tests (especially in the queue test suite) fail quite frequently. This is due to setTimeout timing issues. We could sort of address this by increasing the delays but that isn't a proper solution. These tests probably just need rewrites.

These issues have been observed several times in the past

• Browser CI Setup #1152 (blocking sauce integration)
• CITGM test failing on s390 #1303, Asyc failing with citgm on linux #1311 (citgm fails intermittently)
• frequently in travis build failures where we have to manually retrigger the build

I created PR #1321 as a stopgap, to alleviate some of the issues a bit by retrying the flakier tests when they fail. All retries we implement should be removed along with this fix for this issue as retries in unit tests are really ugly and may be masking other issues with the library.

Note: this may be difficult to reproduce on your local system. I've mainly noticed it in simulated environments with slower systems causing setTimeout calls to be delayed. Systems like travis/sauce fail quite frequently

[aearly]

The mapValues tests were written in such a way that they verify concurrency without using timeouts. Someone could use a similar strategy for refactoring tests that use timeouts.

[aearly]

Got some interesting new failures here: https://travis-ci.org/caolan/async/builds/220394697

Don't know what happened in the first applyEach test. Also fascinating to see that setTimeout(50) + setTimeout(50) = 99 in the retry test.

1) applyEach applyEach:
      Uncaught AssertionError: expected [ 'one', 'two', 'three' ] to deeply equal [ 'two', 'one', 'three' ]
      + expected - actual
       [
      +  "two"
         "one"
      -  "two"
         "three"
       ]
      at mocha_test/applyEach.js:32:35

  1) retry retry with interval when all attempts fail:
     Uncaught AssertionError: expected 99 to be above 99
      at mocha_test/retry.js:72:36

[aearly]

  asyncify
    promisified
      bluebird
        ✔ resolve
WARN: '%cWarning: a promise was rejected with a non-error: [object String]
    (No stack trace)

[aearly]

  parallel
    ✖ parallel object
      Firefox 52.0.0 (Linux 0.0.0)
    Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[aearly]

FAILED TESTS:
  cargo
    ✖ without callback
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(3) ] to deeply equal [ Array(4) ] (:0)

[hargasinski]

FAILED TESTS:
  cargo
    ✖ bulk task
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(4) ] to deeply equal [ Array(6) ] (:0)

[hargasinski]

FAILED TESTS:
  cargo
    ✖ drain twice
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected 1 to equal 2 (:0)

[hargasinski]

The concat error might be on me?

FAILED TESTS:
  concat
    concat
      ✖ basics
        Firefox 52.0.0 (Linux 0.0.0)
      Timeout of 250ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
      require<["/home/travis/build/caolan/async/mocha_test/concat.js"]</</</</<@/tmp/mocha_test/concat.js:22:16 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:7703:17
      require<[10]</exports.default/<@/tmp/lib/concatLimit.js:41:15 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:873:16
      _asyncMap/<@/tmp/lib/internal/map.js:18:8 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:3503:9
      once/<@/tmp/lib/internal/once.js:6:8 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:3530:9
      iterateeCallback@/tmp/lib/internal/eachOfLimit.js:27:23 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:3266:24
      onlyOnce/<@/tmp/lib/internal/onlyOnce.js:6:8 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:3546:9
      _asyncMap/</<@/tmp/lib/internal/map.js:15:12 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:3500:13
      require<[10]</exports.default/</<@/tmp/lib/concatLimit.js:30:19 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:862:20
      concatIteratee/<@/tmp/mocha_test/concat.js:11:12 <- /tmp/96399286a833e1592b04287ff5e22018.browserify:7692:13
      
  queue
    ✖ bulk task
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(8) ] to deeply equal [ Array(8) ] (:0)

[aearly]

  concat
    concatLimit
      ✖ does not continue replenishing after error
        Firefox 52.0.0 (Linux 0.0.0)
      Timeout of 250ms exceeded. 

[aearly]

I'm this close 👌 to just disabling the firefox tests. They're the only tests that are flakey these days, probably due to limited CPU resources in Travis causing timeouts to not fire predictably.

[aearly]

  cargo
    ✖ without callback
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(3) ] to deeply equal [ Array(4) ] (:0)

[aearly]

  concat
    concat
      ✖ preserves order
        Firefox 52.0.0 (Linux 0.0.0)
      Timeout of 250ms exceeded.

[hargasinski]

FAILED TESTS:
  cargo
    ✖ without callback
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(3) ] to deeply equal [ Array(4) ] (:0)

[hargasinski]

  concat
    concat
      ✖ preserves order
        Firefox 52.0.0 (Linux 0.0.0)
      Timeout of 250ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[hargasinski]

FAILED TESTS:
  queue
    ✖ basics
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(8) ] to deeply equal [ Array(8) ] (:0)
    ✖ bulk task
      Firefox 52.0.0 (Linux 0.0.0)
    AssertionError: expected [ Array(8) ] to deeply equal [ Array(8) ] (:0)

[aearly]

Tests have been pretty solid after that browser CI PR. We can re-open if we find other flaky tests.