Provide a way for custom commands to retry on failed should()

[alexkrolick]

Current behavior:

cy.get('#me').should('not.exist') // retries chain
cy.customGetById('me').should('not.exist') // does not retry

#1210 (comment)

Desired behavior:

cy.customGetById('me').should('not.exist') // retries custom get if "should" fails

Versions

Cypress 3.1.4

Related

• Cypress.Commands.add needs option to force retry on that command #2670
• getBy* with .should('not.exist') testing-library/cypress-testing-library#30
• getAllBy* commands chained by should don't retry correctly testing-library/cypress-testing-library#23

[kuceb]

@alexkrolick can you provide the code for customGetById? I'm not able reproduce this issue

[alexkrolick]

See testing-library/cypress-testing-library#30 - there are a number of custom commands

[kuceb]

@alexkrolick I see, you're asking for plugin authors to have access to the retry mechanism, custom commands using cy.get under the hood will still work fine

[alexkrolick]

Yes exactly

[lukeapage]

This workaround works:

cy.getByTestId = (id) => cy.get(`[data-test-id="${id}"]`);

[alexkrolick]

The custom queries we are trying to support aren't wrappers around jQuery attribute selectors; some of them use multiple DOM traversals to match labels to inputs, for example. cy.get isn't a workaround unfortunately. See linked issues.

[Lakitna]

I've been looking at this for a bit now and digging into the source code gave me the command verifyUpcomingAssertions. That's how you can wait for a should to be resolved.

However, this command is not documented and can, therefore, be a bit iffy to work with at the moment.

The way I was able to figure out how to use it was again by looking to the source of the default commands found here: https://github.com/cypress-io/cypress/tree/develop/packages/driver/src/cy/commands

When you get it working it works great, but expect to spend a lot of time tinkering. It helps to use the same basic structure that's used in default commands.

I'm at a point where I got it working perfectly except for the log.end() of the first upcoming assertion, the steps remains 'pending' (blue with spinner) when it should be 'passed' (green).

edit: I got it working now! The basic format should look something like the following. I left out logging, options etc for clarity.

Cypress.Commands.add('aThing', (element, options={}) => {
    /**
     * This function is recursively called untill the timeout passes or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolveAThing() {
        // Resolve a thing
        const aThing = $(element).attr('aThing');

        // Test the upcomming assertion where aThing is the value used to assert.
        return cy.verifyUpcomingAssertions(aThing, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolveAThing,
        });
    }

    return resolveAThing();
});

[kuceb]

You can make an arbitrary function retry and pass along its return value using a small hack that combines should and invoke

Here's an example of a custom command that makes sure all h2's are in alphabetical order, and it retries:

cy.get('h2')
.should(($els) => $els.fn = () => {
  return $els.toArray().map(el => el.innerText)
})
.invoke('fn')
.should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

or

better yet, add a new command that does this called try:

Cypress.Commands.addAll({
  prevSubject: 'optional',
},
{
  try: (subject, fn) => {
    return cy.wrap({ try: () => fn(subject) }, { log: false })
    .invoke('try')

  },
})

and use it like:

cy.get('h2').try(($els) => {
  return $els.toArray().map(el => el.innerText)
})
.should(headers => {
  const sortedHeaders = headers.concat().sort()

  expect(headers).deep.eq(sortedHeaders)
})

without a cypress parent command:

cy.try(() => {
  const h2s = cy.state('document').querySelectorAll('h2')
  return Array.from(h2s).map(el => el.innerText)
}).should(headers => {
  const sortedHeaders = headers.concat().sort()
  expect(headers).deep.eq(sortedHeaders)
})

[Lakitna]

It's a nice idea to add a command that is basically then + retry. I've run into situation where it would be usefull. Though I don't think try is the best name. It implies error handling and that you need a catch to go with it.

Couple of ideas:

Name it retry

cy.get('foo')
    .retry((elem) => {
        // ...
    });

Name it thenTry

cy.get('foo')
    .thenTry((elem) => {
        // ...
    });

Overwrite then
When you overwrite then you can add an option to enable retrying and make it false by default, something like below. Though try might be one of the few commands that can't be overwritten, I'm not sure.

cy.get('foo')
    .then((elem) => {
        // ...
    }, {retry: true});

[bohendo]

What's the status on this? I've got a two-part cy.get that I'd like to retry until it resolves to a non-zero value. My first attempt:

const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    // cy.wait(2000)
    getBalance().should('not.equal', '0.00')
  })
})

The above runs getBalance() once and then waits on the return value to pass the assertion. It fails via timeout even though my UI eventually has a non-zero balance because getBalance() isn't rerun when it's return value fails the assertion.

I can get the above test to pass by uncommenting the cy.wait(2000) but I'd rather not introduce any potential race conditions.

My second attempt, inspired by the conversation between @Lakitna and @bkucera above.

Cypress.Commands.addAll({ prevSubject: 'optional' }, {
  retry: (subject, fn) => {
    return cy.wrap({ retry: () => fn(subject) }, { log: false }).invoke('retry')
  },
})

describe('Test', () => {
  cy.retry(getBalance).should('not.equal', '0.00')
})

The above errors out with CypressError: Cypress detected that you returned a promise from a command while also invoking one or more cy commands in that promise. and I've tinkered with it for a while but can't figure out how to avoid this error while still testing what I want to test..

[Lakitna]

There is no official stance on this yet. But that won't stop us!

I've made a quick implementation of this idea using the retry name and tried to document it a bit so you can alter it. I only tested if it retries, I have no idea what its limitations are.

const _ = Cypress._;
const $ = Cypress.$;

/**
 * Basically `then`, but will retry if any upcoming assertion fails
 * @param {JQuery} subject
 * @param {function} fn
 * @param {boolean} [options.log=true]
 * Log to Cypress bar
 *
 * @return {*}
 */
Cypress.Commands.add('retry', {prevSubject: 'optional'}, (subject, fn, options={}) => {
    _.defaults(options, {
        log: true,
    });

    // Setup logging
    const consoleProps = {
        'Applied to': $(subject),
    };
    if (options.log) {
        options._log = Cypress.log({
            $el: $(subject),
            name: 'retry',
            message: fn.name,
            consoleProps: () => {
                return consoleProps;
            },
        });
    }

    /**
     * This function is recursively called untill timeout or the upcomming
     * assertion passes. Keep this function as fast as possible.
     *
     * @return {Promise}
     */
    function resolve() {
        const result = fn(subject);

        // Update logging
        if (options.log) {
            consoleProps.Yielded = result;
        }

        // Test the upcomming assertion where result is the value used to assert.
        return cy.verifyUpcomingAssertions(result, options, {
            // When the upcoming assertion failes first onFail is called
            // onFail: () => {},
            // When onFail resolves onRetry is called
            onRetry: resolve,
        });
    }

    return resolve();
});

The full extend of my tests:

it('retries', function() {
    let c = 0;
    cy.retry(() => ++c)
        .should('equal', 5);
});

[bohendo]

Thank you @Lakitna 🙏 An(other) example using cy.verifyUpcomingAssertions was just what I needed. I think cy.retry is a reserved command though (at least in v3.2.0) because if I try to add it, unrelated parts of my code start blowing up.

But I tinkered a bit, got it working & was able to remove the race condition (ie a cy.wait) I was relying on earlier. Hope the following example is useful for others who are in a similar situation:

// cy.resolve(fn).should(blah) will re-run the promise-returning fn until
// the value it resolves to passes the assertion
Cypress.Commands.add('resolve', { prevSubject: 'optional' }, (subject, fn, opts={}) => {
  const resolve = () => {
    fn(subject).then(res => cy.verifyUpcomingAssertions(res, opts, { onRetry: resolve }))
  }
  return resolve();
});

// an example function that returns a Cypress.Promise
const getBalance = () => {
  return cy.wrap(new Cypress.Promise((resolve, reject) => {
    cy.get('h1').children('span').invoke('text').then(whole => {
      cy.get('h3').children('span').invoke('text').then(fraction => {
        cy.log(`Got balance: ${whole}${fraction}`)
        resolve(`${whole}${fraction}`)
      })
    })
  }))
}

describe('Test', () => {
  it(`Should wait until balance is non-zero`, () => {
    cy.resolve(getBalance).should('not.contain', '0.00')
  })
})

[Lakitna]

You are correct, retry is an existing, undocumented cy command (see screenshot).

I would personally fall back to thentry until I made a way to overwrite then with a retry option.

[Lakitna]

Behold, the birth of the module cypress-commands!

I've made the extension on the then command and published it in a repo (and on npm) where I will add more commands in the future. I have a few commands laying on the shelf I could add.

For more details see the repo at https://github.com/Lakitna/cypress-commands

[jennifer-shehane]

Hey @Lakitna, can you add your commands library to our docs on the plugins page?

[Manuel-Suarez-Abascal]

Hi, I was using the @Lakitna example to implement my own retry mechanism to assert three different modals on our app. I'm using the verifyUpcomingAssertion method, but it seems it retries right away & I can't make it to wait until the assertions fail or pass. This is my code:

Cypress.Commands.add(
  'assertModal',
  ({ selector = '[data-test=popup-container]', headerTitle, shouldClose = true } = {}) => {
    const assertADSModal = () => {
      cy.get('[data-test=acmodal]')
      // a bunch of assertions here
      .........
    };

    const assertDefaultModal = () => {
      cy.get(selector)
       // a bunch of assertions here
      .........
    };

    const assertLegacyModal = () => {
      cy.get('.ui-dialog')
      // a bunch of assertions here
      ......... 
    };

    const triggerModalAssertions = () => {
      cy.get('body').then($body => {
        const isAnyDefaultModalVisible = $body.find(selector).length > 0;
        const isAnyAdsModalVisible = $body.find('[data-test=acmodal]').length > 0;
        const isAnyLegacyModalVisible = $body.find('.ui-dialog').length > 0;

        if (isAnyAdsModalVisible) {
          assertADSModal();
        }
        if (isAnyDefaultModalVisible) {
          assertDefaultModal();
        }
        if (isAnyLegacyModalVisible || selector === '.ui-dialog') {
          assertLegacyModal();
        }

        return cy.verifyUpcomingAssertions(
          $body,
          {},
          {
            onRetry: triggerModalAssertions,
          }
        );
      });
    };

    return triggerModalAssertions();
  }
);

How can I make the verifyUpcomingAssertions to wait until the assertion pass or fail until retries? I have tried with setTimeout & specify the amount of time, but it didn't work.

[pathurs]

For this problem, I ended up creating my own implementation of "retry-ability" (in TypeScript):

interface RetryOptions<T> extends Cypress.Timeoutable {
    interval: number;
    default?: T;
    throw: boolean;
}

export function retry<T>(checker: () => T, confirmer: (result: T) => boolean, options?: Partial<RetryOptions<T>>): Cypress.Chainable<T>;
export function retry<T>(checker: () => T, confirmer: (result: T) => boolean, originalOptions?: Partial<RetryOptions<T>>): Cypress.Chainable<T> {
    const options: RetryOptions<T> = {
        ...{ timeout: Cypress.config('defaultCommandTimeout'), interval: 200, throw: true },
        ...originalOptions
    };

    return cy.wrap(
        new Promise<T>((resolve, reject) => {
            const startTime = Date.now();
            const result = checker();

            if (confirmer(result)) {
                resolve(result);

                return;
            }

            const intervalId = setInterval(() => {
                const currentTime = Date.now();
                const endTime = startTime + options.timeout;

                if (currentTime >= endTime) {
                    if (options.throw) {
                        reject(new Error(`Timed out while retrying after ${options.timeout}ms`));
                    } else if ('default' in options) {
                        resolve(options.default);
                    } else {
                        resolve();
                    }

                    clearInterval(intervalId);

                    return;
                }

                const result = checker();

                if (confirmer(result)) {
                    resolve(result);

                    clearInterval(intervalId);

                    return;
                }
            }, options.interval);
        }),
        { log: false, timeout: options.timeout + options.interval }
    );
}

Usage Example:

retry(
    () => {
        const selector = 'your selector';

        return Cypress.$(selector);
    },
    elements => elements.length > 0,
    { throw: false, default: Cypress.$() }
);

[yktoo]

Here's my naive implementation (Typescript, but can easily be converted to Javascript):

const retryableResolver = (getter: (...x: any) => any, ...args) => {
    const end = Date.now() + config('defaultCommandTimeout');
    const resolve = () => Cypress.Promise
        .try(() => getter(...args))
        .then(value => Date.now() > end ? value : cy.verifyUpcomingAssertions(value, {}, {onRetry: resolve}));
    return resolve();
};

And then use it when adding a command, here's an example that collects texts of passed subject elements and returns them as a string array:

const texts = elements => elements.map((i, e) => $(e).text()).get();
Cypress.Commands.add(
    'texts',
    {prevSubject: 'element'},
    elements => retryableResolver(texts, elements));

[DanaGoyette]

Does anyone have a nice way to effectively replace cy.get with simpler logging, while keeping the element highlighting and all that? I want something like the following, but I want the logging behavior to be synchronous the same way as the real cy.get:

export function getByTestId<E extends HTMLElement>(
  testId: string,
  options?: Partial<Cypress.Loggable & Cypress.Timeoutable & Cypress.Withinable & Cypress.Shadow>
): Cypress.Chainable<JQuery<E>> {
  const props = {
    Subject: null as JQuery<E>,
    Yielded: null as unknown,
  }
  const logConf: Partial<Cypress.LogConfig> = {
    $el: undefined,
    name: 'getByTestId',
    displayName: 'get(testid)',
    message: testId,
    consoleProps: () => props,
  }
  const log = Cypress.log(logConf)
  return cy.get(`[data-testid=${testId}]`, {log: false})

All my attempts to retry either end up with an infinite loop, or failing due to element not existing when it's expected that the element doesn't exist.

[maxkoretskyi]

This could work, retrying for 5 times within the interval of 1s

Cypress.Commands.add('custom', () => {
  let index = 0;
  const promise = new Promise((resolve) => {
    const interval = setInterval(() => {
      if (index++ > 5) {
        clearInterval(interval);
        resolve(1);
      } else {
        console.log('retry attempt ' + index);
      }
    }, 1000);
  });

  return cy.wrap(promise, {timeout: 10000});
});

[egucciar]

I am also unable to get the above solutions to work consistently. I have the following assertion:

    cy.retryGet('[data-cy="form-back"]').safeClick();

And the following code:

/**
 * allow for promise to verify upcoming assertions to resolve
 * this differs from the above commands by running the future assertions
 * and retrying until those assertions also pass
 *
 * Inspired by https://github.com/cypress-io/cypress/issues/3109#issuecomment-477037752
 */
Cypress.Commands.add(
  'resolve',
  { prevSubject: 'optional' },
  (subject, fn, opts = {}) => {
    const resolve = () => {
      fn(subject).then((res) =>
        cy.verifyUpcomingAssertions(res, opts, { onRetry: resolve })
      );
    };
    return resolve();
  }
);

/**
 * get selector with retryability on the selector
 * check that element exists and is not undefiend
 */
Cypress.Commands.add('retryGet', (selector) => {
  const fn = () =>
    cy.wrap(
      new Cypress.Promise((resolve) => {
        cy.get(selector).then(resolve);
      })
    );

  // When retryGet is used as the first command in a test,
  // There can be a race condition where cypress see no commands
  // due to the leading promsie. Fix by forcing one command
  // with no interactions just to register it into cypress
  cy.get(selector);

  return cy
    .resolve(fn)
    .should((el) => expect(Cypress.dom.isAttached(el)).to.eq(true))
    .should((el) => expect(el === undefined).to.eq(false));
});

// Retryable click selector to be used in tandem with retryGet
// https://github.com/testing-library/cypress-testing-library/issues/153#issuecomment-897867503
Cypress.Commands.add('safeClick', { prevSubject: 'element' }, ($element) => {
  const click = ($el) => $el.click();
  return cy
    .wrap($element)
    .pipe(click)
    .should(($el) => expect(Cypress.dom.isDetached($el)).to.be.false);
});

@jennifer-shehane, @Lakitna , @bohendo , is there any reason why the above code would fail like so?

I would expect since that final assertion failed, it should retry the whole chain, but it appears not to do that.

Another shot of the command log:

I'm hoping to get a solution without intervals or timeout using Cypress methods. Also, the only way our team has been avoiding this issue has been by adding cy.wait(500) before interacting with elements we presume to be rerendering based on flakey tests.

[BlueWinds]

Hello! I just wanted to let you all know that we're adding this in Cypress 12.0.0. It turns out that "ability to retry commands" was pretty central to resolving #7306 , and as part of that effort, we're exposing the addQuery interface publicly.

This will be going out in a couple of weeks with Cypress 12, but if you want a bit of a preview, here's the PR we have open with cypress-testing-library to update them to use the new API: https://github.com/testing-library/cypress-testing-library/pull/238/files

As an easier way to get started, here's a preview of the API docs: https://deploy-preview-4835--cypress-docs.netlify.app/api/cypress-api/custom-queries, and the re-written guide on retry-ability that discusses queries vs. other commands: https://deploy-preview-4835--cypress-docs.netlify.app/guides/core-concepts/retry-ability#Commands-Queries-and-Assertions

The docs are still in review, but I'd welcome any comments or questions on them if people want to read it / try out the pre-release builds of Cypress 12 (latest as of now: b9d053e#comments).

[BlueWinds]

Cypress 12.0.0 is going out today, which should address these needs. If there's still anything lacking once you take a look at Cypress.Commands.addQuery, please feel free to open a new issue and we can start some new discussions and look at improvements!

[mjhenkes]

This was release in 12.0.0