Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve functional testing by adding condition validation support function #1294

Closed
4 tasks
schrolla opened this issue Aug 29, 2024 · 1 comment
Closed
4 tasks
Labels
enhancement This issue or pull request will add new or improve existing functionality Testing This issue or task involves testing the automation tool function
Milestone

Comments

@schrolla
Copy link
Collaborator

💡 Summary

Functional testing often requires changing tenant configuration settings in order to validate certain use cases. Some changes, however, may be executed asynchronously or the new value is reflected with delay causing a test to intermittently fail depending on the load on the API service at any given time.

The idea is to add a support helper method that would just loop given a command/expression until the expression returned true or timed out given a timeout period. That way, you could set a pre-condition and have it wait until that condition became true or a pre-defined time period elapsed.

Motivation and context

Having an easy to reference and use pre-defined function to validate that a config item shows as changed or matches a given state prior to running a test will be convenient for developers to build robust tests that return predictable results.

Implementation notes

Please provide details for implementation, such as:

  • Create a new PowerShell function that accepts an expression or function to execute and a timeout value
  • Make sure that the timeout parameter defaults to a reasonable time frame to prevent test run delays
  • Timeout period should also be bounded to prevent delays in execution, if timeout period is exceeded, fail the continue test but issue warning
  • Convert several candidate functional tests (existing or new) to use the new function
  • Run existing functional tests to validate they work as intended, including running as part of existing testing workflows

Acceptance criteria

How do we know when this work is done?

  • Successfully defined new support function that delays test until an expression or function returns true or timeout expires
  • Tests to run new functional tests using support function run successfully
  • New function is documented via PowerShell docstring to help developers understand usage
  • Functional testing README updated with details and examples of how to use the support function when building tests
@schrolla schrolla added enhancement This issue or pull request will add new or improve existing functionality Testing This issue or task involves testing the automation tool function labels Aug 29, 2024
@schrolla schrolla added this to the Kraken milestone Sep 16, 2024
@schrolla
Copy link
Collaborator Author

This is a duplicate of #1364. Closing as duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This issue or pull request will add new or improve existing functionality Testing This issue or task involves testing the automation tool function
Projects
None yet
Development

No branches or pull requests

3 participants
@rgbrow1949 @schrolla and others