💡 Summary

Some functional tests fail when a pre-condition to the test requires setting a tenant configuration value and the change is not reflected in time for the test. This results in a failed functional test that is the result of a false positive and wastes developer time verifying the results manually. The goal of this enhancement is to build a utility function that can check that a specific value has been set and wait until the change is reflected in API calls. The function should not block indefinitely, but should have a configurable timeout parameter after which it retries setting the value.
This "Set and Check" function should also be able to retry setting the tenant value after a configurable timeout and then restart its checking. Recommend defaulting the number of settings before failure to 3.
So, if you set the timeout to 10 seconds and the number of retries to 3, then the function would return with a failure after 30 seconds if the value did not change

Motivation and context

Using this function could prevent intermittent functional test failures from marking tests as failed incorrectly and provide a more stable signal of code quality back to the developers.

Implementation notes

Please provide details for implementation, such as:

• The function should not be a core part of ScubaGear and may best be added to FunctionalTestUtils.ps1.
• The function could take two ScriptBlocks, one to set the value and another to check it. try/catch should be used to deal with invalid script blocks and produce reasonable errors and stacktraces as feedback to users when functional tests are run.
• The result from the function should be a boolean indicating test success or failure after all retries have been exhausted
• Both the retry and timeout parameters should be optional with defined defaullts (recommend timeout=30 seconds, retry = 3)

Acceptance criteria

How do we know when this work is done?

• New function is available in scope of functional testplans and can be called successfully as part of a pre or post condition
• The new function has been applied to several of the Defender and SharePoint tests that most frequently fail intermittently as defined in Defender functional tests fail unexpectedly in test tenants #1343 and SharePoint functional test plan for the spo variant is failing/erroring for multiple policies #1093