Discovery right before run, on file by file basis

[nohwnd]

I am thinking about moving discovery and run closer together: Instead of Discovery, Discovery, Discovery, Run, Run, Run, it would do Discovery, Run, Discovery, Run, Discovery, Run, on file by file basis.

This should help with global side effects:

• Especially for InModuleScope, which now has some issues that are very hard to debug when you force import module in the top of your file. Which is a great strategy for having files that run interactively.
• Functions and variables from top level BeforeDiscovery (or naked statement) could be shared with tests, avoiding the need for double definitions.
• TestDrive could optionally initialize before discovery, and we would not have to keep 100 directories (or however many test files you have) in your temp, to finish discovery and then do run.
• Test results come back sooner, becuase you only need to do 1 discovery before you run something, instead of all.
• If we add native parallelization then it can start as soon as we know all the files that will be run.

What we lose:

• We won't be able to re-introduce the -Focus functionality that runs only focused tests when there is at least 1 focused test, no matter how many files are provided. This is a nifty little feature, but can be largerly replaced by adding "breakpoint" like functionality for tests in VSCode extension, or by code lens that we already have - both of those features would use the line filter that is built into Pester. Using code edits to narrow down the run is also inferior to a GUI approach, because a gui approach is impossible to accidentally commit into codebase, which end up running just a single test in CI. Which is why I hiddent that -Focus functionality from Pester5 release.
• We will be only able to split tests for potential parallelization on file by file basis. Which is imho the most logical granularity anyway, and is relatively easy to workaround for the user.
• The output that summarizes how many tests we discovered and how many we are going to run will be hidden to avoid printing it out in between files.

[nohwnd]

I don't know if this will even be technically feasible, but it seems like it would make global side effects easier to grasp for people who need them in order to generate tests. I will keep digging into this to see if that change is warranted.

[fflaten]

Something to explore for v6?

[nohwnd]

#1885 (comment) This describes the module issue in more detail.

[nohwnd]

This describes the need for global setup: #1645 (comment)

[fflaten]

Global testdrive would be a risk for future parallelization though.

[nohwnd]

Yeah, I think there is more info in that thread than I meant to share. What I am thinking about is that discovery that will happen right before run will limit the global side effects to mostly make sense. Like you import-module on the top of the file, and the module is still the same module in your Run. In that case we can also move figuring out TestDrive directory before discovery, and optionally make it actually setup the folder before discovery. I am not sure if TestDrive setting up before discovery warrants this change. But the other issue with modules is really hard to debug.

The second point is that I am occasionally missing a global setup, or maybe rather a "beforeEachFile", for which I don't have a good solution now, because I miss it for interactive scenarios, where I want to have each file to act like self-contained unit, instead of relying on some run.ps1 to setup the state for me.

[fflaten]

Sounds interesting, a middle ground between Pester 4 and 5. Feels more streamlined for any future parallelization support.

Run would feel slower though so we'd have to include discovery time in the container output to explain the repeating pauses.

Never used to Focus-mode so maybe I'm missing something. Why would this block Focus? Wasn't that always a switch parameter put on tests, which means at least parts of discovery had to run on all containers to look for it? Wouldn't a tag be just as quick?

[nohwnd]

Never used to Focus-mode so maybe I'm missing something. Why would this block Focus?

Say you give it test1.tests.ps1, and test2.tests.ps1. You put -focus on a test in test2.tests.ps1. test1.tests.ps1 should be discovered but none of its tests should run. Only the focused test should run. Putting a tag e.g. "focus" on a test, and then doing Inovke-Pester -Tag "focus" is very similar. But it requires 2 changes to your codebase.

It is a minor feature, that is easily replaced by other means.

[nohwnd]

Run would feel slower though so we'd have to include discovery time in the container output to explain the repeating pauses.

That is maybe true, a file that is not well behaved can take few seconds to discover. But the same would be true for the discovery happening at the start, and is probably even worse because there all the "delays" add up into one big pause.

Normally files are discovered in couple of milliseconds, which I am not sure if the user would even notice.

[fflaten]

Yeah, I was thinking about data-driven tests with external lookups etc. Still agree it's better though as long as there's some output detailing why it suddenly paused for >1sec etc mid-run. 👍