Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update TESTGUIDE.md #1234

Merged
merged 1 commit into from
Jun 2, 2016
Merged

Update TESTGUIDE.md #1234

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions TESTGUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

## Prerequisites

In order to run the FSharpQA auite, you will need to install [Perl](http://www.perl.org/get.html) (ActiveState Perl 5.16.3 is known to work fine).
Perl must be included in the `%PATH%` for the below steps to work. It is also recommended that you run tests from an elevated command prompt, as there are a couple of test cases which require administrative privileges.
In order to run the FSharpQA suite, you will need to install [Perl](http://www.perl.org/get.html) (ActiveState Perl 5.16.3 is known to work fine).
Perl must be included in the `%PATH%` for the below steps to work. It is also recommended that you run tests from an elevated command prompt, as there are a couple of test cases which require administrative privileges.

The Perl requirement is gradually being removed.

Expand All @@ -19,7 +19,7 @@ To run tests, use variations such as the following, depending on which test suit
build.cmd debug,ci
build.cmd debug,all

Default is ``ci`
Default is `ci`

* ``ci`` = the build and tests done by continuous integration
* ``compiler`` = build the compiler
Expand All @@ -33,7 +33,7 @@ Default is ``ci`

The F# tests are split as follows:

* [FSharp Suite](tests\fsharp). Older suite with broad coverage of mainline compiler and runtime scenarios.
* [FSharp Suite](tests\fsharp) - Older suite with broad coverage of mainline compiler and runtime scenarios.

* [FSharpQA Suite](tests\fsharpqa\source) - Broad and deep coverage of a variety of compiler, runtime, and syntax scenarios.

Expand All @@ -49,15 +49,15 @@ The F# tests are split as follows:

### FSharp Suite

This is now compiled usinng [tests\fsharp\FSharp.Tests.fsproj] to a unit test DLL which acts as a driver script.
This is now compiled using [tests\fsharp\FSharp.Tests.fsproj] to a unit test DLL which acts as a driver script.

This compiles and execute the `test.fsx` file using some combination of compiler or fsi flags.
This compiles and executes the `test.fsx` file using some combination of compiler or FSI flags.
If the compilation and execution encounter no errors, the test is considered to have passed.

### FSharpQA Suite

These tests require use of the `RunAll.pl` framework to execute.
Test area directories in this suite will contain a number of source code files and a single `env.lst` file. The `env.lst` file defines a series of test cases, one per line.
Test area directories in this suite will contain a number of source code files and a single `env.lst` file. The `env.lst` file defines a series of test cases, one per line.
Test cases will run an optional "pre command," compile some set of source files using some set of flags, optionally run the resulting binary, then optionally run a final "post command."
If all of these steps complete without issue, the test is considered to have passed.

Expand All @@ -71,7 +71,7 @@ using the '--x86' flag to `nunit3-console.exe`

### RunTests.cmd

The script `tests\RunTests.cmd` is used to execute the suites. It's used like this:
The script `tests\RunTests.cmd` is used to execute the suites. It's used like this:

RunTests.cmd <debug|release> fsharp [tags to run] [tags not to run]
RunTests.cmd <debug|release> fsharpqa [tags to run] [tags not to run]
Expand Down Expand Up @@ -103,12 +103,11 @@ From a Powershell environment, make sure the double quotes are passed literally,
`RunTests.cmd` is mostly just a simple wrapper over `tests\fsharpqa\testenv\bin\RunAll.pl`, which has capabilities not discussed here. More advanced test execution scenarios can be achieved by invoking `RunAll.pl` directly.
Run `perl tests\fsharpqa\testenv\bin\RunAll.pl -?` to see a full list of flags and options.


### Other Tips

* Run as admin, or a handful of tests will fail
* Run as Administrator, or a handful of tests will fail

* Making the tests run faster
* NGen-ing the F# bits (fsc, fsi, FSharp.Core, etc) will result in tests executing much faster. Make sure you run `src\update.cmd` with the `-ngen` flag before running tests.
* The Fsharp and FsharpQA suites will run test cases in parallel by default. You can comment out the relevant line in `RunTests.cmd` (look for `PARALLEL_ARG`) to disable this.
* By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line in `RunTests.cmd` (look for `HOSTED_COMPILER`).
* NGen-ing the F# bits (fsc, fsi, FSharp.Core, etc) will result in tests executing much faster. Make sure you run `src\update.cmd` with the `-ngen` flag before running tests.
* The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line in `RunTests.cmd` (look for `PARALLEL_ARG`) to disable this.
* By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line in `RunTests.cmd` (look for `HOSTED_COMPILER`).