Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleanup and simplify test runner logic, remove RunTests.cmd #1628

Merged
merged 14 commits into from
Oct 19, 2016
Merged

cleanup and simplify test runner logic, remove RunTests.cmd #1628

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
f649cfc
remove RunTests.cmd
dsyme Oct 17, 2016
60964c0
more cleanup
dsyme Oct 17, 2016
363f843
allow tags in build.cmd
dsyme Oct 18, 2016
cdaec29
Merge branch 'cleanup-9' into cleanup-10
dsyme Oct 18, 2016
fa44188
cleanup and simplify tests
dsyme Oct 18, 2016
8d60372
test fixes
dsyme Oct 18, 2016
540136d
disable long running test
dsyme Oct 18, 2016
c0ba4af
Merge branch 'cleanup-9' of https://github.com/dsyme/visualfsharp int…
dsyme Oct 18, 2016
db4c230
disable expensive tests that weren't previously being run
dsyme Oct 18, 2016
25d40c6
Merge branch 'cleanup-9' of https://github.com/dsyme/visualfsharp int…
dsyme Oct 18, 2016
a6a01f4
more cleanup
dsyme Oct 18, 2016
3ea88f5
more cleanup
dsyme Oct 18, 2016
62c1cba
integrate
dsyme Oct 19, 2016
4376ff1
integrate
dsyme Oct 19, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions DEVGUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ To learn what F# is and why it's interesting, go to [fsharp.org](http://fsharp.o

The primary technical documents for the F# compiler code are

* [The F# Language and Core Library RFC Process](http://fsharp.github.io/2016/09/26/fsharp-rfc-process.html)

* [The F# Language Specification](http://fsharp.org/specs/language-spec/)

* [The F# Compiler Technical Guide](http://fsharp.github.io/2015/09/29/fsharp-compiler-guide.html)
Expand Down Expand Up @@ -34,7 +36,7 @@ There are various qualifiers:
build.cmd release -- build release (the default)
build.cmd debug -- build debug instead of release

build.cmd net40 -- build .NET Framework compiler
build.cmd net40 -- build .NET Framework compiler (the default)
build.cmd coreclr -- build .NET Core compiler
build.cmd vs -- build the Visual F# IDE Tools
build.cmd pcls -- build the PCL FSharp.Core libraries
Expand All @@ -49,11 +51,8 @@ There are various qualifiers:
build.cmd all test -- build all, run all tests

build.cmd test-smoke -- build, run smoke tests
build.cmd test-coreunit -- build, run FSharp.Core tests
build.cmd test-coreclr -- build, run .NET Core tests
build.cmd test-pcls -- build, run PCL tests
build.cmd test-fsharp -- build, run tests\fsharp suite
build.cmd test-fsharpqa -- build, run tests\fsharpqa suite
build.cmd test-net40-fsharp -- build, run tests\fsharp suite for .NET Framework
build.cmd test-net40-fsharpqa -- build, run tests\fsharpqa suite for .NET Framework

**Notes**
To build and test Visual F# IDE Tools, you must use [Visual Studio "vNext" (aka "Dev15")](https://www.visualstudio.com/en-us/downloads/visual-studio-next-downloads-vs.aspx). This is the one after Visual Studio 2015 (aka "Dev 14"). You must also install Visual Studio SDK (also called _Visual Studio Extensibility SDK_ on the Visual Studio installer) before building Visual F# IDE Tools.
Expand Down
8 changes: 4 additions & 4 deletions TESTGUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ For the FSharp and FSharpQA suites, the list of test areas and their associated
tests\test.lst // FSharp suite
tests\fsharpqa\source\test.lst // FSharpQA suite

Tags are in the left column, paths to to corresponding test folders are in the right column. If no tags are specified to `RunTests.cmd`, all tests will be run.
Tags are in the left column, paths to to corresponding test folders are in the right column. If no tags are specifie, all tests will be run.

If you want to re-run a particular test area, the easiest way to do so is to set a temporary tag for that area in test.lst (e.g. "RERUN"), then call `RunTests.cmd <debug|release> <fsharp|fsharpqa> RERUN`.
If you want to re-run a particular test area, the easiest way to do so is to set a temporary tag for that area in test.lst (e.g. "RERUN").

### Logs and output

Expand All @@ -83,5 +83,5 @@ All test execution logs and result files will be dropped into the `tests\TestRes

* Making the tests run faster
* NGen-ing the F# bits (fsc, fsi, FSharp.Core, etc) will result in tests executing much faster. Make sure you run `src\update.cmd` with the `-ngen` flag before running tests.
* The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line in `RunTests.cmd` (look for `PARALLEL_ARG`) to disable this.
* By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line in `RunTests.cmd` (look for `HOSTED_COMPILER`).
* The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line (look for `PARALLEL_ARG`) to disable this.
* By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line (look for `HOSTED_COMPILER`).
97 changes: 79 additions & 18 deletions VisualFSharp.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 15
VisualStudioVersion = 15.0.25618.0
# Visual Studio 14
VisualStudioVersion = 14.0.25420.1
MinimumVisualStudioVersion = 10.0.40219.1
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharp.Compiler", "src\fsharp\FSharp.Compiler\FSharp.Compiler.fsproj", "{2E4D67B4-522D-4CF7-97E4-BA940F0B18F3}"
EndProject
Expand Down Expand Up @@ -131,6 +131,16 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VisualFSharpWeb", "vsintegr
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VisualFSharpOpenSource", "vsintegration\Vsix\VisualFSharpOpenSource\VisualFSharpOpenSource.csproj", "{E6A45CDF-B408-420F-B475-74611BEFC52B}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharp.Compiler.Hosted", "tests\fsharpqa\testenv\src\FSharp.Compiler.Hosted\FSharp.Compiler.Hosted.fsproj", "{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "HostedCompilerServer", "tests\fsharpqa\testenv\src\HostedCompilerServer\HostedCompilerServer.fsproj", "{4239EFEA-E746-446A-BF7A-51FCBAB13946}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "diff", "tests\fsharpqa\testenv\src\diff\diff.fsproj", "{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "ILComparer", "tests\fsharpqa\testenv\src\ILComparer\ILComparer.fsproj", "{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}"
EndProject
Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharpQA.Tests", "tests\fsharpqa\Source\FSharpQA.Tests.fsproj", "{21958DEB-133B-4760-A238-1293ED9687EC}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -696,10 +706,8 @@ Global
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Debug|x86.ActiveCfg = Debug|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Debug|x86.Build.0 = Debug|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|Any CPU.Build.0 = Proto|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|x86.ActiveCfg = Proto|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|x86.Build.0 = Proto|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|x86.ActiveCfg = Release|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Release|Any CPU.Build.0 = Release|Any CPU
{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Release|x86.ActiveCfg = Release|Any CPU
Expand All @@ -708,10 +716,8 @@ Global
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Debug|x86.ActiveCfg = Debug|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Debug|x86.Build.0 = Debug|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|Any CPU.Build.0 = Proto|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|x86.ActiveCfg = Proto|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|x86.Build.0 = Proto|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|x86.ActiveCfg = Release|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Release|Any CPU.Build.0 = Release|Any CPU
{59ADCE46-9740-4079-834D-9A03A3494EBC}.Release|x86.ActiveCfg = Release|Any CPU
Expand All @@ -720,10 +726,8 @@ Global
{58730C8B-16F5-4956-9291-BB68E17C9142}.Debug|Any CPU.Build.0 = Debug|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Debug|x86.ActiveCfg = Debug|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Debug|x86.Build.0 = Debug|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|Any CPU.Build.0 = Proto|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|x86.ActiveCfg = Proto|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|x86.Build.0 = Proto|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|x86.ActiveCfg = Release|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Release|Any CPU.ActiveCfg = Release|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Release|Any CPU.Build.0 = Release|Any CPU
{58730C8B-16F5-4956-9291-BB68E17C9142}.Release|x86.ActiveCfg = Release|Any CPU
Expand All @@ -732,14 +736,66 @@ Global
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Debug|x86.ActiveCfg = Debug|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Debug|x86.Build.0 = Debug|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|Any CPU.Build.0 = Proto|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|x86.ActiveCfg = Proto|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|x86.Build.0 = Proto|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|x86.ActiveCfg = Release|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|Any CPU.Build.0 = Release|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|x86.ActiveCfg = Release|Any CPU
{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|x86.Build.0 = Release|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|Any CPU.Build.0 = Debug|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|x86.ActiveCfg = Debug|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|x86.Build.0 = Debug|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|Any CPU.Build.0 = Proto|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|x86.ActiveCfg = Proto|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|x86.Build.0 = Proto|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|Any CPU.ActiveCfg = Release|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|Any CPU.Build.0 = Release|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|x86.ActiveCfg = Release|Any CPU
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|x86.Build.0 = Release|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|Any CPU.Build.0 = Debug|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|x86.ActiveCfg = Debug|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|x86.Build.0 = Debug|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Proto|x86.ActiveCfg = Release|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|Any CPU.ActiveCfg = Release|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|Any CPU.Build.0 = Release|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|x86.ActiveCfg = Release|Any CPU
{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|x86.Build.0 = Release|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|Any CPU.Build.0 = Debug|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|x86.ActiveCfg = Debug|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|x86.Build.0 = Debug|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Proto|x86.ActiveCfg = Release|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|Any CPU.ActiveCfg = Release|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|Any CPU.Build.0 = Release|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|x86.ActiveCfg = Release|Any CPU
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|x86.Build.0 = Release|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|x86.ActiveCfg = Debug|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|x86.Build.0 = Debug|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Proto|Any CPU.ActiveCfg = Release|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Proto|x86.ActiveCfg = Release|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|Any CPU.ActiveCfg = Release|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|Any CPU.Build.0 = Release|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|x86.ActiveCfg = Release|Any CPU
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|x86.Build.0 = Release|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|x86.ActiveCfg = Debug|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|x86.Build.0 = Debug|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|Any CPU.Build.0 = Proto|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|x86.ActiveCfg = Proto|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|x86.Build.0 = Proto|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Release|Any CPU.Build.0 = Release|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Release|x86.ActiveCfg = Release|Any CPU
{21958DEB-133B-4760-A238-1293ED9687EC}.Release|x86.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -800,5 +856,10 @@ Global
{59ADCE46-9740-4079-834D-9A03A3494EBC} = {141F6C23-E1B1-4D89-9F10-F0B8AD58E71F}
{58730C8B-16F5-4956-9291-BB68E17C9142} = {141F6C23-E1B1-4D89-9F10-F0B8AD58E71F}
{E6A45CDF-B408-420F-B475-74611BEFC52B} = {141F6C23-E1B1-4D89-9F10-F0B8AD58E71F}
{4FAE092E-5366-43F5-85FA-71EAF66FA4A8} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
{4239EFEA-E746-446A-BF7A-51FCBAB13946} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
{BF5C6D92-D053-4216-9C42-B62F5F5C5E91} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
{2E60864A-E3FF-4BCC-810F-DC7C34E6B236} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
{21958DEB-133B-4760-A238-1293ED9687EC} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
EndGlobalSection
EndGlobal
Loading