cleanup and simplify test runner logic, remove RunTests.cmd (#1628)

* remove RunTests.cmd * more cleanup * allow tags in build.cmd * cleanup and simplify tests * test fixes * disable long running test * disable expensive tests that weren't previously being run * more cleanup * more cleanup * integrate
dotnet · Oct 19, 2016 · 07fd501 · 07fd501
1 parent 21c95d2
commit 07fd501
Show file tree

Hide file tree

Showing 61 changed files with 3,897 additions and 9,859 deletions.
diff --git a/DEVGUIDE.md b/DEVGUIDE.md
@@ -7,6 +7,8 @@ To learn what F# is and why it's interesting, go to [fsharp.org](http://fsharp.o
 
 The primary technical documents for the F# compiler code are
 
+* [The F# Language and Core Library RFC Process](http://fsharp.github.io/2016/09/26/fsharp-rfc-process.html)
+
 * [The F# Language Specification](http://fsharp.org/specs/language-spec/)
 
 * [The F# Compiler Technical Guide](http://fsharp.github.io/2015/09/29/fsharp-compiler-guide.html) 
@@ -34,7 +36,7 @@ There are various qualifiers:
     build.cmd release         -- build release (the default)
     build.cmd debug           -- build debug instead of release
 
-    build.cmd net40           -- build .NET Framework compiler 
+    build.cmd net40           -- build .NET Framework compiler (the default)
     build.cmd coreclr         -- build .NET Core compiler 
     build.cmd vs              -- build the Visual F# IDE Tools
     build.cmd pcls            -- build the PCL FSharp.Core libraries
@@ -49,11 +51,8 @@ There are various qualifiers:
     build.cmd all test        -- build all, run all tests
 
     build.cmd test-smoke      -- build, run smoke tests
-    build.cmd test-coreunit   -- build, run FSharp.Core tests
-    build.cmd test-coreclr    -- build, run .NET Core tests
-    build.cmd test-pcls       -- build, run PCL tests
-    build.cmd test-fsharp     -- build, run tests\fsharp suite
-    build.cmd test-fsharpqa   -- build, run tests\fsharpqa suite
+    build.cmd test-net40-fsharp     -- build, run tests\fsharp suite for .NET Framework
+    build.cmd test-net40-fsharpqa   -- build, run tests\fsharpqa suite for .NET Framework
 
 **Notes**
 To build and test Visual F# IDE Tools, you must use [Visual Studio "vNext" (aka "Dev15")](https://www.visualstudio.com/en-us/downloads/visual-studio-next-downloads-vs.aspx). This is the one after Visual Studio 2015 (aka "Dev 14").  You must also install Visual Studio SDK (also called _Visual Studio Extensibility SDK_ on the Visual Studio installer) before building Visual F# IDE Tools.

diff --git a/TESTGUIDE.md b/TESTGUIDE.md
@@ -63,9 +63,9 @@ For the FSharp and FSharpQA suites, the list of test areas and their associated
     tests\test.lst                   // FSharp suite
     tests\fsharpqa\source\test.lst   // FSharpQA suite
 
-Tags are in the left column, paths to to corresponding test folders are in the right column.  If no tags are specified to `RunTests.cmd`, all tests will be run.
+Tags are in the left column, paths to to corresponding test folders are in the right column.  If no tags are specifie, all tests will be run.
 
-If you want to re-run a particular test area, the easiest way to do so is to set a temporary tag for that area in test.lst (e.g. "RERUN"), then call `RunTests.cmd <debug|release> <fsharp|fsharpqa> RERUN`.
+If you want to re-run a particular test area, the easiest way to do so is to set a temporary tag for that area in test.lst (e.g. "RERUN").
 
 ### Logs and output
 
@@ -83,5 +83,5 @@ All test execution logs and result files will be dropped into the `tests\TestRes
 
 * Making the tests run faster
   * NGen-ing the F# bits (fsc, fsi, FSharp.Core, etc) will result in tests executing much faster. Make sure you run `src\update.cmd` with the `-ngen` flag before running tests.
-  * The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line in `RunTests.cmd` (look for `PARALLEL_ARG`) to disable this.
-  * By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line in `RunTests.cmd` (look for `HOSTED_COMPILER`).
+  * The FSharp and FSharpQA suites will run test cases in parallel by default. You can comment out the relevant line (look for `PARALLEL_ARG`) to disable this.
+  * By default, tests from the FSharpQA suite are run using a persistent, hosted version of the compiler. This speeds up test execution, as there is no need for the `fsc.exe` process to spin up repeatedly. To disable this, uncomment the relevant line (look for `HOSTED_COMPILER`).
diff --git a/VisualFSharp.sln b/VisualFSharp.sln
@@ -1,7 +1,7 @@
 
 Microsoft Visual Studio Solution File, Format Version 12.00
-# Visual Studio 15
-VisualStudioVersion = 15.0.25618.0
+# Visual Studio 14
+VisualStudioVersion = 14.0.25420.1
 MinimumVisualStudioVersion = 10.0.40219.1
 Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharp.Compiler", "src\fsharp\FSharp.Compiler\FSharp.Compiler.fsproj", "{2E4D67B4-522D-4CF7-97E4-BA940F0B18F3}"
 EndProject
@@ -131,6 +131,16 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VisualFSharpWeb", "vsintegr
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "VisualFSharpOpenSource", "vsintegration\Vsix\VisualFSharpOpenSource\VisualFSharpOpenSource.csproj", "{E6A45CDF-B408-420F-B475-74611BEFC52B}"
 EndProject
+Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharp.Compiler.Hosted", "tests\fsharpqa\testenv\src\FSharp.Compiler.Hosted\FSharp.Compiler.Hosted.fsproj", "{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}"
+EndProject
+Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "HostedCompilerServer", "tests\fsharpqa\testenv\src\HostedCompilerServer\HostedCompilerServer.fsproj", "{4239EFEA-E746-446A-BF7A-51FCBAB13946}"
+EndProject
+Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "diff", "tests\fsharpqa\testenv\src\diff\diff.fsproj", "{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}"
+EndProject
+Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "ILComparer", "tests\fsharpqa\testenv\src\ILComparer\ILComparer.fsproj", "{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}"
+EndProject
+Project("{F2A71F9B-5D33-465A-A702-920D77279786}") = "FSharpQA.Tests", "tests\fsharpqa\Source\FSharpQA.Tests.fsproj", "{21958DEB-133B-4760-A238-1293ED9687EC}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -696,10 +706,8 @@ Global
 		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Debug|x86.ActiveCfg = Debug|Any CPU
 		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Debug|x86.Build.0 = Debug|Any CPU
-		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
-		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|Any CPU.Build.0 = Proto|Any CPU
-		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|x86.ActiveCfg = Proto|Any CPU
-		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|x86.Build.0 = Proto|Any CPU
+		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Proto|x86.ActiveCfg = Release|Any CPU
 		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Release|Any CPU.Build.0 = Release|Any CPU
 		{E7EC4A3E-9D57-45B8-83F5-EDDFD6CEF090}.Release|x86.ActiveCfg = Release|Any CPU
@@ -708,10 +716,8 @@ Global
 		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Debug|x86.ActiveCfg = Debug|Any CPU
 		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Debug|x86.Build.0 = Debug|Any CPU
-		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
-		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|Any CPU.Build.0 = Proto|Any CPU
-		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|x86.ActiveCfg = Proto|Any CPU
-		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|x86.Build.0 = Proto|Any CPU
+		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Proto|x86.ActiveCfg = Release|Any CPU
 		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Release|Any CPU.Build.0 = Release|Any CPU
 		{59ADCE46-9740-4079-834D-9A03A3494EBC}.Release|x86.ActiveCfg = Release|Any CPU
@@ -720,10 +726,8 @@ Global
 		{58730C8B-16F5-4956-9291-BB68E17C9142}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{58730C8B-16F5-4956-9291-BB68E17C9142}.Debug|x86.ActiveCfg = Debug|Any CPU
 		{58730C8B-16F5-4956-9291-BB68E17C9142}.Debug|x86.Build.0 = Debug|Any CPU
-		{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
-		{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|Any CPU.Build.0 = Proto|Any CPU
-		{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|x86.ActiveCfg = Proto|Any CPU
-		{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|x86.Build.0 = Proto|Any CPU
+		{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{58730C8B-16F5-4956-9291-BB68E17C9142}.Proto|x86.ActiveCfg = Release|Any CPU
 		{58730C8B-16F5-4956-9291-BB68E17C9142}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{58730C8B-16F5-4956-9291-BB68E17C9142}.Release|Any CPU.Build.0 = Release|Any CPU
 		{58730C8B-16F5-4956-9291-BB68E17C9142}.Release|x86.ActiveCfg = Release|Any CPU
@@ -732,14 +736,66 @@ Global
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Debug|x86.ActiveCfg = Debug|Any CPU
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Debug|x86.Build.0 = Debug|Any CPU
-		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
-		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|Any CPU.Build.0 = Proto|Any CPU
-		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|x86.ActiveCfg = Proto|Any CPU
-		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|x86.Build.0 = Proto|Any CPU
+		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Proto|x86.ActiveCfg = Release|Any CPU
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|Any CPU.Build.0 = Release|Any CPU
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|x86.ActiveCfg = Release|Any CPU
 		{E6A45CDF-B408-420F-B475-74611BEFC52B}.Release|x86.Build.0 = Release|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Debug|x86.Build.0 = Debug|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|Any CPU.Build.0 = Proto|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|x86.ActiveCfg = Proto|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Proto|x86.Build.0 = Proto|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|Any CPU.Build.0 = Release|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|x86.ActiveCfg = Release|Any CPU
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8}.Release|x86.Build.0 = Release|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Debug|x86.Build.0 = Debug|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Proto|x86.ActiveCfg = Release|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|Any CPU.Build.0 = Release|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|x86.ActiveCfg = Release|Any CPU
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946}.Release|x86.Build.0 = Release|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Debug|x86.Build.0 = Debug|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Proto|x86.ActiveCfg = Release|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|Any CPU.Build.0 = Release|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|x86.ActiveCfg = Release|Any CPU
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91}.Release|x86.Build.0 = Release|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Debug|x86.Build.0 = Debug|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Proto|Any CPU.ActiveCfg = Release|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Proto|x86.ActiveCfg = Release|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|Any CPU.Build.0 = Release|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|x86.ActiveCfg = Release|Any CPU
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236}.Release|x86.Build.0 = Release|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Debug|x86.Build.0 = Debug|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|Any CPU.ActiveCfg = Proto|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|Any CPU.Build.0 = Proto|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|x86.ActiveCfg = Proto|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Proto|x86.Build.0 = Proto|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Release|Any CPU.Build.0 = Release|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Release|x86.ActiveCfg = Release|Any CPU
+		{21958DEB-133B-4760-A238-1293ED9687EC}.Release|x86.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -800,5 +856,10 @@ Global
 		{59ADCE46-9740-4079-834D-9A03A3494EBC} = {141F6C23-E1B1-4D89-9F10-F0B8AD58E71F}
 		{58730C8B-16F5-4956-9291-BB68E17C9142} = {141F6C23-E1B1-4D89-9F10-F0B8AD58E71F}
 		{E6A45CDF-B408-420F-B475-74611BEFC52B} = {141F6C23-E1B1-4D89-9F10-F0B8AD58E71F}
+		{4FAE092E-5366-43F5-85FA-71EAF66FA4A8} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
+		{4239EFEA-E746-446A-BF7A-51FCBAB13946} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
+		{BF5C6D92-D053-4216-9C42-B62F5F5C5E91} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
+		{2E60864A-E3FF-4BCC-810F-DC7C34E6B236} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
+		{21958DEB-133B-4760-A238-1293ED9687EC} = {CFE3259A-2D30-4EB0-80D5-E8B5F3D01449}
 	EndGlobalSection
 EndGlobal