Failing tests on master #915

latkin · 2016-01-29T23:36:11Z

I see the following failures on master:

fsc usage baseline files have not been updated with new response file flag

+++ CompilerOptions\fsc\help (NetFx40 -?) +++
PRECMD: [fsc >help40.txt -?     2>&1]
Microsoft (R) F# Compiler version (private)
Copyright (c) Microsoft Corporation. All Rights Reserved.
POSTCMD: [fsiAnyCPU --nologo --quiet --exec comparer.fsx help40.txt help40.437.1033.bsl]
Fail to execute the POSTCMD. FAIL
+++ CompilerOptions\fsc\help (NetFx40 --help) +++
PRECMD: [fsc >help40.txt --help 2>&1]
Microsoft (R) F# Compiler version (private)
Copyright (c) Microsoft Corporation. All Rights Reserved.
POSTCMD: [fsiAnyCPU --nologo --quiet --exec comparer.fsx help40.txt help40.437.1033.bsl]
Fail to execute the POSTCMD. FAIL
+++ CompilerOptions\fsc\help (NetFx40 /?) +++
PRECMD: [fsc >help40.txt /? 2>&1]
Microsoft (R) F# Compiler version (private)
Copyright (c) Microsoft Corporation. All Rights Reserved.
POSTCMD: [fsiAnyCPU --nologo --quiet --exec comparer.fsx help40.txt help40.437.1033.bsl]
Fail to execute the POSTCMD. FAIL

Various warning numbers don't match actual warning numbers.

+++ CompilerOptions\fsc\responsefile (E_responsefile_not_found.fs error if response file does not exists) +++
Microsoft (R) F# Compiler version (private)
Copyright (c) Microsoft Corporation. All Rights Reserved.
(0,0): error FS3194: Response file '@not_exists' not found in 'C:\src\latkinfsharp\tests\fsharpqa\Source\CompilerOptions\fsc\responsefile\not_exists'

*** The following necessary lines were never matched:
*** error FS3192:.+

+++ CompilerOptions\fsc\responsefile (E_responsefile_path_invalid.fs error if response file name is empty or invalid) +++
Microsoft (R) F# Compiler version (private)
Copyright (c) Microsoft Corporation. All Rights Reserved.
(0,0): error FS3195: Response file name '@' is empty, contains invalid characters, has a drive specification without an absolute path, or is too long

*** The following necessary lines were never matched:
*** error FS3193:.+

The text was updated successfully, but these errors were encountered:

enricosada · 2016-01-30T13:56:59Z

I'll fix test failures, thx @latkin

dsyme · 2016-02-07T11:07:40Z

Here's the full list of failures in "fsharpqa" based on a run last night on my machine.

I'm concerned about the failures in GenericComparison in particular though regardless we have to get back to the point where we are running clean, and stay there.

fail001     CompilerOptions\fsc\help    #  (NetFx40 -?)
fail002     CompilerOptions\fsc\help    #  (NetFx40 --help)
fail003     CompilerOptions\fsc\help    #  (NetFx40 /?)
fail004     CompilerOptions\fsc\responsefile    #  (E_responsefile_not_found.fs error if response file does not exists)
fail005     CompilerOptions\fsc\responsefile    #  (E_responsefile_path_invalid.fs error if response file name is empty or invalid)
fail006     Conformance\DeclarationElements\CustomAttributes\AttributeUsage #  (X_AssemblyVersion01.fs)
fail007     Conformance\DeclarationElements\CustomAttributes\AttributeUsage #  (X_AssemblyVersion02.fs)
fail008     Conformance\DeclarationElements\MemberDefinitions\MethodsAndProperties  #  (E_DuplicateProperty01.fs)
fail009     Conformance\DeclarationElements\MemberDefinitions\MethodsAndProperties  #  (E_AbstractProperties03.fs)
fail010     Conformance\DeclarationElements\MemberDefinitions\MethodsAndProperties  #  (E_AbstractAndConcereteProp.fs)
fail011     Conformance\ObjectOrientedTypeDefinitions\ClassTypes\MemberDeclarations #  (E_PropertySetterUnit01.fs)
fail012     Conformance\ObjectOrientedTypeDefinitions\ClassTypes\MemberDeclarations #  (E_PropertyInvalidGetterSetter01.fs)
fail013     Conformance\ObjectOrientedTypeDefinitions\ClassTypes\MemberDeclarations #  (E_PropertySameNameDiffArity.fs)
fail014     TypeProviders\RuntimeDesignTimeAssembly\LoadingLogic    #  (NFYY)
fail015     TypeProviders\RuntimeDesignTimeAssembly\LoadingLogic    #  (NFYN)
fail016     Optimizations\GenericComparison #  (Compare07.fs - NetFx40)
fail017     Optimizations\GenericComparison #  (Compare10.fs - NetFx40)
fail018     Optimizations\GenericComparison #  (Equals06.fs - NetFx40)
fail019     Optimizations\GenericComparison #  (Equals09.fs - NetFx40)
fail020     Optimizations\GenericComparison #  (Hash09.fs - NetFx40)
fail021     Optimizations\GenericComparison #  (Hash12.fs - NetFx40)

dsyme · 2016-02-07T11:26:14Z

After #937 the current failures will be like this:

fail014 TypeProviders\RuntimeDesignTimeAssembly\LoadingLogic # (NFYY)
fail015 TypeProviders\RuntimeDesignTimeAssembly\LoadingLogic # (NFYN)
fail016 Optimizations\GenericComparison # (Compare07.fs - NetFx40)
fail017 Optimizations\GenericComparison # (Compare10.fs - NetFx40)
fail018 Optimizations\GenericComparison # (Equals06.fs - NetFx40)
fail019 Optimizations\GenericComparison # (Equals09.fs - NetFx40)
fail020 Optimizations\GenericComparison # (Hash09.fs - NetFx40)
fail021 Optimizations\GenericComparison # (Hash12.fs - NetFx40)

manofstick · 2016-02-07T23:21:14Z

I commented on this in #944 (the gist being that we should probably restore the tail calls. Agree?)

dsyme · 2016-02-07T23:51:49Z

These are the failing tests on my machine for the "FSHARP" suite:

Errors and Failures

1) Failed : FSharp-Tests-Core+Printing.printing("","z.output.test.default.txt","
z.output.test.default.bsl")
2) Failed : FSharp-Tests-Core+Printing.printing("--use:preludePrintSize1000.fsx"
,"z.output.test.1000.txt","z.output.test.1000.bsl")
3) Failed : FSharp-Tests-Core+Printing.printing("--use:preludePrintSize200.fsx",
"z.output.test.200.txt","z.output.test.200.bsl")
4) Failed : FSharp-Tests-Core+Printing.printing("--use:preludeShowDeclarationVal
uesFal...","z.output.test.off.txt","z.output.test.off.bsl")
5) Failed : FSharp-Tests-Typecheck+Sigs.sigs

dsyme · 2016-02-08T00:39:35Z

All failing tests should now be fixed in #944

dsyme · 2016-02-08T17:46:22Z

I'm still seeing failing tests when running tests using the VS tets runner from VisualFSharp.sln:

core\members\basics
core\load-script
negTests("neg1a")
negTests("neg1_a")
negTests("ProviderAttribute_EmptyConsume")
negTests("providerAttributeErrorConsume")
printing tests

I'm not sure why/if these appear to be passing outside VS. Some of the issues seem to do with timing of output. Will look into it and keep this thread open.

dsyme · 2016-02-08T18:03:42Z

We're also seeing FSHARP suite failures on AppVeyor, however the AppVeyor build itself is not failing when that suite has errors....

[00:33:34] 1) Failed : FSharp-Tests-Core+Interop.interop
[00:33:34] 2) Failed : FSharp-Tests-Core+Printing.printing("--use:preludePrintSize200.fsx","z.output.test.200.txt","z.output.test.200.bsl")
[00:33:34] 3) Failed : FSharp-Tests-Core+Printing.printing("--use:preludeShowDeclarationValuesFal...","z.output.test.off.txt","z.output.test.off.bsl")
[00:33:34] 4) Failed : FSharp-Tests-Core+Verify.verify
[00:33:34] 5) Failed : FSharp-Tests-Typecheck+Sigs.sigs
[00:33:34] 6) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(FSC_BASIC)
[00:33:34] 7) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(FSC_O3)
[00:33:34] 8) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(FSC_OPT_MINUS_DEBUG)
[00:33:34] 9) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(FSC_OPT_PLUS_DEBUG)
[00:33:34] 10) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(FRENCH)
[00:33:34] 11) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(SPANISH)
[00:33:34] 12) Failed : FSharp-Tests-TypeProviders+Builtin+EdmxFile.EdmxFile(AS_DLL)
[00:33:34] 13) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(FSC_BASIC)
[00:33:34] 14) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(FSC_O3)
[00:33:34] 15) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(FSC_OPT_MINUS_DEBUG)
[00:33:34] 16) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(FSC_OPT_PLUS_DEBUG)
[00:33:34] 17) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(FRENCH)
[00:33:34] 18) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(SPANISH)
[00:33:34] 19) Failed : FSharp-Tests-TypeProviders+Builtin+SqlDataConnection.sqlDataConnection(AS_DLL)
[00:33:34] 20) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(FSC_BASIC)
[00:33:34] 21) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(FSC_O3)
[00:33:34] 22) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(FSC_OPT_MINUS_DEBUG)
[00:33:34] 23) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(FSC_OPT_PLUS_DEBUG)
[00:33:34] 24) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(FRENCH)
[00:33:34] 25) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(SPANISH)
[00:33:34] 26) Failed : FSharp-Tests-TypeProviders+Builtin+ODataService.oDataService(AS_DLL)
[00:33:34] 27) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(FSC_BASIC)
[00:33:34] 28) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(FSC_O3)
[00:33:34] ERRORLEVEL 1 ERRORLEVEL 1
[00:33:34] 29) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(FSC_OPT_MINUS_DEBUG)
[00:33:34] 30) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(FSC_OPT_PLUS_DEBUG)
[00:33:34] 31) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(FRENCH)
[00:33:34] 32) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(SPANISH)
[00:33:34] 33) Failed : FSharp-Tests-TypeProviders+Builtin+WsdlService.wsdlService(AS_DLL)

enricosada · 2016-02-08T19:59:28Z

@dsyme some like core/member/basic are disabled in appveyor becuase of NO_CI in https://github.com/Microsoft/visualfsharp/blob/master/tests/test.lst

enricosada · 2016-02-08T20:05:38Z

Ref #813 (comment)

dsyme · 2016-02-09T00:36:52Z

I think https://github.com/Microsoft/visualfsharp/pulls/948 should address all test failures. We're getting close to having all tests passing. core\printing will still fail on non-US culture

dsyme · 2016-02-09T08:45:17Z

Status of CI after merging #948

FSHARP and FSHARPQA are both enanbled
One remaining FSHARP suite failure. 1) Failed : FSharp-Tests-Typecheck+Sigs.sigs
Failures in FSHARP are still not triggering an overall failure of the build, nor uploading results correctly
The VisualFSharp.Unittests suite is not yet enabled
Overall appveyor build+test times are at around 55 minutes. (2x27min)
VisualFSharp.Unittests has these failures based on a "release" local run on my machine.

Error : Tests.LanguageService.General.UsingMSBuild.PublicSurfaceArea.DotNetRe
flection
Error : Tests.LanguageService.General.UsingProjectSystem.PublicSurfaceArea.Do
tNetReflection
Error : Tests.ProjectSystem.Miscellaneous.Miscellaneous.CreateServices
Error : Tests.ProjectSystem.References.PathReferences.Existing
Error : Tests.ProjectSystem.References.ReferenceResolution.Bug650591.Automati
onReference.Add.FullPath
Failed : Tests.ProjectSystem.UpToDate.ItemInputs

enricosada · 2016-02-09T08:53:21Z

Failures in FSHARP are still not triggering an overall failure of the build, nor uploading results correctly

i'll fix that now, pr wip in #949 (i'll make suite fail, and fix error)

dsyme · 2016-02-09T10:15:05Z

@enricosada it's fixed in #950 , but good to double check with #949

enricosada · 2016-02-09T10:28:09Z

@dsyme i think the problem is another, i'll try to fix it with my pr

call RunTests.cmd %BUILD_CONFIG_LOWER% fsharp %TEST_TAGS% 

@if ERRORLEVEL 1 type testresults\fsharp_failures.log && echo Error: 'RunTests.cmd %BUILD_CONFIG_LOWER% fsharp %TEST_TAGS%' failed && goto :failure

with type testresults\fsharp_failures.log if fsharp_failures.log doesnt exists, there is no goto :failure. the fsharp nunit runner write FSharpNunit_Output.log

enricosada · 2016-02-09T10:29:14Z

btw @dsyme amazing job fixing all failures, and the code cleanup in fsharp nunit 👍

dsyme · 2016-02-09T10:43:41Z

ok, thanks

dsyme · 2016-02-09T11:06:54Z

@enricosada I've fixed the fsharp_failures.log problem in #950

enricosada · 2016-02-09T11:09:14Z

@dsyme ok, closed mine

dsyme · 2016-02-09T11:11:39Z

@enricosada A couple of comments on the test runner code for the FSHARP quite

Awesome job on the 1:1 translation. It's made it feasible to track down these bugs. I can't believe how you persevered with that: it really is a huge contribution especially in the long term
Now that the translation is done and stable (apart from enabling it for CoreCLR) we should aim to simplify the code. For example I spent an hour staring at some "pipe to FSI sed.fsx script" code trying to work out what was going on, and in the end replaced it with a File.ReadAllLines+String.Replace+File.WriteAllLines in 3 minutes :) So we can likely look for similar places where we are needlessly shelling out or where we have inherited complexity we no longer need.
It would be good to split the "sigs" test into multiple individual tests.

enricosada · 2016-02-09T11:58:05Z

btw @dsyme i think you can ask @KevinRansom to make you collaborator in appveyor, so you can cancel/stop useless builds.
I use stop build a lot in my personal appveyor

dsyme · 2016-02-09T17:40:04Z

Thanks @enricosada , have asked

dsyme · 2016-02-09T17:43:40Z

We have a GREEN on master https://ci.appveyor.com/project/KevinRansom/visualfsharp-radou/build/0.0.1.1511 that includes running all FSHARP and FSHARPQA tests (apart from the relatively few "Expensive" ones)

Am closing this!!!!

forki · 2016-02-09T17:58:25Z

Kudos!

dsyme changed the title ~~Response file tests fail~~ Failing tests on master Feb 7, 2016

dsyme mentioned this issue Feb 7, 2016

Various codegen tests failing #918

Closed

dsyme added Regression Impact-High (Internal MS Team use only) Describes an issue with extreme impact on existing code. labels Feb 7, 2016

This was referenced Feb 7, 2016

separate smoke and CI tests #937

Merged

Fix failing tests (including changes codegen for failing IL generation tests for generic comparison) #944

Closed

dsyme mentioned this issue Feb 9, 2016

Failing tests in cambridge suite ( disabled now ) #813

Closed

4 tasks

dsyme mentioned this issue Feb 9, 2016

More fixes for FSHARP suite tests #950

Merged

dsyme closed this as completed Feb 9, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing tests on master #915

Failing tests on master #915

latkin commented Jan 29, 2016

enricosada commented Jan 30, 2016

dsyme commented Feb 7, 2016

dsyme commented Feb 7, 2016

manofstick commented Feb 7, 2016

dsyme commented Feb 7, 2016

dsyme commented Feb 8, 2016

dsyme commented Feb 8, 2016

dsyme commented Feb 8, 2016

enricosada commented Feb 8, 2016

enricosada commented Feb 8, 2016

dsyme commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

dsyme commented Feb 9, 2016

forki commented Feb 9, 2016

Failing tests on master #915

Failing tests on master #915

Comments

latkin commented Jan 29, 2016

enricosada commented Jan 30, 2016

dsyme commented Feb 7, 2016

dsyme commented Feb 7, 2016

manofstick commented Feb 7, 2016

dsyme commented Feb 7, 2016

dsyme commented Feb 8, 2016

dsyme commented Feb 8, 2016

dsyme commented Feb 8, 2016

enricosada commented Feb 8, 2016

enricosada commented Feb 8, 2016

dsyme commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

enricosada commented Feb 9, 2016

dsyme commented Feb 9, 2016

dsyme commented Feb 9, 2016

forki commented Feb 9, 2016