setup-kdc.sh is not installing Kerberos on some distros causing System.Net.Security.Tests.NegotiateStreamTests to no-op

[ghost]

Opened on behalf of @JiayiLi1

The test System.Net.Security.Tests.NegotiateStreamTest/NegotiateStream_EchoServer_NTLM_ClientWriteRead_Successive_Async_Success has failed.

System.ComponentModel.Win32Exception : Invalid argument

    Stack Trace:

       at Interop.Sys.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Int32& lpChildPid, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean shouldThrow) in /root/corefx-987254/src/Common/src/Interop/Unix/System.Native/Interop.ForkAndExecProcess.cs:line 43
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo) in /root/corefx-987254/src/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 291
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /root/corefx-987254/src/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1248
   at System.Net.Security.Tests.KDCSetup.CheckAndClearCredentials(ITestOutputHelper output) in /root/corefx-987254/src/System.Net.Security/tests/FunctionalTests/NegotiateStreamTestForUnix.cs:line 78
   at System.Net.Security.Tests.NegotiateStreamTest..ctor(KDCSetup fixture, ITestOutputHelper output) in /root/corefx-987254/src/System.Net.Security/tests/FunctionalTests/NegotiateStreamTestForUnix.cs:line 151

Build : Master - 20170913.01 (Core Tests)
Failing configurations:

• suse.422.amd64-x64
  • Release

Detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170913.01/workItem/System.Net.Security.Tests/analysis/xunit/System.Net.Security.Tests.NegotiateStreamTest~2FNegotiateStream_EchoServer_NTLM_ClientWriteRead_Successive_Async_Success

[KristinXie1]

failed tests:
NegotiateStream_EchoServer_ClientWriteRead_Successive_Async_Success
NegotiateStream_EchoServer_ClientWriteRead_Successive_Sync_Success
NegotiateStream_EchoServer_NTLM_ClientWriteRead_Successive_Async_Success
NegotiateStream_EchoServer_NTLM_ClientWriteRead_Successive_Sync_Success
NegotiateStream_StreamToStream_AuthToHttpTarget_Success
NegotiateStream_StreamToStream_KerberosAuthDefaultCredentialsNoSeed_Failure
NegotiateStream_StreamToStream_KerberosAuthDefaultCredentials_Success
NegotiateStream_StreamToStream_KerberosAuthInvalidPassword_Failure
NegotiateStream_StreamToStream_KerberosAuthInvalidTarget_Failure
NegotiateStream_StreamToStream_KerberosAuthInvalidUser_Failure
NegotiateStream_StreamToStream_KerberosAuthWithoutRealm_Success
NegotiateStream_StreamToStream_KerberosAuthentication_Success
NegotiateStream_StreamToStream_NtlmAuthentication_Fallback_Success
NegotiateStream_StreamToStream_NtlmAuthentication_KerberosCreds_Success
NegotiateStream_StreamToStream_NtlmAuthentication_NtlmUser_InvalidCredentials_Fail
NegotiateStream_StreamToStream_NtlmAuthentication_ValidCredentials_Success

[davidsh]

cc: @karelz @danmosemsft

System.Net.Security.NegotiateStream seems broken on Linux.

[danmoseley]

It may be as simple as "kdestroy" not being available on suse 4.22.
@karelz someone should either update the images, or cause this test to skip on suse, or find an alternative way to achieve whatever it's doing. Perhaps some apt-get type command needs to go into the dockerfile.

            var startInfo = new ProcessStartInfo(KDestroyCmd);
            startInfo.UseShellExecute = true;
            startInfo.CreateNoWindow = true;
            startInfo.Arguments = "-A";
            using (Process clearCreds = Process.Start(startInfo))

[davidsh]

or cause this test to skip on suse,

These tests have been failing on Ubuntu 17.10 as well. I thought @Priya91 was looking into that.

[Priya91]

From the callstack, it looks like these tests are failing due to some wrong arguments to fork and exec.. This is not the same issue as the one on ubuntu 17.10, that's a different bug.

[danmoseley]

This is causing 350 tests to fail. It is only on Centos, Debian, Fedora, Ubuntu 17.xx and SUSE. Not RH, SLES, Ubuntu 16.xx,

https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170916.01/workItem/System.Net.Security.Tests

On my personal Centos box, I had to yum install krb5-workstation to get it. On my Ubuntu box, I had to get the krb5-client package. It seems like src\System.Net.Security\tests\Scripts\Unix\setup-kdc.sh is supposed to do this.

Next step: someone SSH into one of these and debug the script.

[karelz]

@danmosemsft I don't see any test failures on the link you pasted.
Where can I see the 350 tests affected? Where can I find how long they are failing? (is that a recent regression?) ... Or do we still lack that kind of info about test failures?

[karelz]

Oh, I wasn't signed in (and the page didn't indicate that well :( ... @Chrisboh is it something that can be improved please?)
I still see only 25 tests failing on the page (on 7 OS versions). That does not add up to 350 ... what am I missing?

[karelz]

@wfurt can you please look at this high-impactful test failure?

[danmoseley]

@karelz if you look at https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170916.01 on the left you see 350, this is the number of failures. The failures began in https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170915.02 . I don't know why : the script didn't change. It would probably be easy for @wfurt to figure out by running it locally.

I'll modify this PR - not a problem

[KristinXie1]

This issue is also repro in CI, outerloop_netcoreapp_centos7.1_release, detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_centos7.1_release/170/testReport/

[karelz]

on the left you see 350, this is the number of failures

Ah, in the higher-level report, I didn't notice that -- it counts each failure twice (automatic rerun I assume?)

[danmoseley]

@karelz there are no reruns currently. The x2 is chk and ret. They don't have columns for both, for lack of space.

[KristinXie1]

Now it failed with "Xunit.Sdk.TrueException", detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20170920.02/workItem/System.Net.Security.Tests/analysis/xunit/System.Net.Security.Tests.NegotiateStreamTest~2FNegotiateStream_EchoServer_ClientWriteRead_Successive_Async_Success

The test System.Net.Security.Tests.NegotiateStreamTest/NegotiateStream_EchoServer_ClientWriteRead_Successive_Async_Success has failed.

Assert.True() Failure
Expected: True
Actual: False

    Stack Trace:

       at System.Net.Security.Tests.KDCSetup.CheckAndClearCredentials(ITestOutputHelper output) in /root/corefx-1003914/src/System.Net.Security/tests/FunctionalTests/NegotiateStreamTestForUnix.cs:line 93
   at System.Net.Security.Tests.NegotiateStreamTest..ctor(KDCSetup fixture, ITestOutputHelper output) in /root/corefx-1003914/src/System.Net.Security/tests/FunctionalTests/NegotiateStreamTestForUnix.cs:line 168

Build : Master - 20170920.02 (Core Tests)
Failing configurations:

• Ubuntu.1604.Amd64-x64
  • Debug

[KristinXie1]

These tests all failed with "Xunit.Sdk.TrueException" in outerloop_netcoreapp_ubuntu16.04_debug, detail: https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_netcoreapp_ubuntu16.04_debug/180/testReport/

[KristinXie1]

failed again in Master branch, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20171009.01/workItem/System.Net.Security.Tests

[wfurt]

It seems like this got broken by dotnet/corefx#24017
When I revert the change the test can pass OK on Ubuntu17.
It seems like with the change we cannot longer execute binaries on PATH.

The negotiate tests seems problematic as they try to install packages during test runs as well as they can mark test as Pass even if the test was not executed.
I'm inclined to track that as separate issue.

[wfurt]

actually, it may have been dotnet/corefx#23705 . The execution on Unix with shell seems broken. I'll do more testing.

[KristinXie1]

still failed in 20171018.01, detail: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20171018.01/workItem/System.Net.Security.Tests

[Priya91]

This should be fixed by dotnet/corefx#24682