Skip to content

Seeing intermittent failures in the 'DotNetInterfacesShouldEqualJavaInterfaces' runtime test #1534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pjcollins opened this issue Apr 10, 2018 · 0 comments
Closed

Seeing intermittent failures in the 'DotNetInterfacesShouldEqualJavaInterfaces' runtime test #1534

pjcollins opened this issue Apr 10, 2018 · 0 comments
Assignees
@grendello
Labels
Area: Unit Tests Issues with repo unit tests.

Comments

@pjcollins
Copy link
Member

pjcollins commented Apr 10, 2018 •
edited
Loading

Copying over https://bugzilla.xamarin.com/show_bug.cgi?id=59871

I'm seeing an intermittent failure in DotNetInterfacesShouldEqualJavaInterfaces which fails at different points across different devices. In some of these cases there were network connectivity issues present, however I don't believe interface discovery should be affected by network connectivity state. Rebooting the affected devices did not resolve the negative behavior. I have hit this a couple of times during d15-7 testing as well.

Oneplus One

https://gist.github.com/pjcollins/50b3036dd4a0b4a4d881a24fe49ab30d

09-25 12:36:18.234 I/mono-stdout(26038): [FAIL] : #6.3 (dummy0 not found in Java interfaces)

Samsung S8

https://gist.github.com/pjcollins/5782e1bc225cdf105a75491e23f257d6

09-29 10:26:41.754 I/mono-stdout(18006): [FAIL] : #3.1
09-29 10:26:41.754 I/mono-stdout(18006): Expected: 19
09-29 10:26:41.754 I/mono-stdout(18006): But was: 5

Nexus 6p (API 25)

https://gist.github.com/pjcollins/062f2124e1e76561809193450f97e768

09-29 10:26:05.053 I/mono-stdout(32432): [FAIL] : #3.1
09-29 10:26:05.053 I/mono-stdout(32432): Expected: 22
09-29 10:26:05.053 I/mono-stdout(32432): But was: 4

Asus Zenphone

https://gist.github.com/pjcollins/f8dd5ad7c685d537d2ef512625320cc3

09-29 15:17:45.971 I/mono-stdout(10500): [FAIL] : #3.1
09-29 15:17:45.971 I/mono-stdout(10500): Expected: 24
09-29 15:17:45.971 I/mono-stdout(10500): But was: 7

Environment

Operating System: Mac OS X 10.12.3
monodroid/d15-5/12e1ed307ba15f186f2a2363815a5bf195c41370
@pjcollins pjcollins added the Area: Unit Tests Issues with repo unit tests. label Apr 10, 2018
pjcollins added a commit to pjcollins/android that referenced this issue Jul 13, 2018
@pjcollins
[Mono.Android-Tests] Ignore DotNetJavaNetworkIfaces test
06ed18f 
This test is failing rather consistently on many physical devices. See
dotnet#1534 for more info.
@pjcollins pjcollins mentioned this issue Jul 13, 2018
Merged
jonpryor pushed a commit that referenced this issue Jul 26, 2018
@pjcollins @jonpryor
[tests] Category support in TestRunner.NUnit (#1963)
f81225d 
Context: #1534

The `DotNetInterfacesShouldEqualJavaInterfaces()` unit test
*occasionally* fails on certain Android devices, such as:

  * [Oneplus One](https://gist.github.com/pjcollins/50b3036dd4a0b4a4d881a24fe49ab30d):

        09-25 12:36:18.234 I/mono-stdout(26038): [FAIL] : #6.3 (dummy0 not found in Java interfaces)

  * [Samsung S8](https://gist.github.com/pjcollins/5782e1bc225cdf105a75491e23f257d6)

        09-29 10:26:41.754 I/mono-stdout(18006): Expected: 19
        09-29 10:26:41.754 I/mono-stdout(18006): But was: 5

  * [Nexus 6p (API 25)](https://gist.github.com/pjcollins/062f2124e1e76561809193450f97e768)

        09-29 10:26:05.053 I/mono-stdout(32432): Expected: 22
        09-29 10:26:05.053 I/mono-stdout(32432): But was: 4

We do not yet understand what is causing these failures.

That said, QA doesn't want to continue seeing these errors in QA
builds, as reviewing "errors for which we've already filed bugs" is
tedious and annoying, and makes it easier to overlook new issues.

Extend the `[CategoryAttribute]` support in 096210c so that the new
`TestRunner.NUnit` test runner (c4e8165) can also use the same
`$(ExcludeCategories)` MSBuild property to cause tests to be skipped.

Add `[Category("NetworkInterfaces")]` to the
`DotNetInterfacesShouldEqualJavaInterfaces()` test.

Together, these two changes will allow QA to selectively skip the
`DotNetInterfacesShouldEqualJavaInterfaces()` test.
jonpryor pushed a commit that referenced this issue Aug 3, 2018
@pjcollins @jonpryor
[tests] Category support in TestRunner.NUnit (#1963)
32d9484 
Context: #1534

The `DotNetInterfacesShouldEqualJavaInterfaces()` unit test
*occasionally* fails on certain Android devices, such as:

  * [Oneplus One](https://gist.github.com/pjcollins/50b3036dd4a0b4a4d881a24fe49ab30d):

        09-25 12:36:18.234 I/mono-stdout(26038): [FAIL] : #6.3 (dummy0 not found in Java interfaces)

  * [Samsung S8](https://gist.github.com/pjcollins/5782e1bc225cdf105a75491e23f257d6)

        09-29 10:26:41.754 I/mono-stdout(18006): Expected: 19
        09-29 10:26:41.754 I/mono-stdout(18006): But was: 5

  * [Nexus 6p (API 25)](https://gist.github.com/pjcollins/062f2124e1e76561809193450f97e768)

        09-29 10:26:05.053 I/mono-stdout(32432): Expected: 22
        09-29 10:26:05.053 I/mono-stdout(32432): But was: 4

We do not yet understand what is causing these failures.

That said, QA doesn't want to continue seeing these errors in QA
builds, as reviewing "errors for which we've already filed bugs" is
tedious and annoying, and makes it easier to overlook new issues.

Extend the `[CategoryAttribute]` support in 096210c so that the new
`TestRunner.NUnit` test runner (c4e8165) can also use the same
`$(ExcludeCategories)` MSBuild property to cause tests to be skipped.

Add `[Category("NetworkInterfaces")]` to the
`DotNetInterfacesShouldEqualJavaInterfaces()` test.

Together, these two changes will allow QA to selectively skip the
`DotNetInterfacesShouldEqualJavaInterfaces()` test.
@ghost ghost locked as resolved and limited conversation to collaborators Jun 8, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@grendello grendello

Labels
Area: Unit Tests Issues with repo unit tests.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@grendello @pjcollins