Use sysconf(_SC_NPROCESSORS_CONF) in PAL_GetLogicalCpuCountFromOS

[echesakov]

As we found in #17851 sysconf(_SC_NPROCESSORS_ONLN) should not be used in GetLogicalCpuCountFromOS since it could return arbitrary number between 1 and actual number of logical CPU cores which causes VM to make wrong decision about which JIT_WriteBarrier to be used (SP vs MP).

When I tested sysconf(_SC_NPROCESSORS_ONLN) on NVIDIA Jetson TK1 I got all the values 1,2,3,4 depending on how busy the processor is.

As suggested in https://github.com/dotnet/coreclr/issues/17851#issuecomment-390333775 sysconf(_SC_NPROCESSORS_CONF) and seanmonstar/num_cpus#34 should be used instead.

[janvorli]

LGTM, thank you for figuring it out!

[RussKeldorph]

@janvorli Would you make this specific to ARM/ARM64 if we port it to release/2.1?

[janvorli]

@RussKeldorph yes, that would be better.

[danmoseley]

Is it OK here?

coreclr/src/gc/unix/gcenv.unix.cpp

Line 70 in cb73944

int cpuCount = sysconf(_SC_NPROCESSORS_ONLN);

[janvorli]

No, it is the same issue.

[sandreenko]

cc @dotnet/jit-contrib

[echesakov]

Fixed coreclr/src/gc/unix/gcenv.unix.cpp
@danmosemsft Thanks for catching this!

[echesakov]

@dotnet-bot test Ubuntu arm Cross Checked corefx_baseline Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstress1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstress2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs0x10 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs0x1000 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs0x80 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs3 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs4 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_jitstressregs8 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_minopts Build and Test
@dotnet-bot test Ubuntu arm Cross Checked corefx_tieredcompilation Build and Test
@dotnet-bot test Ubuntu arm Cross Checked forcerelocs Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0x3 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc_jitstress1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc_jitstress2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc_minopts_heapverify1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc_zapdisable Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc_zapdisable_heapverify1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked gcstress0xc_zapdisable_jitstress2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked heapverify1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs0x10 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs0x1000 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs0x80 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs3 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs4 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstress2_jitstressregs8 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs0x10 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs0x1000 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs0x80 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs3 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs4 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked jitstressregs8 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked minopts Build and Test
@dotnet-bot test Ubuntu arm Cross Checked normal Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_gcstress15 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitforcerelocs Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitminopts Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstress1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstress2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs0x10 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs0x1000 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs0x80 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs2 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs3 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs4 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs8 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked tailcallstress Build and Test
@dotnet-bot test Ubuntu arm Cross Checked tieredcompilation Build and Test
@dotnet-bot test Ubuntu arm Cross Checked zapdisable Build and Test
@dotnet-bot test Ubuntu arm Cross Release normal Build and Test
@dotnet-bot test Ubuntu arm Cross Release r2r Build and Test

[echesakov]

@dotnet-bot test Ubuntu arm64 Cross Checked normal Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_gcstress15 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitforcerelocs Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitminopts Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstress1 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstress2 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs0x10 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs0x1000 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs0x80 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs1 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs2 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs3 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs4 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Checked r2r_jitstressregs8 Build and Test
@dotnet-bot test Ubuntu arm64 Cross Release normal Build and Test
@dotnet-bot test Ubuntu arm64 Cross Release r2r Build and Test

[echesakov]

@dotnet-bot test CentOS7.1 x64 Build and Test
@dotnet-bot test CentOS7.1 x64 Checked r2r
@dotnet-bot test CentOS7.1 x64 Checked r2r_gcstress15
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitforcerelocs
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitminopts
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstress1
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstress2
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs0x10
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs0x1000
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs0x80
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs1
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs2
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs3
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs4
@dotnet-bot test CentOS7.1 x64 Checked r2r_jitstressregs8
@dotnet-bot test CentOS7.1 x64 Release r2r

[echesakov]

@dotnet-bot test Ubuntu x64 Build and Test
@dotnet-bot test Ubuntu x64 Checked corefx_baseline
@dotnet-bot test Ubuntu x64 Checked corefx_baseline
@dotnet-bot test Ubuntu x64 Checked corefx_jitstress1
@dotnet-bot test Ubuntu x64 Checked corefx_jitstress1
@dotnet-bot test Ubuntu x64 Checked corefx_jitstress2
@dotnet-bot test Ubuntu x64 Checked corefx_jitstress2
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs0x10
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs0x10
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs0x1000
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs0x1000
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs0x80
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs0x80
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs1
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs1
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs2
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs2
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs3
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs3
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs4
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs4
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs8
@dotnet-bot test Ubuntu x64 Checked corefx_jitstressregs8
@dotnet-bot test Ubuntu x64 Checked corefx_minopts
@dotnet-bot test Ubuntu x64 Checked corefx_minopts
@dotnet-bot test Ubuntu x64 Checked corefx_tieredcompilation
@dotnet-bot test Ubuntu x64 Checked corefx_tieredcompilation
@dotnet-bot test Ubuntu x64 Checked forcerelocs
@dotnet-bot test Ubuntu x64 Checked gcstress0x3
@dotnet-bot test Ubuntu x64 Checked gcstress0xc
@dotnet-bot test Ubuntu x64 Checked gcstress0xc_jitstress1
@dotnet-bot test Ubuntu x64 Checked gcstress0xc_jitstress2
@dotnet-bot test Ubuntu x64 Checked gcstress0xc_minopts_heapverify1
@dotnet-bot test Ubuntu x64 Checked gcstress0xc_zapdisable
@dotnet-bot test Ubuntu x64 Checked gcstress0xc_zapdisable_heapverify1
@dotnet-bot test Ubuntu x64 Checked gcstress0xc_zapdisable_jitstress2
@dotnet-bot test Ubuntu x64 Checked heapverify1
@dotnet-bot test Ubuntu x64 Checked illink
@dotnet-bot test Ubuntu x64 Checked illink
@dotnet-bot test Ubuntu x64 Checked jitincompletehwintrinsic
@dotnet-bot test Ubuntu x64 Checked jitnosimd
@dotnet-bot test Ubuntu x64 Checked jitnox86hwintrinsic
@dotnet-bot test Ubuntu x64 Checked jitsse2only
@dotnet-bot test Ubuntu x64 Checked jitstress1
@dotnet-bot test Ubuntu x64 Checked jitstress2
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs0x10
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs0x1000
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs0x80
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs1
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs2
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs3
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs4
@dotnet-bot test Ubuntu x64 Checked jitstress2_jitstressregs8
@dotnet-bot test Ubuntu x64 Checked jitstressregs0x10
@dotnet-bot test Ubuntu x64 Checked jitstressregs0x1000
@dotnet-bot test Ubuntu x64 Checked jitstressregs0x80
@dotnet-bot test Ubuntu x64 Checked jitstressregs1
@dotnet-bot test Ubuntu x64 Checked jitstressregs2
@dotnet-bot test Ubuntu x64 Checked jitstressregs3
@dotnet-bot test Ubuntu x64 Checked jitstressregs4
@dotnet-bot test Ubuntu x64 Checked jitstressregs8
@dotnet-bot test Ubuntu x64 Checked jitx86hwintrinsicnoavx
@dotnet-bot test Ubuntu x64 Checked jitx86hwintrinsicnoavx2
@dotnet-bot test Ubuntu x64 Checked jitx86hwintrinsicnosimd
@dotnet-bot test Ubuntu x64 Checked minopts
@dotnet-bot test Ubuntu x64 Checked r2r
@dotnet-bot test Ubuntu x64 Checked r2r_gcstress15
@dotnet-bot test Ubuntu x64 Checked r2r_jitforcerelocs
@dotnet-bot test Ubuntu x64 Checked r2r_jitminopts
@dotnet-bot test Ubuntu x64 Checked r2r_jitstress1
@dotnet-bot test Ubuntu x64 Checked r2r_jitstress2
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs0x10
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs0x1000
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs0x80
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs1
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs2
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs3
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs4
@dotnet-bot test Ubuntu x64 Checked r2r_jitstressregs8
@dotnet-bot test Ubuntu x64 Checked tailcallstress
@dotnet-bot test Ubuntu x64 Checked tieredcompilation
@dotnet-bot test Ubuntu x64 Checked zapdisable

[BruceForstall]

The Ubuntu arm64 will fail because we don't have any such machines in the CI currently (despite having the jobs defined).

[stephentoub]

Presumably we also need to fix the test in corefx:
https://github.com/dotnet/corefx/blob/07cafe74d5c1e5b7657c13fea4e586a05ff90e1a/src/System.Runtime.Extensions/tests/System/Environment.ProcessorCount.cs#L29-L51

[echesakov]

Re-running these jobs failed due to CompareExchangeTString and got fixed in #18075

@dotnet-bot test Ubuntu arm Cross Checked normal Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstress1 Build and Test
@dotnet-bot test Ubuntu arm Cross Checked r2r_jitstressregs0x80 Build and Test

[echesakov]

@dotnet-bot test Ubuntu x64 Formatting

[echesakov]

@RussKeldorph I think I have created new issues or commented on existing issues in coreclr on all the test failures I saw during the PR testing. I also created corefx test issue dotnet/corefx#29902
Are you approving this to be merged in?

[akoeplinger]

Oh my, this rings some memories where we hit exactly the same problems in Mono about two years ago: mono/mono@2debfb8 😄

We decided back then to only use _SC_NPROCESSORS_CONF on ARM platforms since otherwise you'd run into Docker/taskset not being able to constrain the processor count (back then Docker only existed on x86 targets).