Process.ProcessorAffinity returns a 32-bit value on ARM64

[adamsitnik]

When we run the following code on ARM64 machine with 48 cores (Ubuntu) without setting the CPU affinity in explicit way:

Console.WriteLine(System.Diagnostics.Process.GetCurrentProcess().ProcessorAffinity);

We get 00000000FFFFFFFF which is 2^32 - 1 while it should be 2^48 - 1

I don't know if this is specific to ARM64 or Ubuntu or 64 bit in general. I just don't have an access to a machine with more than 32 core to test.

@AndyAyersMS have hit this issue when he was benchmarking .NET Core 3.0 on ARM

[danmoseley]

Was it possible that 16 of the cores were asleep? As mentioned here
https://github.com/mono/mono/blob/17aa73d0ae216529b59d5aa3d0ee68007bb035c4/mono/utils/mono-proclib.c#L717-L720

[danmoseley]

Looks like the CLR already fixed this (when getting core count for GC)

dotnet/coreclr#18053
dotnet/coreclr#18289

As for Environment.GetProcessorCount it eventually gets to
https://github.com/dotnet/coreclr/blob/e54dffef08c22a94962aacd93d4793b377cac632/src/classlibnative/bcltype/system.cpp#L237
and then it gets complicated. I am not sure whether it is susceptible to this problem on ARM and needs a fix also.

[danmoseley]

@janvorli

[AndyAyersMS]

One place this shows up is in the spectralnorm-3 perf test, which uses Environment.ProcessorCount to figure out how to partition work. It currently runs 700x slower on the arm64 HW than on an 8 core Skylake... I think this is a combination of restrictive affinity, higher synchronization costs (and maybe excessive spinning?), and more frequent synchronization.

Console.WriteLine($"ProcessArchitecture={System.Runtime.InteropServices.RuntimeInformation.ProcessArchitecture}");
Console.WriteLine($"IntPtrSize = {IntPtr.Size}");
ulong a = (ulong) System.Diagnostics.Process.GetCurrentProcess().ProcessorAffinity;
Console.WriteLine("ProcessorAffinity = " + a.ToString("X16"));
Console.WriteLine($"ProcessorCount = {Environment.ProcessorCount}");

gives

ProcessArchitecture = Arm64
IntPtrSize = 8
ProcessorAffinity = 00000000FFFFFFFF
ProcessorCount = 46

[janvorli]

I can see what's going on. The System.Diagnostics.Process.ProcessorAffinity is implemented on Linux in CoreFX in SystemNative_SchedGetAffinity function. And there is a bug in there. See the 1u in bits |= (1u << cpu); below. The u suffix means 32 bit and so the value returned will always have max 32 bits set.

        intptr_t bits = 0;
        for (int cpu = 0; cpu < maxCpu; cpu++)
        {
            if (CPU_ISSET(cpu, &set))
            {
                bits |= (1u << cpu);
            }
        }

It should be e.g. bits |= ((intptr_t)1) << cpu; instead.

[janvorli]

I've created a PR with a fix: dotnet/corefx#33825

[adamsitnik]

One place this shows up is in the spectralnorm-3 perf test, which uses Environment.ProcessorCount to figure out how to partition work. It currently runs 700x slower on the arm64 HW than on an 8 core Skylake... I think this is a combination of restrictive affinity, higher synchronization costs (and maybe excessive spinning?), and more frequent synchronization.

With the fix from @janvorli we can remove "restrictive affinity" from the list.

I just run spectralnorm-3 on my 12 core x64 Windows machine (6 real cores, HTx2) with Concurrency Visualizer Profiler enabled ( --profiler CV in the perf repo).

The CV tells me that on Windows we use up to 5 cores out of 12:

And that 90% of the time is spent for synchronization:

So @AndyAyersMS you are most probably right about higher synchronization costs on ARM.

[adamsitnik]

I just read the code of the benchmark. As of today, it creates an array of just 100 doubles and divides it into Environment.ProcessorCount-many parallel tasks.

100 is a very small input here. When I set it to 10000 the CPU consumption is totally different:

and it spends less time for synchronizaiton:

So it looks like that as of today this benchmark is measuring the perf of synchronization?

[adamsitnik]

Was it possible that 16 of the cores were asleep?

@danmosemsft I did not know that this is possible. I guess that this is why ARM is having lower energy consumption.

@brianrob if we ever run the perf tests for ARM we need to make sure this setting is off

[danmoseley]

I opened dotnet/corefx#33838 for the other issue

[danmoseley]

So @AndyAyersMS you are most probably right about higher synchronization costs on ARM.

@kouvel are you/have you looked at synchronization costs on ARM? I am not famliar with this benchmark but 90% of the time is spent for synchronization sounds not ideal.

[janvorli]

@danmosemsft this is arm64, not arm. I guess you meant that, but wanted to make it clear for others.

[danmoseley]

@janvorli thanks, it's confusing that we often call 32 bit ARM just ARM. Is it common for 32 and 64 bit ARM to have quite different performance issues?

[janvorli]

@danmosemsft they have different instruction set with different performance characteristics, so I would expect them to be quite different. Btw, the official naming is ARM and ARM64.

[kouvel]

I haven't looked at synchronization costs on arm64 and I'm not aware of issues related to synchronization there. I suspect that spectralnorm-3 would be mostly measuring barrier.SignalAndWait() and that would scale worse with number of processors but 700x seems odd. Would have to try a repro to see.

[kouvel]

If anyone has ready access to an Ubuntu arm64 machine, could you please run the following code? Just want to make sure the thread pool is also getting the correct number of processors, otherwise it would explain why it would be so slow.

        Console.WriteLine($"ProcessorCount: {Environment.ProcessorCount}");
        int w, c;
        ThreadPool.GetMinThreads(out w, out c);
        Console.WriteLine($"ThreadPool min thread counts: {w}, {c}");

[AndyAyersMS]

Last line is w & c

ProcessArchitecture = Arm64
IntPtrSize = 8
ProcessorAffinity = 00000000FFFFFFFF
ProcessorCount = 46
46 46

[kouvel]

Thanks, that looks correct, I'll take a closer look.

[danmoseley]

@adamsitnik it might make sense to move this to a new issue