Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NativeAOT on Graviton2 As armv8.2-a Fails to Execute #89937

Closed
chrisoverzero opened this issue Aug 3, 2023 · 19 comments · Fixed by #89991
Closed

NativeAOT on Graviton2 As armv8.2-a Fails to Execute #89937

chrisoverzero opened this issue Aug 3, 2023 · 19 comments · Fixed by #89991

Comments

@chrisoverzero
Copy link

chrisoverzero commented Aug 3, 2023

Description

When NativeAOT-compiling an application for deployment onto AWS Lambda, specifying IlcInstructionSet with the known, documented supported instruction sets for Graviton2 processors fails to execute with the error message:

The required instruction sets are not supported by the current CPU.

Reproduction Steps

For a minimal reproduction, I'm using the code from maxday's lambda-perf cold start–testing repository, but targeting .NET 8's NativeAOT. The meaningful parts of that are:

public class Function
{
    private static async Task Main()
    {
        await LambdaBootstrapBuilder.Create(FunctionHandler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    public static StatusResponse FunctionHandler()
    {
        return new StatusResponse(statusCode: 200);
    }
}

public record StatusResponse(int statusCode);

[JsonSerializable(typeof(StatusResponse))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

…and for .csproj settings:

    <TargetFramework>net8.0</TargetFramework>
    <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
    <PublishAot>true</PublishAot>
    <AssemblyName>bootstrap</AssemblyName>

    <IlcInstructionSet>armv8.2-a</IlcInstructionSet>

Expected behavior

Because Amazon's Graviton2 processor is documented as revision 8.2-a with support for features "fp16, rcpc, dotprod, crypto" and ilc doesn't support "fp16" or "crypto", I expected this:

<IlcInstructionSet>armv8.2-a,rcpc,dotprod</IlcInstructionSet>

…to compile and run. It compiled, but did not run. Figuring that maybe I was trying to add too much sauce, I tried:

<IlcInstructionSet>armv8.2-a</IlcInstructionSet>

…expecting this to compile and run. It compiled, but did not run.

Actual behavior

Upon execution, this error message is received:

The required instruction sets are not supported by the current CPU.

Regression?

No response

Known Workarounds

Leave off IlcInstructionSet, implicitly targeting revision 8.0. This does compile (natch) and run.

Configuration

I am cross-compiling for arm64 in the Docker image mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64:

~/.dotnet/dotnet publish -p:SysRoot=/crossrootfs/arm64 -p:LinkerFlavor=lld

dotnet --version is "8.0.100-rc.1.23402.12"

Other information

I admit that I'm not certain this is a problem with the NativeAOT compiler, since I don't know how to look into a binary and say "Aha, here is the incorrect instruction."

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 3, 2023
@ghost
Copy link

ghost commented Aug 3, 2023

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

Description

When NativeAOT-compiling an application for deployment onto AWS Lambda, specifying IlcInstructionSet with the known, documented supported instruction sets for Graviton2 processors fails to execute with the error message:

The required instruction sets are not supported by the current CPU.

Reproduction Steps

For a minimal reproduction, I'm using the code from maxday's lambda-perf cold start–testing repository, but targeting .NET 8's NativeAOT. The meaningful parts of that are:

public class Function
{
    private static async Task Main()
    {
        await LambdaBootstrapBuilder.Create(FunctionHandler, new SourceGeneratorLambdaJsonSerializer<LambdaFunctionJsonSerializerContext>())
            .Build()
            .RunAsync();
    }

    public static StatusResponse FunctionHandler()
    {
        return new StatusResponse(statusCode: 200);
    }
}

public record StatusResponse(int statusCode);

[JsonSerializable(typeof(StatusResponse))]
public partial class LambdaFunctionJsonSerializerContext : JsonSerializerContext
{
}

…and for .csproj settings:

    <TargetFramework>net8.0</TargetFramework>
    <RuntimeIdentifier>linux-arm64</RuntimeIdentifier>
    <PublishAot>true</PublishAot>
    <AssemblyName>bootstrap</AssemblyName>

    <IlcInstructionSet>armv8.2-a</IlcInstructionSet>

Expected behavior

Because Amazon's Graviton2 processor is documented as revision 8.2-a with support for features "fp16, rcpc, dotprod, crypto" and ilc doesn't support "fp16" or "crypto", I expected this:

<IlcInstructionSet>armv8.2-a,rcpc,dotprod</IlcInstructionSet>

…to compile and run. It compiled, but did not run. Figuring that maybe I was trying to add too much sauce, I tried:

<IlcInstructionSet>armv8.2-a</IlcInstructionSet>

…expecting this to compile and run. It compiled, but did not run.

Actual behavior

Upon execution, this error message is received:

The required instruction sets are not supported by the current CPU.

Regression?

No response

Known Workarounds

Leave off IlcInstructionSet, implicitly targeting revision 8.0.

Configuration

I am cross-compiling for arm64 in the Docker image mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64:

~/.dotnet/dotnet publish -p:SysRoot=/crossrootfs/arm64 -p:LinkerFlavor=lld

dotnet --version is "8.0.100-rc.1.23402.12"

Other information

I admit that I'm not certain this is a problem with the NativeAOT compiler, since I don't know how to look into a binary and say "Aha, here is the incorrect instruction."

Author: chrisoverzero
Assignees: -
Labels:

area-NativeAOT-coreclr

Milestone: -

@chrisoverzero chrisoverzero changed the title NativeAOT on Graviton2 As armv8.2-a Fails NativeAOT on Graviton2 As armv8.2-a Fails to Execute Aug 3, 2023
@EgorBo
Copy link
Member

EgorBo commented Aug 3, 2023

I'll try on my M1, does it work if you don't specifcy any ISA at all? armv8.2-a should include neon (always enabled), lse (atomicts, e.g. all Interlocked.* APIs) and crc + rdma, the last two are unlikely to be hit?

@chrisoverzero
Copy link
Author

[…] does it work if you don't specifcy any ISA at all?

Do you mean leaving off IlcInstructionSet? That does compile and run, yes. I'll update "Workarounds" to be clearer about that.

@am11
Copy link
Member

am11 commented Aug 3, 2023

It is working on M1 with RC1 daily build. You can also specify native (since you are on new enough SDK).

cpufeatures.{c,h}, cpuid.h and minipalconfig.h from https://github.com/dotnet/runtime/tree/main/src/native/minipal participate in CPU feature detection and they can be complied independently with this kind of a glue code:

#define TARGET_UNIX
#define TARGET_ARM64

#include <stdio.h>
#include <cpufeatures.h>

int main(void) {
  int actualFeatures = minipal_getcpufeatures();
  printf("actual features: %d\n", actualFeatures);

  // TODO: unpack and print individual feature flags defined in cpufeatures.h

  return 0;
}

@chrisoverzero
Copy link
Author

You can also specify native […]

Even if I'm not building on ARM?

@am11
Copy link
Member

am11 commented Aug 3, 2023

Ah cross compilation 🙈
Maybe try using this build container without cross: #88971 (comment).

@am11
Copy link
Member

am11 commented Aug 3, 2023

I admit that I'm not certain this is a problem with the NativeAOT compiler, since I don't know how to look into a binary and say "Aha, here is the incorrect instruction."

@EgorBo, @MichalStrehovsky, what do you think about dumping all the supported instruction sets in this error path:

if ((g_cpuFeatures & g_requiredCpuFeatures) != g_requiredCpuFeatures)
{
PalPrintFatalError("\nThe required instruction sets are not supported by the current CPU.\n");
RhFailFast();
}

"Supported instruction sets are: ..."

@chrisoverzero
Copy link
Author

chrisoverzero commented Aug 3, 2023

Setting cross-compilation aside, this also does not work when compiled and run on the Graviton2 machine I allocated for testing:

sh-5.2$ ./src/bin/Release/net8.0/linux-arm64/publish/bootstrap 

The required instruction sets are not supported by the current CPU.
Aborted (core dumped)

What can I do to get you more information?

ETA: Oh, I just remembered native. Trying that, now that I'm on Graviton2 natively, does work. So there's something defined in ilc's "armv8.2-a" which shouldn't be present? Or maybe Amazon is mistaken about the specs of Graviton2? Either feels unlikely, but I'll go with whatever.

Or the feature detection is missing features?

sh-5.2$ grep ^Features /proc/cpuinfo
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs

@MichalPetryka
Copy link
Contributor

all the supported instruction sets

Listing required ones too would make sense.

@chrisoverzero
Copy link
Author

chrisoverzero commented Aug 3, 2023

Trying the supported options one at a time, the ones which cause failures are:

  • rcpc
  • dotprod
  • rdma

…but I think these are covered in the features line from my previous comment, as "lrcpc", "asimddp", and "asimdrdm". The names are different, but these look like they correspond to HWCAP_LRCPC, HWCAP_ASIMDDP, and HWCAP_ASIMDRDM from here.

I'm well over my head on this, so I don't know where those are #defined, or how they're used or detected or anything, really. I do not know what the next steps are.

@MichalPetryka
Copy link
Contributor

Could you run some software that can list out supported CPU features on the machine (like for example https://github.com/google/cpu_features#quickstart)?

@chrisoverzero
Copy link
Author

The output of list_cpu_features:

{
  "arch": "aarch64",
  "implementer": 65,
  "variant": 3,
  "part": 3340,
  "revision": 1,
  "flags": ["aes", "asimd", "asimddp", "asimdhp", "asimdrdm", "atomics", "cpuid", "crc32", "dcpop", "evtstrm", "fp", "fphp", "lrcpc", "pmull", "sha1", "sha2", "ssbs"]
}

@MichalStrehovsky
Copy link
Member

ETA: Oh, I just remembered native. Trying that, now that I'm on Graviton2 natively, does work.

If you're native on Graviton right now, try adding <ItemGroup><IlcArg Include="--verbose" /></ItemGroup> with native.

It will dump the list of instruction sets it expanded native into at compile time. We can crosscheck with what's expected.

@MichalStrehovsky
Copy link
Member

@EgorBo, @MichalStrehovsky, what do you think about dumping all the supported instruction sets in this error path:

Yes, I wanted to change this to PalPrintFatalError("\nThe required instruction sets are not supported by the current CPU (Guru Meditation %d %d).\n", g_cpuFeatures, g_requiredCpuFeatures);. We should do that.

@chrisoverzero
Copy link
Author

chrisoverzero commented Aug 3, 2023

If you're native on Graviton right now, try adding <ItemGroup><IlcArg Include="--verbose" /></ItemGroup> with native.

Done.

ILC: The 'native' instruction set expanded to +aes,+crc,+lse,+neon,+sha1,+sha2

MichalStrehovsky added a commit that referenced this issue Aug 4, 2023
Testing a theory for #89937
@MichalStrehovsky
Copy link
Member

I did a quick check in #89988 and this is a product-build-time issue. I think this is a general product issue, not Native AOT specific. If this #define is not set, we would not generate a check for this at all.

/__w/1/s/src/native/minipal/cpufeatures.c:359:2: error: No HWCAP_LRCPC
#error No HWCAP_LRCPC
 ^
1 error generated.

@MichalStrehovsky
Copy link
Member

I'll try on my M1

It would likely not reproduce on M1 because Apple has its own way to detect these. This would be Linux specific.

jkotas added a commit to jkotas/runtime that referenced this issue Aug 4, 2023
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 4, 2023
@ghost ghost removed in-pr There is an active PR which will close this issue when it is merged untriaged New issue has not been triaged by the area owner labels Aug 4, 2023
@chrisoverzero
Copy link
Author

Fantastic. I look forward to this landing in the daily build. Thanks to all for your efforts.

@chrisoverzero
Copy link
Author

chrisoverzero commented Aug 7, 2023

I confirm that with SDK version 8.0.100-rc.1.23407.1 and:

<PackageReference Include="Microsoft.DotNet.ILCompiler" Version="8.0.0-rc.1.23406.6" />
<PackageReference Include="runtime.linux-x64.Microsoft.DotNet.ILCompiler" Version="8.0.0-rc.1.23406.6" />

…the problem is resolved. I don't know what daily build will incorporate these versions into the SDK, but I can keep working with this. Thanks again.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants