bicep build fails when run from within amd64/linux docker container on M1 Mac

[ellismg]

Bicep version
Bicep CLI version 0.17.1 (d423d61)

Describe the bug
When trying to run bicep build <path-to-a-bicep-file> --stdout from within a linux docker container running on an M1 mac, there's a null reference exception during what appears to be initialization of bicep itself.

root ➜ ~ $ ./bicep build ./todo-nodejs-mongo/infra/main.bicep 
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Text.RegularExpressions.Regex.CreateRunner()
   at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode, Int32, String, Int32, Int32, Int32)
   at System.Text.RegularExpressions.Regex.Match(String)
   at Bicep.Core.Resources.ResourceTypeReference.TryParse(String resourceType)
   at Bicep.Core.Resources.ResourceTypeReference.Parse(String resourceType)
   at Bicep.Core.TypeSystem.Az.AzResourceTypeLoader.<>c.<.ctor>b__4_0(KeyValuePair`2 kvp)
   at System.Collections.Immutable.ImmutableDictionary.<>c__DisplayClass9_0`3.<ToImmutableDictionary>b__0(TSource)
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1, MutationInput, KeyCollisionBehavior )
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1, Boolean)
   at System.Collections.Immutable.ImmutableDictionary`2.AddRange(IEnumerable`1 )
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1, Func`2, Func`2, IEqualityComparer`1 , IEqualityComparer`1 )
   at System.Collections.Immutable.ImmutableDictionary.ToImmutableDictionary[TSource,TKey,TValue](IEnumerable`1, Func`2, Func`2)
   at Bicep.Core.TypeSystem.Az.AzResourceTypeLoader..ctor()
   at System.RuntimeMethodHandle.InvokeMethod(Object, Void**, Signature, Boolean)
   at System.Reflection.ConstructorInvoker.Invoke(Object, IntPtr*, BindingFlags)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags, Binder, Object[], CultureInfo)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite, RuntimeResolverContext)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite, TArgument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey, Func`2)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type, ServiceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider, Type)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider)
   at Bicep.Cli.Program.RunAsync(String[] args)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)
Aborted
root ➜ ~ $

Strangely the null reference looks to be coming from within the .NET Core Runtime itself. Also, other, simpler gestures, eg bicep --version work fine.

To Reproduce
Steps to reproduce the behavior:

The simplest way I've found to reproduce this behavior is to do the following:

1. Install Docker Desktop on an ARM Mac
2. Do docker run -it --platform linux/amd64 mcr.microsoft.com/devcontainers/base:bullseye to launch and amd64 container running on Mac.
3. Inside the container, download bicep via curl: curl -OL https://github.com/Azure/bicep/releases/download/v0.17.1/bicep-linux-x64
4. chmod +x bicep-linux-x64
5. touch main.bicep (it seems like it crashes before it even tries to read the file, so I don't think you need anything in main.bicep)
6. ./bicep-linux-x64 build ./main.bicep --stdout

Additional context
Instead of the stack trace I showed above, you might instead see this error:

qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

I was getting this before I told docker to use Rosetta for x86/amd64 emulation on Apple Silicon (something you can configure with the Features In Development page in the Settings part of the Docker Desktop app)

It's possible this is related to #10245, but that was focused on the language server and this is related to just running the bicep CLI directly.

From the stack trace, it's not obvious that this is a bicep issue and not some underlying runtime issue and dotnet/runtime#78340 shows a similar shape of issue with regex weirdness on amd64 when run via an emulator, but I figured I'd open an issue here for folks that stumble upon this via bicep.

I also saw this in 0.16.1 and 0.16.2, so it's not a recent regression.

I suspect that the issue may be tied to the move to using .NET 7. If I pick up v0.14.85, which from my reading of the Git history was the last version built using .NET 6.0, things work fine. If I move to v0.15.31, then I start to see this failure (and that looks like the first official release after adopting .NET 7.0)

[stephaniezyen]

Can you please provide a reproducible bicep file so we can repro the error on our end?

[ghost]

Hi ellismg, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! 😄 🦾

[anthony-c-martin]

This looks like a similar root cause to #10245. Would you mind trying to upgrade qemu (as described under "Known possible causes") and confirming whether or not that helps?

[ellismg]

@anthony-c-martin

I attempted to update the qemu version by following the docker run commands in the linked issues, but it provided no help. I also attempted to update to the latest Docker Desktop (4.20.1) and that didn't help the issue at all. However, I noticed that in the Docker Desktop changelog they mentioned that they reverted from 7.0.0 to 6.2.0 of qemu in 4.14.0

Note - I'm not 100% sure the docker commands that I ran from issues linked off of #10245 ended up actually upgrading the qemu version that Docker Desktop uses for emulation, because I don't have a great mental model of where the qemu that is being used for emulation is installed (either on the host or inside the container, and all the commands I have found linked from #10245 feel to me like they are upgrading things inside the container).

To get the above stack trace I shared, I actually enabled the "Use Rosetta for x86/amd64 emulation on Apple Silicon" check box under the "Features In Development" tab of the setting pane in Docker Desktop on my mac. In this case, I don't believe that qemu is even in the mix and yet I still see issues. My reading of #10245 makes it seem like @kurt-mueller-osumc was facing the same issue and also using Rosetta, not qemu for emulation.

Happy to try more things, but I am unfortunately not a Docker Desktop on macOS expert.

To answer, @stephaniezyen's question, as I called out in my repo steps, this crash happens even on a completely blank main.bicep (created via touch). But, that sort of makes sense since from my reading of the stack trace the crash is happening very early in bicep's startup, as it is resolving services from its IoC container.

I did verify that when running the same steps but using a native arm container things work as expected.

I do expect that the root cause here is some issue in .NET 7, not bicep, since things were working fine in v0.14.85, the last official release compiled with .NET 6 and then broke in the first release built using .NET 7.0 (i.e v0.15.31).

If there's some easy way to build HEAD using .NET 6, I'm happy to try that, but looking at #8963 where you adopted .NET 7.0 it was not obvious to me that there would be a quick way for me to build HEAD with .NET 6 when it landed, and I'm sure that since then you've adopted more .NET 7.0 features.

[anthony-c-martin]

If there's some easy way to build HEAD using .NET 6, I'm happy to try that, but looking at #8963 where you adopted .NET 7.0 it was not obvious to me that there would be a quick way for me to build HEAD with .NET 6 when it landed, and I'm sure that since then you've adopted more .NET 7.0 features.

I think switching the current HEAD back to .NET 6 may be tricky, but a quick way to prove your hypothesis could be to try and repro with the commit immediately before and after #8963 was merged.

I'm not sure how comfortable you are with C#, but another possibility for trying to get a minimal repro could be creating a .NET 7 project with something like the following (basically a copy+paste of the code that's throwing): https://dotnetfiddle.net/OAWLxC

using System.Text.RegularExpressions;
					
public class Program
{
    private const string TypeSegmentPattern = "[a-z0-9][a-z0-9-.]*";
    private const string VersionPattern = "[a-z0-9][a-z0-9-]+";

    private const RegexOptions PatternRegexOptions = RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture | RegexOptions.Compiled | RegexOptions.CultureInvariant;
    private static readonly Regex ResourceTypePattern = new Regex(@$"^(?<type>{TypeSegmentPattern})(/(?<type>{TypeSegmentPattern}))*(@(?<version>{VersionPattern}))?$", PatternRegexOptions);
	
    public static void Main()
    {
        ResourceTypePattern.Match("Microsoft.Compute/virtualMachines@2022-01-01");
    }
}

[microsoft-github-policy-service]

Hi @ellismg, this issue has been marked as stale because it was labeled as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 3 days of this comment. Thanks for contributing to bicep! 😄 🦾