CreateAppHost fails on file systems that cannot mmap with MAP_SHARED

[loop-evgeny]

I created a new console application in VS 2017 using .NET Core 2.1. Published it as a self-contained linux-x64 app without any changes and tried to run it on an Ubuntu 18.04 VM. Got this error:

 ./LinuxTest 
This executable is not bound to a managed DLL to execute. The binding value is: 'c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2'
A fatal error was encountered. This executable was not bound to load a managed DLL.

c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2 is apparently the SHA256 hash of "foobar", so whatever the real problem is, this is a very weird error message, too!

LinuxTest.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

</Project>

Program.cs:

using System;

namespace LinuxTest
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

Resulting self-contained binary: LinuxTest.zip

If I install dotnet-runtime-2.1 on the Linux VM and run dotnet LinuxTest.dll it works.

[livarcocc]

@peterhuene can you take a look when you have a chance?

[peterhuene]

Hi @loop-evgeny. Thanks for reporting this to us. It appears that a non-modified apphost executable got copied to your publish output somehow. The c3ab8ff13720e8ad9047dd39466b3c8974e592c2fa383d4a3960714caef0c4f2 value is a placeholder in the generic apphost that should get replaced with the name of your dll to load.

I'll investigate and report back with my findings.

[peterhuene]

@loop-evgeny I'm unable to reproduce this with SDK 2.1.504.

Could you share with me your exact repro steps and the dotnet --info output?

The steps I took to reproduce:

1. Create a .NET Core Console application named LinuxTest in VS 2017.
2. Since I have a 2.2 SDK installed, I had to take the following steps to lock the SDK to 2.1:
   a. Closed the solution.
   b. Ran dotnet new global --sdk-version 2.1.504 in the solution directory.
   c. Edited the project file to retarget the project at netcoreapp2.1.
   d. Opened the solution.
3. Published the project with the following publish properties:
   a. Deployment Mode set to Self-contained.
   b. Target Runtime set to linux-x64.
   c. Target Location set to bin\Release\netcoreapp2.1\linux-x64\publish.
4. Checked that the placeholder was correctly replaced with the name of my dll with:

cd bin\Release\netcoreapp2.1\linux-x64\publish
xxd -u LinuxTest | grep 000193e0

The output was the expected 000193e0: 4C69 6E75 7854 6573 742E 646C 6C00 0000 LinuxTest.dll.... To reproduce your issue, this should have printed the placeholder value instead.

I was also unable to reproduce it with a 2.2 SDK targeting netcoreapp2.2.

Thanks!

[loop-evgeny]

OK, so the repro is a little more complicated! I actually have 2 Linux VM on my Windows host: a Linux Mint 19 one with the .NET Core SDK installed and an Ubuntu 18.04 one, without .NET Core installed. The Windows host also has .NET Core SDK installed. (dotnet --info for both attached.)

I initially tried to publish on the Mint VM, got an error, re-ran the same command and it appeared to work:

evgeny@emmint:/media/sf_work/Play/LinuxTest$ dotnet publish -r linux-x64 -o publish-linux-from-mint
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for /media/sf_work/Play/LinuxTest/LinuxTest.csproj...
  Generating MSBuild file /media/sf_work/Play/LinuxTest/obj/LinuxTest.csproj.nuget.g.props.
  Generating MSBuild file /media/sf_work/Play/LinuxTest/obj/LinuxTest.csproj.nuget.g.targets.
  Restore completed in 476.51 ms for /media/sf_work/Play/LinuxTest/LinuxTest.csproj.
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018: The "EmbedAppNameInHost" task failed unexpectedly. [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018: System.IO.IOException: Invalid argument [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 requestedOffset, Int64 requestedSize) [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewAccessor(Int64 offset, Int64 size, MemoryMappedFileAccess access) [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewAccessor() [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at Microsoft.NET.Build.Tasks.AppHost.Create(String appHostSourceFilePath, String appHostDestinationFilePath, String appBinaryFilePath, Boolean overwriteExisting) [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at Microsoft.NET.Build.Tasks.EmbedAppNameInHost.ExecuteCore() [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at Microsoft.NET.Build.Tasks.TaskBase.Execute() [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]
/usr/share/dotnet/sdk/2.1.503/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(308,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/media/sf_work/Play/LinuxTest/LinuxTest.csproj]

evgeny@emmint:/media/sf_work/Play/LinuxTest$ dotnet publish -r linux-x64 -o publish-linux-from-mint
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 64.45 ms for /media/sf_work/Play/LinuxTest/LinuxTest.csproj.
  LinuxTest -> /media/sf_work/Play/LinuxTest/bin/Debug/netcoreapp2.1/linux-x64/LinuxTest.dll
  LinuxTest -> /media/sf_work/Play/LinuxTest/publish-linux-from-mint/

(/media/sf_work here is the host NTFS directory C:\W ork mounted inside the VM)

However, when I ran the output I got the error in the original issue.

I then tried to publish on the Windows host:

C:\W ork\Play\LinuxTest>dotnet publish -r linux-x64 -o publish-linux-from-windows-cmd
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\W ork\Play\LinuxTest\LinuxTest.csproj...
  Generating MSBuild file C:\W ork\Play\LinuxTest\obj\LinuxTest.csproj.nuget.g.props.
  Generating MSBuild file C:\W ork\Play\LinuxTest\obj\LinuxTest.csproj.nuget.g.targets.
  Restore completed in 233.6 ms for C:\W ork\Play\LinuxTest\LinuxTest.csproj.
  LinuxTest -> C:\W ork\Play\LinuxTest\bin\Debug\netcoreapp2.1\linux-x64\LinuxTest.dll
  LinuxTest -> C:\W ork\Play\LinuxTest\publish-linux-from-windows-cmd\

This appears to succeed, but the resulting executable gives the same error (even though I'm publishing to a different directory!) Even if I run dotnet clean on the Windows host and try to publish again it keeps happening. Only if I delete the obj directory and publish again does it work!

So it seems like there is a number of bugs here:

1. The original The "EmbedAppNameInHost" task failed unexpectedly error
2. The fact that running publish a second time in the Mint VM gives no error, but produces a non-working EXE.
3. That this affects future attempts to publish, even to a different directory.
4. That even dotnet clean does not actually cause dotnet publish to start from a clean state, as the user would expect.

dotnet-info-Linux-Mint.txt
dotnet-info-Windows.txt

[peterhuene]

Hi @loop-evgeny. Thanks for the detailed information.

It looks like the mmap syscall failed when attempting to customize the apphost executable.

Do you know how /media/sf_work is mounted? If I had to guess since you mentioned this being a VM, it's being mounted via a network share, probably automatically by your VM software.

From the .NET Core Framework sources, it is using MAP_SHARED. I know there are limitations of file system drivers to support MAP_SHARED where they cannot guarantee coherency; networked file systems would inherently be one such case.

We might be able to fix this by either asking the framework to support specifying whether or not we want a private mapping; I can also investigate alternative methods to mapping the file since the original apphost file is not very big.

Regardless of that fix, the SDK should be deleting the output file in obj whenever we fail to modify the apphost to avoid leaving a useless apphost in the output. When you publish a second time, the file already exists and is "up-to-date" with respect to its inputs. Thus the second publish operation performed on Windows doesn't "fix" the apphost.

You should be able to work around the issue for now by deleting LinuxTest under obj/ before doing the publish on Windows or by touching one of the source files to cause the application dll to be rebuilt, and then running it under Linux. Alternatively, if you copied the sources to another mount point on the Linux VM that would support MAP_SHARED (e.g. $HOME/LinuxTest), you should be able to publish and run the application without issue.

[loop-evgeny]

/media/sf_work is not mounted as a network share, it's using the VirtualBox guest driver to mount a filesystem of type vboxsf via /etc/fstab.

[peterhuene]

Looks like the same root cause though: MAP_SHARED is not supported. However, it appears that someone is working on adding support, but I have no context as to what the status of this is getting into vboxsf.

[peterhuene]

@jwrdegoede out of curiosity regarding your work of supporting MAP_SHARED in vboxsf, is that something that has been or will be merged to upstream?

[loop-evgeny]

Seems like it would be safer to just use regular file access - as you say, there will be other filesystems that don't support memory-mapped files, even if vboxsf eventually does.

[peterhuene]

That will likely be my fix if I have to make one. There aren't that many reads we have to do to search through the file to find what we're looking for, so any performance benefit of memory mapping the file is likely negligible, even if mapping the file does reduce the complexity of the search code since it can just treat it as linear memory.

[jwrdegoede]

I'm working on getting a cleaned up version of vboxsf into the mainline linux kernel and that version will have the commit you point to. This project has been on halt for a while, but I actually hope to resume work on it soonish.

[peterhuene]

@jwrdegoede thanks for following up to my random question! I look forward to seeing it in mainline.

[nguerrera]

Side note: the original code did readallbytes/search and replace/writeallbytes. Memory map was just the straight forward way to stop putting several hundred KB on LOH.

I argued to the original implementers of the feature that the apphost should come with a descriptor that says where the offset is, and we could just seek to it, but they felt this was too much trouble.

[peterhuene]

Re-opening as dotnet/sdk#3427 only fixed part of the issue: deleting the output apphost upon failure so it isn't left in a semi-customized state.

[peterhuene]

@livarcocc can I get a bar check on this?

I think to properly address this issue we would have to stop using memory mapped I/O in the HostWriter (now in core-setup) and either pollute the LOH with the apphost's contents or use an incremental search for the path marker.

Both seem like pretty risky changes for 3.0 P9. This affects users that are building on a filesystem that can't support shared memory mapped I/O.

[nguerrera]

https://github.com/dotnet/core-setup/issues/8764 tracks having the apphost pack supply the offset so that the SDK won't need to search at all. This would fix this and be more efficient.

[wli3]

Updated title per @swaroop-sridhar request

[nguerrera]

I had already renamed it. I think my title is more accurate so I renamed it back while keeping CreateAppHost. This is not limited to scenarios involving mapping a windows drive, it can be on any file system on unix that does not allow MAP_SHARED.

I do think https://github.com/dotnet/core-setup/issues/8764 is the best solution.

[bustroker]

Is this fixed already? I can't run a hello world dotnet new webapi with this error, using vagrant with ubuntu... any way to solve this at all?
Thx

"error MSB4018: The "CreateAppHost" task failed unexpectedly."

[btull89]

I'm still seeing these issues as of NET Core 3.1.201 inside of Virtual Box's synced folders.
My host OS is Windows 10 and my guest OS is CentOS.
Vagrant/VirtualBox Synced Folders

On the guest OS inside of a synced folder, I run dotnet new webapi then run dotnet build and it fails inside of the synced folder with the error: The "CreateAppHost" task failed unexpectedly. The same command works outside of the synced folder inside the same environment.

Update: I updated VirtualBox from 6.0.0 to 6.1.4 and dotnet build is now working in the synced folder.

[Atihinen]

I get same error with:
$ dotnet --version
3.1.201
$ VBoxManage --version
6.1.6r137129
$ vagrant --version
Vagrant 2.2.7

And I'm running on host macOS and ubuntu 18.04 as guest.

[kaBit76]

Hi @peterhuene @nguerrera

Any solution/update for this? as on Linux 19.10 issue still exists.

[nguerrera]

Adding @dsplaisted as neither Peter nor I work on this project anymore.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. Please help me learn by adding exactly one area label.

[dsplaisted]

@swaroop-sridhar @jeffschwMSFT We think this would be a fix to the CreateAppHost code which lives in the runtime repo. So I've moved it to dotnet/runtime. Please triage. Thanks!

[ghost]

Tagging subscribers to this area: @vitek-karas, @swaroop-sridhar
Notify danmosemsft if you want to be subscribed.

[swaroop-sridhar]

Sure @dsplaisted.
This issue depends on implementing #3829, which would be a good thing to implement in .net 5. Thanks.

[loop-evgeny]

Why does it depend on that (providing the offset)? Can't the search and replace still be done using normal file read/write APIs (instead of a memory-mapped file)?

[swaroop-sridhar]

@loop-evgeny "depends" was probably too strong a constraint.
Fixing #3829 would eliminate the need to search the pattern, and is a better fix than using search/replace via read/write. Please see Nick's comments above.

,

[agocke]

This should be fixed as we no longer use MAP_SHARED to write the apphost