mono-symbolicate errors (on iOS only): "Warning: MVID directory does not exist: ..."

[kingces95]

Description

We have several Xamarin.Forms apps that are instrumented with the HockeyApp SDK to send crash reports. The build environment is Visual Studio 2017 on Windows 10 with local builds for android and remote builds to a Mac for iOS builds. We've recently gotten the android versions to build in such a way that they produce the needed .mSym files and have been successful in using mono-symbolicate to view the filenames and line numbers from stack traces sent to HockeyApp for production builds. For iOS, on the build server in the "~/Library/Caches/Xamarin/mtbs/builds/<ourApp>/<someGuid>/bin/iPhone/Ad-Hoc/<ourApp>.mSYM" directory, there are a bunch of sub-directories. But when we run mono-symbolicate, every sub-directory that it's looking for isn't found. So each line of the input stack trace produces an error "Warning: MVID directory does not exist: <someSubDirectory>". Example error line:

Warning: MVID directory does not exist: /Users/derek.wheeler/Library/Caches/Xamarin/mtbs/builds/FieldSheet.iOS/b5f6c442719e634bde4b19906949135b/bin/iPhone/Ad-Hoc/FieldSheet.app.mSYM/1861ef8a8a87467c9c37202d50ff68b2 at FieldSheet.Pages.ConnectionSettingsPage.TestAppCrash () <0x100925550 + 0x0001f> in <1861ef8a8a87467c9c37202d50ff68b2#d25ebbac98c60d1ed35df16528c08be6>:0

Of note too, we get a build warning that sounds like a problem producing the mSYM stuff:

1>C:\Program Files (x86)\MSBuild\Xamarin\iOS\Xamarin.iOS.Common.targets(777,3): warning : Aot files could not be copied to the destination directory /Users/derek.wheeler/Library/Caches/Xamarin/mtbs/builds/FieldSheet.iOS/b5f6c442719e634bde4b19906949135b/obj/iPhone/Ad-Hoc/mtouch-cache/64/Build/Msym/Msym/tmp: Could not start process.

We've tried a bunch of different build options but couldn't find a definitive source for this so we were basically just guessing. The latest version from the iOS csproj:

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Ad-Hoc|iPhone' "> <DebugType>pdbonly</DebugType> <Optimize>True</Optimize> <OutputPath>bin\iPhone\Ad-Hoc</OutputPath> <ErrorReport>prompt</ErrorReport> <WarningLevel>4</WarningLevel> <ConsolePause>False</ConsolePause> <MtouchArch>ARMv7, ARM64</MtouchArch> <BuildIpa>True</BuildIpa> <CodesignKey>...redacted...</CodesignKey> <CodesignEntitlements>Entitlements.plist</CodesignEntitlements> <GenerateSerializationAssemblies>Off</GenerateSerializationAssemblies> <DebugSymbols>True</DebugSymbols> <MonoSymbolArchive>True</MonoSymbolArchive> <MtouchEnableSGenConc>false</MtouchEnableSGenConc> </PropertyGroup>

Steps to Reproduce

see Description

Expected Behavior

mono-symbolicate should be able to show filenames and line numbers for Ad-Hoc iOS builds

Actual Behavior

mono-symbolicate shows errors

Basic Information

• Version with issue: Xamarin=4.8.0.760
• Last known good version: This is the first time we're trying this.
• IDE: Visual Studio Enterprise 2015, Version 14.0.25425.01 Update 3
• Platform Target Frameworks:
  • iOS: Xamarin.iOS and Xamarin.Mac SDK = 11.6.1.4 (db807ec)

Screenshots

Reproduction Link

VS bug #588941, VS bug #591266

[kingces95]

Copied from xamarin/Xamarin.Forms#1925

[mandel-macaque]

This looks like the mono-symbolicate tool that is used to copy the .Msym files from the tmp dir to the final directory is not found.

Can you please check the location of 'mono-symbolicate' in the path of the building machine? The mtouch process uses the tool to copy the files and looking at that error, it seems that the tool is not found.

[derekwheeler-tylertech]

On the build machine:

$ find /Library/Frameworks/ -name mono-symbolicate

/Library/Frameworks//Xamarin.Android.framework/Versions/8.0.0-33/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.0.2-42/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.4.5-1/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.2.0-7/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.3.0-6/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/8.0.2-1/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.1.0-43/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.0.2-37/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.0.1-3/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.3.1-2/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/7.0.0-18/bin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/8.1.0-25/lib/xamarin.android/xbuild/Xamarin/Android/Darwin/mono-symbolicate
/Library/Frameworks//Xamarin.Android.framework/Versions/8.1.0-24/lib/xamarin.android/xbuild/Xamarin/Android/Darwin/mono-symbolicate
/Library/Frameworks//Mono.framework/Versions/5.4.1/bin/mono-symbolicate
/Library/Frameworks//Mono.framework/Versions/5.4.1/Commands/mono-symbolicate
/Library/Frameworks//Mono.framework/Versions/4.8.1/bin/mono-symbolicate
/Library/Frameworks//Mono.framework/Versions/4.8.1/Commands/mono-symbolicate

[mandel-macaque]

Ok, I need to check with the VS on Windows people to try and reproduce it. It is strange that the execution says it was not found.

[derekwheeler-tylertech]

I can confirm that it's not finding mono-symbolicate. I added "-v -v -v -v" to the "Additional mtouch arguments" and get this in the build log now:

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Xamarin\iOS\Xamarin.iOS.Common.targets(777,3): warning : Aot files could not be copied to the destination directory /Users/derek.wheeler/Library/Caches/Xamarin/mtbs/builds/FieldSheet.iOS/b5f6c442719e634bde4b19906949135b/obj/iPhone/Ad-Hoc/mtouch-cache/64/Build/Msym/Msym/tmp: Could not start process.
1> --- inner exception
1> System.ComponentModel.Win32Exception (0x80004005): ApplicationName='mono-symbolicate', CommandLine='store-symbols "/Users/derek.wheeler/Library/Caches/Xamarin/mtbs/builds/FieldSheet.iOS/b5f6c442719e634bde4b19906949135b/bin/iPhone/Ad-Hoc/FieldSheet.app.mSYM" "/Users/derek.wheeler/Library/Caches/Xamarin/mtbs/builds/FieldSheet.iOS/b5f6c442719e634bde4b19906949135b/obj/iPhone/Ad-Hoc/mtouch-cache/64/Build/Msym/Msym/tmp"', CurrentDirectory='', Native error= Cannot find the specified file
1> at System.Diagnostics.Process.StartWithCreateProcess (System.Diagnostics.ProcessStartInfo startInfo) [0x002dc] in /Users/builder/data/lanes/5533/mono-mac-sdk/external/bockbuild/builds/mono-x64/mcs/class/System/System.Diagnostics/Process.cs:775
1> at System.Diagnostics.Process.Start () [0x0003a] in /Users/builder/data/lanes/5533/mono-mac-sdk/external/bockbuild/builds/mono-x64/mcs/class/referencesource/System/services/monitoring/system/diagnosticts/Process.cs:2005
1> at (wrapper remoting-invoke-with-check) System.Diagnostics.Process.Start()
1> at Xamarin.Bundler.Application.CopyMSymData (System.String src, System.String dest) [0x00077] in /Users/builder/data/lanes/5808/1c6f300e/source/xamarin-macios/tools/mtouch/Application.cs:1506

If I just log into the Mac and look at my path from a terminal window it looks like it should find it:

$> echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Xamarin Workbooks.app/Contents/SharedSupport/path-bin
$> ls -l /Library/Frameworks/Mono.framework/Versions/Current/Commands/mono-symbolicate
lrwxr-xr-x 1 root admin 60 Mar 14 14:01 /Library/Frameworks/Mono.framework/Versions/Current/Commands/mono-symbolicate -> /Library/Frameworks/Mono.framework/Commands/mono-symbolicate
$> ls -l /Library/Frameworks/Mono.framework/Commands/mono-symbolicate
-rwxr-xr-x 1 root admin 177 Feb 23 21:19 /Library/Frameworks/Mono.framework/Commands/mono-symbolicate
$> mono-symbolicate -h
Usage: symbolicate [options] <msym dir> <input file>
...snip...

[derekwheeler-tylertech]

I looked at CopyMSymData() in Application.cs and noted it's creating a Process with StartInfo.UseShellExecute set to false so the shell's path won't be searched. I figured out where the build agent runs from and copied mono-symbolicate there:

$> cp /Library/Frameworks/Mono.framework//Commands/mono-symbolicate /Users/derek.wheeler/Library/Caches/Xamarin/XMA/Agents/Build/4.8.0.760/

...and now the build doesn't get the error. I confirmed I could run mono-symbolicate against a stack dump from HockeyApp (or AppCenter which we're now beta testing) and I can now see file names and line numbers:

$> cat > /tmp/stackDump.txt
at Tyler.Xamarin.Pages.ConnectionSettingsPage.CrashTheApp () <0x10220bbc0 + 0x0001f> in <1e036c3e606542e4af7679b5e9484746#b91dd0ed214840933a2978672fd2ea22>:0
at Tyler.Xamarin.Pages.ConnectionSettingsPage+<urlEntry_Unfocused>d__15.MoveNext () <0x102218d00 + 0x001d3> in <1e036c3e606542e4af7679b5e9484746#b91dd0ed214840933a2978672fd2ea22>:0
$> /Library/Frameworks/Mono.framework//Commands/mono-symbolicate ~/Library/Caches/Xamarin/mtbs/builds/FieldSheet.iOS/b5f6c442719e634bde4b19906949135b/bin/iPhone/Ad-Hoc/FieldSheet.app.mSYM/ /tmp/stackDump.txt
at Tyler.Xamarin.Pages.ConnectionSettingsPage.CrashTheApp () <0x10220bbc0 + 0x0001f> in C:\src\Tyler.XamarinProduction\Tyler.Xamarin\Pages\ConnectionSettingsPage.xaml.cs:72
at Tyler.Xamarin.Pages.ConnectionSettingsPage+<urlEntry_Unfocused>d__15.MoveNext () <0x102218d00 + 0x001d3> in C:\src\Tyler.XamarinProduction\Tyler.Xamarin\Pages\ConnectionSettingsPage.xaml.cs:83

So this is obviously a work around solution but the real fix doesn't look too hard.

[mandel-macaque]

I have tested this with master, and without the change I get the following in the build (on a mac):

MandelsWorkMachine:TestSymv mandel$ tree . -f | grep Msym
    │       │   │   ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym
    │       │   │   │   ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/33cce96150953d11041c418348e2247f
    │       │   │   │   │   └── ./obj/iPhone/Release/mtouch-cache/Build/Msym/33cce96150953d11041c418348e2247f/Xamarin.iOS.dll.msym
    │       │   │   │   ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/601c76e952739021f35b68593355fb42
    │       │   │   │   │   └── ./obj/iPhone/Release/mtouch-cache/Build/Msym/601c76e952739021f35b68593355fb42/mscorlib.dll.msym
    │       │   │   │   ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym
    │       │   │   │   │   └── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/TestSymv.aotdata.arm64
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/TestSymv.exe
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/TestSymv.pdb
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/Xamarin.iOS.aotdata.arm64
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/Xamarin.iOS.dll
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/Xamarin.iOS.pdb
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/mscorlib.aotdata.arm64
    │       │   │   │   │       ├── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/mscorlib.dll
    │       │   │   │   │       └── ./obj/iPhone/Release/mtouch-cache/Build/Msym/Msym/tmp/mscorlib.pdb
    │       │   │   │   └── ./obj/iPhone/Release/mtouch-cache/Build/Msym/c7cac9e90b862cba44c96f1d9202ccba
    │       │   │   │       └── ./obj/iPhone/Release/mtouch-cache/Build/Msym/c7cac9e90b862cba44c96f1d9202ccba/TestSymv.exe.msym
MandelsWorkMachine:TestSymv mandel$

But that might be due to my setup. @derekwheeler-tylertech I'll update the CopyMSym to use the shell, retest with master and will confirm if it works, if that is the case, I'll do a back port to 15-7 with the change and will pose here a package for you to test.

[mandel-macaque]

Closing after landing #3781

[mandel-macaque]

A back port of the fix will be available in 15-6.