Add local build for System.Globalization shim (#72896)

* Add local build for System.Globalization shim

Bundle the System.Globalization.Native shim sources into the native AOT toolchain package and compile them automatically using C/C++ compiler on the target machine.

Compiling of the shim sources using C/C++ compiler is done during dotnet publish as follows:
  dotnet publish  /p:LocalSystemGlobalizationNative=1 -bl -r linux-x64

This change is 'opt-in', it is enabled only by using /p:LocalSystemGlobalizationNative=1

* Incorporate review feedback

* Copy shim source code into users Obj directory at IntermediateOutputPath and compile
** $ ls -al obj/Debug/net7.0/linux-x64/shim_source_code/libs/System.Globalization.Native/build/libSystem.Globalization.Native-Static.a

* Add shell script for readability

* Add copyright headers

* Fix IlCompilerPackagePath

Add change that was part of previous commit

* Address review feedback

* Include existing pal_icushim_static.c by using STATIC_ICU define
* Use IlcSdkPath to copy shim source code

* Incorporate review feedback

* Remove USE_ICUSHIM_STATIC_BUILD in pal_icushim.c
* Rename shim_source_code to source_code in Microsoft.DotNet.ILCompiler.pkgproj and script
* Regular config is used, config-standalone.h is removed
* IlcPath/.. is replaced to make the ".." go away

* Address feedback from 08/04

* Rename option to ICUStaticLinking
* Remove <Message ..
* Compile without copying sources
* Trim list of defines
* Emit error when run on Linux in CMakeLists.txt
* Move StaticICULibs to near licucore on Mac
* Use only IlcHostPackagePath

* Fix DirectPInvoke for library built by user

* Feedback on 08/25: rename static-build, "source_code" and cleanup compile defs

* Remove IlcHostPackagePath check
* StaticICULinking property is true/false
* Remove -g and -fPIC from definitions
* Use minimal warning level
* Remove -DHOST_AMD64
* Remove -O0
* Remove -fms-extensions -fwrapv
* Standardize on "local build"
* Rename build-nativeaot-shim-standalone.sh to build.sh
* Script name in the error message should match the actual script name
* Rename source_code to native/src

Test code:
$ cat Program.cs
using System.Globalization;

namespace CurrentCulture
{
	class Program
	{
		static void Main(string[] args)
		{
			double val = 1235.56;

			Console.WriteLine($"Current culture: {CultureInfo.CurrentCulture.Name}");
			Console.WriteLine(val);

			CultureInfo.DefaultThreadCurrentCulture = new CultureInfo("en-US");

			Console.WriteLine($"Current culture: {CultureInfo.CurrentCulture.Name}");
			Console.WriteLine(val);
		}
	}
}

$ cat hello-4.csproj
<Project Sdk="Microsoft.NET.Sdk">
 <PropertyGroup>
  <OutputType>Exe</OutputType>
  <TargetFramework>net7.0</TargetFramework>
  <RootNamespace>hello_4</RootNamespace>
  <ImplicitUsings>enable</ImplicitUsings>
  <Nullable>enable</Nullable>
 </PropertyGroup>

$ cat nuget.config
<?xml version="1.0" encoding="utf-8"?>
<configuration>
 <packageSources>
  <!--To inherit the global NuGet package sources remove the <clear/> line below -->
  <clear />
    <add key="repositoryPath" value="/home/ubuntu/runtime/artifacts/packages/Debug/Shipping" />
    <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
 </packageSources>
</configuration>

Test code build:
rm ./bin/Debug/net7.0/linux-arm64/native/hello-4
~/dotnet-sdk/dotnet clean
~/dotnet-sdk/dotnet publish /p:PublishAot=true /p:StaticICULinking=true -bl -r linux-arm64 --self-contained -v d 2>&1 > log
./bin/Debug/net7.0/linux-arm64/native/hello-4

Tested on x86-64 and arm64

* Feedback on 09/02

*  Incorporated change to use only src/native/libs/System.Globalization.Native/CMakeLists.txt
*  Conditionalized CMakeLists.txt for local build
*  Moved the build.sh into parent as local_build.sh and removed local-build directory.
*  Rename STATIC_SHIM_COMPILE to LOCAL_BUILD

* Move file includes to previous itemgroup

Move the includes in Microsoft.DotNet.ILCompiler.pkgproj to
<ItemGroup Condition="'$(PackageTargetRuntime)' != ''">

* Move file includes to previous itemgroup

Move the includes in Microsoft.DotNet.ILCompiler.pkgproj to
<ItemGroup Condition="'$(PackageTargetRuntime)' != ''">

* Update section in compiling.md

* Add "Using statically linked ICU" to compiling.md
* Remove define check for LOCAL_BUILD in CMakeLists.txt

* Update src/coreclr/nativeaot/docs/compiling.md

* Update src/coreclr/nativeaot/docs/compiling.md

* Update src/coreclr/nativeaot/docs/compiling.md

* Update src/coreclr/nativeaot/docs/compiling.md

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

* Update src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/Microsoft.DotNet.ILCompiler.pkgproj

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

Co-authored-by: Samiullah Mohammed <ubuntu@ip-172-31-24-113>
Co-authored-by: Ubuntu <ubuntu@ip-10-0-0-223.us-west-1.compute.internal>
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

Author: 5 people

src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets

@@ -52,7 +52,7 @@ The .NET Foundation licenses this file to you under the MIT license.
 
     <ItemGroup>
       <NetCoreAppNativeLibrary Include="System.Native" />
-      <NetCoreAppNativeLibrary Include="System.Globalization.Native" />
+      <NetCoreAppNativeLibrary Include="System.Globalization.Native" Condition="'$(StaticICULinking)' != 'true'" />
       <NetCoreAppNativeLibrary Include="System.IO.Compression.Native" />
       <NetCoreAppNativeLibrary Include="System.Net.Security.Native" />
       <NetCoreAppNativeLibrary Include="System.Security.Cryptography.Native.Apple" Condition="'$(TargetOS)' == 'OSX'" />
@@ -67,13 +67,21 @@ The .NET Foundation licenses this file to you under the MIT license.
       <NativeLibrary Include="@(NetCoreAppNativeLibrary->'%(EscapedPath)')" />
     </ItemGroup>
 
+    <ItemGroup Condition="'$(StaticICULinking)' == 'true'">
+      <NativeLibrary Include="$(IntermediateOutputPath)/libs/System.Globalization.Native/build/libSystem.Globalization.Native.a"/>
+      <DirectPInvoke Include="libSystem.Globalization.Native" />
+      <StaticICULibs Include="-Wl,-Bstatic -licuio -licutu -licui18n -licuuc -licudata -Wl,-Bdynamic" />
+    </ItemGroup>
+
     <ItemGroup Condition="'$(TargetOS)' == 'OSX'">
       <NativeFramework Include="CoreFoundation" />
       <NativeFramework Include="Foundation" />
       <NativeFramework Include="Security" />
       <NativeFramework Include="GSS" />
     </ItemGroup>
 
+    <Exec Command="$(IlcHostPackagePath)/native/src/libs/System.Globalization.Native/local_build.sh $(IlcHostPackagePath)/ $(IntermediateOutputPath)" Condition="'$(StaticICULinking)' == 'true'"/>
+
     <ItemGroup>
       <LinkerArg Include="@(NativeLibrary)" />
       <LinkerArg Include="--sysroot=$(SysRoot)" Condition="'$(SysRoot)' != ''" />
@@ -91,6 +99,7 @@ The .NET Foundation licenses this file to you under the MIT license.
       <LinkerArg Include="-lz" />
       <LinkerArg Include="-lrt" Condition="'$(TargetOS)' != 'OSX'" />
       <LinkerArg Include="-licucore" Condition="'$(TargetOS)' == 'OSX'" />
+      <LinkerArg Include="@(StaticICULibs)" Condition="'$(StaticICULinking)' == 'true'" />
       <LinkerArg Include="-dynamiclib" Condition="'$(TargetOS)' == 'OSX' and '$(NativeLib)' == 'Shared'" />
       <LinkerArg Include="-shared" Condition="'$(TargetOS)' != 'OSX' and '$(NativeLib)' == 'Shared'" />
       <!-- binskim warning BA3001 PIE disabled on executable -->

src/coreclr/nativeaot/docs/compiling.md

@@ -68,3 +68,24 @@ You also need to specify the sysroot directory for Clang using the `SysRoot` pro
 ```
 
 You may also follow [cross-building instructions](../../../../docs/workflow/building/coreclr/cross-building.md) to create your own sysroot directory.
+
+## Using statically linked ICU
+This feature can statically link libicu libraries (such as libicui18n.a) into your applications at build time.
+NativeAOT binaries built with this feature can run even when libicu libraries are not installed.
+
+You can use this feature by adding the `StaticICULinking` property to your project file as follows:
+
+```xml
+    <PropertyGroup>
+      <StaticICULinking>true</StaticICULinking>
+    </PropertyGroup>
+```
+
+This feature is only supported on Linux. This feature is not supported when crosscompiling.
+
+### Prerequisites
+
+Ubuntu (20.04+)
+```
+sudo apt-get install libicu-dev cmake
+```

src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/Microsoft.DotNet.ILCompiler.pkgproj

@@ -16,6 +16,11 @@
       <File Include="$(CoreCLRAotSdkDir)*" TargetPath="sdk" />
       <File Include="$(MibcOptimizationDataDir)/$(TargetOS)/$(TargetArchitecture)/**/*.mibc" TargetPath="mibc" />
     </ItemGroup>
+    <ItemGroup Condition="'$(PackageTargetRuntime)' != '' and '$(TargetOS)' == 'Linux'">
+      <File Include="$(MSBuildThisFileDirectory)\..\..\..\..\native\libs\System.Globalization.Native\*" TargetPath="native/src/libs/System.Globalization.Native"/>
+      <File Include="$(MSBuildThisFileDirectory)\..\..\..\..\native\minipal\*" TargetPath="native/src/minipal"/>
+      <File Include="$(MSBuildThisFileDirectory)\..\..\..\..\native\libs\Common\*" TargetPath="native/src/libs/Common"/>
+    </ItemGroup>
 
     <ItemGroup>
       <File Include="$(CoreCLRBuildIntegrationDir)*" TargetPath="build" />

src/native/libs/System.Globalization.Native/CMakeLists.txt

@@ -57,9 +57,21 @@ set(NATIVEGLOBALIZATION_SOURCES
     pal_localeNumberData.c
     pal_localeStringData.c
     pal_normalization.c
-    pal_icushim.c
 )
 
+if (LOCAL_BUILD)
+    set(NATIVEGLOBALIZATION_SOURCES ${NATIVEGLOBALIZATION_SOURCES} pal_icushim_static.c)
+    add_definitions(-DLOCAL_BUILD)
+    add_definitions(-DSTATIC_ICU)
+    add_definitions(-DPALEXPORT=EXTERN_C)
+    add_definitions(-DTARGET_UNIX)
+    # For minipal files
+    include_directories(../../)
+    include_directories(${CMAKE_CURRENT_BINARY_DIR})
+else()
+    set(NATIVEGLOBALIZATION_SOURCES ${NATIVEGLOBALIZATION_SOURCES} pal_icushim.c)
+endif()
+
 if (CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_MACCATALYST OR CLR_CMAKE_TARGET_IOS OR CLR_CMAKE_TARGET_TVOS)
     set(NATIVEGLOBALIZATION_SOURCES ${NATIVEGLOBALIZATION_SOURCES} pal_locale.m)
 endif()

src/native/libs/System.Globalization.Native/local_build.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+
+# Licensed to the .NET Foundation under one or more agreements.
+# The .NET Foundation licenses this file to you under the MIT license.
+#
+
+# This script is used only for building libSystem.Globalization.Native.a
+# in the end-user's computer for NativeAOT purposes (static linking).
+# This file is not used during the dotnet runtime build.
+
+# Currently, only Linux is supported
+
+SHIM_SOURCE_DIR=$1/native/src
+INTERMEDIATE_OUTPUT_PATH=$2
+
+if [ -d "$SHIM_SOURCE_DIR" ]; then
+    LOCAL_SHIM_DIR="$INTERMEDIATE_OUTPUT_PATH"/libs/System.Globalization.Native/build
+    mkdir -p "$LOCAL_SHIM_DIR" && cd "$LOCAL_SHIM_DIR"
+    if [ $? -ne 0 ]; then echo "local_build.sh::ERROR: Cannot use local build directory"; exit 1; fi
+    cmake -S "$SHIM_SOURCE_DIR/libs/System.Globalization.Native/" -DLOCAL_BUILD:STRING=1 -DCLR_CMAKE_TARGET_UNIX:STRING=1
+    if [ $? -ne 0 ]; then echo "local_build.sh::ERROR: cmake failed"; exit 1; fi
+    make -j
+    if [ $? -ne 0 ]; then echo "local_build.sh::ERROR: Build failed"; exit 1; fi
+fi
+
+exit 0

src/native/libs/System.Globalization.Native/pal_icushim_internal.h

@@ -15,7 +15,9 @@
 #include "pal_icushim_internal_android.h"
 #else
 
+#if !defined(LOCAL_BUILD)
 #define U_DISABLE_RENAMING 1
+#endif
 
 // All ICU headers need to be included here so that all function prototypes are
 // available before the function pointers are declared below.

src/native/libs/System.Globalization.Native/pal_icushim_static.c

@@ -217,12 +217,17 @@ const char* GlobalizationNative_GetICUDTName(const char* culture)
 
 int32_t GlobalizationNative_LoadICU(void)
 {
+#if !defined(LOCAL_BUILD)
+// Static NativeAOT compilation does not have
+// GlobalizationNative_LoadICUData() as entrypoint
     if (!isDataSet)
     {
         // don't try to locate icudt.dat automatically if mono_wasm_load_icu_data wasn't called
         // and fallback to invariant mode
         return 0;
     }
+#endif
+
     UErrorCode status = 0;
     UVersionInfo version;
     // Request the CLDR version to perform basic ICU initialization and find out