Integrating into Unreal Engine (UE5.1) on Mac

[laurie71]

I'm trying to integrate the SDK into a UE5 plugin on macOS.

I used this blog post as a starting point, but needed to make a number of changes to get things hooked up correctly, including adding dependencies on the Foundation, CoreFoundation, Security and SystemConfiguration frameworks. I also had to build a custom version of UnrealBuildTool (UBT) to prevent linker warnings from failing the build.

Now I have everything compiling, but the project crashes on launch whilst loading the plugin containing the AWS SDK.

Per the blog post I used vcpkg to build the SDK, then pulled in the includes and libs in my plugin's .Build.cs (see below), as well as adding the required frameworks. Compilation failed with lots of linker warnings like this:

ld: warning: object file (/Users/.../Plugins/AmazonAWS/Source/SDK/vcpkg/installed/x64-osx/lib/libcrypto.a(libcrypto-lib-siphash.o)) was built for newer macOS version (12.0) than being linked (10.15)

I couldn't find a way to get vcpkg to target the SDK build onto 10.15, to to move past this I needed to remove the -fatal_warnings linker flag. After modifying UBT to do that everything compiles, but there are lots of linker warnings about duplicate symbols which look to be a conflict between the OpenSSL bundled in the SDK and the OpenSSL bundled in Unreal:

ld: warning: duplicate symbol '_OPENSSL_instrument_bus2' in: /Users/.../Plugins/AmazonAWS/Source/SDK/vcpkg/installed/x64-osx/lib/libcrypto.a(libcrypto-lib-x86_64cpuid.o) /Users/.../UnrealEngine/Engine/Source/ThirdParty/OpenSSL/1.1.1n/lib/Mac/libcrypto.a(x86_64cpuid.o)

On launching the project execution fails on a segfault during module initialization. The is the call stack at the point of failure:

pthread_rwlock_wrlock (@pthread_rwlock_wrlock:3) CRYPTO_THREAD_write_lock (@CRYPTO_THREAD_write_lock:6) ENGINE_finish (@ENGINE_finish:13) evp_md_init_internal (@evp_md_init_internal:68) EVP_DigestInit_ex (@EVP_Digest:34) EVP_Digest (@EVP_Digest:29) ossl_asn1_item_digest_ex (@ossl_asn1_item_digest_ex:52) X509_digest (@X509_digest:42) ossl_x509v3_cache_extensions (@ossl_x509v3_cache_extensions:31) X509_check_purpose (@X509_check_purpose:14) X509_cmp (@X509_cmp:17) FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8::operator()(x509_st*) const (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/SSL/Private/SslCertificateManager.cpp:457) decltype(Forward<FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8&>(fp)(Forward<x509_st*&>(fp0))) Invoke<FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8&, x509_st*&>(FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8&, x509_st*&) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Public/Templates/Invoke.h:47) x509_st** TArray<x509_st*, TSizedDefaultAllocator<32>>::FindByPredicate<FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8>(FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Public/Containers/Array.h:1139) x509_st* const* TArray<x509_st*, TSizedDefaultAllocator<32>>::FindByPredicate<FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8>(FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8) const (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Public/Containers/Array.h:1125) bool TArray<x509_st*, TSizedDefaultAllocator<32>>::ContainsByPredicate<FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8>(FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*)::$_8) const (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Public/Containers/Array.h:1199) FSslCertificateManager::AddCertificateToRootCertificateArray(x509_st*) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/SSL/Private/SslCertificateManager.cpp:454) FSslCertificateManager::AddPEMFileToRootCertificateArray(FString const&) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/SSL/Private/SslCertificateManager.cpp:419) FSslCertificateManager::BuildRootCertificateArray() (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/SSL/Private/SslCertificateManager.cpp:351) FSslModule::StartupModule() (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/SSL/Private/SslModule.cpp:47) FModuleManager::LoadModuleWithFailureReason(FName, EModuleLoadResult&, ELoadModuleFlags) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:495) FModuleManager::LoadModule(FName, ELoadModuleFlags) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:381) FModuleManager::LoadModuleChecked(FName) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:392) FSslModule& FModuleManager::LoadModuleChecked<FSslModule>(FName) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Public/Modules/ModuleManager.h:338) FSslModule::Get() (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/SSL/Private/SslModule.cpp:87) FApplePlatformHttp::Init() (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/HTTP/Private/Apple/ApplePlatformHttp.cpp:15) FHttpModule::StartupModule() (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Online/HTTP/Private/HttpModule.cpp:90) FModuleManager::LoadModuleWithFailureReason(FName, EModuleLoadResult&, ELoadModuleFlags) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:495) FModuleManager::LoadModule(FName, ELoadModuleFlags) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:381) FOnlineSubsystemModule::StartupModule() (/Users/laurie/Projects/UnrealEngine/Engine/Plugins/Online/OnlineSubsystem/Source/Private/OnlineSubsystemModule.cpp:80) FModuleManager::LoadModuleWithFailureReason(FName, EModuleLoadResult&, ELoadModuleFlags) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:495) FModuleDescriptor::LoadModulesForPhase(ELoadingPhase::Type, TArray<FModuleDescriptor, TSizedDefaultAllocator<32>> const&, TMap<FName, EModuleLoadResult, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FName, EModuleLoadResult, false>>&) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Projects/Private/ModuleDescriptor.cpp:695) FPluginManager::TryLoadModulesForPlugin(FPlugin const&, ELoadingPhase::Type) const (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Projects/Private/PluginManager.cpp:2076) FPluginManager::LoadModulesForEnabledPlugins(ELoadingPhase::Type) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Projects/Private/PluginManager.cpp:2154) FEngineLoop::AppInit() (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:6122) FEngineLoop::PreInitPreStartupScreen(char16_t const*) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:2486) FEngineLoop::PreInit(char16_t const*) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:3941) EnginePreInit(char16_t const*) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Launch/Private/Launch.cpp:46) GuardedMain(char16_t const*) (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Launch/Private/Launch.cpp:154) -[UEAppDelegate runGameThread:] (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Launch/Private/Mac/LaunchMac.cpp:237) -[FCocoaGameThread main] (/Users/laurie/Projects/UnrealEngine/Engine/Source/Runtime/Core/Private/Mac/CocoaThread.cpp:383) __NSThread__start__ (@__NSThread__start__:212) _pthread_start (@_pthread_start:32) thread_start (@thread_start:8)

And that's as far as I've been able to get. Here's my current .Build.cs file for reference:

`using System;
using System.Collections.Generic;
using System.IO;
using UnrealBuildTool;

public class AmazonAWS : ModuleRules
{
// list every library you plan to use here
private List LibraryNames = new List()
{
"libaws-c-auth",
"libaws-c-cal",
"libaws-c-common",
"libaws-c-compression",
"libaws-c-event-stream",
"libaws-c-http",
"libaws-c-io",
"libaws-c-mqtt",
"libaws-c-s3",
"libaws-c-sdkutils",
"libaws-checksums",
"libaws-cpp-sdk-access-management",
"libaws-cpp-sdk-cognito-identity",
"libaws-cpp-sdk-core",
"libaws-cpp-sdk-iam",
"libaws-cpp-sdk-kinesis",
"libaws-crt-cpp",
"libcrypto",
"libcurl",
"libs2n",
"libssl",
"libz",
};

public AmazonAWS(ReadOnlyTargetRules Target) : base(Target)
{
    // ModuleType.External tells the engine not to look for (or compile) any source code.
    // Type = ModuleType.External;
    PCHUsage = ModuleRules.PCHUsageMode.UseExplicitOrSharedPCHs;

    // add the header files for reference
    PublicIncludePaths.Add(Path.Combine(ModuleDirectory, "../SDK/vcpkg/installed/x64-osx/include"));

    // AWS SDK relies on certain identifiers being undefined, so this produces warnings
    // Unreal engine treats certain warnings as errors and fails the build
    // this line will disable those warnings:
    bEnableUndefinedIdentifierWarnings = false;

    // Dynamically linking to the SDK requires us to define the
    // USE_IMPORT_EXPORT symbol for all build targets using the
    // SDK. Source: https://github.com/aws/aws-sdk-cpp/blob/main/Docs/SDK_usage_guide.md#build-defines
    PublicDefinitions.Add("USE_IMPORT_EXPORT");
    PublicDefinitions.Add("AWS_CRT_CPP_USE_IMPORT_EXPORT");

    if (Target.Platform == UnrealTargetPlatform.Win64)
    {
        PublicDefinitions.Add("USE_WINDOWS_DLL_SEMANTICS");
    }

    // do this for each lib and dll
    foreach (string libName in LibraryNames)
    {
        // string LibraryPath = Path.Combine(ModuleDirectory, "../SDK/vcpkg/installed/" + Target.Platform.ToString() + "/lib");
        string LibraryPath = Path.Combine(ModuleDirectory, "../SDK/vcpkg/installed/x64-osx/lib");
        // add a new section for each platform you plan to compile for (only Win64 is included for this example)
        if (Target.Platform == UnrealTargetPlatform.Win64)
        {
            PublicAdditionalLibraries.Add(Path.Combine(LibraryPath, libName + ".lib"));
            // Stage the library along with the target, so it can be loaded at runtime.
            RuntimeDependencies.Add("$(BinaryOutputDir)/" + libName + ".dll", Path.Combine(LibraryPath, libName + ".dll"));
        }
        else if (Target.Platform == UnrealTargetPlatform.Mac)
        {
            PublicAdditionalLibraries.Add(Path.Combine(LibraryPath, libName + ".a"));
            // RuntimeDependencies.Add("$(BinaryOutputDir)/" + libName + ".dll", Path.Combine(LibraryPath, libName + ".dll"));
        }
        else if (Target.Platform == UnrealTargetPlatform.Linux)
        {
            // add linux libraries
        }
    }

	PublicFrameworks.AddRange(
		new string[]
		{
			"Foundation",
			"CoreFoundation",
			"Security",
			"SystemConfiguration",
		}
	);

	PublicDependencyModuleNames.AddRange(
		new string[]
		{
			"Core",
		}
	);

    PrivateDependencyModuleNames.AddRange(
		new string[] {
					"CoreUObject",
					"Engine",
					"Slate",
					"SlateCore",
		}
	);
}

}`

[jmklix]

We have a feature request open for adding Unreal support. The first response from iniside might be helpful to you right now

[laurie71]

Thanks, I'll add a watch to the feature request and add updates there if I'm able to make further progress.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.