Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[compilers] Use a pre-configure setup step #856

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Steelskin
Copy link
Collaborator

@Steelskin Steelskin commented Dec 2, 2024

  • Add a preliminary step to export the context for the compiler
    configuration step. This makes it easier to debug the compiler
    configuration step by seeing the full command line invocation on the
    GHA page.

@Steelskin Steelskin force-pushed the fabrice/mac-build-compilers-alt branch from fde4ec2 to 3243bb7 Compare December 3, 2024 21:01
@Steelskin Steelskin force-pushed the fabrice/mac-build-compilers-alt branch from 3243bb7 to 824eea7 Compare December 16, 2024 17:15
@Steelskin Steelskin changed the title [Mac Build] Add compilers to the Mac Build with extra config step [compilers] Use a pre-configure setup step Dec 16, 2024
@Steelskin Steelskin force-pushed the fabrice/mac-build-compilers-alt branch 7 times, most recently from 4d4d026 to 62027d9 Compare December 18, 2024 23:27
* Add a preliminary step to export the context for the compiler
  configuration step. This makes it easier to debug the compiler
  configuration step by seeing the full command line invocation on the
  GHA page.
@Steelskin Steelskin force-pushed the fabrice/mac-build-compilers-alt branch from 62027d9 to b5f43a6 Compare January 6, 2025 18:59
@Steelskin Steelskin marked this pull request as ready for review January 7, 2025 01:03
@Steelskin
Copy link
Collaborator Author

Steelskin commented Jan 7, 2025

PTAL. This is how the compilers configure step shows up as before this change:

$CxxFlags = "/GS- /Gw /Gy /Oi /Oy /Z7 /Zc:inline /Zc:preprocessor /Zc:__cplusplus /D_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR"
  $SwiftFlags = ""
  $EXTRA_FLAGS = @()
  
  if ( "Windows" -eq "Windows" ) {
    $SWIFTC = cygpath -m (Get-Command swiftc).Source
    # Use toolchain clang to avoid broken __prefetch intrinsic on arm64 in Clang 18.
    # TODO: Use llvm-19 when available. See https://github.com/compnerd/swift-build/issues/846
    $CLANG_LOCATION = Split-Path (Get-Command swiftc).Source
    if ( "amd64" -eq "arm64" ) {
      $EXTRA_FLAGS += "-D CMAKE_SYSTEM_PROCESSOR=AMD64".Split()
      $EXTRA_FLAGS += @("-D", "CMAKE_SYSTEM_NAME=Windows")
      $CACHE="C:\a\swift-build\swift-build/SourceCache/swift/cmake/caches/Windows-aarch64.cmake"
  
      # FIXME(compnerd) re-enable runtimes after we sort out compiler-rt
      (Get-Content C:\a\swift-build\swift-build/SourceCache/swift/cmake/caches/Windows-aarch64.cmake).Replace(' runtimes', '') | Set-Content C:\a\swift-build\swift-build/SourceCache/swift/cmake/caches/Windows-aarch64.cmake
    } else {
      $CACHE="C:\a\swift-build\swift-build/SourceCache/swift/cmake/caches/Windows-x86_64.cmake"
    }
    $CC = "cl"
    $CXX = "cl"
  
    # Workaround for compiler and STL version mismatch on Windows.
    $CxxFlags += " -D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH"
    $SwiftFlags += " -Xcc -D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH"
  
    $SDKROOT = cygpath -m ${env:SDKROOT}
    $CLANG_LOCATION = cygpath -m $CLANG_LOCATION
    $LIBPYTHON_PATH = "${env:PYTHON_LOCATION_amd64}/libs/python[3](https://github.com/thebrowsercompany/swift-build/actions/runs/12587696230/job/35084714847#step:23:3)9.lib"
    $PYTHON_INCLUDE_DIR = "${env:PYTHON_LOCATION_amd6[4](https://github.com/thebrowsercompany/swift-build/actions/runs/12587696230/job/35084714847#step:23:4)}/include"
    $PYTHON_BINARY="python.exe"
    $ExeSuffix = ".exe"
    Remove-Item env:\SDKROOT
  } elseif ( "Windows" -eq "Darwin" ) {
    # Default swiftc comes from /usr/bin and is not compatible with the toolchain.
    $CLANG_LOCATION = "${env:HOME}/Library/Developer/Toolchains/swift-6.0.1-RELEASE.xctoolchain/usr/bin"
    $SWIFTC = Join-Path $CLANG_LOCATION "swiftc"
  
    # We need to use llvm-17 to build the compiler on macOS. We get it from the Swift toolchain.
    $CC = Join-Path $CLANG_LOCATION "clang"
    $CXX = Join-Path $CLANG_LOCATION "clang++"
    $CACHE = "C:\a\swift-build\swift-build/SourceCache/swift/cmake/caches/Darwin-amd64.cmake"
    $SDKROOT = xcrun --sdk macosx --show-sdk-path
    $EXTRA_FLAGS += "-D CMAKE_SYSTEM_PROCESSOR=AMD64".Split()
    $EXTRA_FLAGS += @("-D", "CMAKE_SYSTEM_NAME=Darwin")
    # TODO: Use early-swift-driver on Windows too.
    $EXTRA_FLAGS += @("-D", "SWIFT_EARLY_SWIFT_DRIVER_BUILD=C:\a\swift-build\swift-build/BinaryCache/swift-driver/bin")
    $LIBPYTHON_PATH = "${env:pythonLocation}/lib/python3.9/config-3.9-darwin/libpython3.9.a"
    $PYTHON_INCLUDE_DIR = "${env:pythonLocation}/include/python3.9"
    $PYTHON_BINARY="python3"
    $ExeSuffix = ""
  }
  
  $SwiftFlags += " -sdk `"${SDKROOT}`""
  
  cmake -B C:\a\swift-build\swift-build/BinaryCache/1 `
        -C "${CACHE}" `
        -D CMAKE_BUILD_TYPE=Release `
        -D CMAKE_C_COMPILER="${CC}" `
        -D CMAKE_C_COMPILER_LAUNCHER=sccache `
        -D CMAKE_C_FLAGS="/GS- /Gw /Gy /Oi /Oy /Z7 /Zc:inline /Zc:preprocessor" `
        -D CMAKE_CXX_COMPILER="${CXX}" `
        -D CMAKE_CXX_COMPILER_LAUNCHER=sccache `
        -D CMAKE_CXX_FLAGS="${CxxFlags}" `
        -D CMAKE_Swift_COMPILER="${SWIFTC}" `
        -D CMAKE_Swift_COMPILER_WORKS=YES `
        -D CMAKE_Swift_FLAGS="${SwiftFlags}" `
        -D CMAKE_EXE_LINKER_FLAGS="-incremental:no -debug -opt:ref -opt:icf" -D CMAKE_SHARED_LINKER_FLAGS="-incremental:no -debug -opt:ref -opt:icf" `
        -D CMAKE_FIND_PACKAGE_PREFER_CONFIG=YES `
        -D CMAKE_INSTALL_PREFIX=C:\a\swift-build\swift-build/BuildRoot/Library/Developer/Toolchains/unknown-Asserts-development.xctoolchain/usr `
        @EXTRA_FLAGS `
        -G Ninja `
        -S C:\a\swift-build\swift-build/SourceCache/llvm-project/llvm `
        -D CLANG_TABLEGEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/clang-tblgen${ExeSuffix}" `
        -D CLANG_TIDY_CONFUSABLE_CHARS_GEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/clang-tidy-confusable-chars-gen${ExeSuffix}" `
        -D LLDB_TABLEGEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/lldb-tblgen${ExeSuffix}" `
        -D LLVM_CONFIG_PATH="C:\a\swift-build\swift-build/BinaryCache/0/bin/llvm-config${ExeSuffix}" `
        -D LLVM_EXTERNAL_SWIFT_SOURCE_DIR=C:\a\swift-build\swift-build/SourceCache/swift `
        -D LLVM_NATIVE_TOOL_DIR=C:\a\swift-build\swift-build/BinaryCache/0/bin `
        -D LLVM_TABLEGEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/llvm-tblgen${ExeSuffix}" `
        -D LLVM_USE_HOST_TOOLS=NO `
        -D SWIFT_BUILD_DYNAMIC_SDK_OVERLAY=NO `
        -D SWIFT_BUILD_DYNAMIC_STDLIB=NO `
        -D SWIFT_BUILD_REMOTE_MIRROR=NO `
        -D SWIFT_BUILD_SWIFT_SYNTAX=YES `
        -D SWIFT_CLANG_LOCATION=${CLANG_LOCATION} `
        -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_CXX_INTEROP=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_OBSERVATION=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING=YES `
        -D SWIFT_ENABLE_SYNCHRONIZATION=YES `
        -D SWIFT_NATIVE_SWIFT_TOOLS_PATH=C:\a\swift-build\swift-build/BinaryCache/0/bin `
        -D SWIFT_PATH_TO_LIBDISPATCH_SOURCE=C:\a\swift-build\swift-build/SourceCache/swift-corelibs-libdispatch `
        -D SWIFT_PATH_TO_SWIFT_SYNTAX_SOURCE=C:\a\swift-build\swift-build/SourceCache/swift-syntax `
        -D SWIFT_PATH_TO_STRING_PROCESSING_SOURCE=C:\a\swift-build\swift-build/SourceCache/swift-experimental-string-processing `
        -D SWIFT_PATH_TO_SWIFT_SDK="${SDKROOT}" `
        -D CLANG_VENDOR=compnerd.org `
        -D CLANG_VENDOR_UTI=org.compnerd.dt `
        -D cmark-gfm_DIR=C:\a\swift-build\swift-build/BinaryCache/Library/cmark-gfm-0.29.0.gfm.13/usr/lib/cmake `
        -D LibXml2_DIR=C:\a\swift-build\swift-build/BinaryCache/Library/libxml2-2.11.5/usr/lib/cmake/libxml2-2.11.5 `
        -D PACKAGE_VENDOR=compnerd.org `
        -D SWIFT_VENDOR=compnerd.org `
        -D LLVM_PARALLEL_LINK_JOBS=8 `
        -D SWIFT_PARALLEL_LINK_JOBS=8 `
        -D LLVM_APPEND_VC_REV=NO `
        -D LLVM_VERSION_SUFFIX="" `
        -D LLDB_PYTHON_EXE_RELATIVE_PATH=${PYTHON_BINARY} `
        -D LLDB_PYTHON_EXT_SUFFIX=.pyd `
        -D LLDB_PYTHON_RELATIVE_PATH=lib/site-packages `
        -D Python3_EXECUTABLE=C:\hostedtoolcache\windows\Python\3.9.10\x64\python.exe `
        -D Python3_INCLUDE_DIR=$PYTHON_INCLUDE_DIR `
        -D Python3_LIBRARY=$LIBPYTHON_PATH `
        -D Python3_ROOT_DIR=$env:pythonLocation

And after:

if ( "Windows" -eq "Windows" ) {
    $ExeSuffix = ".exe"
    Remove-Item env:\SDKROOT
  } elseif ( "Windows" -eq "Darwin" ) {
    $ExeSuffix = ""
  }
  
  cmake -B C:\a\swift-build\swift-build/BinaryCache/1 `
        -C "C:\a\swift-build\swift-build/SourceCache/swift/cmake/caches/Windows-x86_64.cmake" `
        -D CMAKE_BUILD_TYPE=Release `
        -D CMAKE_C_COMPILER="cl" `
        -D CMAKE_C_COMPILER_LAUNCHER=sccache `
        -D CMAKE_C_FLAGS="/GS- /Gw /Gy /Oi /Oy /Zc:inline /Zc:preprocessor" `
        -D CMAKE_CXX_COMPILER="cl" `
        -D CMAKE_CXX_COMPILER_LAUNCHER=sccache `
        -D CMAKE_CXX_FLAGS="/GS- /Gw /Gy /Oi /Oy /Zc:inline /Zc:preprocessor /Zc:__cplusplus /D_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR -D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH" `
        -D CMAKE_Swift_COMPILER="C:/Users/runneradmin/AppData/Local/Programs/Swift/Toolchains/6.0.0+Asserts/usr/bin/swiftc.exe" `
        -D CMAKE_Swift_COMPILER_WORKS=YES `
        -D CMAKE_Swift_FLAGS=" -Xcc -D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH -sdk "C:/Users/runneradmin/AppData/Local/Programs/Swift/Platforms/6.0.0/Windows.platform/Developer/SDKs/Windows.sdk/"" `
        -D CMAKE_EXE_LINKER_FLAGS="" -D CMAKE_SHARED_LINKER_FLAGS="" `
        -D CMAKE_FIND_PACKAGE_PREFER_CONFIG=YES `
        -D CMAKE_INSTALL_PREFIX=C:\a\swift-build\swift-build/BuildRoot/Library/Developer/Toolchains/unknown-Asserts-development.xctoolchain/usr `
         `
        -G Ninja `
        -S C:\a\swift-build\swift-build/SourceCache/llvm-project/llvm `
        -D CLANG_TABLEGEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/clang-tblgen${ExeSuffix}" `
        -D CLANG_TIDY_CONFUSABLE_CHARS_GEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/clang-tidy-confusable-chars-gen${ExeSuffix}" `
        -D LLDB_TABLEGEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/lldb-tblgen${ExeSuffix}" `
        -D LLVM_CONFIG_PATH="C:\a\swift-build\swift-build/BinaryCache/0/bin/llvm-config${ExeSuffix}" `
        -D LLVM_EXTERNAL_SWIFT_SOURCE_DIR=C:\a\swift-build\swift-build/SourceCache/swift `
        -D LLVM_NATIVE_TOOL_DIR=C:\a\swift-build\swift-build/BinaryCache/0/bin `
        -D LLVM_TABLEGEN="C:\a\swift-build\swift-build/BinaryCache/0/bin/llvm-tblgen${ExeSuffix}" `
        -D LLVM_USE_HOST_TOOLS=NO `
        -D SWIFT_BUILD_DYNAMIC_SDK_OVERLAY=NO `
        -D SWIFT_BUILD_DYNAMIC_STDLIB=NO `
        -D SWIFT_BUILD_REMOTE_MIRROR=NO `
        -D SWIFT_BUILD_SWIFT_SYNTAX=YES `
        -D SWIFT_CLANG_LOCATION="C:/Users/runneradmin/AppData/Local/Programs/Swift/Toolchains/6.0.0+Asserts/usr/bin" `
        -D SWIFT_ENABLE_EXPERIMENTAL_CONCURRENCY=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_CXX_INTEROP=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_DISTRIBUTED=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_DIFFERENTIABLE_PROGRAMMING=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_OBSERVATION=YES `
        -D SWIFT_ENABLE_EXPERIMENTAL_STRING_PROCESSING=YES `
        -D SWIFT_ENABLE_SYNCHRONIZATION=YES `
        -D SWIFT_NATIVE_SWIFT_TOOLS_PATH=C:\a\swift-build\swift-build/BinaryCache/0/bin `
        -D SWIFT_PATH_TO_LIBDISPATCH_SOURCE=C:\a\swift-build\swift-build/SourceCache/swift-corelibs-libdispatch `
        -D SWIFT_PATH_TO_SWIFT_SYNTAX_SOURCE=C:\a\swift-build\swift-build/SourceCache/swift-syntax `
        -D SWIFT_PATH_TO_STRING_PROCESSING_SOURCE=C:\a\swift-build\swift-build/SourceCache/swift-experimental-string-processing `
        -D SWIFT_PATH_TO_SWIFT_SDK="C:/Users/runneradmin/AppData/Local/Programs/Swift/Platforms/6.0.0/Windows.platform/Developer/SDKs/Windows.sdk/" `
        -D CLANG_VENDOR=compnerd.org `
        -D CLANG_VENDOR_UTI=org.compnerd.dt `
        -D cmark-gfm_DIR=C:\a\swift-build\swift-build/BinaryCache/Library/cmark-gfm-0.29.0.gfm.13/usr/lib/cmake `
        -D LibXml2_DIR=C:\a\swift-build\swift-build/BinaryCache/Library/libxml2-2.11.5/usr/lib/cmake/libxml2-2.11.5 `
        -D PACKAGE_VENDOR=compnerd.org `
        -D SWIFT_VENDOR=compnerd.org `
        -D LLVM_PARALLEL_LINK_JOBS=2 `
        -D SWIFT_PARALLEL_LINK_JOBS=2 `
        -D LLVM_APPEND_VC_REV=NO `
        -D LLVM_VERSION_SUFFIX="" `
        -D LLDB_PYTHON_EXE_RELATIVE_PATH=python.exe `
        -D LLDB_PYTHON_EXT_SUFFIX=.pyd `
        -D LLDB_PYTHON_RELATIVE_PATH=lib/site-packages `
        -D Python3_EXECUTABLE=C:\hostedtoolcache\windows\Python\3.9.10\x64\python.exe `
        -D Python3_INCLUDE_DIR=C:\hostedtoolcache\windows\Python\3.9.10\x64/include `
        -D Python3_LIBRARY=C:\hostedtoolcache\windows\Python\3.9.10\x64/libs/python39.lib `
        -D Python3_ROOT_DIR=$env:pythonLocation

@Steelskin Steelskin requested review from compnerd and hjyamauchi and removed request for compnerd January 7, 2025 01:07
$CACHE="${{ github.workspace }}/SourceCache/swift/cmake/caches/Windows-aarch64.cmake"

# FIXME(compnerd) re-enable runtimes after we sort out compiler-rt
(Get-Content ${{ github.workspace }}/SourceCache/swift/cmake/caches/Windows-aarch64.cmake).Replace(' runtimes', '') | Set-Content ${{ github.workspace }}/SourceCache/swift/cmake/caches/Windows-aarch64.cmake
} else {
# FIXME(steelskin) Setting `CMAKE_SYSTEM_NAME and `CMAKE_SYSTEM_PROCESSOR` breaks the compiler-rt build
$ExtraFlags = ""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the impression that this PR doesn't intend to change the behavior, but does this line 998 change the behavior?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I slightly changed the way ExtraFlags is populated, compared to the EXTRA_FLAGS array. On line 983, ExtraFlags is now initialized to ${{ matrix.extra_flags }}. Before, it was initialized to an empty array and then ${{ matrix.extra_flags }} was added in all cases, except for Windows x64. I am now adding ${{ matrix.extra_flags }} unconditionally and then resetting it for Windows x64. In the long term, we'll want to fix the issue in CMake and remove this line, which will also make this script simpler.

While this is a change, it is functionally identical as the final command line invocation is the same as before.

Comment on lines +1036 to +1047
Write-Output "cc=${CC}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "cxx=${CXX}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "swiftc=${SWIFTC}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "cxxflags=${CxxFlags}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "swiftflags=${SwiftFlags}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "extra_flags=${ExtraFlags}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "cache=${CACHE}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "clang_location=${CLANG_LOCATION}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "libpython_path=${LIBPYTHON_PATH}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "python_include_dir=${PYTHON_INCLUDE_DIR}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "python_binary=${PYTHON_BINARY}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Write-Output "sdkroot=${SDKROOT}" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you change this to a heredoc instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants