MSYS2 Bash shell seems to override PATH settings

[chrisd8088]

On the Git LFS project we have an older Windows CI workflow which uses the legacy actions/setup-ruby@v1 action, and which we might like to upgrade to ruby/setup-ruby@v1 as it would address some build problems on Windows Server 2022.

However, simply replacing one for the other results in many CI test failures, apparently because a different Bash shell is used and our PATH environment variable settings are not respected, the result being that when our tests run git lfs a system default Git LFS binary is found by Git and executed instead of the newly-built git-lfs.exe which we want to test, leading to lots of unexpected results (new tests run by an older binary which then, of course, fail).

Here are two versions of a small demo CI workflow, one which runs actions/setup-ruby@v1 and one which runs ruby/setup-ruby@v1. In both cases the workflow then builds a small demo Go executable in a subdirectory; we name this executable git-lfs.exe. By default, if we run either git-lfs version or git lfs version, one of the system's default git-lfs.exe binaries is found and run.

If we prepend our build subdirectory's path to the PATH environment variable, we expect that our demo git-lfs.exe should be run instead, and that is what happens in both cases if we run it directly with git-lfs version, which can be seen in the workflow's logs. However, only in the actions/setup-ruby@v1 script does git lfs version find and run our demo executable; in the ruby/setup-ruby@v1 case the git.exe binary instead finds only the system's git-lfs.exe. This may be due to some interaction with the MSYS2 installation and its Bash shell, and possibly the MSYS2_PATH_TYPE configuration.

First, here's the full workflow on a windows-2022 Actions runner which demonstrates how actions/setup-ruby@v1 does not cause PATH to be ignored, which is as we would expect:

name: CI
on: [push, pull_request]

jobs:
  build-windows:
    name: Build on Windows
    runs-on: windows-2022
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-ruby@v1
    - run: |
        git version
        git-lfs version
        git lfs version
        echo "============"
        where git
        echo "============"
        where git-lfs
        echo "============"
        echo "Building Git LFS demo ..."
        cd build
        go build -o git-lfs.exe
        cd ..
        PATH="$(pwd)/build:$PATH"
        export PATH
        git-lfs version
        git lfs version
        echo "============"
        where git-lfs
        echo "============"
      shell: bash

The output looks like:

▼ Run git version
  shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
git version 2.35.1.windows.2
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
============
C:\Program Files\Git\mingw64\bin\git.exe
C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe
============
C:\Program Files\Git\mingw64\bin\git-lfs.exe
C:\Program Files\Git\cmd\git-lfs.exe
============
Building Git LFS demo ...
Git LFS demo
Git LFS demo
============
D:\a\git-lfs\git-lfs\build\git-lfs.exe
C:\Program Files\Git\mingw64\bin\git-lfs.exe
C:\Program Files\Git\cmd\git-lfs.exe
============

Notice how when we run git-lfs version and git lfs version the first time, before we change PATH, one of the system default git-lfs.exe binaries runs and outputs git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2). However, once we build our demo git-lfs.exe and put it in the PATH, both of those commands now return Git LFS demo because our demo binary runs instead. That makes sense, given that it's now found first by where, as shown.

Next, here is the another workflow, again on a windows-2022 runner, where all we've changed is to replace actions/setup-ruby@v1 with ruby/setup-ruby@v1 and specified ruby-version: 3.1.0 (and we've also tried 3.0.3 and seen the same result):

name: CI
on: [push, pull_request]

jobs:
  build-windows:
    name: Build on Windows
    runs-on: windows-2022
    steps:
    - uses: actions/checkout@v1
    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: 3.1.0
    - run: |
        git version
        git-lfs version
        git lfs version
        echo "============"
        where git
        echo "============"
        where git-lfs
        echo "============"
        echo "Building Git LFS demo ..."
        cd build
        go build -o git-lfs.exe
        cd ..
        PATH="$(pwd)/build:$PATH"
        export PATH
        git-lfs version
        git lfs version
        echo "============"
        where git-lfs
        echo "============"
      shell: bash

The output looks like:

▼ Run git version
  shell: C:\msys64\usr\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
  env:
    TMPDIR: D:\a\_temp
    HOME: C:\Users\runneradmin
    MSYS2_PATH_TYPE: inherit
    MAKE: make.exe
    Path: C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\2.7.3\x64;C:\cabal\bin;C:\\ghcup\bin;C:\tools\ghc-9.2.1\bin;C:\Program Files\dotnet;C:\mysql\bin;C:\Program Files\R\R-4.1.2\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\hostedtoolcache\windows\go\1.16.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.10\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.10\x64;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.322-6\x64\bin;C:\npm\prefix;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\Program Files\Microsoft\jdk-11.0.12.7-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\ProgramData\Chocolatey\bin;C:\Program Files\Docker;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\nodejs\;C:\Program Files\LLVM\bin;C:\Program Files\OpenSSL\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.4\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\SeleniumWebDrivers\ChromeDriver\;C:\SeleniumWebDrivers\EdgeDriver\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Users\runneradmin\.dotnet\tools;C:\Users\runneradmin\.cargo\bin;C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
    ACLOCAL_PATH: /ucrt64/share/aclocal:/usr/share/aclocal
    LANG: en_US.UTF-8
    MANPATH: /ucrt64/share/man
    MINGW_PACKAGE_PREFIX: mingw-w64-ucrt-x86_64
    MINGW_PREFIX: /ucrt64
    MSYSTEM: UCRT64
    MSYSTEM_CARCH: x86_64
    MSYSTEM_CHOST: x86_64-w64-mingw32
    MSYSTEM_PREFIX: /ucrt64
    PKG_CONFIG_PATH: /ucrt64/lib/pkgconfig:/ucrt64/share/pkgconfig
    PROMPT: $P$G
    RI_DEVKIT: c:\msys64
git version 2.35.1.windows.2
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
============
C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe
C:\Program Files\Git\mingw64\bin\git.exe
============
C:\Program Files\Git\cmd\git-lfs.exe
C:\Program Files\Git\mingw64\bin\git-lfs.exe
============
Building Git LFS demo ...
Git LFS demo
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
============
D:\a\git-lfs\git-lfs\build\git-lfs.exe
C:\Program Files\Git\cmd\git-lfs.exe
C:\Program Files\Git\mingw64\bin\git-lfs.exe
============

With this second workflow, notice that now, after changing PATH, we only see our demo git-lfs.exe run by the git-lfs version command, whereas the git lfs version one finds and runs a system default git-lfs.exe. This can be seen by the difference in output; the former returns Git LFS demo but the latter returns git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2). And that's despite the fact that where correctly finds our demo git-lfs.exe first.

A few things are also different, of course. A different bash.exe is run; the first workflow runs C:\Program Files\Git\bin\bash.EXE but the second workflow runs C:\msys64\usr\bin\bash.EXE, and that reports a set of environment variables, including MSYS2_PATH_TYPE: inherit and a PATH in which C:\Program Files\Git\bin is ahead of C:\Program Files\Git\mingw64\bin.

That accounts for another obvious difference, which is that the first workflow finds C:\Program Files\Git\mingw64\bin\git.exe when running git while the second finds C:\Program Files\Git\bin\git.exe and runs that instead. Whichever git.exe is run, though, should not make a difference; both are version 2.35.1.windows.2. Both should respond to the change in PATH which puts our local build subdirectory first.

We are suspicious that there may be some way in which the MSYS2_PATH_TYPE: inherit setting is not taking effect here. We're not familiar with MSYS2 at all, but skimming through the code it appears that this is intended to make it work the "normal" way, with PATH being inherited by any processes invoked by a parent process. But the default appears to be to not follow this convention, and msys2/MSYS2-packages#2140 suggests that perhaps there are ways this setting reverts to the default in various situations. Is it possible that that is what is occurring here, and the Git process we invoke gets its PATH reset back to the one Bash is started with? That would certainly account for Git not finding our demo git-lfs.exe.

At any rate, the overall situation is one which seems akin to the issue raised in #19 and #20, namely, that highly unexpected behaviour results from running ruby/setup-ruby@v1. This issue has taken two days of time to debug just to the point where we could report it in a somewhat detailed fashion, and we still don't understand exactly what's going on.

For the time being we're going to stick with actions/setup-ruby@v1 because it continues to work without breaking our entire Git LFS test suite, which is what happens if we replace it with ruby/setup-ruby@v1 due to this issue of local PATH settings apparently being ignored by child processes.

[MSP-Greg]

What happens if you add another step right after the setup-ruby step that adds build to the env? Some thing like:

echo "$PWD/build" >> $GITHUB_PATH

bash

    - run: echo "$PWD/build" >> $GITHUB_PATH

ps1

    - run: echo "$pwd\build" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

[MSP-Greg]

@chrisd8088

Forget the above. The issue seems to be the bash shell. So, add the following to your above workflow, which will use the Git bash shell instead of the msys2 bash shell:

      - run: echo "C:/Program Files/Git/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

Also, Ruby 3.1.0 will install the MSYS2 ucrt64 gcc tools, where 3.0.3 will install the MSYS2 mingw64 gcc tools.

See workflow run here

[chrisd8088]

Hey, thanks for looking at this! I confess I remain baffled as to what is fundamentally happening here.

I'd actually tried the - run: echo "C:/Program Files/Git/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append idea once before while debugging and I've tried it again now, and it definitely resolves the immediate issue; however, for reasons I don't understand, it causes our actual real CI job to hang during a Go build step. So I gave up on that approach, although maybe I did so too quickly. (Here's a run of a slimmed-down version of that job I've been using to debug; it's stalled now as I write this.)

I had an idea overnight which I wanted to try, which was to explicitly use the "old" Git, i.e., C:\Program Files\Git\mingw64\bin\git.exe rather than C:\Program Files\Git\bin\git.exe. That, it appears, makes a key difference; here's the relevant bit of workflow:

    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: 3.1.0
    - run: |
        git version
        git-lfs version
        git lfs version
        echo "============"
        where git
        echo "============"
        where git-lfs
        echo "============"
        echo "Building Git LFS demo ..."
        cd build
        go build -o git-lfs.exe
        cd ..
        PATH="$(pwd)/build:$PATH"
        export PATH
        git-lfs version
        git lfs version
        '/c/Program Files/Git/mingw64/bin/git' lfs version
        echo "============"
        where git-lfs
        echo "============"
      shell: bash

and here's the output:

Building Git LFS demo ...
Git LFS demo
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
Git LFS demo

So it almost seems as though the issue might lie in the Git binary and/or the binary in combination with the Bash shell.

As a non-Windows developer I'm in deep waters here and can't yet see how a different Git binary (of the same version of Git!) could end up searching different paths when running a command, but I thought I'd try to trace the code a bit and see if anything jumps out. (Trying with GIT_TRACE2=1 hasn't yielded any differences, though, alas.)

[chrisd8088]

One update -- I realized that trying to prepend C:/Program Files/Git/bin to the $GITHUB_PATH is actually not quite what I think we want to do, and instead we want to prepend C:/Program Files/Git/mingw64/bin.

In the "old" version with actions/setup-ruby@v1, a where git command returns:

C:\Program Files\Git\mingw64\bin\git.exe
C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe

In the "new" version with ruby/setup-ruby@v1, it returns:

C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe
C:\Program Files\Git\mingw64\bin\git.exe

I wrote last night:

That accounts for another obvious difference, which is that the first workflow finds C:\Program Files\Git\mingw64\bin\git.exe when running git while the second finds C:\Program Files\Git\bin\git.exe and runs that instead. Whichever git.exe is run, though, should not make a difference; both are version 2.35.1.windows.2. Both should respond to the change in PATH which puts our local build subdirectory first.

But as I noted in the previous commit, it actually seems that which Git binary runs makes a big difference (again, for reasons I don't understand). Adding this step after ruby/setup-ruby@v1 allowed a debug run to make significant progress:

- run: echo "C:/Program Files/Git/mingw64/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

I'm going to try a full CI job next with that in place. I do wish I had some inkling why this matters, though.

[MSP-Greg]

Ok. First of all, Windows exe's find library files in a much different manner than *nix. There's newer functions that often aren't used, and there's the legacy method which relies on PATH and other things.

C:/Program Files/Git/mingw64/bin appears to be where the GIT bash files are and there are a lot of library files (dll's) in that folder. C:\Program Files\Git\bin has only three 'stub' exe's files, bash, git, and sh. My guess is the whole reason for the C:\Program Files\Git\bin folder is so PATH doesn't include all the library files in C:/Program Files/Git/mingw64/bin.

Also, I thought I saw a command gem install ronn in the main repo. That command/step should happen before any PATH changes, but after the setup-ruby step.

[dscho]

I realized that trying to prepend C:/Program Files/Git/bin to the $GITHUB_PATH is actually not quite what I think we want to do, and instead we want to prepend C:/Program Files/Git/mingw64/bin.

That's not quite correct. There can occur some problems with adding the complete /mingw64/bin/ to the PATH, therefore we recommend C:/Program Files/Git/cmd (or, for legacy reasons, C:/Program Files/Git/bin.

The "wrong" git-lfs.exe that you see being used is the one living in C:/Program Files/Git/mingw64/bin/. The reason is that Git for Windows purposefully prepends that directory to the PATH. In the CI that wants to override Git LFS, you are probably best served by calling something like

git -c alias.delete-lfs='!for d in /mingw64/bin /cmd; do test ! -f "$d/git-lfs.exe" || rm "$d/git-lfs.exe"; done' delete-lfs

[dscho]

FWIW this is related to git-for-windows/git#3668.

[chrisd8088]

Just as an update, we're likely to try to bypass this issue in the Git LFS CI suite by dropping the use of ronn, and therefore dodging the need to install Ruby at all. That's take us some time, however, as we'll have to rewrite the manual pages' ronn-format source files.

In the meantime, if I find some spare time I'll try to revisit the issue and see if I can make your suggestions work for us; it would be nice to upgrade to the Windows 2022 runners, as we bumped into the problem of not using them again during a recent Git LFS patch release.

Anyway, thanks for the advice and I'll report back if I have any success.

[chrisd8088]

A small update: we're upgrading to ronn-ng (see git-lfs/git-lfs#4992 and git-lfs/git-lfs#4997) which is going to allow us to stick with the deprecated actions/setup-ruby@v1 action for now while still upgrading to the Windows 2022 runners.

We tried also upgrading to the ruby/setup-ruby@v1 action but ran into the same problem with PATH documented above, so we're going to continue with the older action for now, since at least we're unblocked on moving to Windows 2022.