Skip to content

MSYS2 Bash shell seems to override PATH settings #293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chrisd8088 opened this issue Feb 18, 2022 · 19 comments
Closed

MSYS2 Bash shell seems to override PATH settings #293

chrisd8088 opened this issue Feb 18, 2022 · 19 comments
Assignees

Comments

@chrisd8088
Copy link

chrisd8088 commented Feb 18, 2022

On the Git LFS project we have an older Windows CI workflow which uses the legacy actions/setup-ruby@v1 action, and which we might like to upgrade to ruby/setup-ruby@v1 as it would address some build problems on Windows Server 2022.

However, simply replacing one for the other results in many CI test failures, apparently because a different Bash shell is used and our PATH environment variable settings are not respected, the result being that when our tests run git lfs a system default Git LFS binary is found by Git and executed instead of the newly-built git-lfs.exe which we want to test, leading to lots of unexpected results (new tests run by an older binary which then, of course, fail).

Here are two versions of a small demo CI workflow, one which runs actions/setup-ruby@v1 and one which runs ruby/setup-ruby@v1. In both cases the workflow then builds a small demo Go executable in a subdirectory; we name this executable git-lfs.exe. By default, if we run either git-lfs version or git lfs version, one of the system's default git-lfs.exe binaries is found and run.

If we prepend our build subdirectory's path to the PATH environment variable, we expect that our demo git-lfs.exe should be run instead, and that is what happens in both cases if we run it directly with git-lfs version, which can be seen in the workflow's logs. However, only in the actions/setup-ruby@v1 script does git lfs version find and run our demo executable; in the ruby/setup-ruby@v1 case the git.exe binary instead finds only the system's git-lfs.exe. This may be due to some interaction with the MSYS2 installation and its Bash shell, and possibly the MSYS2_PATH_TYPE configuration.

First, here's the full workflow on a windows-2022 Actions runner which demonstrates how actions/setup-ruby@v1 does not cause PATH to be ignored, which is as we would expect:

name: CI
on: [push, pull_request]

jobs:
  build-windows:
    name: Build on Windows
    runs-on: windows-2022
    steps:
    - uses: actions/checkout@v1
    - uses: actions/setup-ruby@v1
    - run: |
        git version
        git-lfs version
        git lfs version
        echo "============"
        where git
        echo "============"
        where git-lfs
        echo "============"
        echo "Building Git LFS demo ..."
        cd build
        go build -o git-lfs.exe
        cd ..
        PATH="$(pwd)/build:$PATH"
        export PATH
        git-lfs version
        git lfs version
        echo "============"
        where git-lfs
        echo "============"
      shell: bash

The output looks like:

▼ Run git version
  shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
git version 2.35.1.windows.2
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
============
C:\Program Files\Git\mingw64\bin\git.exe
C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe
============
C:\Program Files\Git\mingw64\bin\git-lfs.exe
C:\Program Files\Git\cmd\git-lfs.exe
============
Building Git LFS demo ...
Git LFS demo
Git LFS demo
============
D:\a\git-lfs\git-lfs\build\git-lfs.exe
C:\Program Files\Git\mingw64\bin\git-lfs.exe
C:\Program Files\Git\cmd\git-lfs.exe
============

Notice how when we run git-lfs version and git lfs version the first time, before we change PATH, one of the system default git-lfs.exe binaries runs and outputs git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2). However, once we build our demo git-lfs.exe and put it in the PATH, both of those commands now return Git LFS demo because our demo binary runs instead. That makes sense, given that it's now found first by where, as shown.

Next, here is the another workflow, again on a windows-2022 runner, where all we've changed is to replace actions/setup-ruby@v1 with ruby/setup-ruby@v1 and specified ruby-version: 3.1.0 (and we've also tried 3.0.3 and seen the same result):

name: CI
on: [push, pull_request]

jobs:
  build-windows:
    name: Build on Windows
    runs-on: windows-2022
    steps:
    - uses: actions/checkout@v1
    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: 3.1.0
    - run: |
        git version
        git-lfs version
        git lfs version
        echo "============"
        where git
        echo "============"
        where git-lfs
        echo "============"
        echo "Building Git LFS demo ..."
        cd build
        go build -o git-lfs.exe
        cd ..
        PATH="$(pwd)/build:$PATH"
        export PATH
        git-lfs version
        git lfs version
        echo "============"
        where git-lfs
        echo "============"
      shell: bash

The output looks like:

▼ Run git version
  shell: C:\msys64\usr\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
  env:
    TMPDIR: D:\a\_temp
    HOME: C:\Users\runneradmin
    MSYS2_PATH_TYPE: inherit
    MAKE: make.exe
    Path: C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\2.7.3\x64;C:\cabal\bin;C:\\ghcup\bin;C:\tools\ghc-9.2.1\bin;C:\Program Files\dotnet;C:\mysql\bin;C:\Program Files\R\R-4.1.2\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\hostedtoolcache\windows\go\1.16.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.10\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.10\x64;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.322-6\x64\bin;C:\npm\prefix;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\Program Files\Microsoft\jdk-11.0.12.7-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\ProgramData\Chocolatey\bin;C:\Program Files\Docker;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\nodejs\;C:\Program Files\LLVM\bin;C:\Program Files\OpenSSL\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.4\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\SeleniumWebDrivers\ChromeDriver\;C:\SeleniumWebDrivers\EdgeDriver\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Users\runneradmin\.dotnet\tools;C:\Users\runneradmin\.cargo\bin;C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
    ACLOCAL_PATH: /ucrt64/share/aclocal:/usr/share/aclocal
    LANG: en_US.UTF-8
    MANPATH: /ucrt64/share/man
    MINGW_PACKAGE_PREFIX: mingw-w64-ucrt-x86_64
    MINGW_PREFIX: /ucrt64
    MSYSTEM: UCRT64
    MSYSTEM_CARCH: x86_64
    MSYSTEM_CHOST: x86_64-w64-mingw32
    MSYSTEM_PREFIX: /ucrt64
    PKG_CONFIG_PATH: /ucrt64/lib/pkgconfig:/ucrt64/share/pkgconfig
    PROMPT: $P$G
    RI_DEVKIT: c:\msys64
git version 2.35.1.windows.2
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
============
C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe
C:\Program Files\Git\mingw64\bin\git.exe
============
C:\Program Files\Git\cmd\git-lfs.exe
C:\Program Files\Git\mingw64\bin\git-lfs.exe
============
Building Git LFS demo ...
Git LFS demo
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
============
D:\a\git-lfs\git-lfs\build\git-lfs.exe
C:\Program Files\Git\cmd\git-lfs.exe
C:\Program Files\Git\mingw64\bin\git-lfs.exe
============

With this second workflow, notice that now, after changing PATH, we only see our demo git-lfs.exe run by the git-lfs version command, whereas the git lfs version one finds and runs a system default git-lfs.exe. This can be seen by the difference in output; the former returns Git LFS demo but the latter returns git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2). And that's despite the fact that where correctly finds our demo git-lfs.exe first.

A few things are also different, of course. A different bash.exe is run; the first workflow runs C:\Program Files\Git\bin\bash.EXE but the second workflow runs C:\msys64\usr\bin\bash.EXE, and that reports a set of environment variables, including MSYS2_PATH_TYPE: inherit and a PATH in which C:\Program Files\Git\bin is ahead of C:\Program Files\Git\mingw64\bin.

That accounts for another obvious difference, which is that the first workflow finds C:\Program Files\Git\mingw64\bin\git.exe when running git while the second finds C:\Program Files\Git\bin\git.exe and runs that instead. Whichever git.exe is run, though, should not make a difference; both are version 2.35.1.windows.2. Both should respond to the change in PATH which puts our local build subdirectory first.

We are suspicious that there may be some way in which the MSYS2_PATH_TYPE: inherit setting is not taking effect here. We're not familiar with MSYS2 at all, but skimming through the code it appears that this is intended to make it work the "normal" way, with PATH being inherited by any processes invoked by a parent process. But the default appears to be to not follow this convention, and msys2/MSYS2-packages#2140 suggests that perhaps there are ways this setting reverts to the default in various situations. Is it possible that that is what is occurring here, and the Git process we invoke gets its PATH reset back to the one Bash is started with? That would certainly account for Git not finding our demo git-lfs.exe.

At any rate, the overall situation is one which seems akin to the issue raised in #19 and #20, namely, that highly unexpected behaviour results from running ruby/setup-ruby@v1. This issue has taken two days of time to debug just to the point where we could report it in a somewhat detailed fashion, and we still don't understand exactly what's going on.

For the time being we're going to stick with actions/setup-ruby@v1 because it continues to work without breaking our entire Git LFS test suite, which is what happens if we replace it with ruby/setup-ruby@v1 due to this issue of local PATH settings apparently being ignored by child processes.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Feb 18, 2022

What happens if you add another step right after the setup-ruby step that adds build to the env? Some thing like:

echo "$PWD/build" >> $GITHUB_PATH

bash

    - run: echo "$PWD/build" >> $GITHUB_PATH

ps1

    - run: echo "$pwd\build" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

@MSP-Greg
Copy link
Collaborator

@chrisd8088

Forget the above. The issue seems to be the bash shell. So, add the following to your above workflow, which will use the Git bash shell instead of the msys2 bash shell:

      - run: echo "C:/Program Files/Git/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

Also, Ruby 3.1.0 will install the MSYS2 ucrt64 gcc tools, where 3.0.3 will install the MSYS2 mingw64 gcc tools.

See workflow run here

@chrisd8088
Copy link
Author

Hey, thanks for looking at this! I confess I remain baffled as to what is fundamentally happening here.

I'd actually tried the - run: echo "C:/Program Files/Git/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append idea once before while debugging and I've tried it again now, and it definitely resolves the immediate issue; however, for reasons I don't understand, it causes our actual real CI job to hang during a Go build step. So I gave up on that approach, although maybe I did so too quickly. (Here's a run of a slimmed-down version of that job I've been using to debug; it's stalled now as I write this.)

I had an idea overnight which I wanted to try, which was to explicitly use the "old" Git, i.e., C:\Program Files\Git\mingw64\bin\git.exe rather than C:\Program Files\Git\bin\git.exe. That, it appears, makes a key difference; here's the relevant bit of workflow:

    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: 3.1.0
    - run: |
        git version
        git-lfs version
        git lfs version
        echo "============"
        where git
        echo "============"
        where git-lfs
        echo "============"
        echo "Building Git LFS demo ..."
        cd build
        go build -o git-lfs.exe
        cd ..
        PATH="$(pwd)/build:$PATH"
        export PATH
        git-lfs version
        git lfs version
        '/c/Program Files/Git/mingw64/bin/git' lfs version
        echo "============"
        where git-lfs
        echo "============"
      shell: bash

and here's the output:

Building Git LFS demo ...
Git LFS demo
git-lfs/3.0.2 (GitHub; windows amd64; go 1.17.2)
Git LFS demo

So it almost seems as though the issue might lie in the Git binary and/or the binary in combination with the Bash shell.

As a non-Windows developer I'm in deep waters here and can't yet see how a different Git binary (of the same version of Git!) could end up searching different paths when running a command, but I thought I'd try to trace the code a bit and see if anything jumps out. (Trying with GIT_TRACE2=1 hasn't yielded any differences, though, alas.)

@chrisd8088
Copy link
Author

One update -- I realized that trying to prepend C:/Program Files/Git/bin to the $GITHUB_PATH is actually not quite what I think we want to do, and instead we want to prepend C:/Program Files/Git/mingw64/bin.

In the "old" version with actions/setup-ruby@v1, a where git command returns:

C:\Program Files\Git\mingw64\bin\git.exe
C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe

In the "new" version with ruby/setup-ruby@v1, it returns:

C:\Program Files\Git\bin\git.exe
C:\Program Files\Git\cmd\git.exe
C:\Program Files\Git\mingw64\bin\git.exe

I wrote last night:

That accounts for another obvious difference, which is that the first workflow finds C:\Program Files\Git\mingw64\bin\git.exe when running git while the second finds C:\Program Files\Git\bin\git.exe and runs that instead. Whichever git.exe is run, though, should not make a difference; both are version 2.35.1.windows.2. Both should respond to the change in PATH which puts our local build subdirectory first.

But as I noted in the previous commit, it actually seems that which Git binary runs makes a big difference (again, for reasons I don't understand). Adding this step after ruby/setup-ruby@v1 allowed a debug run to make significant progress:

- run: echo "C:/Program Files/Git/mingw64/bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

I'm going to try a full CI job next with that in place. I do wish I had some inkling why this matters, though.

@MSP-Greg
Copy link
Collaborator

Ok. First of all, Windows exe's find library files in a much different manner than *nix. There's newer functions that often aren't used, and there's the legacy method which relies on PATH and other things.

C:/Program Files/Git/mingw64/bin appears to be where the GIT bash files are and there are a lot of library files (dll's) in that folder. C:\Program Files\Git\bin has only three 'stub' exe's files, bash, git, and sh. My guess is the whole reason for the C:\Program Files\Git\bin folder is so PATH doesn't include all the library files in C:/Program Files/Git/mingw64/bin.

Also, I thought I saw a command gem install ronn in the main repo. That command/step should happen before any PATH changes, but after the setup-ruby step.

@dscho
Copy link

dscho commented Feb 20, 2022

I realized that trying to prepend C:/Program Files/Git/bin to the $GITHUB_PATH is actually not quite what I think we want to do, and instead we want to prepend C:/Program Files/Git/mingw64/bin.

That's not quite correct. There can occur some problems with adding the complete /mingw64/bin/ to the PATH, therefore we recommend C:/Program Files/Git/cmd (or, for legacy reasons, C:/Program Files/Git/bin.

The "wrong" git-lfs.exe that you see being used is the one living in C:/Program Files/Git/mingw64/bin/. The reason is that Git for Windows purposefully prepends that directory to the PATH. In the CI that wants to override Git LFS, you are probably best served by calling something like

git -c alias.delete-lfs='!for d in /mingw64/bin /cmd; do test ! -f "$d/git-lfs.exe" || rm "$d/git-lfs.exe"; done' delete-lfs

@dscho
Copy link

dscho commented Feb 20, 2022

FWIW this is related to git-for-windows/git#3668.

@chrisd8088
Copy link
Author

Just as an update, we're likely to try to bypass this issue in the Git LFS CI suite by dropping the use of ronn, and therefore dodging the need to install Ruby at all. That's take us some time, however, as we'll have to rewrite the manual pages' ronn-format source files.

In the meantime, if I find some spare time I'll try to revisit the issue and see if I can make your suggestions work for us; it would be nice to upgrade to the Windows 2022 runners, as we bumped into the problem of not using them again during a recent Git LFS patch release.

Anyway, thanks for the advice and I'll report back if I have any success.

chrisd8088 added a commit to chrisd8088/git-lfs that referenced this issue May 7, 2022
In commit f4ab4e7 of PR git-lfs#4883 we
noted that the "gem install ronn" step in our Windows CI workflows
currently fails on newer Windows Server 2022 GitHub Actions runners,
due to possibly some missing C headers or libraries.

While this could be resolved by upgrading to the "ruby/setup-ruby"
action from the deprecated "actions/setup-ruby" one, that introduced a
series of issues with PATH lookups as documented in ruby/setup-ruby#293.

We therefore chose to continue using Windows 2019 runners until a
resolution or workaround was found.

Fortunately, as of commit 912b607
in PR git-lfs#4992 we are now using the ronn-ng Ruby gem instead of the
unmaintained ronn gem, and the "gem install ronn-ng" workflow step
succeeds on Windows 2022 while still using the "actions/setup-ruby"
action to install a Ruby build environment.  This allows us to
upgrade to the latest Windows runners in GitHub Actions, although
we can't yet upgrade to the "ruby/setup-ruby" workflow action.

h/t to @bk2204 for finding ronn-ng!
@chrisd8088
Copy link
Author

A small update: we're upgrading to ronn-ng (see git-lfs/git-lfs#4992 and git-lfs/git-lfs#4997) which is going to allow us to stick with the deprecated actions/setup-ruby@v1 action for now while still upgrading to the Windows 2022 runners.

We tried also upgrading to the ruby/setup-ruby@v1 action but ran into the same problem with PATH documented above, so we're going to continue with the older action for now, since at least we're unblocked on moving to Windows 2022.

dhiwakarK pushed a commit to dhiwakarK/expert-octo-doodle that referenced this issue Nov 3, 2022
In commit dd03e52 [formerly f4ab4e7445c04fe5d663fc79ddb2ebf30a5a870f] of PR #4883 we
noted that the "gem install ronn" step in our Windows CI workflows
currently fails on newer Windows Server 2022 GitHub Actions runners,
due to possibly some missing C headers or libraries.

While this could be resolved by upgrading to the "ruby/setup-ruby"
action from the deprecated "actions/setup-ruby" one, that introduced a
series of issues with PATH lookups as documented in ruby/setup-ruby#293.

We therefore chose to continue using Windows 2019 runners until a
resolution or workaround was found.

Fortunately, as of commit ccefc90 [formerly 912b607acb162ea324d0ff00fea1a1f47657e1d4]
in PR #4992 we are now using the ronn-ng Ruby gem instead of the
unmaintained ronn gem, and the "gem install ronn-ng" workflow step
succeeds on Windows 2022 while still using the "actions/setup-ruby"
action to install a Ruby build environment.  This allows us to
upgrade to the latest Windows runners in GitHub Actions, although
we can't yet upgrade to the "ruby/setup-ruby" workflow action.

h/t to @bk2204 for finding ronn-ng!


Former-commit-id: 6c234a29a2a2b646b4b3596cd0a386b0df7a4f32
swsnr added a commit to swsnr/mdcat that referenced this issue Dec 1, 2022
Avoid ruby/setup-ruby because it messes with $PATH, see
ruby/setup-ruby#293
swsnr added a commit to swsnr/mdcat that referenced this issue Dec 1, 2022
Avoid ruby/setup-ruby because it messes with $PATH, see
ruby/setup-ruby#293
swsnr added a commit to swsnr/mdcat that referenced this issue Dec 1, 2022
Avoid ruby/setup-ruby because it messes with $PATH, see
ruby/setup-ruby#293
@chrisd8088
Copy link
Author

chrisd8088 commented Dec 21, 2022

A couple of followup notes:

First, I realized that the Actions step suggested in #293 (comment) appends, not prepends, the provided path to the Git binaries to the GITHUB_PATH environment file. That approach of adjusting which Git binary is run, which I discussed further in #293 (comment), can actually be achieved with something like the following:

- run: Write-Output ('C:\Program Files\Git\mingw64\bin' + (Get-Content $env:GITHUB_PATH -Encoding utf8)) | Set-Content $env:GITHUB_PATH -Encoding utf8

That "works" in the sense that the Git binary in the C:\Program Files\Git\mingw64\bin location respects the PATH we then set in our Bash CI scripts, and finds our built version of git-lfs.exe instead of the system-provided one.

However, @dscho noted in #293 (comment) that:

There can occur some problems with adding the complete /mingw64/bin/ to the PATH, therefore we recommend C:/Program Files/Git/cmd (or, for legacy reasons, C:/Program Files/Git/bin).

Unfortunately, the C:/Program Files/Git/cmd Git binary doesn't respect out PATH settings, and always finds the system-provided git-lfs.exe instead of our built one, so that won't work for us.

In the same commit, @dscho's suggested that an alternative would be to remove the system-installed git-lfs.exe binaries. I used this Action step, which does work to ensure our built git-lfs.exe is found first:

- run: Get-Command git-lfs -All | Remove-Item -Verbose

So that might be what we end up using.


Unfortunately, though, that's not quite the end of the story; there are also problems caused by MSYS2 mangling command arguments due to its automatic path translation. One example is that the legitimate Git LFS command git lfs track '**/*.dat' receives its argument as **C:/msys64/*.dat.

So we're experimenting with setting a global MSYS2_ARG_CONV_EXCL=* environment variable value, e.g., with the following, but that leads to other problems, it seems.

- run: Write-Output 'MSYS2_ARG_CONV_EXCL=*' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append

In brief, it would be great to have a way to retain the Bash shell and environment used in actions/setup-ruby@v1, if at all possible.

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 21, 2022

First, sorry about that mix up between prepend & append.

Anyway, re the which git-lfs.exe issue, you're removing the existing and then some PATH gymnastics, maybe use a symlink?

Re the bash issue, can you rename C:/msys64/usr? I think the Git shell is still in PATH, with the MSYS2 shell appended, not sure.

As to the reasons for the MSYS2 path changes, this is messy. Ruby needs MSYS2 build tools, which are installed in directories within C:/msys64.

But, some CI scripts use bash commands. Also, some 'platform agnostic' build tools are located in C:/msys64/usr, so those two uses are the reason for the bash shell change.

We certainly would strongly prefer to not add an input for this, but I suppose we could add a switch based on an ENV setting that could affect it.

@chrisd8088
Copy link
Author

Thanks for the thoughts!

Anyway, re the which git-lfs.exe issue, you're removing the existing and then some PATH gymnastics, maybe use a symlink?

We could try that, but just removing the system-installed git-lfs.exe files, or forcing C:\Program Files\Git\mingw64\bin to be at the front of GITHUB_PATH, both work OK, in the sense that when we run git lfs, the Git which runs doesn't find the system-installed git-lfs.exe but only the one our CI job has just built, and which we've put at the front of PATH.

(To recap the original issue: we build a local git-lfs.exe, put the path to it at the front of PATH, and then expect Git to locate that binary by searching PATH. But this doesn't work with the Git programs in C:/Program Files/Git/cmd or C:/Program Files/Git/bin -- they do something extra, and always find the system-installed git-lfs.exe even though it's not what they should find, given what PATH is set to. So it works for us to either (a) force the use of the Git in C:\Program Files\Git\mingw64\bin, which does obey PATH, or (b) remove all the other git-lfs.exe programs so only ours is findable.)

Re the bash issue, can you rename C:/msys64/usr? I think the Git shell is still in PATH, with the MSYS2 shell appended, not sure.

Great idea, thank you! 🙇 Trying that with the following step works really well, and leaves just a couple of tests (out of about 700) failing:

- run: Rename-Item -Path C:\msys64 -NewName msys64-tmp -Force

It does indeed let the Bash shell in C:\Program Files\Git\bin\bash.EXE run by default, and that lets almost everything that was failing before succeed.

Well, that's definitely progress -- thank you for the great suggestion!

@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Dec 21, 2022

Glad things are working a bit better.

But this doesn't work with the Git programs in C:/Program Files/Git/cmd or C:/Program Files/Git/bin

The symlink might fix that. Windows 'location resolution' for library files (*.dll) has a few mechanisms that take precedence over PATH settings...

Two things.

First, Ruby 3.1 and later (3.2 will be released this month) are compiled with the MSYS2 ucrt tool chain. setup-ruby will install that tool chain and add it to PATH. Ruby 3.0 uses the mingw toolchain. I believe git and git-lfs link to msvcrt.dll, so the mingw chain is more appropriate. At present, the default Ruby used by Actions is Ruby 3.0, but that might change soon also.

This could be causing issues. If so, you could rename C:/msys64/mingw64, but that would need to be done after all gems are installed.

Secondly, things may get strange when MSYS2 switches to OpenSSL 3.0.x, as it's currently using 1.1.1. Ping me if that causes problems.

@chrisd8088
Copy link
Author

One other thing I've run into is the need to unset the TMPDIR environment variable, which I believe is set for all subsequent steps by the ruby/setup-ruby@v1 step.

For us, that produced some CI failures because directories created using mktemp(1) ended up on one volume, but other tools used Go's internal os.TempDir() function, which uses TMP, TEMP, etc., and so created directories on another volume. Our CI tests then failed because some Git LFS commands were unable to move files between the two locations. (Specifically, one would be under D:\a\_temp\... and the other under C:\Users\RUNNER~1\AppData\Local\Temp\....)

Using env -u TMPDIR to run our scripts resolved this issue.

@eregon
Copy link
Member

eregon commented Feb 11, 2023

Is this still an issue or could we close this?

@chrisd8088
Copy link
Author

We've worked around it for Git LFS, but so far as I know it's still an issue.

@MSP-Greg
Copy link
Collaborator

One option would be to add an input (or env setting) that turns off the MSYS2 additions to PATH.

Many workflows would break if it was changed to 'opt-in', but allowing 'opt-out' would be ok...

@eregon
Copy link
Member

eregon commented Feb 14, 2023

I think we should close this as not planned.
Removing MSYS2 from PATH is likely to cause far more problems than help, and Ruby on Windows basically relies on this (in my understanding).
So opting out seems of almost no value.

@eregon eregon closed this as not planned Won't fix, can't repro, duplicate, stale Feb 14, 2023
@MSP-Greg
Copy link
Collaborator

MSP-Greg commented Feb 14, 2023

Just to make clear for non-Windows users:

  1. When installing extension gems with either RubyGems or Bundler, RubyInstaller2 builds will add MSYS2 to PATH. Using rake compile in any form will not, and it occurs in a workflow step after the setup-ruby step is finished.

  2. Some repo's CI code assumes the existence of a 'bash' shell, and the Git shell may not provide all the expected functionality.

Because of the above, and to make Windows CI easier for non-Windows users, the MSYS2 build tools (gcc & packages) needed to build Ruby are installed and added to PATH, and the MSYS2 bash shell is also added to PATH. Note that some of build tools are located in the bash shell path (usr/bin).

These PATH additions should remain the default. Whether an option to 'opt-out' of the PATH additions is added, I don't have a strong opinion...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

4 participants