Switch Unix tools from Git internal tools to MSYS2 tools on Windows images #1525
Comments
AlenaSviridenko
changed the title
|
These changes will remove some Unix tools (that come with Git) from PATH on Windows machines. See the full list of tools in post above. This change won't affect such tools like bash and tar. @bryanmacfarlane @konradpabjan , @damccorm do you know if any of these tools can be used on Windows by |
|
As far as I know we should be good with this change from the Azure Pipelines side |
@maxim-lobanov I believe Windows 2016 relies on the Git Unix tar since only Windows 2019 has the Windows tar (see https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows).
|
|
Besides tar on earlier versions of windows, to the best of my knowledge I think we should be good for the toolkit. We will definitely want to manually install that before we remove this from path. Individual actions or users may be using these utilities though, there isn't a great way to audit that. We should plan on rolling out slowly and rolling back in case we start seeing issues. We do use stuff like |
|
@joshmgross , that is good point. The reason for this change is that we shouldn't rely on Unix tools that come from Git, they can be removed in future. |
|
@maxim-lobanov similar to |
As long as you do not try to call it via |
|
Re: If you want to make sure that those are shipped, and officially supported, it will need to be handled via One plan, for example, is to use BusyBox at some stage in the future, which does have optional support for |
|
Hello everyone, Summary & Details:
Per discussion with Possible options:
Tools with changes versions (3 tools)
Tools that will be removed (54 tools)
Tools without changes (223 tools)
We don't have much experience with Unix tools on Windows but the second option looks like a good way to keep the most of the tools to work as previously and stop relying on Git Unix binaries cc: @dscho as |
|
I don't see any problems with this. ruby/setup-ruby and MSP-Greg/setup-ruby-pkgs are enabling MSYS2 early in their code, and have switched to using the MSYS2 bash shell. Re option 2 above, you may need to also add Note that there are two issues, first, what exe is run from a command/ps1 prompt, and the second, does this affect the use of bash shell with Windows runners. |
|
I am in favor of replacing |
|
Thanks. What is the suggested way to add Git to path, EDIT: I believe the current path contains: then, later in the path: |
|
If you want to be able to use Git for Windows' Bash, you need the |
@maxim-lobanov, IMHO, this is not correct. Instead, I'd say it is because of lack of expertise, and specially lack of attention. I've been following this repository since it was created, and there were discussions about this same issue before (related to MSYS2 and/or WSL). This has been going around for about a year already (before Actions were "officially" out of beta). Overall, many different employees seem to be doing their best effort, but I feel that none of them is being scheduled the time for properly understanding the issue, providing a stable solution and maintaining it. Until managers understand the severe impact of all these arbitrary "fixes", the product is going to suffer (even more), regardless of all the effort that engineers and contributors are devoting. As the maintainer of Git for Windows said, none of the However, given that the mechanism for communicating these breaking changes seems to be "good luck finding an specific changelog", it might be better to do nothing until that is addressed. From a wider point of view, most if not all the users that are relying on Git for Windows' internal GNU/Linux tools, do likely want to use MSYS2 instead. In MSYS2, all those utilities are explicitly expected to be used (#1525 (comment)). Plus, there is a plethora of packages available through Very fortunately, some months ago actions/toolkit#318 was partially addressed, and it is now possible to preset a custom shell, even before the file/script exists. See, for instance: defaults:
run:
# We define a custom shell script here, although `msys2.cmd` does neither exist nor is it available in the PATH yet
shell: msys2 {0}
steps:
# We use a JS Action, which calls the system terminal or other custom terminals directly, if required
- uses: msys2/setup-msys2@v2
with:
update: true
install: base-devel git
# We want to change the configuration of the git command that actions/checkout will be using (since it is not possible to set autocrlf through the action yet, see actions/checkout#226).
- run: git config --global core.autocrlf input
shell: bash
# This uses the default git in the environment, thus Git for Windows
- uses: actions/checkout@v2
# This uses the default shell, which is msys2 (defined above, and "installed" by msys2/setup-msys2)
- run: git describe --dirtyAt the same time, maintainers of MSYS2 were really pro-active and put lots of love into the Action. Updates and fixes are addressed in few hours, and all their knowledge about the environment is seen in the Action. From user's perspective, even if 0.5-1 GB of ad-hoc packages need to be installed, startup takes no longer than 90s. Summarizing, I believe that both Git for Windows and MSYS2 can and should be dropped from the environment. There should be an Action for installing Git for Windows. That Action should also be available as a library, in case other Actions rely on it. Nevertheless, I believe that Actions such as actions/checkout should be compatible with any git (i.e. with the one available through msys/setup-msys2 and/or in WSL too). Alternatively, Git for Windows, MSYS2 and WSL might coexist, along with handy built-in setup scripts and/or custom entry shells. That would be ideal. Unfortunately, that needs a thoughful proposal and discussion, instead of fixing partial issues as they come by and looking at limited scopes. Disclaimer: I am not an MSYS2 expert by any means. I am an average user who maintains a couple of unofficial PKGBUILD scripts. My involvement with GitHub Actions and MSYS2 was the following so far:
During that period, I maintained my fork of setup-msys2 alive, in the hope that the "official" solution would get to be good enough for ditching the Action. That didn't happen. At the end of May, msys2/msys2-installer#5 was opened. It is worth mentioning that @MSP-Greg is apparently not aware of the most basic netiquette. He/she tries to sneak by splitting the discussions in multiple unnecessary threads and NOT pinging users which he/she knows would provide arguments discussing his/her proposals. I tried to explain what taking care of a codebase implies: #916 (comment). He/she responded with, msys2/msys2-installer#5, another effort to mess the discussion. So, at some point, I was fed up with trying to have an (at least) decent official MSYS2 solution on GitHub's side. Fortunately, the reaction of MSYS2 maintainers was lovely. In a couple of weeks, there was an official Action and automatic caching was enabled. In July, lots of enhancements were contributed. Now, I help maintain msys2/setup-msys2, and I understand the codebase, but I'm not an expert on the technical details. Note that the enhancements in setup-msys2 are compatible with the built-in MSYS2 install provided in this repo (see https://github.com/msys2/setup-msys2#release), but I am not aware of anyone using it since the startup time is typically worse than just ignoring it (see msys2/setup-msys2#23 (comment)). Please, please, someone (employee, contributor, maintainer...) step back and think about the complexity of "having bash and git in Windows". If you are not familiar with some or any of the tools, that's ok. I believe that maintainers of Git for Windows, MSYS2, Chocolatey, etc. all will be delighted to have some public and clarifying discussion about how should all of them coexist in GitHub Actions (both, in the environments and in the ecosystem). But, please, stop doing fast and not properly thought modifications (after this issue, which is NEEDED). Users can solve their own typically complex ad-hoc problems faster than you (or anyone else) can even understand them. IMHO, you should focus on providing an stable environment that makes sense (any sense, you define it). |
I could say the same. Re removing MSYS2:
Hence, I feel that MSYS2 should remain installed. My name is Greg, and I identify myself as male. |
|
@eine @MSP-Greg I really value seeing both of your perspectives spelled out clearly. Maybe we could meet real quick on https://meet.jit.si/GHA-MSYS2 to hash some things out "in person"? |
I'm going to log off for the day, but maybe we can agree on a time to meet face to face, to resolve any conflicts and work on a mutually satisfactory solution? |
|
Re meet.jit.si, never used it, just installed on my phone. Obviously, I feel strongly that MSYS2 should be installed on the Actions Windows image(s). Past that, I'm not sure. I've worked with MSYS2 for a while, build packages locally (most recently OpenSSL 3.0.0), but I'm not a strong c type. I've also used it to build Ruby master/main, and the build has been widely used in Ruby related CI for a few years. It was originally built and stored on AppVeyor, now it's also built on Actions. I think you, @eine, and myself have dealt with MSYS2 enough to always 'make it work'. With custom actions, we can match the needs of other users and preset the environment for them. @eine and I are involved with separate actions to accomplish that. Assuming that having MSYS2 installed but not enabled is ok, the issue here is what makes sense for the majority of Actions users. I'm not sure I know the answer to that. To me, two main issues. How to configure the bash shell, and conflicts. JFYI, I'm -0500 (central US) |
@MSP-Greg, sure, you are entitled. However, I explained why I thought it and what is the implication for other users. This is not about you and me, but about how our actions affect other users. I'd be so glad if you could warn them properly. Right now it's a childish "you too", again.
This is a false premise that has been discussed and replied before. Please, bring the relevent references instead of trying to tendentiously make a null point, again. Basic netiquette.
Again, false assertions without referencing relevant discussion which you are aware of. Bring them. Basic netiquette. I did not mention, it is tested by tens, hundreds, thousands of users. Because of your decisions, the built-it solution needs up to 7-12min. It was not my decision/suggestion to ditch the built-in installation. A maintainer of MSYS2 suggested it after being surprised by all the packages installed by default: msys2/setup-msys2#23 (comment). Nonetheless, I agree with the decision, of course. Note to readers, "for many repos CI" and "all the other code" do mean "his many repos" and "all his other code". There has been no analysis, at least he has not done any public, about the truthfulness of those mantras.
Ideally, MSYS2 would be available as a built-in resource. I said that. I did never propose that CI shouldn't have any software. That's something that you seem to have proposed out of nowhere. What I said, and I still propose, is that no one seems to be taking care of the elephant in the room. Then, better open the garage door and let it run free.
Some people NEED to use the MSYS2 Action for avoiding your opinionated and arbitrary decisions. In order to reduce those 90s for your specific use case, you are forcing a +300s penalty in the workflows of all the users which want to use an updated solution. Your reply is: msys2/setup-msys2#23 (comment). And you were explicitly told that partial updates are NOT supported. This one is not about netiquette, but about basic etiquette, since it's other people's money that you are wasting. Fortunately, thanks to MSYS2 maintainers, the Action can partially alleviate this severe design flaw.
This is the premise of this issue. Here, "users" and "the Actions team" are discussing how to adjust the PATH for meeting their needs. We agree that removing everything from the PATH (and from the environment) is a solution for avoiding conflicts. However, I believe that's not the solution (integration) we are aiming for.
As explained several times, what you think is irrelevant, given your purposely limited point of view. Let me just place an counterexample: - uses: msys2/setup-msys2@v2
with:
msystem: MINGW32
update: true
- uses: ghdl/setup-ghdl-ci@nightly
with:
backend: mcode
- shell: msys2 {0}
run: |
ghdl --versionCurrently, GHDL is the only open source solution for testing and synthesis of hardware designs written in VHDL.
I disagree. MSYS2 should only remain installed if properly taken care of.
Thanks. @dscho, I prefer not to use either video or voice. However, I am open to engaging the discussion in any chat, pad, wiki... Nonetheless, as said, I believe I am not a relevant agent to take decisions. The discussion needs to be public and open to maintainers of other projects. Otherwise, we will be having this scope limited conversation again and again. Two months ago, I proposed Greg to translate part of setup-msys2 to Powershell, so that it could be integrated in this environment (see msys2/msys2-installer#5 (comment)) and other environments. He passed, again.
@MSP-Greg, as explained in #1525 (comment), I am involved with providing a usable environment in GitHub Actions that allows using MSYS2 without favouring a very specific subset of users and severily penalizing others. This is specially relevant because GitHub Actions are paid per minute, not per run. The fact that I am involved in a general purpose Action is just a result of the poor built-in solution. I wish setup-msys2 did not exist. I am a maintainer of other projects, and I'd prefer to spend time on those. setup-msys2 is just a tool that I (and others) need for testing software on Windows. It is quite tendencious on your side to put on the same page my involvement in setup-msys2 and your involvement in any of the actions you maintain. Neither the scope nor the motivation have any relation. Moreover, you were invited multiple times to help extending setup-msys2 for suiting your specific needs, instead of creating one or multiple custom Actions. You ignored those. |
@eine fair enough. I only proposed it because I see that emotions run unnecessarily high, and in the past I found that a quick face-to-face conversation helped alleviate similar situations. There is just so much lost when using text-only communication, and it is very easy to make much more aggressive arguments than intended. I am convinced that we can resolve this issue in a way that meets the needs of all (currently) involved parties. |
|
@dscho, just to clarify: I am not a native speaker, and my comments might sound harder than they should. I want them to be direct, and clear, not rude. I am neither angry, nor do I have any personal issue with Greg. I would love to collaborate and find a technical solution that fits all the use cases with as little conflicts as possible. That includes discussing technical details and objective decisions with Greg. My frustration is motivated by how little attention is GitHub as a company paying to their GitHub Actions product. It came out of beta almost a year ago and the UX is still awful. They had (and still have) lots of references to learn from and to copy from. Yet, there is no captain sailing this boat. That is, Greg pushed his PRs and the environment is as it is now because of GitHub, not because of Greg. In any regular open source project, the review process would have arised all of this problems, and those would have been prevented. EDIT Of course, Greg (and any other) is absolutely entitled to replying "I don't want to do that", "I don't want to spend more time on this" or "I believe that employees should do it instead of me". Those are all valid points/replies in a discussion. |
|
I followed the references in #1525 (comment) and I didn't identify those as self-hosted agents. Could you please clarify how you reached that conclusion? My understanding is that self-hosted agents don't pull the virtual environment provided in this repo. |
|
Sorry, I meant to say that they're trying to run their workflows in their own laptops and VMs as if they were self-hosted agents.
That's exactly it! If your workflows don't require MSYS2, you shouldn't need it. But that's the exact opposite of what you suggested. It's still a bad idea, in my opinion, to try to convince users that they have to use MSYS2 when they don't actually want to. |
|
I think there is a misunderstanding. I did not say that they have to use MSYS2. I suggested that their workflow might be significantly simplified if they used MSYS2 instead of Git for Windows. At least 30 lines can be removed straightaway. That is because Git for Windows does not suffice and they need to install an external package manager as well as setting up custom paths for the dependencies installed through that manager. Since the missing dependency is |
|
The target date in 0th post is set to Nov 2, is the |
|
Using: bash shell is listed in the log as |
|
I.e. target date from 0-th post is incorrect... |
@igagis, I think it was set on date, but then reverted: #1525 (comment). |
|
I have updated the first post to be clear: |
|
Just for information for those who want to use To switch After this step |
|
@igagis, @ericsampson, @pepijndevos, that was the reason for the change to be reverted, and what is delaying this issue being fixed. Please, see options The easiest and most robust solution for using the built-in MSYS2 installation is the following: defaults:
run:
shell: msys2 {0}
steps:
- uses: msys2/setup-msys2@v2
with:
update: false
release: false
- run: |
Your commands, already on msys2
- run: |
Your commands, NOT in msys2
shell: bash
- run: Some other command, also on msys2 (by defaults above)See Usage for guidelines about how to execute either one command or a sequence of commands in MSYS2. EDIT An example using numpy, matplotlib and openssl inside MSYS2: https://github.com/ghdl/ghdl-cosim/blob/master/.github/workflows/test.yml#L67-L87 |
|
As a further note, the reference workflow above should remain almost unaffected, regardless of the modifications to the virtual environments in this repo. That's because all MSYS2/MINGW32/MINGW64 shells are isolated from the system PATH by default. Users need to explicitly set an option for making it visible. That avoids most of the reported conflicts, per se. In fact, maintainers of this repo nicely avoided introducing aliases which would conflict with the The second run command ( My hope is that, at some point, this repository will borrow the entrypoint only from setup-msys2, or it will present it as the canonical action to be used for running MSYS2 shells. |
The main purpose of the `minimal-sdk` artifact is to serve in automated Azure Pipelines and GitHub workflows. Let's make sure that we are not picking up any unintended executables along the way. One such unintended executable would be Git for Windows' gpg.exe: Git for Windows is installed in the build agents of Azure Pipelines and GitHub Actions, and its `/usr/bin/` is sadly in the `PATH` and will most likely remain there indefinitely, for details, see actions/runner-images#1525. However, this can lead to `gpg.exe` being picked up by Git's test suite, and unfortunately the GPG agent tends to hang in the VMs in which the tests are run. So why does this not happen all the time? Most likely due to mismatches of the MSYS2 runtime, which let `gpg.exe` fail to load (and which Git's test suite takes as sign that "we looked all around but no GPG could be found"). This is a fragile situation, of course: at any point, outside of Git's (or Git for Windows') control, `gpg.exe` can start working again. By specifically overriding the `PATH` to only include the `bin/` directories of the minimal SDK, plus Windows' system directories, we can avoid this from ever happening again. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Okay. Well, we tried. And it seems that we simply have to add Git's internal tools to the I guess I will have to keep the build agents in mind going forward, which binds my hands when it comes to e.g. removing Bash in favor of Dash or trying to use BusyBox' ash instead. An unideal situation for me as Git for Windows maintainer! But it apparently cannot be helped, that ship has now sailed. |
|
Is there a solution for using the MSYS2 tools in Azure Pipelines? I'm having some issues compiling a package on CI because the git tools do not correctly handle windows paths - |
AFAICT you simply have to add |
|
@planetmarshall I'd say it can handle Windows paths, but it cannot handle paths with spaces (without quoting them). So: |
It is better to use special action to switch to msys shell, as described in this comment: #1525 (comment) |
Is that applicable to Azure Pipelines? |
|
@planetmarshall I suppose, it is applicable as long as we are talking about github actions. |
|
@igagis I think that they can use an entrypoint such as https://github.com/msys2/setup-msys2/blob/master/main.js#L164-L168 in Azure Pipelines, but not the msys2/setup-msys2 Action. Nevertheless, the error seems to be related to an unquoted path containing an space. |
|
Hello everyone! Unfortunately, there is no easy way to switch default Bash for existing images and don't affect customers.
@dscho , JFYI some other CI systems use Git for Windows as default Bash too and depend on pre-installed tools, for example Travis CI. |
|
FTR, how to setup |
|
I wonder if there shouldn't just be an Pretending they're NOT completely different environments hasn't really been to anyone's benefit so far, seems to me... And if the Windows people really want a |
I'm no expert, but I believe that a "Windows bash" exists in |
and others
Breaking changes
"Git For Windows" Unix tools will be replaced by "MSYS2" Unix tools on Windows images.
Currently, folders
C:\Program Files\Git\binandC:\Program Files\Git\usr\binandC:\Program Files\Git\mingw64\binare added to PATH and it expose a ton of Unix tools (Git For Windows Internal tools) to PATH. We would like to remove those folders from PATH because one day they can be removed by Git maintainers.A lot of people already depend on those tools so we would like to replace Git Unix tools by MSYS2 Unix tools to reduce impact.
Target date
Deployment will start on November, 2 and changes propagation will take 2-3 days.This change was rolled back and should be reevaluated
The motivation for the changes
Per communication with Git For Windows maintainers, we should not add Git Internal tools to PATH and rely on it. They could be removed in future and break a lot of people.
More details can be found in this PR by @dscho #1435.
All discussions can be found in this issue, also in MSYS2 issue #1572 and in the final PR #1648
Possible impact
The most of the tools are identical between "Git For Windows" Internal and MSYS2 but some minor versions could be different and it can affect some builds.
Mitigation ways
You will be able to return "Git For Windows" Unix tools to PATH using the following powershell step:
The text was updated successfully, but these errors were encountered: