MSY2/MINGW example workflow

[eine]

See msys2/setup-msys2 for an up-to-date and actively maintained solution.

---

Please, provide an example workflow to build MSYS2/MINGW packages with makepkg-mingw. I am currently using the following steps:

    - name: Install MSYS2
      run: choco install msys2
    - uses: actions/checkout@v1
    - name: Build and (hopefully) install package
      run: C:\tools\msys64\usr\bin\bash -lc "./msys2-mingw/run.sh -b"
      env:
        HOME: ${{ runner.workspace }}/myrepo
        MINGW_INSTALLS: ${{ matrix.task.installs }}
        CI: true
    - name: Test package
      run: C:\tools\msys64\usr\bin\bash -lc "source /usr/bin/shell $MINGW_INSTALLS; ./msys2-mingw/run.sh -t"
      env:
        HOME: ${{ runner.workspace }}/myrepo
        MINGW_INSTALLS: ${{ matrix.task.installs }}
        CI: true

However, this is not ideal, because:

• Installing MSYS2 takes 4-5 minutes.
• Repeating the envvars in both tasks is redundant.
• I did not find how to use the following syntax:

     run: |
        commandA
        commandB

[eine]

This is the workflow: https://github.com/1138-4EB/ghdl-packaging/blob/master/.github/workflows/mingw.yml and this is the main script: https://github.com/1138-4EB/ghdl-packaging/blob/master/msys2-mingw/run.sh. As you see, I'm not using any *.bat file, but *.sh instead.

examples of how one might pulling various mingw dependency libraries (SDL, or something common, or even just the syntax).

pacman -S --noconfirm <THE_PACKAGE_YOU_WANT>

[eine]

Should anyone be considering to propose a PR, please note that action https://github.com/1138-4EB/setup-msys2/tree/master exists. Unlike the example above, the action caches MSYS2, instead of installing it through chocolatey. This reduces startup time from 5min to 30s.

Maybe setup-msys2 can be adopted in github.com/actions?

[eine]

@krcroft, I'm afraid that either you are confused or I am confused, but I don't understand what you mean with @andymckay's approach. AFAIK, I am the only one that proposed multiple approaches in this issue. The only thing @andymckay did was to add a label yesterday.

[eine]

@1138-4eb, it would make sense to provide msys2 and mingw as first-class canned development environments under Windows system choices

I completely agree. I don't understand why do they provide such a limited bash (Git for Windows, I assume), given that MSYS2 and WSL are available. Furthermore, I found the tests to run at least two times slower than AppVeyor; which is nonsense to me, since GitHub is using Azure under the hood, isn't it?

Likewise, the commands here: 1138-4EB/setup-msys2@master are obfuscated.

I do not agree here. Commands in setup-msys2 are as clear and easy to find as it could be. It is a 50 line script, with three specially relevant commands only:

https://github.com/1138-4EB/setup-msys2/blob/494967ab5f95f19ba276a004babe37b1ecdc2e61/main.js#L37-L39

The fact that it is written in JS doesn't make it more obfuscated than writing in bash or YAML.

An ideal CI workflow is transparent and transportable out of its hosting environment: one that can be readily reproduce on any user's system by simply copying and pasting commands into the users' shell.

The proposed CI workflows, be it with or without setup-msys2, are as transparent and transportable as they can be. This is because in this issue we are only covering how to install a full-featured MSYS2 environment in the windows VMs provided by GitHub. Either the YAML file or the action are specific to this environment, because I do not consider the installation procedure of MSYS2 to be transportable; neither to be worth transporting it if it was. Nevertheless, all the relevant logic is defined in a helper bash script written by the user (see ./msys2-mingw/run.sh in the first comment above, or test.sh in https://github.com/1138-4EB/setup-msys2/blob/master/.github/workflows/test.yml). The bash script is portable across GNU/Linux, windows and macOS. Precisely, the motivation of this issue and of setup-msys2 is to keep existing test/build scripts untouched.

The steps you provive above can be dropped into any user's desktop and run (assuming they have Chocolatey installed).

This is to assume too much. I don't have chocolatey installed on any of my windows workstations. With a standard MSYS2 and Docker I have everything I need. Plus, I wouldn't suggest anyone to copy and paste the following on a cmd on their desktop:

export HOME=whatever
export MINGW_INSTALLS=whatever
C:\tools\msys64\usr\bin\bash -lc "./msys2-mingw/run.sh -b"
C:\tools\msys64\usr\bin\bash -lc "source /usr/bin/shell $MINGW_INSTALLS; ./msys2-mingw/run.sh -t"

Instead, I'd suggest to open a MINGW64 shell and do:

./msys2-mingw/run.sh -b
./msys2-mingw/run.sh -t

Where as commands like numworks/setup-msys2@v1 and msys2do are internal to GitHub and contribute to vendor (GitHub in this case) lock-in.

Of course the action is locked to the vendor, but just the same as the YAML file. That's why the action is supposed to be used in the YAML file and nowhere else. By the same token, the YAML is expected to be used in GitHub Actions, and nowhere else.

msys2do is generated/created by the action. It is neither internal nor exclusive to GitHub. It is the helper bat file you need to execute a command in any MSYS2 shell programmatically from a windows cmd terminal. The point is that most windows users will start a MINGW64 shell graphically, instead of opening a cmd terminal to then call MINGW64 as a child.

Note that, as commented above, none of this would be required if Actions supported shell: msys2 or if shell: bash was a decent MSYS2 installation.

So even though I appreciate the speed improvements, I like your above transparent approach better because I can ultimately wrap it in a .bat file and run it locally or move it to GitLab, etc..

I feel that you are overlooking the fact that the .bat file you suggest is exactly the same as msys2do + test.sh. Nevertheless, although my comments might sound strongly opinionated, please feel free to further discuss. I think that this dialogue is very interesting.

I like the externally-portage approach better, but the 5-minute install time is brutal. Is there a way to achieve both?

This is mostly because setup-msys2 downloads a tarball and saves it in the cache. Then, it is just extracted with tar:

https://github.com/1138-4EB/setup-msys2/blob/494967ab5f95f19ba276a004babe37b1ecdc2e61/main.js#L26-L32

Hence, it is possible to replace the chocolatey command with:

curl -fsSL http://repo.msys2.org/distrib/x86_64/msys2-base-x86_64-20190524.tar.xz | tar -x -J --force-local -C <DEST_DIR>

However, I don't think it is possible to use the cache without @actions/tool-cache. Precisely, this is the main motivation for me to prefer setup-msys2 over the YAML file in the first comment. I don't like having to use JS at all. But since I'm forced to, I prefer to put as much of the logic specific to GitHub Actions in it.

EDIT

Clarification regarding "Of course the action is locked to the vendor":

setup-msys2 can be executed locally by running node main.js. However, I don't think it is useful, because I would expect users to follow any other procedure to install MSY2 on their workstations.

[eine]

I was comparing it to running choco <...>, which only involves installing chocolatey.

Gotcha, and agree. When I said that obfuscation was the same, I was thinking on steps such as C:\tools\msys64\usr\bin\bash -lc "source /usr/bin/shell $MINGW_INSTALLS; ./msys2-mingw/run.sh -t".

Thanks for expanding upon how all this works though - it's interesting, but yes, I would also prefer just a clean and useable shell: bash as you've described.

I think we should focus on making this message be heard loud. You might want to comment in https://github.saobby.my.eu.orgmunity/t5/GitHub-Actions/Windows-MSYS2-Ruby/m-p/33946#M1727

Is your work being fed into or being reviewed by GitHub's workflow team? (what do I know.. maybe you're already a github employee?)

Oh, not at all. I am a regular user trying to migrate existing workflows from Travis CI and AppVeyor to GitHub Actions. I opened a few other issues (https://github.com/actions/starter-workflows/issues?utf8=%E2%9C%93&q=author%3A1138-4EB+, https://github.com/actions/toolkit/issues?utf8=%E2%9C%93&q=author%3A1138-4EB+), but that's all.

In anycase - your effort and ideas here are where GitHub should be going, so I hope there's a mechanism for you to contribute and see this through to a tidy solution!

Unfortunately, employees are not being as responsive as in other projects. I feel they are overwhelmed. Let's hope GitHub managers can put more resources to bring GitHub Actions to the hype they built.

the client-side javascript involved in showing that step in the UI absolutely locks up for 30+ seconds.

I found this behaviour too. Some steps consistenly lock for up to 5-10 mins, and all the output is shown after the step or job is finished.

I now only use the 'raw log viewer' instead of the UI. Is it possible to redirect its stdout to a file or even NULL? (I'm not sure what command line parser it's using, so unsure of its redirection syntax).

The default terminal on windows-latest is cmd. However, you should be able to execute choco install msys2 with either shell: cmd, shell: powershell or shell: bash. Therefore, it is up to you which redirection mechanism to use.

Edit: just found we can append --no-progress to choco install PKG1 PKG2

I will focus on setup-msys2, because it better suits my needs. Nevertheless, should you wish to, I'm ok with replacing the code block in the first commit of this issue with a link/reference to gist of yours. That would allow you to add this and hopefully other variants, along with comments about that workflow syntax. This also because, should I contribute a workflow to this repo, it would probably be based on setup-msys2 since it is more idiomatic in this context.

[eine]

Features from 1138-4EB/setup-msys2 are now merged upstream (numworks/setup-msys2).

[Rkbhasan1]

Please, provide an example workflow to build MSYS2/MINGW packages with makepkg-mingw. I am currently using the following steps:

    - name: Install MSYS2
      run: choco install msys2
    - uses: actions/checkout@v1
    - name: Build and (hopefully) install package
      run: C:\tools\msys64\usr\bin\bash -lc "./msys2-mingw/run.sh -b"
      env:
        HOME: ${{ runner.workspace }}/myrepo
        MINGW_INSTALLS: ${{ matrix.task.installs }}
        CI: true
    - name: Test package
      run: C:\tools\msys64\usr\bin\bash -lc "source /usr/bin/shell $MINGW_INSTALLS; ./msys2-mingw/run.sh -t"
      env:
        HOME: ${{ runner.workspace }}/myrepo
        MINGW_INSTALLS: ${{ matrix.task.installs }}
        CI: true

However, this is not ideal, because:

• Installing MSYS2 takes 4-5 minutes.
• Repeating the envvars in both tasks is redundant.
• I did not find how to use the following syntax:

     run: by http://
        commandA
        commandB

[Rkbhasan1]

Good work.

[Termux-squad]

Please, provide an example workflow to build MSYS2/MINGW packages with makepkg-mingw. I am currently using the following steps:

    - name: Install MSYS2
      run: choco install msys2
    - uses: actions/checkout@v1
    - name: Build and (hopefully) install package
      run: C:\tools\msys64\usr\bin\bash -lc "./msys2-mingw/run.sh -b"
      env:
        HOME: ${{ runner.workspace }}/myrepo
        MINGW_INSTALLS: ${{ matrix.task.installs }}
        CI: true
    - name: Test package
      run: C:\tools\msys64\usr\bin\bash -lc "source /usr/bin/shell $MINGW_INSTALLS; ./msys2-mingw/run.sh -t"
      env:
        HOME: ${{ runner.workspace }}/myrepo
        MINGW_INSTALLS: ${{ matrix.task.installs }}
        CI: true

However, this is not ideal, because:

• Installing MSYS2 takes 4-5 minutes.
• Repeating the envvars in both tasks is redundant.
• I did not find how to use the following syntax:

     run: |
        commandA
        commandB

[randy3k]

I use a more direct approach at here: https://github.com/randy3k/winpty/blob/b3be9209ae7fc8597a34a256583dc3c01ff08391/.github/workflows/build.yml#L37

It involves downloading msys2 and invoking bash directly.

      - name: Pacman
        run: |
          $env:MSYSTEM = "MINGW${{ matrix.arch }}"
          $env:MSYS2_PATH_TYPE = "strict"
          $script = @'
            pacman --noconfirm -S msys/gcc
            if [ "$MSYSTEM" = "MINGW32" ]; then
              pacman --noconfirm -S mingw32/mingw-w64-i686-gcc
            else
              pacman --noconfirm -S mingw64/mingw-w64-x86_64-gcc
            fi
            pacman --noconfirm -S make
          '@
          & "$env:GITHUB_WORKSPACE\msys${{ matrix.arch }}\usr\bin\bash" -c -l $script

[eine]

@randy3k, you might want to use https://github.com/numworks/setup-msys2 instead.

[randy3k]

I tried, but the implementation is kind of weird. I prefer a more direct approach and have more control how the shell is called.

[eine]

IMHO, the implementation is exactly the same as yours. The only difference is that setup-msys2 is coded in JavaScript/cmd and you coded in YAML/bash. Should you want finer control, it is up to you to replace script msys2do.cmd with your own, or to ignore it and call bash directly. Alternatively, you can just fork setup-msys2. The point is that your approach requires to C&P at least 50 lines (including multiple steps) to every new workflow. I'd say that the UX experience is much worse, unless you only need to set it up for a single project.

I think that both implementations (setup-msys2 and yours) are weird because of actions/toolkit#232.

[randy3k]

It's fair enough. The only relevant lines are
https://github.com/randy3k/winpty/blob/b3be9209ae7fc8597a34a256583dc3c01ff08391/.github/workflows/build.yml#L20-L28
and
https://github.com/randy3k/winpty/blob/b3be9209ae7fc8597a34a256583dc3c01ff08391/.github/workflows/build.yml#L52

So in total, there are only 10-ish lines needed. One plus thing for my setup is that it also handles i686/x86_64 sys2 and it's more transparent how I could cache the msys2 setup because I have the control of the download position of msys2. Of course, one could argue that it is also possible for setup-msys2 to add supports i686 and caching. And you are right, I only need this for this project, so I call it done now.

[Nil1995-15]

Random chat

[github-actions]

This issue has become stale and will be closed automatically within a period of time. Sorry about that.

[eine]

See msys2/setup-msys2 for an up-to-date and actively maintained solution.