PKG_CONFIG_PATH not exported from windows to msys2

[eyal0]

This is a new error that I'm getting in GitHub CI. It started about 2 weeks ago. Seems like cygpath might be provided by git for windows, which I have never explicitly installed... Maybe GitHub changed something about their environment and cygpath is no longer in there?

/usr/bin/bash: line 1: cygpath: command not found
/usr/bin/bash: line 1: source: filename argument required
source: usage: source filename [arguments]
Error: Process completed with exit code 1.

[Biswa96]

Please provide your GHA yml file or minimal sample to reproduce your issue.

[eyal0]

It's probably due to the craziness that I try to make msys behave like bash in my scripts. Getting a single script to work for both unix and windows is a pain.

It would be nice is GITHUB_ENV and PKG_CONFIG_PATH just plain worked instead of all the workarounds needed for github ci.

[eine]

With path-type: inherit, those envvars should work.

[eyal0]

I'll try it again. I had a very convoluted solution to my problem many months ago that stopped working 2-3 weeks ago. I'll go back to the simply solution and see if it is working for me.

[eine]

That might be because the default installation path changed (#163). If you were relying on absolute and hardcoded paths, you might need to fix those.

[Biswa96]

Getting a single script to work for both unix and windows is a pain.

You could check msys2 specific environment variables in the shell script. Like MSYSTEM=mingw64 or OSTYPE=msys etc.

[eine]

@Biswa96, see #104 for context.

[eyal0]

Yeah, I had a really bad solution. I'm going to try to make it work cleanly. Maybe all that I ever needed was path-inherit? I thought that I had done the first time that I tried and I didn't succeed. But maybe I didn't try it enough or maybe path-inherit works better now?

Either way, I'll give it a go!

[eine]

There were some changes in the virtual environments provided by GitHub. Previously MSYS2's PATH was added by default, and several packages were preinstalled. Therefore, inheriting that in the clean installation might produce conflicts. Now, in windows-2022 no extra packages are preinstalled, and I'm unsure about having it pre-added to the PATH.
Anyway, I would give it a try again.

[eyal0]

@eine I've tried using path-inherit but I'm not getting the result that I expect.

In my CI, I have a place where I'm printing out the entire environment. Here's what it looks like:

You can see where I'm adding the variables for PKG_CONFIG_PATH, LD_LIBRARY_PATH, and PATH.

Here's where I'm printing the env by running the env command in msys2:

You can see that the CI is honoring my environment changes. But that's in the environment to windows, not to Unix! It's up to msys to "inherit" that.

Here's the output of env for the PATH variable:

I can see my $HOME/.local in there, so that's good! What about PKG_CONFIG_PATH?

Nope. Unlike PATH, where the variable is copied from the windows environment into the msys environment, PKG_CONFIG_PATH is not! This is why I had to do that big kludge. It would be nice if PKG_CONFIG_PATH got the same treatment as PATH. I've even settle for having all the windows environment variables copied over into new names so that I could at least use them inside of msys and do it myself!

For now, I have no solution. I could add source $GITHUB_ENV to every single rule in the CI but that's pretty ugly! I tried to add that into the shell but it's really difficult, too, because GitHub has expectations about how that shell command will look.

What can I do to fix this?

[eyal0]

With path-type: inherit, those envvars should work.

They don't. :-( Do you have a CI output that I can look at which proves otherwise?

[eyal0]

You can see here where PATH gets special handling: https://github.com/msys2/MSYS2-packages/blob/915946a637e1f2b7e26e32782f3af322009293db/filesystem/profile#L28-L45

PKG_CONFIG_PATH does not: https://github.com/msys2/MSYS2-packages/blob/915946a637e1f2b7e26e32782f3af322009293db/filesystem/profile#L53

[eyal0]

@eine Okay, I found a solution to my problem. I'll describe it here and the different things that I tried. Hopefully it will save someone time in the future. If you don't have time to read it, here's the trick: Use ~/.bash_profile instead of $GITHUB_ENV for everything.

Problem

Attempt 1: $GITHUB_ENV

We want to use a single GitHub CI for all builds, macos, Windows, and Ubuntu. This makes our CI smaller and more manageable. macos and unix are pretty easy because they are mostly compatible but Windows is a little trickier, especially about the environment variables! The issue is that if you put something like this in your CI:

- name: Set env variable FOO to bar
  run: |
      echo "FOO=bar" >> $GITHUB_ENV

It will set the variable in the environments for macos and for ubuntu, but it won't work the same for Windows. That's because it is setting the environment variable in the windows environment, which is, like the powershell. But you want the environment variable set inside of msys2.

Attempt 2: path-type: inherit

The solution should be to use path-type: inherit. You can read about how it works. If you look at the code for it here and here, you'll see that inherit only affects PATH and not other things that you might set, like PKG_CONFIG_PATH. That's a bummer because we'd like those to work, too.

Attempt 3: source a different file

One solution would be to instead write to a different file, maybe called ~/my_env and then source it at each command. So you would need to modify every single command in your CI file to have a source line at the top. Like this:

       - run: |
         source ~/.my_env
         do_the_rest.sh

That's annoying! We don't want to have to do that.

Attempt 4: source it in the shell

GitHub will let you change the shell command for your platform. So you might try something like this:

    defaults:
      run:
        shell: source ~/my_env; /bin/bash {0}

Now it should source your file before running the shell each time. I tried that but it didn't work. GitHub was complaining that source wasn't an executable file on the disk. That's true, source is a command that bash know about, it's not a file that can be executed.

Attempt 5: Put it in .bashrc

We know that bash will always run .bashrc before it starts. So we could just put the lines that we want in .bashrc. However, one problem: .bashrc is only run for interactive shells. This is described in the man page for bash:

       When an interactive shell that is not a login shell is started, bash reads and executes commands from /etc/bash.bashrc and ~/.bashrc, if these files exist.  This may be inhibited by  us‐
       ing the --norc option.  The --rcfile file option will force bash to read and execute commands from file instead of /etc/bash.bashrc and ~/.bashrc.

So we need to make bash interactive. So we need to modify the shell to be interactive.

    defaults:
      run:
        shell: source ~/my_env; /bin/bash -i {0}

This causes other problems, though, because if the shell is interactive then some of the stuff that you do in your CI will behave differently and your tests will fail in a weird way. So this is no good.

Attempt 6: Put it in .bash_login.

~/.bash_login is run only for login shells. And we can fake that by running bash with a -l. So that's a good technique for us!

Solution

So in the end, here's what the CI script could looks like:

jobs:
  my_job:
    name: ${{ matrix.os }}_job
    strategy:
      matrix:
        os: [ubuntu, macos, windows]
        include:
          - os: windows
            shell: msys2 {0}
          - os: ubuntu
            shell: '/usr/bin/bash -l -e -o pipefail {0}'
          - os: macos
            shell: '/bin/bash -l -e -o pipefail {0}'
    runs-on: ${{ matrix.os }}-latest
    defaults:
      run:
        shell: ${{ matrix.shell }}
    steps:
    - name: Setup paths and env
      run: |
        rm -f ~/.bash_profile
        echo "export FOO=bar" >> ~/.bash_profile

What this does is:

• Create a matrix of jobs, where the variable os is one of ubuntu, windows, and macos. So matrix, if we converted it to json, would be 3 jobs, each with a different value of os. Here's what the job matrix looks like in yaml:

          - os: windows
          - os: ubuntu
          - os: macos

• Now we have the include lines, which attach the shell variable to all elements of the matrix with os that matches. This way, if you have an even more complex matrix with lots of rows you don't have to attach the shell to each one. This is what we have now in yaml for our matrix.

          - os: windows
            shell: msys2 {0}
          - os: ubuntu
            shell: '/usr/bin/bash -l -e -o pipefail {0}'
          - os: macos
            shell: '/bin/bash -l -e -o pipefail {0}'

• Then we set runs on an the default shell. Those come from the matrix so we'll have a job running windows with the windows shell, a job running ubuntu with the ubuntu shell, etc.
• Our first step is to delete the ~/.bash_profile. It starts as all comments except for a line that causes it to exit when we are not interactive. Any lines that we place after that won't be executed because we are not interactive, so we need to remove that. Might as well just remove it all, there is nothing useful in there. (If you want to check for yourself, put cat .bash_profile into a CI command and see it in the logs.)
• Now we are ready to add environment variables. We append them to the ~/.bash_profile. I used export but I'm not totally sure if that is necessary. No harm in it anyway.

That's it! Now a single CI will work for windows and ubuntu and macos. The pcb2gcode project is using this successfully with a fairly complicated build for all three platforms and they all use the same steps, more or less.