Create actions for Windows and Mac OS

[evandrocoan]

I saw you recently created the uses: ankitects/setup-protoc@master, because setup-protoc was not working. I also had the same problem when I was creating the GitHub actions for Mac OS. The solution come after opening a issue on (actions/runner-images#602 - API rate limit exceeded for Mac OS hosted by GitHub itself). The problem was that the README.md of the setup-protoc project was not updated to use the correct version of setup-protoc. I opened a issue there, asking for their README.md to be updated (arduino/actions#23 - Mark the repository as deprecated). Meanwhile, on this pull request, I am already using the correct version of the setup-protoc actions as suggested by actions/runner-images#602.

1. I created an Mac OS tests, after fixing (API rate limit exceeded for Mac OS hosted by GitHub itself actions/runner-images#602 - API rate limit exceeded for Mac OS hosted by GitHub itself).
2. I added to also run build develop because while I was building this I got a case where make check build was working/passing, but make develop not. I put the develop rule first because it should be the most important, i.e., a user can run the develop build without errors. Then, I check whether Anki can pack itself into python wheels and finally I run all unit tests. Also, by putting develop before check allows to each one of them build half of things. If check is the first rule, it build all rust libraries at once instead of just part of them.
3. I separated make develop, make check and make build in different rules just to:
   1. Know more easily in which step the build failed
   2. Know how much time each one of them is taking to complete
4. I created caches for everything I could, to speed up the build process. GitHub automatically deletes the caches every week and the caches are also regenerated on certain conditions:
   1. For the cache of pyenv directory, they are rebuild when you edit a Makefile or some requirements.txt file.
   2. For the cache of package manager pacman, they are rebuild every time you edit the windows_checks.yml file.
   3. To build the project using all caches should take:
      1. About 7 minutes on Linux and Mac (without cache about 12 to 20 minutes)
      2. About 10 minutes on Windows (without cache about 22 minutes)
5. Additionally to all these, you can always for the cache to be rebuilt by editing the line key: and increment its the number at the end of the line or just by changing anything else on this composition.
6. I created the special environment variable SHELLFLAGS to allow the user to easily debug installation errors by defining the environment variable SHELLFLAGS=-x before running anything. The flag -x will cause bash to print out every command it is running. This is vital to figure out how to make the GitHub Actions to work.
7. I also added by default the -u flag to all shell scripts. All Makefiles were already using it, then, I decided to normalize it and keep everybody using it. The -u flags causes the shell script to fail if it attempts to use a variable which is not defined. If a variable can be undefined, we need to explicitly say it by adding this before using it:

   if [[ -z "${variablename+x}" ]]; then
       variablename=;
   fi

8. It was difficult to use the perl rename script on Windows because the cmd terminal has a builtin command called rename, this it was always taking precedence over the perl script rename. To fix it, I directly call it with perl rename on windows. I also could not figure out where should I put this rename script, then, I just added it to .gitignore and used a simple curl to download it.
9. After several times trying to use Cygwin or the msys2 packages, the only way to make it working was using the Unix Environment already shipped with Git for Windows. The only problem is that it does not have a package manager. Then, I installed one (pacman, Update/Add Git for windows SDK actions/runner-images#596 - Update/Add Git for windows SDK).
10. I moved some declarations on the Makefiles as MAKEFLAGS += --warn-undefined-variables to the top of the Makefiles because they only start working after they are defined.
11. I have removed the @set -eo pipefail && which the main Makefile had because it was redundant. You were already setting the make setting .SHELLFLAGS := -eu -o pipefail which makes all command to be run through shell with the flags -eu -o pipefail.
12. Only now I had run for the first the the make build command (on Windows), and I figure out it was not working due this problem (Expected python to be a python interpreter inside a virtualenv PyO3/maturin#283 - Expected python to be a python interpreter inside a virtualenv). Then, to fix it I had to manually edit the anki/pyenv/Scripts/activate script. Also, I am not sure why, but the Python interpreter on the GitHub Actions Windows was create the file anki/pyenv/Scripts/activate with CRLF line endings, which I had to convert to LF, other the shell script would never be sourced.

    ...
    "${PYTHON_BIN}" -m venv pyenv && \
    case "$$(uname -s)" in CYGWIN*|MINGW*|MSYS*) \
        dos2unix "${ACTIVATE_SCRIPT}" && \
        sed -i -- "s@VIRTUAL_ENV=\".*\"@VIRTUAL_ENV=\"$$(pwd)/pyenv\"@g" "${ACTIVATE_SCRIPT}"; \
        ;; esac && \
    . "${ACTIVATE_SCRIPT}" && \
    ...

13. The cache jobs Cache cargo rslib and Cache cargo rspy are doing nothing as of now. I just put them there in case these directories are generated some day.
14. Now, you can access the checks page and download the Python Wheels generated by make build: https://github.com/evandroforks/anki/actions/runs/62647001
    

[evandrocoan]

Do you know why this test is failing? https://github.com/ankitects/anki/runs/532323156?check_suite_focus=true#step:18:477

________________________________ test_learn_day ________________________________

    def test_learn_day():
        d = getEmptyCol()
        # add a note
        f = d.newNote()
        f["Front"] = "one"
        f = d.addNote(f)
        d.sched.reset()
        c = d.sched.getCard()
        d.sched._cardConf(c)["new"]["delays"] = [1, 10, 1440, 2880]
        # pass it
        d.sched.answerCard(c, 2)
        # two reps to graduate, 1 more today
        assert c.left % 1000 == 3
>       assert c.left // 1000 == 1
E       AssertionError: assert (2003 // 1000) == 1
E        +  where 2003 = {'data': '', 'did': 1, 'due': 1585097436, 'factor': 0, 
            'flags': 0, 'id': 1585096836649, 'ivl': 0, 'lapses': 0, 'left':...836, 
             'nid': 1585096836648, 'odid': 0, 'odue': 0, 'ord': 0, 'queue': 1, 
             'reps': 1, 'type': 1, 'usn': -1, 'wasNew': True}.left

Perhaps it is related to the order I call the make rules, i.e., first develop, then, build and lastly check?

This error is happening in all 3 platforms, including Mac OS. The Mac OS build is not failing because the Mac OS make is not stopping errors: https://github.com/evandroforks/anki/runs/532403941?check_suite_focus=true#step:18:632

FAILED tests/test_schedv1.py::test_learn_day - AssertionError: assert (2003 /...
====================== 1 failed, 126 passed in -7194.51s =======================
make[1]: *** [.build/test] Error 1
make[1]: Leaving directory `/Users/runner/runners/2.165.2/work/anki/anki/pylib'
make[1]: Entering directory `/Users/runner/runners/2.165.2/work/anki/anki/qt'
./tools/typecheck-setup.sh
python -m mypy --warn-unused-ignores aqt
Success: no issues found in 92 source files
python -m pytest -s
============================= test session starts ==============================
platform darwin -- Python 3.7.6, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /Users/runner/runners/2.165.2/work/anki/anki/qt
collected 11 items

tests/test_addons.py ..........

I am going to look into why Mac OS make is not stop on errors.

[evandrocoan]

After I figure out what is wrong, I am going to squash into a single commit all these Attempt to fix build errors commits and force push this branch.

[dae]

The macOS make is too old to support SHELLFLAGS, which is why the set -eo lines are necessary

I haven't had a chance to review this all yet, but it sounds great - thank you for all your hard work on this.

[dae]

re the test failing, perhaps it's another issue related to the time of day - it's right around 4am in GMT time

[evandrocoan]

Ah thanks! Then, I just need to ask brew to update make or find a way to update it.

I am about to go to sleep now. Tomorrow I should continue digging into it.

[dae]

I think it's best if the set -eo commands are returned instead of requiring Mac users to install make as well.

Sleep well :-)

[evandrocoan]

Thanks!

I think it's best if the set -eo commands are returned instead of requiring Mac users to install make as well.

You are right. I will keep the set -e thing. (It is already back on my commits few minutes ago).

Now all tests are passing. I am going to try revert back a few things to see if they crash again.

[dae]

Thanks for all your hard work on this! It's looking pretty good already, but the checks.yml could do with some tweaks - I've added some comments below.

[evandrocoan]

These are the results of the lastest tests (no cache https://github.com/ankitects/anki/actions/runs/64172038) vs (with full cache https://github.com/ankitects/anki/actions/runs/64185052)

1. Ubuntu (fully cached)

   pip wheels ~164 MB
   cargo index ~9 MB
   cargo registry ~146 MB
   total ~319 MB
   
   pyenv ~187 MB
   cargo target (check) ~525 MB
   cargo target (build) ~184 MB
   total ~1215 MB (saved 896 MB)
   check 05:12 minutes (with no cache 10:12 minutes)
   build 03:46 minutes (with no cache 12:29 minutes)

2. Mac OS (fully cached)

   pip wheels ~131 MB
   cargo index ~9 MB
   cargo registry ~85 MB
   total ~225 MB
   
   pyenv ~131 MB
   cargo target (check) ~362 + 439 MB
   cargo target (build) ~137 MB
   total ~1294 MB (saved 1069 MB)
   check 07:11 minutes (with no cache 15:02 minutes)
   build 04:48 minutes (with no cache 10:40 minutes)

3. Windows (fully cached)

   pip wheels ~133 MB
   cargo index ~15 MB
   cargo registry ~145 MB
   total ~293 MB
   
   pacman ~147 MB
   pyenv ~139 MB
   cargo target (check) ~497 MB
   cargo target (build) ~183 MB
   total ~1259 MB (saved 966 MB)
   check 06:55 minutes (with no cache 18:14 minutes)
   build 06:22 minutes (with no cache 18:32 minutes)

4. Total 3768 / 837 MB (saved 2931 MB)

As of now, I do not think that the caches of wheels, index and registry are being used. Then, I am going to disable them by commenting them out because it was quite hard to set them up. And should be simple to reenable them back if we decide to switch back.

By disabling them, it would save us 837 MB from our total of 3768 MB of cache used on all machines (of a maximum of 5000 MB per repository). Also, it will save the time of unpacking them, so the build times should become faster than it is already.

[evandrocoan]

These are the new results of the lastest tests compared with the first batch (no cache https://github.com/ankitects/anki/actions/runs/64240676) vs (with full cache https://github.com/ankitects/anki/actions/runs/64256407)

1. Ubuntu (fully cached)

   pyenv ~187 MB
   cargo target (check) ~525 MB
   cargo target (build) ~184 MB
   total ~896 MB
   
   before/after
   check 05:12 minutes (with no cache 10:12 minutes)
   check 05:03 minutes (with no cache 11:35 minutes)
   
   build 03:46 minutes (with no cache 12:29 minutes)
   build 04:44 minutes (with no cache 14:34 minutes)

2. Mac OS (fully cached)

   pyenv ~131 MB
   cargo target (check) ~362 + 439 MB
   cargo target (build) ~137 MB
   total ~1069 MB
   
   before/after
   check 07:11 minutes (with no cache 15:02 minutes)
   check 07:31 minutes (with no cache 12:16 minutes)
   
   build 04:48 minutes (with no cache 10:40 minutes)
   build 07:10 minutes (with no cache 11:08 minutes)

3. Windows (fully cached)

   pacman ~147 MB
   pyenv ~139 MB
   cargo target (check) ~497 MB
   cargo target (build) ~183 MB
   total ~966 MB
   
   before/after
   check 06:55 minutes (with no cache 18:14 minutes)
   check 07:44 minutes (with no cache 14:48 minutes)
   
   build 06:22 minutes (with no cache 18:32 minutes)
   build 07:20 minutes (with no cache 17:26 minutes)

4. Total 2931 / 5000 MB (remaining 2069 MB)

Looking at them, the times seem worst than before. After checking, I noticed rust was wasting time downloading all the crates (just for nothing because it was not compiling them). Hopefully Python pip was not that much dumb. So, I am going to re-enable the cargo index/registry caches, but keep the Python pip wheels disabled for now.

[evandrocoan]

Haha. Now, it is time for that failing test to come back and haunt us again:

        d.sched.answerCard(c, 2)
        # two reps to graduate, 1 more today
        assert c.left % 1000 == 3
>       assert c.left // 1000 == 1
E       AssertionError: assert (2003 // 1000) == 1
E        +  where 2003 = {'data': '', 'did': 1, 'due': 1585268120, 'factor': 0, 
            'flags': 0, 'id': 1585267520336, 'ivl': 0, 'lapses': 0, 'left':...520,
            'nid': 1585267520335, 'odid': 0, 'odue': 0, 'ord': 0, 'queue': 1, 
            'reps': 1, 'type': 1, 'usn': -1, 'wasNew': True}.left

I am going to disable it until this is merged.

[evandrocoan]

Just taking a screenshot for recordation before I force-push this branch again:

These are the results of the lastest tests, after re-enabling the rust cargo index/registry caches (no cache https://github.com/ankitects/anki/actions/runs/64280921) vs (with full cache https://github.com/ankitects/anki/actions/runs/64287568)

1. Ubuntu (fully cached)

   pyenv ~187 MB
   cargo index ~9 MB
   cargo registry ~146 MB
   cargo target (check) ~525 MB
   cargo target (build) ~184 MB
   total ~1051 MB
   
   before/after/math
   check 05:12 minutes (with no cache 10:12 minutes)
   check 05:03 minutes (with no cache 11:35 minutes)
   check 04:27 minutes (with no cache 12:04 minutes)
   
   build 03:46 minutes (with no cache 12:29 minutes)
   build 04:44 minutes (with no cache 14:34 minutes)
   build 03:36 minutes (with no cache 12:42 minutes)

2. Mac OS (fully cached)

   pyenv ~131 MB
   cargo index ~9 MB
   cargo registry ~85 MB
   cargo target (check) ~362 + 439 MB
   cargo target (build) ~137 MB
   total ~1163 MB
   
   before/after
   check 07:11 minutes (with no cache 15:02 minutes)
   check 07:31 minutes (with no cache 12:16 minutes)
   check 04:47 minutes (with no cache 11:59 minutes)
   
   build 04:48 minutes (with no cache 10:40 minutes)
   build 07:10 minutes (with no cache 11:08 minutes)
   build 05:08 minutes (with no cache 11:59 minutes)

3. Windows (fully cached)

   pacman ~147 MB
   pyenv ~139 MB
   cargo index ~15 MB
   cargo registry ~145 MB
   cargo target (check) ~497 MB
   cargo target (build) ~183 MB
   total ~1126 MB
   
   before/after/math
   check 06:55 minutes (with no cache 18:14 minutes)
   check 07:44 minutes (with no cache 14:48 minutes)
   check 06:52 minutes (with no cache 17:13 minutes)
   
   build 06:22 minutes (with no cache 18:32 minutes)
   build 07:20 minutes (with no cache 17:26 minutes)
   build 06:06 minutes (with no cache 18:27 minutes)

4. Total 3340 / 5000 MB (remaining 1660 MB)

Despite the increment in space usage, now the build times are better than the first time. Of course, these times are just an approximation and not all builds should take them exactly. I would expect that a cached build to variate up to 3 minutes more or less, while a build fully rebuilding the cache should take about 5 minutes to more or less.

Now force-pushing this branch, reenabling that failing test I had removed.

[dae]

(please let me know when you'd like me to do a final review + merge)

[evandrocoan]

I have finished it. I just forced push to remove duplicated jobs (actions/runner#391).

[dae]

Thank you very much for your work on this Evandro!