Miniwdl does not work on ARM macs (e.g. M1)

[corneliusroemer]

I've just hit this issue previously mentioned in a comment and would like to increase visibility by making it an issue of its own.

Would be great if the README could mention that ARM macs are currently not supported to reduce time waste trying things out.

This is @eholdmore's comment:

Hi @mlin and others, thanks for all the hard work here. I just wanted to add that miniwdl run_self_test fails on macOS Venture 13.1 with M1 for me. Here are the steps I followed:

1. Install Docker Desktop and ensure Docker is running with docker ps
2. Set file sharing to osxfs (Legacy) as suggested by macOS: task input files sporadically fail to mount (zero length) #461 (comment)
3. Install Miniconda3
4. Install miniwdl using conda install -c conda-forge miniwdl
5. export TMPDIR=/tmp
6. Finally, miniwdl run_self_test

System Details
macOS: Ventura 13.2
Chip: M1
Docker: 4.16.2
Python: 3.10.8
MiniWDL: 0.10.0

Output
~ % miniwdl run_self_test
test.wdl
workflow hello_caller
scatter name
call hello
if
task hello

• Suggestion: install shellcheck (www.shellcheck.net) to check task commands. (--no-shellcheck suppresses this message)
  2023-01-31 13:40:45.024 miniwdl-run read configuration defaults :: filename: "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/config_templates/default.cfg"
  2023-01-31 13:40:45.024 miniwdl-run searching for configuration files :: filenames: ["/Users/ericah/.config/miniwdl.cfg", "/etc/xdg/miniwdl.cfg"]
  2023-01-31 13:40:45.024 miniwdl-run no configuration file found
  2023-01-31 13:40:45.024 miniwdl-run applying configuration overrides :: logging: {"json": "false"}
  2023-01-31 13:40:45.025 miniwdl-run configuration :: download_awscli: {"host_credentials": "false", "docker": "amazon/aws-cli@sha256:98ec84db99a707708f8bfb9a09ce48cccf827a5ed682917f7bd1b2d4be5c4751"}, call_cache: {"put": "false", "dir": "/Users/ericah/.cache/miniwdl", "get": "false"}, logging: {"json": "false"}, download_cache: {"get": "false", "disable_patterns": "[".php", ".aspx"]", "put": "false", "ignore_query": "false", "enable_patterns": "[""]", "dir": "/tmp/miniwdl_download_cache"}, download_aria2c: {"docker": "hobbsau/aria2@sha256:9d4646b1576e88e4061f509b25b8c0c6e1a5fface4ac37d9ecab67de5f7d9a4b"}, task_runtime: {"as_user": "false", "memory_max": "0", "defaults": "{\n"docker": "ubuntu:20.04"\n}", "cpu_max": "0", "memory_limit_multiplier": "0.0"}, docker_swarm: {"auto_init": "true"}, file_io: {"root": "/", "delete_work": "false", "copy_input_files": "false", "output_hardlinks": "false"}, plugins: {"enable_patterns": "[""]", "disable_patterns": "["miniwdl_task_omnibus_example:*"]"}, download_gsutil: {"docker": "google/cloud-sdk@sha256:64ddc4e5d3f7fdc5a198c8acf1c361702994462dbe79e7de5a6bde6e62e60e6b"}, scheduler: {"call_concurrency": "0", "container_backend": "docker_swarm"}
  Traceback (most recent call last):
  File "/Users/ericah/miniconda3/bin/miniwdl", line 10, in
  sys.exit(main())
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 78, in main
  run_self_test(**vars(args))
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 1209, in run_self_test
  raise exn
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 1191, in run_self_test
  outputs = main(argv)["outputs"]
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 76, in main
  runner(**vars(args))
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 625, in runner
  target, input_env, input_json = runner_input(
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 887, in runner_input
  v = runner_input_value(s_value, decl.type, downloadable, root)
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 1034, in runner_input_value
  s_value = validate_input_path(os.path.expanduser(s_value), directory, downloadable, root)
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 1069, in validate_input_path
  if downloadable and downloadable(path, directory):
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/CLI.py", line 632, in
  downloadable=lambda fn, is_dir: runtime.download.able(cfg, fn, directory=is_dir),
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/download.py", line 68, in able
  return bool(uri and _downloader(cfg, uri, directory=directory) is not None)
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/download.py", line 56, in _downloader
  _load(cfg)
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/download.py", line 45, in _load
  for plugin_name, plugin_fn in config.load_plugins(cfg, "file_download"):
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/config.py", line 355, in load_plugins
  yield from (
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/config.py", line 355, in
  yield from (
  File "/Users/ericah/miniconda3/lib/python3.10/site-packages/WDL/runtime/config.py", line 346, in load_all_plugins
  importlib_metadata.entry_points().get(f"miniwdl.plugin.{group}", [])
  AttributeError: 'EntryPoints' object has no attribute 'get'
• Hint: ensure Docker is installed & running; and on macOS override the environment variable TMPDIR=/tmp/

Originally posted by @eholdmore in #145 (comment)

[mlin]

@corneliusroemer Thanks for raising this, indeed in the first couple years of Apple Silicon, I don't think anyone had much expectations for Docker-based bioinformatics workflows to work on that platform, so our docs did not address it -- but I presume that's changing by now, and you're right to point it out. I have made some updates to the readme and the pinned issue #145, also linking here to continue discussion.

miniwdl itself is purely Python so I don't think it has any fundamental incompatibility with ARM; the questions would be (1) is the right version of everything getting installed and (2) is it able to talk to Docker and use the applicable Docker images. Unfortunately, I'm still using an Intel MacBook and GitHub Actions still only has Intel macOS runners. If somebody with a newer Mac were to hack on it, I doubt it would be heavy lift to get it working so long as the desired Docker images can be run. And the non-runner tools like miniwdl check ought to work, but I'd be curious if anyone can confirm.

The backtrace posted by @eholdmore bears on question 1 because it curiously seems to refer to an issue with a version of miniwdl that was already outdated at the time. The 'EntryPoints' object has no attribute 'get', caused by an unexpected change in a dependent package, had been fixed last October. So I wonder if our conda channel is somehow distributing an older code revision to ARM macs specifically. Is that error among the ones you encountered too?

[adamnovak]

I'm able to run MiniWDL on an M1 Mac.

I installed MiniWDL from pip into a virtual environment, something like:

pip3 install virtualenv
virtualenv venv
. venv/bin/activate
pip install miniwdl

Then I need to export DOCKER_HOST=unix://${HOME}/.docker/run/docker.sock (because that's where Docker Desktop puts its socket), and make sure Docker Desktop is running and has "VirtioFS" set as its file sharing mode, and then I can miniwdl run_self_test just fine.

[mschatz]

@adamnovak I was unable to get this to work. miniwdl run_self_test gets stuck in a loop:

...
2023-09-19 22:01:11.456 wdl.w:hello_caller.download0.t:download-aria2c waiting for local docker swarm manager & worker(s) :: manager: "error", workers: 0
2023-09-19 22:01:13.498 urllib3.connectionpool http://localhost:None "GET /v1.43/info HTTP/1.1" 200 None
2023-09-19 22:01:13.503 wdl.w:hello_caller.download0.t:download-aria2c swarm info :: NodeID: "", NodeAddr: "127.0.0.1", LocalNodeState: "error", ControlAvailable: false, Error: "error while loading TLS certificate in /var/lib/docker/swarm/certificates/swarm-node.crt: certificate (1 - ogip74kecsovzofirtkrm1jqj) not valid after Sat, 24 Dec 2022 23:55:00 UTC, and it is currently Wed, 20 Sep 2023 00:09:06 UTC: x509: certificate has expired or is not yet valid: ", RemoteManagers: null
2023-09-19 22:01:13.505 wdl.w:hello_caller.download0.t:download-aria2c waiting for local docker swarm manager & worker(s) :: manager: "error", workers: 0
2023-09-19 22:01:15.550 urllib3.connectionpool http://localhost:None "GET /v1.43/info HTTP/1.1" 200 None
2023-09-19 22:01:15.551 wdl.w:hello_caller.download0.t:download-aria2c swarm info :: NodeID: "", NodeAddr: "127.0.0.1", LocalNodeState: "error", ControlAvailable: false, Error: "error while loading TLS certificate in /var/lib/docker/swarm/certificates/swarm-node.crt: certificate (1 - ogip74kecsovzofirtkrm1jqj) not valid after Sat, 24 Dec 2022 23:55:00 UTC, and it is currently Wed, 20 Sep 2023 00:09:06 UTC: x509: certificate has expired or is not yet valid: ", RemoteManagers: null
2023-09-19 22:01:15.552 wdl.w:hello_caller.download0.t:download-aria2c waiting for local docker swarm manager & worker(s) :: manager: "error", workers: 0

Any tips on what else to try?

Mike

[adamnovak]

@mschatz It looks like your Docker daemon is set up to be in a swarm (which I think MiniWDL needs), but the swarm's root certificate has expired.

I think you should be able to fix it by destroying the VM image that Docker for Mac is using, and making it create a new one and make a new certificate. I think you can do that by going to the Docker tray menu, hitting "Troubleshoot", and then hitting "Clean / Purge Data".

You might also be able to do docker swarm ca --rotate?

When I do:

docker swarm ca >ca.pem
openssl x509 -in ca.pem -noout -text

it shows me that I am using a certificate that expires in 2043. So I think the certificates used to get a 1 year lifetime and now they get a 20 year lifetime by default.

[mschatz]

Thanks for the tip - you were exactly right! Mike

…

On Thu, Sep 21, 2023 at 11:19 AM Adam Novak ***@***.***> wrote: @mschatz <https://github.com/mschatz> It looks like your Docker daemon is set up to be in a swarm (which I think MiniWDL needs), but the swarm's root certificate has expired. I think you should be able to fix it by destroying the VM image that Docker for Mac is using, and making it create a new one and make a new certificate. I think you can do that by going to the Docker tray menu, hitting "Troubleshoot", and then hitting "Clean / Purge Data". You might also be able to do docker swarm ca --rotate? When I do: docker swarm ca >ca.pem openssl x509 -in ca.pem -noout -text it shows me that I am using a certificate that expires in 2043. So I think the certificates used to get a 1 year lifetime and now they get a 20 year lifetime by default. — Reply to this email directly, view it on GitHub <#652 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABP34ZXLBJX2A4P4EIGX6DX3RLG3ANCNFSM6AAAAAA3BC3SNQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[watronfire]

I had a similar issue on my Macbook with an M2. I installed docker, set the file sharing to osxfs, and installed miniwdl with mamba. run_self_test failed with the same error. But it turns out that conda installed v0.10.0 rather than the current version (at this moment v1.11.1). As @mlin said, I think v0.10.0 is the latest version of miniwdl conda-forge has, whereas for intel macs it has v1.11.1.

Because of this I was able to successfully run the self test after uninstalling miniwdl and reinstalling by temporarily setting my conda subdirectory to osx-64, i.e. CONDA_SUBDIR=osx-64 mamba install miniwdl

[dbacsik]

Another M2 user here, and I was able to successfully run self_test.

As @mlin thought and @watronfire elucidated, conda-forge was serving an old version of v0.10.0. For me, adding the flag @watronfire describes worked with micromamba (my default env manager). I simply had to append CONDA_SUBDIR_osx-64 before all commands.