Contract compilation unusually slow on macOS

[cameel]

I'm preparing a PR that will parallelize CLI tests (#14339) and in particular the via-ir equivalence will now get its own run. Comparing that between Ubuntu and macOS, I see that it runs 14x slower on the latter:

• ~via_ir_equivalence in t_ubu_cli: 30 sec
• ~via_ir_equivalence in t_osx_cli: 7 min

That's even bigger slowdown than we usually see when building solc, which is "only" 3x.

It also stands out compared to soltest, where macOS is about 2x slower:

• t_ubu_clang_soltest: ~8 min (20 parallel runs, ~25 sec each)
• t_osx_soltest: ~16 min

For via IR eqvivalence most of the execution time is taken by solc compiling contracts rather than evmone executing them or unrelated things like running clang. It's possible that it's not CI that's the culprit here but rather that solc is unusually slow on macOS.

The excessive slowdown does not seem to be caused exclusively by IR compilation. The legacy compilation tests are affected to a similar degree:

• ~compilation_tests in t_ubu_cli: 30 sec
• ~compilation_tests in t_osx_cli: 5 min

Reproducing the problem

Here's how to run these particular tests locally:

test/cmdlineTests.sh ~via_ir_equivalence ~compilation_tests

[thedavidmeister]

perhaps related foundry-rs/foundry#5102

in foundry seeing indefinite build times on apple silicon

[cameel]

Interesting. So you're seeing this slowdown outside of CircleCI and also on ARM macs? That's not great, because we still had some hope that maybe it's just bad performance of CircleCI's x86 machines and will resolve itself if we switch mac builds to ARM.

[thedavidmeister]

@cameel i'm not sure if its the same thing, but there's definitely some issue on M1 machines that manifests on some repos and not others

[cameel]

ok, we'll have to investigate and find out. Thanks for the info!

[thedavidmeister]

disabling the model checker made my compilation start running ok, but the other repo linked in the issue on foundry doesn't seem to have the model checker

[cameel]

The slowdown we're seeing in CI is definitely not related to SMTChecker, since the contracts we compile are not enabling it. So this must be different from your case.

[thedavidmeister]

@cameel kk, fyi someone else on the linked issue didn't have the checker enabled and they saw a timeout after 15 mins

[cameel]

After looking at this again today, looks like the difference is much smaller than I claimed in the description. There are 2 tests that are 2x/3x slower, but the rest much closer in reality, because the running time I posted includes dependency installation time, which is significant on macOS (several minutes, see #12925).

Here's t_ubu_cli and t_osx_cli from the most recent run on develop:

test	t_ubu_cli	t_osx_cli	t_ubu_cli (job)	t_osx_cli (job)
remaining tests	1:20	2:20	1:58	7:19
~ast_export_with_stop_after_parsing	4:34	4:47	5:14	9:02
~ast_import_export	7:01	8:20	7:30	12:43
~compilation_tests	0:08	0:09	0:39	4:27
~documentation_examples	0:10	0:18	0:38	4:35
~evmasm_import_export	2:52	7:06	3:22	11:56
~soljson_via_fuzzer	2:30	0:58	3:04	5:23
~via_ir_equivalence	0:15	0:18	0:28	4:36

The first two columns are just the command line tests step and the last two are for the whole parallel run of the job, which includes container setup, dependency installation and other steps.

So looks like in most cases macOS is just a bit slower. It has a 4 CPU machine, while Linux has 1 CPU, but most of these are not multi-threaded. The only exception is the fuzzer, which is reflected in the numbers.

The biggest outliers are "remaining tests" and ~evmasm_import_export. The former is 2x and the later 3x slower on macOS. Seems to be the only one worth looking into.

I think that, when reporting this, I saw the 2x difference in the main batch of tests and must have mistakenly looked at the others taking the whole running time or each job rather than just the time it takes to run the test. Sorry for the confusion.