Docker compilation twice as slow, seems to hang on large contract sets.

[cgewecke]

• I've asked for help in the Truffle Gitter before filing this issue.

---

Issue

5.0.13 significantly reduced the speed of solc docker compilation for large contract sets. Performance when compiling colonyNetwork on a MacBook Air 1.8 ghz:

• 5.0.12 time --> real 0m59.960s
• 5.0.17 time --> real 2m22.203s

The command also hangs before listing the contracts for more than a minute. Believe changes to the imports parsing / profiler in #1913 are the cause. The benefit of the solcjs parsing removed in that PR was that it was fast (see this tcoulter comment from the original implementation).

Steps to Reproduce

# Install colonyNetwork
git clone https://github.com/JoinColony/colonyNetwork.git
cd colonyNetwork
yarn
git submodule update --init

# Install Truffle 5.0.12 and benchmark
npm uninstall -g truffle
npm install truffle@5.0.12
time truffle compile

# Clean build
rm -rf build

# Install Truffle 5.0.17 and benchmark
npm uninstall -g truffle
npm install truffle@5.0.17
time truffle compile

Expected Behavior

Using solc docker is faster than using solc-js

Additional notes / suggestion

#1913 resolved several issues. #1720 is one but it also allowed openzeppelin-solidity to run a solc nightly job. Previous versions of truffle had been unable to meet that use case because they couldn't match a solc-js companion to the commit referenced by the docker nightly tag.

There might be a simple internal fix for this but one option is to go back to using a js parser for imports parsing. solidity-parser-antlr is excellent and widely used across the js eco-system. It would be nice if it received grant support and was treated as a core piece of the stack.

The risks of depending on a parser which lags solidity development (slightly) could be mitigated by E2E testing here, having a strong relationship with the project's maintainer and maybe asking solc to ping solidity-antrl4 with grammar changes. Perhaps they could integrate it with their own testing or even add it as a dependency to the npm published solcjs.

Environment

• Operating System: OSX
• Ethereum client: N/A
• Truffle version (truffle version): 5.0.12 / 5.0.17
• node version (node --version): 10.15.3
• npm version (npm --version): 6.4.1

[CruzMolina]

Hey @cgewecke , thanks for pointing this out!

Since we're still bundling solc-js, it seems like the easy fix is to continue using it for quick import parsing. Will take a look shortly.

[cgewecke]

@CruzMolina Yes that seem reasonable. Might need to throw away the nonImportErrors and let the real compilation pick them up? It's a little weird to compile with different solc versions.

[CruzMolina]

Good call. Looks like there's a few places compilation errors are caught. 👍

[BrendanChou]

I think this can be marked as fixed now? Not sure which version of truffle it's in

[CruzMolina]

Closing since i believe this has been resolved in 5.0.27.

[drortirosh]

Alas, it was not solved.
I'm working with truffle 5.0.27
Below are my tests for truffle compile, for a project with 9 contracts and few referenced libs (https://github.com/tabookey/tabookey-gasless/)

with docker: 34 seconds
without docker (solcjs) : 10 seconds
with solc docker (without truffle) - 1.2 seconds

Note that the solc generates different output format (the above test was with --combined-json with all options set, though truffle generates much more verbose info.

I'm using Mac, where I can see the currently running process.
When running truffle compile with "docker" enabled:

• 8 seconds of "truffle"
• ~10 seconds running "docker"
• ~10 seconds running again "docker"
• few more seconds in "truffle" to "wrap up"

I'm not sure what takes so long, and why the docker image was executed twice (one run can compile all contracts, and anyway, my project had ~10 contracts, including some references to libraries)

[gnidan]

Eek! Alright, re-opening this! Thanks for letting us know!

[CruzMolina]

Hey @drortirosh , I notice the truffle config in the referenced repo doesn't include the new config setting for speedy parsing.

For reference from #2067 :

To enable speedy JS parsing, users will need to specify a parser: "solcjs" key/value pair in their config like so:

module.exports = {
  compilers: {
    solc: {
      parser: "solcjs"
    }
  }  
}

Hope this helps!

[BrendanChou]

I noticed the same problem on Truffle 5.0.28, but the config thing above fixed it 👍 I think you can close this again

[drortirosh]

Yes, indeed the parse: "solcjs" does the job. thanks -
now both truffle compilation methods take ~9 seconds to compile.

but I thought the idea was to get rid of solcjs completely (it takes ~4 seconds just to load, even before it starts doing anything)

[CruzMolina]

Woo! Glad to hear the parser works!

@drortirosh We opted to remove solc-js as a necessary external dependency when installing truffle since it appeared to be the cause of major OS-compatibility issues.

Current behavior is that truffle caches a default solc-js soljson file which gets used by default for compilations. This cached file is what gets used for faster parsing when set in the config and using native or docker solc.

[drortirosh]

I understand that you use solcjs internally. What I don't understand is why you need it at all when working with solc docker: why parse and compile it 2 steps, when compiling does parsing too.

[CruzMolina]

IIRC, solc doesn't support parsing & resolving imports (ethereum/solc-js#114). We use a parser hack to quickly grab necessary imports. Unfortunately this hack works much faster when using solc-js then docker or native.

[drortirosh]

Here's my hack to resolve dependencies for solc.
It compiles in ~1 second my project source tree, compared to ~10 seconds in solcjs

#!/bin/bash -e
pkgs=`cd node_modules;echo */contracts @*/*/contracts`
p=""
for d in $pkgs; do
p="$p $d/=./node_modules/$d/"
done
solc --allow-paths `pwd`/node_modules/ $p "$@"

I agree that this is only a hack, not a general solution. A better solution is to add "search path" to solc