Docker solc standard-json error everytime

[Gildraen]

Hi !

I am quite new in solidity, so i try to setup a dev environment with docker.
So on my test, i try to compile contract with the json way that seems to be the recommended one.

docker run ethereum/solc:0.8.5 --standard-json < ./src/Compiler/001/config.json

and the output is, no matter what is the version of solc, or no matter what i put in my json :

{"errors":[{"component":"general","formattedMessage":"* Line 2, Column 1\n Syntax error: value, object or array expected.\n* Line 1, Column 1\n A valid JSON document must be either an array or an object value.\n","message":"* Line 2, Column 1\n Syntax error: value, object or array expected.\n* Line 1, Column 1\n A valid JSON document must be either an array or an object value.\n","severity":"error","type":"JSONError"}]}

Even with a simple json code like this :

{ "language": "Solidity", "sources": { "": { "content": "contract C { uint[] constant c; }" } } }

[cameel]

Docker by default will not pass standard input to the container it runs. You need to add the --interactive flag if you want it to:

docker run --interactive ethereum/solc:0.8.5 --standard-json < ./src/Compiler/001/config.json

By the way, if you're just running it manually (i.e. not as a part of some server orchestration where you'd want to inspect the dead container if it crashes), it's a good idea to the --rm flag to avoid leaving a stopped container every time you do it (see docker ps --all).

Also, if you want to be able to interact with a program from the terminal (typing text, etc.) instead of giving it a file you should also add --tty. You won't need that with solc, because it never requires interactive user input but might be handy if you ever need that for other programs. For example I'm used to doing docker run -it --rm ... when running containers because most often that's what I need.

This is not a bug in the compiler so I'm closing the issue but feel free to ask if you some other problems :)

[Gildraen]

Yes, that made everything, it was a basic docker thing i forgot.. thanks you !
And yea, nice tip to run --rm everytime. (I just feared myself with the docker ps -all btw..)

now i can start to really learn about the compiler (and i have a lot of work..)

[cameel]

(I just feared myself with the docker ps -all btw..)

Try docker system prune :) Will remove all the temporary stuff like stopped containers, dangling (untagged) images and cache entries, etc.

[Gildraen]

Yea i cleaned everything, it's quite pretty now !

But now, i have a problem with json and sources. Even with base-path or allow-paths, the json cannot find the file (relative or absolute)

I think it's me that does not really understand the doc of all about path/remapping etc..

I'm trying to do a simple way

docker run --interactive --rm ethereum/solc:0.8.4 --base-path ./ --standard-json < ./src/Compiler/001/config.json

with :

{ "language": "Solidity", "sources": { "Contract-001.sol": { "urls": ["src/Contract/Contract-001.sol"] } } }

result to :
{"errors":[{"component":"general","formattedMessage":"Cannot import url (\"src/Contract/Contract-001.sol\"): File outside of allowed directories.","message":"Cannot import url (\"src/Contract/Contract-001.sol\"): File outside of allowed directories.","severity":"error","type":"IOError"}],"sources":{}}

I tried different combo with base-path, allow-paths, url etc.. i can't find the right and simple way to get to the contract

[cameel]

Try this:

docker run \
    --rm \
    --volume "$PWD/src/Contract:/project/src/Contract" \
    --workdir /project \
    ethereum/solc:0.8.4 \
        --standard-json src/Compiler/001/config.json \
        --allow-paths /project | jq --indent 4

Explanation:

• When you run the docker image, it does not have access to any directories in your filesystem so it cannot access the files in your working directory. You can use --volume option to mount a directory inside the container. Here I'm making src/Contract available as /project/src/Contract in the container. You can use any path on the container side - does not have to match your local path (though obviously you'll have to adjust JSON if you change it).
  • I used $PWD because docker requires an absolute path and I don't know what's that your side. You can just insert the full path directly if you prefer.
• You're using a relative path in your JSON file and relative paths are relative to the working directory of the compiler. And since the compiler is in a container, its working directory is not the directory you're running docker from. Actually, I'm not sure what the working directory is (I'd have to check if we're setting it in Dockerfile to something specific) but you can just use --workdir option to change it to anything you need. Here I'm setting it to /project because in the container it's the directory that contains src/Compiler.
  • If the files you are compiling are all inside your working directory, you do not need --base-path.
• You need to use --allow-paths to tell the compiler that it's fine to load files from within /project. If you use solc without --standard-json, the directories containing all files you specify on the command line are automatically added to allowed paths so you do not have to worry about it unless you import something from outside these paths. But in --standard-json mode by default no paths are allowed. In that mode you typically provide source code directly using content key. You're using urls instead so you need to explicitly allow paths used there.
• Also, now that you have the input JSON file in a volume you can drop --interactive and just give solc its path instead redirecting standard input.
• Just in case you're not aware: you can use jq to get more readable JSON output. There's the --pretty-json option but that does not affect Standard JSON. This will likely change though: Support coloured messages in standard json #11507.

Also, an unrelated question: I see you switched from 0.8.5 to 0.8.4. Was it by chance because you ran into #11522 or some other reason?

[Gildraen]

Ooooook, i'll try it, it's because in the documentation there is a mention that standard-json does not need to mount volumes.
Maybe should change it no ? cause it's way easier with mounted volume, at least we are quiet tranquil for the remaining problem.

docker run --rm --volume "$PWD/src:/sources" --workdir /sources ethereum/solc:0.8.4 --standard-json Compiler/001/config.json --allow-paths /sources | jq --indent 4

I assume this is an expected output ? if yes.. Thank you !

So, for docker, i didn't use volumes but since i don't really know how the standard json work, i should have tried this yea ^^

Now i have full relative path but that work since i map all volume correctly in docker and with workdir of course
And you answered one of my question : does it take the position of the workdir, or the one of the file, but it's workdir, and it work nice

So now, it's more for understanding, is that better to use JSON or command-line pure compiler ? i mean Json should be cool to work with, specially if we have a lot of contract.
I mean, when we have just multiple file, json still ok right ?

and for the switch to 0.8.4, it's just that i'm not alone on this, and we agreed to work on the current stable version, since 0.8.5 does not (at least for the moment, regarding the changelog) give us some utility things.
But, i ran into this typically out of reach problem in my previous test, but i think it's the same on 0.8.4 (when i have my fully dev environment set, i will look at it)

I mean, we are at the beginning sooo, i can go up to 0.8.5 easily (thanks to docker)

Next steps are some docker-compose, and makefile to make this easier to work with

[cameel]

Ooooook, i'll try it, it's because in the documentation there is a mention that standard-json does not need to mount volumes.
Maybe should change it no ? cause it's way easier with mounted volume, at least we are quiet tranquil for the remaining problem.

I think that point was written with content (rather than urls) in mind, which is the primary way to use Standard JSON so it's a simplification. But yeah, this should be clarified. I'll fix that.

I assume this is an expected output ? if yes.. Thank you !

Yeah, if you got this, then stuff got compiled successfully.

So now, it's more for understanding, is that better to use JSON or command-line pure compiler ? i mean Json should be cool to work with, specially if we have a lot of contract.
I mean, when we have just multiple file, json still ok right ?

Pure command-line interface is going to be more convenient if you are running the compiler directly. It does have some quirks around --base-path and importing from external libraries that we're currently working to iron out (see #11408 and #11409 if you're interested) but it's still more convenient than writing JSON by hand. The path-related docs you have probably seen (https://docs.soliditylang.org/en/latest/path-resolution.html) are a part of that effort. They're still pretty fresh and any feedback is welcome so if there's anything you think could be explained better, feel free to open an issue or submit a PR (or even just comment on specific lines in the original PR: #11442).

But to be honest most people do not use the compiler directly. They use it through a framework like Truffle, Hardhat or Brownie or an IDE extension like vscode-solidity. The framework typically downloads and manages compiler binaries for you. Under the hood these frameworks use Standard JSON.

and for the switch to 0.8.4, it's just that i'm not alone on this, and we agreed to work on the current stable version, since 0.8.5 does not (at least for the moment, regarding the changelog) give us some utility things.

OK. Makes sense. I'm asking because I'm interested in how widespread that issue is. I.e. are users running into it often in practice. Everything indicates that it is and we're actually planning to release 0.8.6 with a fix pretty soon.

[Gildraen]

Sorry for time, i couldn't put my head on it since last time ^^

So, ok, i will get out of JSON for now and go for a pure command-line (with makefile it won't be so hard to do so automated)

I will of course participate to the documentation, at least if i see some point that need more information.

I understand that ppl use some external framework or others, but as i want to understand how everything work, i prefer not to use any external tools as possible ^^

Ok for the unreachable, i'll look for this update then cause it's quite annoying those unreachable code ^^

[cameel]

Ok for the unreachable, i'll look for this update then cause it's quite annoying those unreachable code ^^

Just FYI: 0.8.6 is already out so it's no longer an issue.

[Gildraen]

Hi, so i worked a bit, and now i don't use json anymore.
So i begin with a simple command :

docker run --rm -v $PWD/src:/sources -w /sources ethereum/solc:0.8.6 -o Compiler/001/Output --abi --bin Contract/Contract-001.sol

First thing is, i don't know if it intended, but, the system can't create folders if they does not exists.
Second based on what you said :

If the files you are compiling are all inside your working directory, you do not need --base-path.
You need to use --allow-paths to tell the compiler that it's fine to load files from within /project. If you use solc without --standard-json, the directories containing all files you specify on the command line are automatically added to allowed paths so you do not have to worry about it unless you import something from outside these paths. But in --standard-json mode by default no paths are allowed. In that mode you typically provide source code directly using content key. You're using urls instead so you need to explicitly allow paths used there.

I still need to '--allow-paths .' to make the usable (so it does not take in action that we are in the right folder..)

And then, after those setted up, the third i can't explain :

can't write file..

[cameel]

First thing is, i don't know if it intended, but, the system can't create folders if they does not exists.

I just checked and it does work for me:

mkdir -p src/Contract/
echo "contract C {}" > src/Contract/Contract-001.sol

docker run \
    --rm \
    --volume $PWD/src:/sources \
    --workdir /sources \
    ethereum/solc:0.8.6 \
        --output-dir Compiler/001/Output \
        --abi \
        --bin \
        Contract/Contract-001.sol

ls src/Compiler/001/Output/

ls prints:

C.abi  C.bin

I still need to '--allow-paths .' to make the usable (so it does not take in action that we are in the right folder..)

Do you have any symlinks in your paths? --allow-paths follows symlinks (by design).
Also, do you still need it if you try the example I posted above?

can't write file..

This looks as if the directory exists but the compiler cannot write there. You should check what are the permissions for that directory and which user owns it.

[Gildraen]

Ok, ok.. you wanna laugh ?

While trying to get rid of "root" problem of docker (you know, when you let the container create folder and file, that make your local folder owned by root and it's annoying)

edit : like now, i can't remove the Compiler folder unless i go with sudo..

and all work now, thanks !

To the next problem now ^^

Edit 2 : Yea, still need allow-path though. i haven't any symlink or anything, i just have some relative-path inside of my files

[cameel]

Edit 2 : Yea, still need allow-path though. i haven't any symlink or anything, i just have some relative-path inside of my files

Even in the example I gave with that same exact code?

If not, maybe you could give me a small self-contained example so that I can reproduce it?

BTW, here's a clarification for the docs about docker volumes: #11576. Could you take a look and tell me if it's better now?

[Gildraen]

With your example, there is no problem, since the contract does not use any imported file.

Here is my workspace atm :

and the zip containe my "src" folder, as i have right now you will see that the Contract_001 import BaseContract, wich import some others files everywhere
src.zip

BTW, here's a clarification for the docs about docker volumes: #11576. Could you take a look and tell me if it's better now?

I think it's already clearer ! With the link to the right chapter of the doc, one could easily understand the difference between the code written inside the JSON, and the code in external file, imported via the url option.

[cameel]

Oh. I thought it was complaining about Contract/Contract-001.sol. If it's complaining only about files you import then it's working as designed.

By default you can only import files that are in the same directory as the files you specified on the command line (or its subdirectories). In your case that's files inside src/Contract/. If you want to import from any "side" directories you have to use --allow-paths to tell the compiler that you're doing it on purpose and it's not some malicious library trying to include something from a random directory on your disk.

[Gildraen]

Yea, that's it so alright then, i'll use --alow-path . with the workdir it's ok, meaning that everything inside the folder is ok to use.

and that's why, it does not complain about the "basecontract" because it's on the same folder, logic.

it's a nice security feature that i did not understand first