-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stack trace tests expectations failing upon introducing new optimization in the Solidity compiler #5443
Comments
Thanks for opening this issue. I see how this test can be problematic. Would it still fail if we removed the unconditional Can you provide a list of the failing cases? Ideally, we change the tests so that they cover both versions. But if we need to, we can specify that some tests should only be run with certain versions of solc. |
No, in that case, the pattern which triggers the new optimization method is no longer present and then it works as expected.
Starting from this point:
The new optimizations will affect only versions after it is introduced, so it could be covered in a specific set of tests. |
Thanks @matheusaaguiar! I think our tests shouldn't unconditionally @fvictorio should we fix these tests here or in the EDR repository? @matheusaaguiar, we moved our network simulation to Rust and now lives in this repository. The tracing logic isn't fully merged, but we are working on it. Your integration with our test suite will need to be updated. We can provide guidance about it. |
Thanks for letting us know, @alcuadrado. We will look into that and appreciate any guidance. |
I opened #5467 which fixes those six test cases. But when I ran this locally (using this build), I got 28 more failures, most of them related to console logs. I don't know why those don't happen in Circle though. I then added 0.8.25 to the test suite, and got those same 28 failures. So I think it's unrelated to the 0.8.27 optimizations, and should be fixed as part of adding support for 0.8.25 and 0.8.26.
Let's do both for now, it's the simplest thing. After the solc CI moves to using EDR, we can stop updating the tests in the Hardhat repo. |
Ok, I think the reason is this: Using |
Thanks everyone! |
That's a bit confusing, since we have to set |
Sorry, maybe I said it backwards. But I meant exactly that: that our tests don't pass with solc 0.8.26 by default. I was surprised that your CI hadn't catched that, and then I saw it was changing the hardfork, which explains it. When we add support for 0.8.26/0.8.27 those tests should pass, and hopefully you can remove that |
Ah sorry, actually I had that backwards 😅 |
I am working on this PR which improves and introduces new optimizations to the Solidity compiler and I have run into some failing stack traces tests. For context, we run hardhat's tests (among other external tools) in our CI for better coverage. The failing tests' log can be seen here.
I am convinced that those failures are false positives and due to a mismatch between expectations and different generated code which is not accounted for.
In short, with the changes introduced by the aforementioned PR, the compiler can produce further optimized code which produces different stack traces from the expected in the failing tests. See the log above.
In the particular case of the first failure in that log, for example, the call to function
test
with param valuefalse
is expected to produce a revert error from arequire
function (line 6 of thec.sol
file in the same folder):hardhat/packages/hardhat-core/test/internal/hardhat-network/stack-traces/test-files/0_8/revert-without-message/modifiers/call-message/multiple-modifiers-require/test.json
Lines 21 to 28 in 53c422c
File
c.sol
:hardhat/packages/hardhat-core/test/internal/hardhat-network/stack-traces/test-files/0_8/revert-without-message/modifiers/call-message/multiple-modifiers-require/c.sol
Lines 1 to 18 in 53c422c
However, with the optimizations introduced by the PR, the code for that is optimized out (which makes sense since
test
always reverts). I explained how that happens in detail in this comment.The other cases are similar situations where there is a mismatch between the expected stack trace and the new code generated by this PR.
I am not sure if it is possible to change those expectations and how that could be done. What is the preferred way to handle this inconsistency?
Sorry if this is lacking more information. Please feel free to request anything I might have forgotten.
The text was updated successfully, but these errors were encountered: