Skip to content

Conversation

@nikola-matic
Copy link
Collaborator

Fixes #14829

@nikola-matic nikola-matic added this to the 0.8.26 milestone Mar 28, 2024
@nikola-matic nikola-matic marked this pull request as ready for review March 28, 2024 15:45
@nikola-matic nikola-matic requested review from cameel and ekpyron March 28, 2024 15:45
@0xalpharush
Copy link
Contributor

Are there any tests for non-determinism like running in a loop and checking that each compilation of the same source unit results in the same bytecode?

@nikola-matic
Copy link
Collaborator Author

Are there any tests for non-determinism like running in a loop and checking that each compilation of the same source unit results in the same bytecode?

What you're describing is fully deterministic already - i.e. the same sources will always compile to the same bytecode - the issue here is of a different sort; we use a ton of associative containers (e.g. a std::set which typically uses a red black tree underneath) to track values (e.g. generated IR variable names) during the optimization stages, and these can sometimes (extremely rarely) change if you insert an unrelated contract that is unused in the rest of the sources (in this case it's an empty dummy contract) which then causes the order of the elements in such containers to be different than without the said unrelated contract/sources. This is what creates the inconsistency in bytecode, and is what this PR fixes.

These inconsistencies don't actually change the behavior of contracts - the code will semantically remain the same, but the bytecode in such cases should none the less be identical, especially when it comes to source verification.

@nikola-matic nikola-matic force-pushed the bytecode-different-with-dummy-file branch from 11b27e3 to b423c2a Compare April 2, 2024 14:15
@cameel
Copy link
Collaborator

cameel commented Apr 2, 2024

Given how easy it is to inadvertently run into this problem when using YulString with associative containers, I wonder if we can do something to prevent them in the future. Maybe we could e.g. create wrappers over set<YulString> and map<YulString> that remove any support for iterating them and add a "style" rule preventing the use of unwrapped containers?

@nikola-matic nikola-matic force-pushed the bytecode-different-with-dummy-file branch from b423c2a to a031490 Compare April 3, 2024 12:59
@nikola-matic nikola-matic force-pushed the bytecode-different-with-dummy-file branch from a031490 to 972c464 Compare April 5, 2024 09:05
@nikola-matic nikola-matic force-pushed the bytecode-different-with-dummy-file branch from 5148ff3 to 7b86576 Compare April 15, 2024 13:44
@nikola-matic nikola-matic force-pushed the bytecode-different-with-dummy-file branch 4 times, most recently from 843b813 to 6822901 Compare April 16, 2024 09:15
Comment on lines 42 to 44
via_ir: bool,
optimize: bool = False,
yul_optimizations: Optional[str] = None) -> FileReport:
Copy link
Member

@r0qs r0qs Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as a note, so we can remember to eventually refactor this. We could pass a dictionary here like what was done in https://github.com/ethereum/solidity/blob/bbb7f58be026fdc51b0b4694a6f25c22a1425586/scripts/externalTests/test_helpers.py#L52 or event have a class for compiler settings so we could reuse in other scripts. I believe this will be desirable when we migrate the external tests to python.

@nikola-matic nikola-matic force-pushed the bytecode-different-with-dummy-file branch 4 times, most recently from b4a8906 to df5229e Compare April 16, 2024 20:05
@nikola-matic
Copy link
Collaborator Author

@cameel this should be ready now.

@nikola-matic nikola-matic requested a review from cameel April 16, 2024 20:06
cameel
cameel previously approved these changes Apr 17, 2024
Copy link
Collaborator

@cameel cameel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good in terms of correctness, just needs two style tweaks.

Changelog

Review comments

Unique vector

Repro test
@nikola-matic nikola-matic merged commit 39af449 into develop Apr 17, 2024
@nikola-matic nikola-matic deleted the bytecode-different-with-dummy-file branch April 17, 2024 18:40
Comment on lines +383 to +387
template<typename T>
void swap(UniqueVector<T>& _lhs, UniqueVector<T>& _rhs)
{
std::swap(_lhs.contents(), _rhs.contents());
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just looking at this to consider using it for #15262 and noticed: this swap function won't work, will it? contents is a const reference, you can't swap these things...
what the std::swap below will do is to use the implicit move constructors of UniqueVector...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you're right, contents() should ideally just return a reference...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have honestly expected a compilation error here, this is really insidious.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, it's currently unused - and C++ only type-checks on instantiation :-). And I mean, it's not unused without reason: we don't need it, so we might as well remove the swap - std::swap will just work based on moving regardless

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shift in AST IDs affects inlining and expression splitting decisions in Yul Optimizer, causing bytecode differences

6 participants