Merge branch 'ethereum:develop' into colors

Author: shikharvashistha

Changelog.md

@@ -14,20 +14,23 @@ Compiler Features:
  * SMTChecker: Add constraints to better correlate ``address(this).balance`` and ``msg.value``.
  * SMTChecker: Support the ``value`` option for external function calls.
  * SMTChecker: Support constants via modules.
+ * SMTChecker: Support user defined value types.
 
 
 Bugfixes:
  * Code Generator: Fix ICE on assigning to calldata structs and statically-sized calldata arrays in inline assembly.
  * Code Generator: Use stable source order for ABI functions.
- * Commandline Interface: Report optimizer options as invalid in Standard JSON and linker modes instead of ignoring them.
  * Commandline Interface: Disallow the ``--experimental-via-ir`` option in Standard JSON, Assembler and Linker modes.
- * Opcode Optimizer: Prevent the optimizer from running multiple times to avoid potential bytecode differences for referenced code.
+ * Commandline Interface: Report optimizer options as invalid in Standard JSON and linker modes instead of ignoring them.
  * Name Resolver: Fix that when importing an aliased symbol using ``import {AliasedName} from "a.sol"`` it would use the original name of the symbol and not the aliased one.
+ * Opcode Optimizer: Prevent the optimizer from running multiple times to avoid potential bytecode differences for referenced code.
  * Parser: Properly check for multiple SPDX license identifiers next to each other and validate them.
+ * SMTChecker: Fix BMC's constraints regarding internal functions.
  * SMTChecker: Fix false negative caused by ``push`` on storage array references returned by internal functions.
  * SMTChecker: Fix false positive in external calls from constructors.
  * SMTChecker: Fix internal error on some multi-source uses of ``abi.*``, cryptographic functions and constants.
- * SMTChecker: Fix BMC's constraints regarding internal functions.
+ * Standard JSON: Fix non-fatal errors in Yul mode being discarded if followed by a fatal error.
+ * Type Checker: Correct wrong error message in inline assembly complaining about ``.slot`` or ``.offset` not valid when actually ``.length`` was used.
  * Type Checker: Disallow modifier declarations and definitions in interfaces.
  * Yul Optimizer: Fix a crash in LoadResolver, when ``keccak256`` has particular non-identifier arguments.
 

README.md

@@ -76,7 +76,7 @@ Please follow the
 if you want to help.
 
 You can find our current feature and bug priorities for forthcoming
-releases [in the projects section](https://github.com/ethereum/solidity/projects).
+releases in the [projects section](https://github.com/ethereum/solidity/projects).
 
 ## Maintainers
 * [@axic](https://github.com/axic)

docs/yul.rst

@@ -174,18 +174,40 @@ whitespace, i.e. there is no terminating ``;`` or newline required.
 Literals
 --------
 
-As literals, you can use integer constants in decimal or hexadecimal notation
-or strings as ASCII (`"abc"`) or HEX strings (`hex"616263"`) of up to
-32 bytes length.
+As literals, you can use:
+
+- Integer constants in decimal or hexadecimal notation.
+
+- ASCII strings (e.g. ``"abc"``), which may contain hex escapes ``\xNN`` and Unicode escapes ``\uNNNN`` where ``N`` are hexadecimal digits.
+
+- Hex strings (e.g. ``hex"616263"``).
+
+In the EVM dialect of Yul, literals represent 256-bit words as follows:
+
+- Decimal or hexadecimal constants must be less than ``2**256``.
+  They represent the 256-bit word with that value as an unsigned integer in big endian encoding.
+
+- An ASCII string is first viewed as a byte sequence, by viewing
+  a non-escape ASCII character as a single byte whose value is the ASCII code,
+  an escape ``\xNN`` as single byte with that value, and
+  an escape ``\uNNNN`` as the UTF-8 sequence of bytes for that code point.
+  The byte sequence must not exceed 32 bytes.
+  The byte sequence is padded with zeros on the right to reach 32 bytes in length;
+  in other words, the string is stored left-aligned.
+  The padded byte sequence represents a 256-bit word whose most significant 8 bits are the ones from the first byte,
+  i.e. the bytes are interpreted in big endian form.
+
+- A hex string is first viewed as a byte sequence, by viewing
+  each pair of contiguous hex digits as a byte.
+  The byte sequence must not exceed 32 bytes (i.e. 64 hex digits), and is treated as above.
 
 When compiling for the EVM, this will be translated into an
 appropriate ``PUSHi`` instruction. In the following example,
 ``3`` and ``2`` are added resulting in 5 and then the
 bitwise ``and`` with the string "abc" is computed.
 The final value is assigned to a local variable called ``x``.
 
-Strings are stored left-aligned and cannot be longer than 32 bytes.
-The limit does not apply to string literals passed to builtin functions that require
+The 32-byte limit above does not apply to string literals passed to builtin functions that require
 literal arguments (e.g. ``setimmutable`` or ``loadimmutable``). Those strings never end up in the
 generated bytecode.
 

libevmasm/Assembly.cpp

@@ -36,8 +36,11 @@
 
 #include <json/json.h>
 
-#include <fstream>
 #include <range/v3/algorithm/any_of.hpp>
+#include <range/v3/view/enumerate.hpp>
+
+#include <fstream>
+#include <limits>
 
 using namespace std;
 using namespace solidity;
@@ -56,7 +59,7 @@ AssemblyItem const& Assembly::append(AssemblyItem const& _i)
 	return m_items.back();
 }
 
-unsigned Assembly::bytesRequired(unsigned subTagSize) const
+unsigned Assembly::codeSize(unsigned subTagSize) const
 {
 	for (unsigned tagSize = subTagSize; true; ++tagSize)
 	{
@@ -66,7 +69,7 @@ unsigned Assembly::bytesRequired(unsigned subTagSize) const
 
 		for (AssemblyItem const& i: m_items)
 			ret += i.bytesRequired(tagSize);
-		if (util::bytesRequired(ret) <= tagSize)
+		if (numberEncodingSize(ret) <= tagSize)
 			return static_cast<unsigned>(ret);
 	}
 }
@@ -177,7 +180,7 @@ void Assembly::assemblyStream(ostream& _out, string const& _prefix, StringMap co
 		_out << _prefix << "stop" << endl;
 		for (auto const& i: m_data)
 			if (u256(i.first) >= m_subs.size())
-				_out << _prefix << "data_" << toHex(u256(i.first)) << " " << toHex(i.second) << endl;
+				_out << _prefix << "data_" << toHex(u256(i.first)) << " " << util::toHex(i.second) << endl;
 
 		for (size_t i = 0; i < m_subs.size(); ++i)
 		{
@@ -188,7 +191,7 @@ void Assembly::assemblyStream(ostream& _out, string const& _prefix, StringMap co
 	}
 
 	if (m_auxiliaryData.size() > 0)
-		_out << endl << _prefix << "auxdata: 0x" << toHex(m_auxiliaryData) << endl;
+		_out << endl << _prefix << "auxdata: 0x" << util::toHex(m_auxiliaryData) << endl;
 }
 
 string Assembly::assemblyString(StringMap const& _sourceCodes) const
@@ -308,7 +311,7 @@ Json::Value Assembly::assemblyJSON(map<string, unsigned> const& _sourceIndices)
 			collection.append(createJsonValue("PUSH data", sourceIndex, i.location().start, i.location().end, toStringInHex(i.data())));
 			break;
 		case VerbatimBytecode:
-			collection.append(createJsonValue("VERBATIM", sourceIndex, i.location().start, i.location().end, toHex(i.verbatimData())));
+			collection.append(createJsonValue("VERBATIM", sourceIndex, i.location().start, i.location().end, util::toHex(i.verbatimData())));
 			break;
 		default:
 			assertThrow(false, InvalidOpcode, "");
@@ -320,7 +323,7 @@ Json::Value Assembly::assemblyJSON(map<string, unsigned> const& _sourceIndices)
 		Json::Value& data = root[".data"] = Json::objectValue;
 		for (auto const& i: m_data)
 			if (u256(i.first) >= m_subs.size())
-				data[toStringInHex((u256)i.first)] = toHex(i.second);
+				data[toStringInHex((u256)i.first)] = util::toHex(i.second);
 
 		for (size_t i = 0; i < m_subs.size(); ++i)
 		{
@@ -331,7 +334,7 @@ Json::Value Assembly::assemblyJSON(map<string, unsigned> const& _sourceIndices)
 	}
 
 	if (m_auxiliaryData.size() > 0)
-		root[".auxdata"] = toHex(m_auxiliaryData);
+		root[".auxdata"] = util::toHex(m_auxiliaryData);
 
 	return root;
 }
@@ -589,20 +592,20 @@ LinkerObject const& Assembly::assemble() const
 			"Cannot push and assign immutables in the same assembly subroutine."
 		);
 
-	unsigned bytesRequiredForCode = bytesRequired(static_cast<unsigned>(subTagSize));
+	unsigned bytesRequiredForCode = codeSize(static_cast<unsigned>(subTagSize));
 	m_tagPositionsInBytecode = vector<size_t>(m_usedTags, numeric_limits<size_t>::max());
 	map<size_t, pair<size_t, size_t>> tagRef;
 	multimap<h256, unsigned> dataRef;
 	multimap<size_t, size_t> subRef;
 	vector<unsigned> sizeRef; ///< Pointers to code locations where the size of the program is inserted
-	unsigned bytesPerTag = util::bytesRequired(bytesRequiredForCode);
+	unsigned bytesPerTag = numberEncodingSize(bytesRequiredForCode);
 	uint8_t tagPush = static_cast<uint8_t>(pushInstruction(bytesPerTag));
 
 	unsigned bytesRequiredIncludingData = bytesRequiredForCode + 1 + static_cast<unsigned>(m_auxiliaryData.size());
 	for (auto const& sub: m_subs)
 		bytesRequiredIncludingData += static_cast<unsigned>(sub->assemble().bytecode.size());
 
-	unsigned bytesPerDataRef = util::bytesRequired(bytesRequiredIncludingData);
+	unsigned bytesPerDataRef = numberEncodingSize(bytesRequiredIncludingData);
 	uint8_t dataRefPush = static_cast<uint8_t>(pushInstruction(bytesPerDataRef));
 	ret.bytecode.reserve(bytesRequiredIncludingData);
 
@@ -619,7 +622,7 @@ LinkerObject const& Assembly::assemble() const
 			break;
 		case Push:
 		{
-			unsigned b = max<unsigned>(1, util::bytesRequired(i.data()));
+			unsigned b = max<unsigned>(1, numberEncodingSize(i.data()));
 			ret.bytecode.push_back(static_cast<uint8_t>(pushInstruction(b)));
 			ret.bytecode.resize(ret.bytecode.size() + b);
 			bytesRef byr(&ret.bytecode.back() + 1 - b, b);
@@ -649,7 +652,7 @@ LinkerObject const& Assembly::assemble() const
 			assertThrow(i.data() <= numeric_limits<size_t>::max(), AssemblyException, "");
 			auto s = subAssemblyById(static_cast<size_t>(i.data()))->assemble().bytecode.size();
 			i.setPushedValue(u256(s));
-			unsigned b = max<unsigned>(1, util::bytesRequired(s));
+			unsigned b = max<unsigned>(1, numberEncodingSize(s));
 			ret.bytecode.push_back(static_cast<uint8_t>(pushInstruction(b)));
 			ret.bytecode.resize(ret.bytecode.size() + b);
 			bytesRef byr(&ret.bytecode.back() + 1 - b, b);
@@ -754,15 +757,23 @@ LinkerObject const& Assembly::assemble() const
 		assertThrow(tagId < tagPositions.size(), AssemblyException, "Reference to non-existing tag.");
 		size_t pos = tagPositions[tagId];
 		assertThrow(pos != numeric_limits<size_t>::max(), AssemblyException, "Reference to tag without position.");
-		assertThrow(util::bytesRequired(pos) <= bytesPerTag, AssemblyException, "Tag too large for reserved space.");
+		assertThrow(numberEncodingSize(pos) <= bytesPerTag, AssemblyException, "Tag too large for reserved space.");
 		bytesRef r(ret.bytecode.data() + i.first, bytesPerTag);
 		toBigEndian(pos, r);
 	}
 	for (auto const& [name, tagInfo]: m_namedTags)
 	{
 		size_t position = m_tagPositionsInBytecode.at(tagInfo.id);
+		optional<size_t> tagIndex;
+		for (auto&& [index, item]: m_items | ranges::views::enumerate)
+			if (item.type() == Tag && static_cast<size_t>(item.data()) == tagInfo.id)
+			{
+				tagIndex = index;
+				break;
+			}
 		ret.functionDebugData[name] = {
 			position == numeric_limits<size_t>::max() ? nullopt : optional<size_t>{position},
+			tagIndex,
 			tagInfo.sourceID,
 			tagInfo.params,
 			tagInfo.returns

libevmasm/Assembly.h

@@ -167,7 +167,7 @@ class Assembly
 	/// that are referenced in a super-assembly.
 	std::map<u256, u256> const& optimiseInternal(OptimiserSettings const& _settings, std::set<size_t> _tagsReferencedFromOutside);
 
-	unsigned bytesRequired(unsigned subTagSize) const;
+	unsigned codeSize(unsigned subTagSize) const;
 
 private:
 	static Json::Value createJsonValue(

libevmasm/AssemblyItem.cpp

@@ -21,11 +21,13 @@
 #include <libevmasm/Assembly.h>
 
 #include <libsolutil/CommonData.h>
+#include <libsolutil/Numeric.h>
 #include <libsolutil/StringUtils.h>
 #include <libsolutil/FixedHash.h>
 #include <liblangutil/SourceLocation.h>
 
 #include <fstream>
+#include <limits>
 
 using namespace std;
 using namespace solidity;
@@ -71,7 +73,7 @@ size_t AssemblyItem::bytesRequired(size_t _addressLength) const
 	case Tag: // 1 byte for the JUMPDEST
 		return 1;
 	case Push:
-		return 1 + max<size_t>(1, util::bytesRequired(data()));
+		return 1 + max<size_t>(1, numberEncodingSize(data()));
 	case PushSubSize:
 	case PushProgramSize:
 		return 1 + 4;		// worst case: a 16MB program
@@ -189,7 +191,7 @@ string AssemblyItem::toAssemblyText(Assembly const& _assembly) const
 		break;
 	}
 	case Push:
-		text = toHex(util::toCompactBigEndian(data(), 1), util::HexPrefix::Add);
+		text = toHex(toCompactBigEndian(data(), 1), util::HexPrefix::Add);
 		break;
 	case PushTag:
 	{
@@ -207,7 +209,7 @@ string AssemblyItem::toAssemblyText(Assembly const& _assembly) const
 		text = string("tag_") + to_string(static_cast<size_t>(data())) + ":";
 		break;
 	case PushData:
-		text = string("data_") + util::toHex(data());
+		text = string("data_") + toHex(data());
 		break;
 	case PushSub:
 	case PushSubSize:
@@ -226,16 +228,16 @@ string AssemblyItem::toAssemblyText(Assembly const& _assembly) const
 		text = string("bytecodeSize");
 		break;
 	case PushLibraryAddress:
-		text = string("linkerSymbol(\"") + util::toHex(data()) + string("\")");
+		text = string("linkerSymbol(\"") + toHex(data()) + string("\")");
 		break;
 	case PushDeployTimeAddress:
 		text = string("deployTimeAddress()");
 		break;
 	case PushImmutable:
-		text = string("immutable(\"") + toHex(util::toCompactBigEndian(data(), 1), util::HexPrefix::Add) + "\")";
+		text = string("immutable(\"") + "0x" + util::toHex(toCompactBigEndian(data(), 1)) + "\")";
 		break;
 	case AssignImmutable:
-		text = string("assignImmutable(\"") + toHex(util::toCompactBigEndian(data(), 1), util::HexPrefix::Add) + "\")";
+		text = string("assignImmutable(\"") + "0x" + util::toHex(toCompactBigEndian(data(), 1)) + "\")";
 		break;
 	case UndefinedItem:
 		assertThrow(false, AssemblyException, "Invalid assembly item.");

libevmasm/BlockDeduplicator.h

@@ -25,6 +25,8 @@
 #pragma once
 
 #include <libsolutil/Common.h>
+#include <libsolutil/Numeric.h>
+
 
 #include <cstddef>
 #include <vector>

libevmasm/ConstantOptimiser.cpp

@@ -133,7 +133,7 @@ bigint LiteralMethod::gasNeeded() const
 	return combineGas(
 		simpleRunGas({Instruction::PUSH1}),
 		// PUSHX plus data
-		(m_params.isCreation ? GasCosts::txDataNonZeroGas(m_params.evmVersion) : GasCosts::createDataGas) + dataGas(util::toCompactBigEndian(m_value, 1)),
+		(m_params.isCreation ? GasCosts::txDataNonZeroGas(m_params.evmVersion) : GasCosts::createDataGas) + dataGas(toCompactBigEndian(m_value, 1)),
 		0
 	);
 }
@@ -146,13 +146,13 @@ bigint CodeCopyMethod::gasNeeded() const
 		// Data gas for copy routines: Some bytes are zero, but we ignore them.
 		bytesRequired(copyRoutine()) * (m_params.isCreation ? GasCosts::txDataNonZeroGas(m_params.evmVersion) : GasCosts::createDataGas),
 		// Data gas for data itself
-		dataGas(util::toBigEndian(m_value))
+		dataGas(toBigEndian(m_value))
 	);
 }
 
 AssemblyItems CodeCopyMethod::execute(Assembly& _assembly) const
 {
-	bytes data = util::toBigEndian(m_value);
+	bytes data = toBigEndian(m_value);
 	assertThrow(data.size() == 32, OptimizerException, "Invalid number encoding.");
 	AssemblyItems actualCopyRoutine = copyRoutine();
 	actualCopyRoutine[4] = _assembly.newData(data);
@@ -192,7 +192,7 @@ AssemblyItems ComputeMethod::findRepresentation(u256 const& _value)
 	if (_value < 0x10000)
 		// Very small value, not worth computing
 		return AssemblyItems{_value};
-	else if (util::bytesRequired(~_value) < util::bytesRequired(_value))
+	else if (numberEncodingSize(~_value) < numberEncodingSize(_value))
 		// Negated is shorter to represent
 		return findRepresentation(~_value) + AssemblyItems{Instruction::NOT};
 	else

libevmasm/ControlFlowGraph.h

@@ -24,12 +24,14 @@
 
 #pragma once
 
-#include <vector>
-#include <memory>
 #include <libsolutil/Common.h>
 #include <libsolutil/Assertions.h>
 #include <libevmasm/ExpressionClasses.h>
 
+#include <vector>
+#include <memory>
+#include <limits>
+
 namespace solidity::evmasm
 {
 

libevmasm/ExpressionClasses.cpp

@@ -29,6 +29,7 @@
 
 #include <functional>
 #include <tuple>
+#include <limits>
 
 using namespace std;
 using namespace solidity;