Merge branch 'main' into update-dependencies-1

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>

CHANGELOG.md

@@ -1,24 +1,61 @@
 # Changelog
 
-## 24.1.0-SNAPSHOT
+## 24.1.1-SNAPSHOT
 
 ### Breaking Changes
 - New `EXECUTION_HALTED` error returned if there is an error executing or simulating a transaction, with the reason for execution being halted. Replaces the generic `INTERNAL_ERROR` return code in certain cases which some applications may be checking for [#6343](https://github.com/hyperledger/besu/pull/6343)
+- The Besu Docker images with `openjdk-latest` tags since 23.10.3 were incorrectly using UID 1001 instead of 1000 for the container's `besu` user. The user now uses 1000 again. Containers created from or migrated to images using UID 1001 will need to chown their persistent database files to UID 1000 [#6360](https://github.com/hyperledger/besu/pull/6360)
+
+### Deprecations
+
+### Additions and Improvements
+- Optimize RocksDB WAL files, allows for faster restart and a more linear disk space utilization [#6328](https://github.com/hyperledger/besu/pull/6328)
+- Disable transaction handling when the node is not in sync, to avoid unnecessary transaction validation work [#6302](https://github.com/hyperledger/besu/pull/6302)
+- Upgrade dependencies [#6377](https://github.com/hyperledger/besu/pull/6377)
+
+### Bug fixes
+- INTERNAL_ERROR from `eth_estimateGas` JSON/RPC calls [#6344](https://github.com/hyperledger/besu/issues/6344)
+- Fix Besu Docker images with `openjdk-latest` tags since 23.10.3 using UID 1001 instead of 1000 for the `besu` user [#6360](https://github.com/hyperledger/besu/pull/6360)
+- Fluent EVM API definition for Tangerine Whistle had incorrect code size validation configured [#6382](https://github.com/hyperledger/besu/pull/6382)
+
+### Download Links
+
+
+## 24.1.0
+
+### Breaking Changes
 
 ### Deprecations
 - Forest pruning (`pruning-enabled` options) is deprecated and will be removed soon. To save disk space consider switching to Bonsai data storage format [#6230](https://github.com/hyperledger/besu/pull/6230)
 
 ### Additions and Improvements
 - Add error messages on authentication failures with username and password [#6212](https://github.com/hyperledger/besu/pull/6212)
-- New `Sequenced` transaction pool. The pool is an evolution of the `legacy` pool and is likely to be more suitable to enterprise or permissioned chains than the `layered` transaction pool. Select to use this pool with `--tx-pool=sequenced`. Supports the same options as the `legacy` pool [#6211](https://github.com/hyperledger/besu/issues/6211)
+- New `Sequenced` transaction pool. The pool is an evolution of the `legacy` pool and is likely to be more suitable to enterprise or permissioned chains than the `layered` transaction pool. Select to use this pool with `--tx-pool=sequenced`. Supports the same options as the `legacy` pool [#6274](https://github.com/hyperledger/besu/issues/6274)
 - Set Ethereum Classic mainnet activation block for Spiral network upgrade [#6267](https://github.com/hyperledger/besu/pull/6267)
 - Add custom genesis file name to config overview if specified [#6297](https://github.com/hyperledger/besu/pull/6297)
 - Update Gradle plugins and replace unmaintained License Gradle Plugin with the actively maintained Gradle License Report [#6275](https://github.com/hyperledger/besu/pull/6275)
-- Optimize RocksDB WAL files, allows for faster restart and a more linear disk space utilization [#6328](https://github.com/hyperledger/besu/pull/6328)
-- Upgrade dependencies [#6377](https://github.com/hyperledger/besu/pull/6377)
 
 ### Bug fixes
-- INTERNAL_ERROR from `eth_estimateGas` JSON/RPC calls [#6344](https://github.com/hyperledger/besu/issues/6344)
+- Hotfix for selfdestruct preimages on bonsai [#6359]((https://github.com/hyperledger/besu/pull/6359)
+- Fix trielog shipping issue during self destruct [#6340]((https://github.com/hyperledger/besu/pull/6340)
+- mitigation for trielog failure [#6315]((https://github.com/hyperledger/besu/pull/6315)
+
+### Download Links
+https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.0/besu-24.1.0.zip / sha256 TBA
+https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/24.1.0/besu-24.1.0.tar.gz / sha256 TBA
+
+
+## 23.10.3-hotfix
+This is a hotfix for a selfdestruct defect that occurred on mainnet at block [18947893](https://etherscan.io/block/18947893)
+
+### Bug fixes
+- Hotfix for selfdestruct preimages on bonsai [#6359]((https://github.com/hyperledger/besu/pull/6359)
+- mitigation for trielog failure [#6315]((https://github.com/hyperledger/besu/pull/6315)
+
+### Download Links
+https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/23.10.3-hotfix/besu-23.10.3-hotfix.zip / sha256 1c37762909858a40eca749fb85b77fb4d1e918f247aff56d518144828bd85378
+https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/23.10.3-hotfix/besu-23.10.3-hotfix.tar.gz / sha256 8e38e9fd0c16e049aa324effc96f9ec31dc06e82ea4995e9dd75d571394667af
+
 
 ## 23.10.3
 
@@ -66,6 +103,7 @@ https://hyperledger.jfrog.io/artifactory/besu-binaries/besu/23.10.3/besu-23.10.3
 - Force tx replacement price bump to zero when zero base fee market is configured or `--min-gas-price` is set to 0. This allows for easier tx replacement in networks where there is not gas price. [#6079](https://github.com/hyperledger/besu/pull/6079)
 - Introduce the possibility to limit the time spent selecting pending transactions during block creation, using the new experimental option `Xblock-txs-selection-max-time` on PoS and PoW networks (by default set to 5000ms) or `Xpoa-block-txs-selection-max-time` on PoA networks (by default 75% of the min block time) [#6044](https://github.com/hyperledger/besu/pull/6044)
 - Remove LowestInvalidNonceCache from `legacy` transaction pool to make it more private networks friendly [#6148](https://github.com/hyperledger/besu/pull/6148)
+- Optimization: Delete leftPad when capturing the stack before and after a frame execution [#6102](https://github.com/hyperledger/besu/pull/6102)
 
 ### Bug fixes
 - Upgrade netty to address CVE-2023-44487, CVE-2023-34462 [#6100](https://github.com/hyperledger/besu/pull/6100)

besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java

@@ -23,6 +23,7 @@
 import static org.hyperledger.besu.cli.config.NetworkName.MAINNET;
 import static org.hyperledger.besu.cli.util.CommandLineUtils.DEPENDENCY_WARNING_MSG;
 import static org.hyperledger.besu.cli.util.CommandLineUtils.DEPRECATION_WARNING_MSG;
+import static org.hyperledger.besu.cli.util.CommandLineUtils.isOptionSet;
 import static org.hyperledger.besu.controller.BesuController.DATABASE_PATH;
 import static org.hyperledger.besu.ethereum.api.graphql.GraphQLConfiguration.DEFAULT_GRAPHQL_HTTP_PORT;
 import static org.hyperledger.besu.ethereum.api.jsonrpc.JsonRpcConfiguration.DEFAULT_ENGINE_JSON_RPC_PORT;
@@ -2027,11 +2028,10 @@ private void issueOptionWarnings() {
             "--p2p-port",
             "--remote-connections-max-percentage"));
 
-    CommandLineUtils.failIfOptionDoesntMeetRequirement(
-        commandLine,
-        "--fast-sync-min-peers can't be used with FULL sync-mode",
-        !SyncMode.isFullSync(getDefaultSyncModeIfNotSet()),
-        singletonList("--fast-sync-min-peers"));
+    if (SyncMode.isFullSync(getDefaultSyncModeIfNotSet())
+        && isOptionSet(commandLine, "--sync-min-peers")) {
+      logger.warn("--sync-min-peers is ignored in FULL sync-mode");
+    }
 
     CommandLineUtils.failIfOptionDoesntMeetRequirement(
         commandLine,

besu/src/test/java/org/hyperledger/besu/cli/BesuCommandTest.java

@@ -5066,8 +5066,7 @@ public void logsSuggestInstallingJemallocWhenEnvVarNotPresent() {
   @Test
   public void logWarnIfFastSyncMinPeersUsedWithFullSync() {
     parseCommand("--sync-mode", "FULL", "--fast-sync-min-peers", "1");
-    assertThat(commandErrorOutput.toString(UTF_8))
-        .contains("--fast-sync-min-peers can't be used with FULL sync-mode");
+    verify(mockLogger).warn("--sync-min-peers is ignored in FULL sync-mode");
   }
 
   @Test

besu/src/test/java/org/hyperledger/besu/services/BesuEventsImplTest.java

@@ -41,7 +41,6 @@
 import org.hyperledger.besu.ethereum.eth.manager.EthMessages;
 import org.hyperledger.besu.ethereum.eth.manager.EthPeer;
 import org.hyperledger.besu.ethereum.eth.manager.EthPeers;
-import org.hyperledger.besu.ethereum.eth.manager.EthScheduler;
 import org.hyperledger.besu.ethereum.eth.sync.BlockBroadcaster;
 import org.hyperledger.besu.ethereum.eth.sync.state.SyncState;
 import org.hyperledger.besu.ethereum.eth.transactions.BlobCache;
@@ -64,6 +63,7 @@
 import org.hyperledger.besu.plugin.data.PropagatedBlockContext;
 import org.hyperledger.besu.plugin.data.SyncStatus;
 import org.hyperledger.besu.services.kvstore.InMemoryKeyValueStorage;
+import org.hyperledger.besu.testutil.DeterministicEthScheduler;
 import org.hyperledger.besu.testutil.TestClock;
 
 import java.math.BigInteger;
@@ -100,7 +100,6 @@ public class BesuEventsImplTest {
   @Mock private EthPeers mockEthPeers;
   @Mock private EthContext mockEthContext;
   @Mock private EthMessages mockEthMessages;
-  @Mock private EthScheduler mockEthScheduler;
 
   @Mock(answer = Answers.RETURNS_DEEP_STUBS)
   private TransactionValidatorFactory mockTransactionValidatorFactory;
@@ -128,7 +127,7 @@ public void setUp() {
 
     when(mockEthContext.getEthMessages()).thenReturn(mockEthMessages);
     when(mockEthContext.getEthPeers()).thenReturn(mockEthPeers);
-    when(mockEthContext.getScheduler()).thenReturn(mockEthScheduler);
+    when(mockEthContext.getScheduler()).thenReturn(new DeterministicEthScheduler());
     lenient().when(mockEthPeers.streamAvailablePeers()).thenAnswer(z -> Stream.empty());
     when(mockProtocolContext.getBlockchain()).thenReturn(blockchain);
     lenient().when(mockProtocolContext.getWorldStateArchive()).thenReturn(mockWorldStateArchive);

build.gradle

@@ -977,6 +977,7 @@ task checkSpdxHeader(type: CheckSpdxHeader) {
     "(.*${sep}.gradle${sep}.*)",
     "(.*${sep}.idea${sep}.*)",
     "(.*${sep}out${sep}.*)",
+    "(.*${sep}bin${sep}.*)",
     "(.*${sep}build${sep}.*)",
     "(.*${sep}src${sep}[^${sep}]+${sep}generated${sep}.*)"
   ].join("|")

docker/openjdk-latest/Dockerfile

@@ -6,7 +6,10 @@ RUN apt-get update && \
  apt-get install --no-install-recommends -q --assume-yes openjdk-21-jre-headless=21* libjemalloc-dev=5.* adduser=3* && \
  apt-get clean  && \
  rm -rf /var/lib/apt/lists/*  && \
- adduser --disabled-password --gecos "" --home /opt/besu besu && \
+ # Ubuntu 23.10 comes with an "ubuntu" user with uid 1000. We need 1000 for besu.
+ userdel ubuntu 2>/dev/null || true && rm -rf /home/ubuntu && \
+ # Ensure we use a stable UID for besu, as file permissions are tied to UIDs.
+ adduser --uid 1000 --disabled-password --gecos "" --home /opt/besu besu && \
     chown besu:besu /opt/besu && \
     chmod 0755 /opt/besu
 

ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/frontier/EthGetFilterChangesIntegrationTest.java

@@ -18,6 +18,7 @@
 import static java.util.Collections.emptyList;
 import static java.util.stream.Collectors.toList;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
 
@@ -107,7 +108,7 @@ public void setUp() {
             blockchain::getChainHeadHeader);
     final ProtocolContext protocolContext = executionContext.getProtocolContext();
 
-    EthContext ethContext = mock(EthContext.class);
+    EthContext ethContext = mock(EthContext.class, RETURNS_DEEP_STUBS);
     EthPeers ethPeers = mock(EthPeers.class);
     when(ethContext.getEthPeers()).thenReturn(ethPeers);
 

ethereum/api/src/integration-test/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/fork/london/EthGetFilterChangesIntegrationTest.java

@@ -18,6 +18,7 @@
 import static java.util.Collections.emptyList;
 import static java.util.stream.Collectors.toList;
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.RETURNS_DEEP_STUBS;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
 
@@ -107,7 +108,7 @@ public void setUp() {
             blockchain::getChainHeadHeader);
     final ProtocolContext protocolContext = executionContext.getProtocolContext();
 
-    EthContext ethContext = mock(EthContext.class);
+    EthContext ethContext = mock(EthContext.class, RETURNS_DEEP_STUBS);
     EthPeers ethPeers = mock(EthPeers.class);
     when(ethContext.getEthPeers()).thenReturn(ethPeers);
 

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/core/Transaction.java

@@ -1007,24 +1007,33 @@ public Transaction detachedCopy() {
             withCommitments ->
                 blobsWithCommitmentsDetachedCopy(withCommitments, detachedVersionedHashes.get()));
 
-    return new Transaction(
-        true,
-        transactionType,
-        nonce,
-        gasPrice,
-        maxPriorityFeePerGas,
-        maxFeePerGas,
-        maxFeePerBlobGas,
-        gasLimit,
-        detachedTo,
-        value,
-        signature,
-        payload.copy(),
-        detachedAccessList,
-        sender,
-        chainId,
-        detachedVersionedHashes,
-        detachedBlobsWithCommitments);
+    final var copiedTx =
+        new Transaction(
+            true,
+            transactionType,
+            nonce,
+            gasPrice,
+            maxPriorityFeePerGas,
+            maxFeePerGas,
+            maxFeePerBlobGas,
+            gasLimit,
+            detachedTo,
+            value,
+            signature,
+            payload.copy(),
+            detachedAccessList,
+            sender,
+            chainId,
+            detachedVersionedHashes,
+            detachedBlobsWithCommitments);
+
+    // copy also the computed fields, to avoid to recompute them
+    copiedTx.sender = this.sender;
+    copiedTx.hash = this.hash;
+    copiedTx.hashNoSignature = this.hashNoSignature;
+    copiedTx.size = this.size;
+
+    return copiedTx;
   }
 
   private AccessListEntry accessListDetachedCopy(final AccessListEntry accessListEntry) {

ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/manager/EthScheduler.java

@@ -23,9 +23,11 @@
 
 import java.time.Duration;
 import java.util.Collection;
+import java.util.Queue;
 import java.util.concurrent.CancellationException;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ConcurrentLinkedDeque;
+import java.util.concurrent.ConcurrentLinkedQueue;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Future;
@@ -34,6 +36,8 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Consumer;
 import java.util.function.Function;
 import java.util.function.Supplier;
 
@@ -295,4 +299,49 @@ public <T> void failAfterTimeout(final CompletableFuture<T> promise, final Durat
         delay,
         unit);
   }
+
+  public <ITEM> OrderedProcessor<ITEM> createOrderedProcessor(final Consumer<ITEM> processor) {
+    return new OrderedProcessor<>(processor);
+  }
+
+  /**
+   * This class is a way to execute a set of tasks, one by one, in a strict order, without blocking
+   * the caller in case there are still previous tasks queued
+   *
+   * @param <ITEM> the class of item to be processed
+   */
+  public class OrderedProcessor<ITEM> {
+    private final Queue<ITEM> blockAddedQueue = new ConcurrentLinkedQueue<>();
+    private final ReentrantLock blockAddedLock = new ReentrantLock();
+    private final Consumer<ITEM> processor;
+
+    private OrderedProcessor(final Consumer<ITEM> processor) {
+      this.processor = processor;
+    }
+
+    public void submit(final ITEM item) {
+      // add the item to the processing queue
+      blockAddedQueue.add(item);
+
+      if (blockAddedLock.hasQueuedThreads()) {
+        // another thread is already waiting to process the queue with our item, there is no need to
+        // schedule another thread
+        LOG.trace(
+            "Block added event queue is already being processed and an already queued thread is present, nothing to do");
+      } else {
+        servicesExecutor.submit(
+            () -> {
+              blockAddedLock.lock();
+              try {
+                // now that we have the lock, process as many items as possible
+                for (ITEM i = blockAddedQueue.poll(); i != null; i = blockAddedQueue.poll()) {
+                  processor.accept(i);
+                }
+              } finally {
+                blockAddedLock.unlock();
+              }
+            });
+      }
+    }
+  }
 }