Merge bitcoin#20323: tests: Create or use existing properly initializ…

…ed NodeContexts

81137c6 test: Add new ChainTestingSetup and use it (Carl Dong)
7e9e7fe qt/test: [FIX] Add forgotten Context setting in RPCNestedTests (Carl Dong)

Pull request description:

  This is part 1/n of the effort to [de-globalize `ChainstateManager`](bitcoin#20158)

  Reviewers: Looking for tested/Code-Review/plain-ACKs

  ### Context

  In many of our tests, we manually instantiate `NodeContext`s or `ChainstateManager`s in the test code, which is error prone. Instead, we should create or use existing references because:
  1. Before we [de-globalize `ChainstateManager`](bitcoin#20158), much of our code still acts on `g_chainman` (our global `ChainstateManager`), sometimes even when you're calling a method on a specific instance of `ChainstateManager`! This means that we may act on two instances of `ChainstateManager`, which is most likely not what we want.
  2. Using existing references (initialized by the `{Basic,}TestingSetup` constructors) means that you're acting on objects which are properly initialized, instead of "just initialized enough for this dang test to pass". Also, they're already there! It's free!
  3. By acting on the right object, we also allow the review-only assertions in future commits of [de-globalize `ChainstateManager`](bitcoin#20158) to work and demonstrate correctness.

  Some more detailed debugging notes can be found in the first commit, reproduced below:
  ```
  Previously, the validation_chainstatemanager_tests test suite
  instantiated its own duplicate ChainstateManager on which tests were
  performed.

  This wasn't a problem for the specific actions performed in
  that suite. However, the existence of this duplicate ChainstateManager
  and the fact that many of our validation static functions reach for
  g_chainman, ::Chain(state|)Active means we may end up acting on two
  different CChainStates should we write more extensive tests in the
  future.

  This change adds a new ChainTestingSetup which performs all
  initialization previously done by TestingSetup except:

  1. Mempool sanity check frequency setting
  2. ChainState initialization
  3. Genesis Activation
  4. {Ban,Conn,Peer}Man initialization

  Means that we will no longer need to initialize a duplicate
  ChainstateManger in order to test the initialization codepaths of
  CChainState and ChainstateManager.

  Lastly, this change has the additional benefit of allowing for
  review-only assertions meant to show correctness to work in future work
  de-globalizing g_chainman.

  In the test chainstatemanager_rebalance_caches, an additional
  LoadGenesisBlock call is added as MaybeReblanaceCaches eventually calls
  FlushBlockFile, which tries to access vinfoBlockFile[nLastBlockFile],
  which is out of bounds when LoadGenesisBlock hasn't been called yet.

  -----

  Note for the future:

  In a previous version of this change, I put ChainTestingSetup between
  BasicTestingSetup and TestingSetup such that TestingSetup inherited from
  ChainTestingSetup.

  This was suboptimal, and showed how the class con/destructor inheritance
  structure we have for these TestingSetup classes is probably not the
  most suitable abstraction. In particular, for both TestingSetup and
  ChainTestingSetup, we need to stop the scheduler first before anything
  else. Otherwise classes depending on the scheduler may be referenced
  by the scheduler after said classes are freed. This means that there's
  no clear parallel between our teardown code and C++'s destructuring
  order for class hierarchies.

  Future work should strive to coalesce (as much as possible) test and
  non-test init codepaths and perhaps structure it in a more fail-proof
  way.
  ```

ACKs for top commit:
  MarcoFalke:
    ACK 81137c6 looking excellent now 🐩
  jnewbery:
    ACK 81137c6
  ryanofsky:
    Code review ACK 81137c6. This change is simpler after the rebase because wallet & bench commits are dropped.

Tree-SHA512: a8d84f08f2db6428b0b88449bdc814c9db35b7559156d536dfebd3225c2707dba65959e76d2152e3f8c96eacbf1e0b0000f745edf1e196deddb97ff1ef360953

src/qt/test/rpcnestedtests.cpp

@@ -43,41 +43,41 @@ void RPCNestedTests::rpcNestedTests()
     tableRPC.appendCommand("rpcNestedTest", &vRPCCommands[0]);
 
     TestingSetup test;
+    m_node.setContext(&test.m_node);
 
     if (RPCIsInWarmup(nullptr)) SetRPCWarmupFinished();
 
     std::string result;
     std::string result2;
     std::string filtered;
-    interfaces::Node* node = &m_node;
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo()[chain]", &filtered); //simple result filtering with path
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo()[chain]", &filtered); //simple result filtering with path
     QVERIFY(result=="main");
     QVERIFY(filtered == "getblockchaininfo()[chain]");
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblock(getbestblockhash())"); //simple 2 level nesting
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblock(getblock(getbestblockhash())[hash], true)");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblock(getbestblockhash())"); //simple 2 level nesting
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblock(getblock(getbestblockhash())[hash], true)");
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblock( getblock( getblock(getbestblockhash())[hash] )[hash], true)"); //4 level nesting with whitespace, filtering path and boolean parameter
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblock( getblock( getblock(getbestblockhash())[hash] )[hash], true)"); //4 level nesting with whitespace, filtering path and boolean parameter
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo");
     QVERIFY(result.substr(0,1) == "{");
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo()");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo()");
     QVERIFY(result.substr(0,1) == "{");
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo "); //whitespace at the end will be tolerated
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo "); //whitespace at the end will be tolerated
     QVERIFY(result.substr(0,1) == "{");
 
-    (RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo()[\"chain\"]")); //Quote path identifier are allowed, but look after a child containing the quotes in the key
+    (RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo()[\"chain\"]")); //Quote path identifier are allowed, but look after a child containing the quotes in the key
     QVERIFY(result == "null");
 
-    (RPCConsole::RPCExecuteCommandLine(*node, result, "createrawtransaction [] {} 0")); //parameter not in brackets are allowed
-    (RPCConsole::RPCExecuteCommandLine(*node, result2, "createrawtransaction([],{},0)")); //parameter in brackets are allowed
+    (RPCConsole::RPCExecuteCommandLine(m_node, result, "createrawtransaction [] {} 0")); //parameter not in brackets are allowed
+    (RPCConsole::RPCExecuteCommandLine(m_node, result2, "createrawtransaction([],{},0)")); //parameter in brackets are allowed
     QVERIFY(result == result2);
-    (RPCConsole::RPCExecuteCommandLine(*node, result2, "createrawtransaction( [],  {} , 0   )")); //whitespace between parameters is allowed
+    (RPCConsole::RPCExecuteCommandLine(m_node, result2, "createrawtransaction( [],  {} , 0   )")); //whitespace between parameters is allowed
     QVERIFY(result == result2);
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "getblock(getbestblockhash())[tx][0]", &filtered);
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "getblock(getbestblockhash())[tx][0]", &filtered);
     QVERIFY(result == "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b");
     QVERIFY(filtered == "getblock(getbestblockhash())[tx][0]");
 
@@ -102,35 +102,35 @@ void RPCNestedTests::rpcNestedTests()
     RPCConsole::RPCParseCommandLine(nullptr, result, "help(importprivkey(abc), walletpassphrase(def))", false, &filtered);
     QVERIFY(filtered == "help(importprivkey(…), walletpassphrase(…))");
 
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest");
     QVERIFY(result == "[]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest ''");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest ''");
     QVERIFY(result == "[\"\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest \"\"");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest \"\"");
     QVERIFY(result == "[\"\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest '' abc");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest '' abc");
     QVERIFY(result == "[\"\",\"abc\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest abc '' abc");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest abc '' abc");
     QVERIFY(result == "[\"abc\",\"\",\"abc\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest abc  abc");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest abc  abc");
     QVERIFY(result == "[\"abc\",\"abc\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest abc\t\tabc");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest abc\t\tabc");
     QVERIFY(result == "[\"abc\",\"abc\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest(abc )");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest(abc )");
     QVERIFY(result == "[\"abc\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest( abc )");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest( abc )");
     QVERIFY(result == "[\"abc\"]");
-    RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest(   abc   ,   cba )");
+    RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest(   abc   ,   cba )");
     QVERIFY(result == "[\"abc\",\"cba\"]");
 
     // do the QVERIFY_EXCEPTION_THROWN checks only with Qt5.3 and higher (QVERIFY_EXCEPTION_THROWN was introduced in Qt5.3)
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo() .\n"), std::runtime_error); //invalid syntax
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo() getblockchaininfo()"), std::runtime_error); //invalid syntax
-    (RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo(")); //tolerate non closing brackets if we have no arguments
-    (RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo()()()")); //tolerate non command brackts
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "getblockchaininfo(True)"), UniValue); //invalid argument
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "a(getblockchaininfo(True))"), UniValue); //method not found
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest abc,,abc"), std::runtime_error); //don't tollerate empty arguments when using ,
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest(abc,,abc)"), std::runtime_error); //don't tollerate empty arguments when using ,
-    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(*node, result, "rpcNestedTest(abc,,)"), std::runtime_error); //don't tollerate empty arguments when using ,
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo() .\n"), std::runtime_error); //invalid syntax
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo() getblockchaininfo()"), std::runtime_error); //invalid syntax
+    (RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo(")); //tolerate non closing brackets if we have no arguments
+    (RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo()()()")); //tolerate non command brackts
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "getblockchaininfo(True)"), UniValue); //invalid argument
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "a(getblockchaininfo(True))"), UniValue); //method not found
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest abc,,abc"), std::runtime_error); //don't tollerate empty arguments when using ,
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest(abc,,abc)"), std::runtime_error); //don't tollerate empty arguments when using ,
+    QVERIFY_EXCEPTION_THROWN(RPCConsole::RPCExecuteCommandLine(m_node, result, "rpcNestedTest(abc,,)"), std::runtime_error); //don't tollerate empty arguments when using ,
 }

src/test/util/setup_common.cpp

@@ -125,18 +125,12 @@ BasicTestingSetup::~BasicTestingSetup()
     ECC_Stop();
 }
 
-TestingSetup::TestingSetup(const std::string& chainName, const std::vector<const char*>& extra_args)
+ChainTestingSetup::ChainTestingSetup(const std::string& chainName, const std::vector<const char*>& extra_args)
     : BasicTestingSetup(chainName, extra_args)
 {
-    const CChainParams& chainparams = Params();
-    // Ideally we'd move all the RPC tests to the functional testing framework
-    // instead of unit tests, but for now we need these here.
-    RegisterAllCoreRPCCommands(tableRPC);
-
-    m_node.scheduler = MakeUnique<CScheduler>();
-
     // We have to run a scheduler thread to prevent ActivateBestChain
     // from blocking due to queue overrun.
+    m_node.scheduler = MakeUnique<CScheduler>();
     threadGroup.create_thread([&] { TraceThread("scheduler", [&] { m_node.scheduler->serviceQueue(); }); });
     GetMainSignals().RegisterBackgroundSignalScheduler(*m_node.scheduler);
 
@@ -146,39 +140,16 @@ TestingSetup::TestingSetup(const std::string& chainName, const std::vector<const
     m_node.mempool = std::make_unique<CTxMemPool>(m_node.fee_estimator.get(), 1);
 
     m_node.chainman = &::g_chainman;
-    m_node.chainman->InitializeChainstate(*m_node.mempool);
-    ::ChainstateActive().InitCoinsDB(
-        /* cache_size_bytes */ 1 << 23, /* in_memory */ true, /* should_wipe */ false);
-    assert(!::ChainstateActive().CanFlushToDisk());
-    ::ChainstateActive().InitCoinsCache(1 << 23);
-    assert(::ChainstateActive().CanFlushToDisk());
-    if (!LoadGenesisBlock(chainparams)) {
-        throw std::runtime_error("LoadGenesisBlock failed.");
-    }
-
-    BlockValidationState state;
-    if (!ActivateBestChain(state, chainparams)) {
-        throw std::runtime_error(strprintf("ActivateBestChain failed. (%s)", state.ToString()));
-    }
 
     // Start script-checking threads. Set g_parallel_script_checks to true so they are used.
     constexpr int script_check_threads = 2;
     for (int i = 0; i < script_check_threads; ++i) {
         threadGroup.create_thread([i]() { return ThreadScriptCheck(i); });
     }
     g_parallel_script_checks = true;
-
-    m_node.banman = MakeUnique<BanMan>(GetDataDir() / "banlist.dat", nullptr, DEFAULT_MISBEHAVING_BANTIME);
-    m_node.connman = MakeUnique<CConnman>(0x1337, 0x1337); // Deterministic randomness for tests.
-    m_node.peerman = MakeUnique<PeerManager>(chainparams, *m_node.connman, m_node.banman.get(), *m_node.scheduler, *m_node.chainman, *m_node.mempool);
-    {
-        CConnman::Options options;
-        options.m_msgproc = m_node.peerman.get();
-        m_node.connman->Init(options);
-    }
 }
 
-TestingSetup::~TestingSetup()
+ChainTestingSetup::~ChainTestingSetup()
 {
     if (m_node.scheduler) m_node.scheduler->stop();
     threadGroup.interrupt_all();
@@ -196,6 +167,39 @@ TestingSetup::~TestingSetup()
     pblocktree.reset();
 }
 
+TestingSetup::TestingSetup(const std::string& chainName, const std::vector<const char*>& extra_args)
+    : ChainTestingSetup(chainName, extra_args)
+{
+    const CChainParams& chainparams = Params();
+    // Ideally we'd move all the RPC tests to the functional testing framework
+    // instead of unit tests, but for now we need these here.
+    RegisterAllCoreRPCCommands(tableRPC);
+
+    m_node.chainman->InitializeChainstate(*m_node.mempool);
+    ::ChainstateActive().InitCoinsDB(
+        /* cache_size_bytes */ 1 << 23, /* in_memory */ true, /* should_wipe */ false);
+    assert(!::ChainstateActive().CanFlushToDisk());
+    ::ChainstateActive().InitCoinsCache(1 << 23);
+    assert(::ChainstateActive().CanFlushToDisk());
+    if (!LoadGenesisBlock(chainparams)) {
+        throw std::runtime_error("LoadGenesisBlock failed.");
+    }
+
+    BlockValidationState state;
+    if (!ActivateBestChain(state, chainparams)) {
+        throw std::runtime_error(strprintf("ActivateBestChain failed. (%s)", state.ToString()));
+    }
+
+    m_node.banman = MakeUnique<BanMan>(GetDataDir() / "banlist.dat", nullptr, DEFAULT_MISBEHAVING_BANTIME);
+    m_node.connman = MakeUnique<CConnman>(0x1337, 0x1337); // Deterministic randomness for tests.
+    m_node.peerman = MakeUnique<PeerManager>(chainparams, *m_node.connman, m_node.banman.get(), *m_node.scheduler, *m_node.chainman, *m_node.mempool);
+    {
+        CConnman::Options options;
+        options.m_msgproc = m_node.peerman.get();
+        m_node.connman->Init(options);
+    }
+}
+
 TestChain100Setup::TestChain100Setup()
 {
     // Generate a 100-block chain:

src/test/util/setup_common.h

@@ -83,14 +83,21 @@ struct BasicTestingSetup {
     const fs::path m_path_root;
 };
 
-/** Testing setup that configures a complete environment.
- * Included are coins database, script check threads setup.
+/** Testing setup that performs all steps up until right before
+ * ChainstateManager gets initialized. Meant for testing ChainstateManager
+ * initialization behaviour.
  */
-struct TestingSetup : public BasicTestingSetup {
+struct ChainTestingSetup : public BasicTestingSetup {
     boost::thread_group threadGroup;
 
+    explicit ChainTestingSetup(const std::string& chainName = CBaseChainParams::MAIN, const std::vector<const char*>& extra_args = {});
+    ~ChainTestingSetup();
+};
+
+/** Testing setup that configures a complete environment.
+ */
+struct TestingSetup : public ChainTestingSetup {
     explicit TestingSetup(const std::string& chainName = CBaseChainParams::MAIN, const std::vector<const char*>& extra_args = {});
-    ~TestingSetup();
 };
 
 /** Identical to TestingSetup, but chain set to regtest */

src/test/validation_chainstatemanager_tests.cpp

@@ -15,15 +15,16 @@
 
 #include <boost/test/unit_test.hpp>
 
-BOOST_FIXTURE_TEST_SUITE(validation_chainstatemanager_tests, TestingSetup)
+BOOST_FIXTURE_TEST_SUITE(validation_chainstatemanager_tests, ChainTestingSetup)
 
 //! Basic tests for ChainstateManager.
 //!
 //! First create a legacy (IBD) chainstate, then create a snapshot chainstate.
 BOOST_AUTO_TEST_CASE(chainstatemanager)
 {
-    ChainstateManager manager;
-    CTxMemPool mempool;
+    ChainstateManager& manager = *m_node.chainman;
+    CTxMemPool& mempool = *m_node.mempool;
+
     std::vector<CChainState*> chainstates;
     const CChainParams& chainparams = Params();
 
@@ -104,8 +105,9 @@ BOOST_AUTO_TEST_CASE(chainstatemanager)
 //! Test rebalancing the caches associated with each chainstate.
 BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches)
 {
-    ChainstateManager manager;
-    CTxMemPool mempool;
+    ChainstateManager& manager = *m_node.chainman;
+    CTxMemPool& mempool = *m_node.mempool;
+
     size_t max_cache = 10000;
     manager.m_total_coinsdb_cache = max_cache;
     manager.m_total_coinstip_cache = max_cache;
@@ -122,6 +124,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches)
     {
         LOCK(::cs_main);
         c1.InitCoinsCache(1 << 23);
+        BOOST_REQUIRE(c1.LoadGenesisBlock(Params()));
         c1.CoinsTip().SetBestBlock(InsecureRand256());
         manager.MaybeRebalanceCaches();
     }
@@ -139,6 +142,7 @@ BOOST_AUTO_TEST_CASE(chainstatemanager_rebalance_caches)
     {
         LOCK(::cs_main);
         c2.InitCoinsCache(1 << 23);
+        BOOST_REQUIRE(c2.LoadGenesisBlock(Params()));
         c2.CoinsTip().SetBestBlock(InsecureRand256());
         manager.MaybeRebalanceCaches();
     }