Enable BucketListDB on Validators by Default

[SirTyson]

Currently, BucketListDB is enabled and running by default in captive-core, but is still disabled by default in validators. While BucketListDB has been running without issue in captive-core, we are holding off on enabling BucketListDB until after Soroban launch on mainnet. BucketListDB has only run on Stellar Classic in production captive-core instances and we want to wait and make sure it works well on Soroban entries before enabling it by default. We will watch how captive-core handles Soroban and enable it after launch.

Currently, captive-core supports persisting indexes on disk, but validators do not. This is due to a bug found during development where loading a persistent index resulted in a corrupted index, causing the DB to load data incorrectly. Because of this issue, validators do not persist indexes. Before enabling BucketListDB for validators, we should resolve #3677 and add safety checks so that validators can persist indexes. After resolving this issue and seeing how captive-core handles Soroban, we should enable BucketListDB.

This is not a pressing issue for now since network performance is not disk bound, but is more of a future-proofing measure and a helpful startup performance increase.

Tasks

Beta Give feedback

1. Run Pubnet Watcher Node with BucketsDB
2. Enable BucketsDB on 1 SDF testnet validator
3. Require BucketsDB flag in config file #4202
   +0
   
4. Drop TxMeta column from TxHistory table #4230
   +0
   
5. Create BucketListDB indexes when upgrade-db is called #4266
   enhancement +1
   
6. Some tables are recreated after BucketListDB migration #4265
   bug +1
   
7. Enable BucketsDB on all testnet validators
8. Enable BucketsDB on one pubnet validator
9. Enable BucketsDB on all pubnet validators

[anupsdf]

Closing this since we have already enabled BL DB in core validators.