[indexer][watermarks][1/n] Modify PruningOptions to point to a toml file of epochs_to_keep and optional per-table overrides

[wlmyng]

Description

Modify PruningOptions to point to a toml file that must specify epochs_to_keep and an optional [overrides] portion of per-table overrides.

pruner.rs file will define prunable tables. the toml file when parsed into Rust config will warn if it tries to name any variants not supported by indexer code. additionally, there's also a check to make sure that prunable tables actually exist in the db

Part of a stack of PRs for watermarks

1. simplify setting up test indexer: [indexer] Simplify setting up a test indexer writer or reader #19663
2. update pruner config: [indexer][watermarks][1/n] Modify PruningOptions to point to a toml file of epochs_to_keep and optional per-table overrides #19637
3. committer writes upper bounds [indexer][watermarks][2/n] committer writes upper bounds to watermarks table #19649
4. pruner writes lower bounds: [indexer][watermarks][3/n] pruner updates watermarks lower bound #19650
5. pruner prunes (wip)

Test plan

How did you test the new or updated feature?

---

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

• Protocol:
• Nodes (Validators and Full nodes):
• Indexer:
• JSON-RPC:
• GraphQL:
• CLI:
• Rust SDK:
• REST API:

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 4, 2024 7:34pm

3 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview		Oct 4, 2024 7:34pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview		Oct 4, 2024 7:34pm
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 4, 2024 7:34pm

[wlmyng]

Whoa.... how'd that happen

[emmazzz]

Left some comments but mostly good. Same comment re objects_history override as Ge and Xun.

[emmazzz]

I find it a bit odd to have both a new and a finalize method when no changes are made between their calls. I understand the purpose might be to reuse the override config map as a complete set, combining both default and override pruning configurations. But what do you think of simplifying this by keeping the struct as just the config, and having a function that takes the config and returns a hashmap with the complete retention policies like this?

pub struct RetentionConfig {
    /// Default retention policy for all tables.
    pub epochs_to_keep: u64,
    /// A user only needs to explicitly list tables that should override the
    /// default retention.
    #[serde(default)]
    pub overrides: HashMap<PrunableTable, u64>,
}

pub fn retention_policies_from_config(config: RetentionConfig
) -> HashMap<PrunableTable, u64> {
    let RetentionConfig {
        epochs_to_keep, overrides
    }
    let res = overrides;
    for table in PrunableTable::iter() {
        res.entry(table).or_insert(self.epochs_to_keep);
    }
    res
}

[emmazzz]

In our chat earlier, I mentioned that this check might not be necessary, but I now think it’s actually useful. Consider a scenario where we add a pruning config for a new table prematurely. Without this check, we might forget to review the config later when the table is officially added, leading to potential issues. Having the check ensures that we verify the pruning configuration at the right time, preventing errors down the line.