feat: Add Atlas configuration via Stacks.toml

[jbencin]

Description

The following parameters can be used in Stacks.toml to configure Atlas database. Example (using defaults):

[atlas]
attachments_max_size = 1048576
max_uninstantiated_attachments = 10000
uninstantiated_attachments_expire_after = 3600
unresolved_attachment_instances_expire_after = 172800

Applicable issues

• fixes #

Additional info (benefits, drawbacks, caveats)

I considered using https://crates.io/crates/validator_derive to implement Atlas config validation, but I know this project has a preference for not adding new dependencies, so I ended up doing it manually. This package could be used in a lot of places, and would result in code that is cleaner and less error prone, so it's worth considering adding it

Made a couple minor code quality improvements also:

• Changed AtlasConfig::default() to AtlasConfig::new() since default() shouldn't be used with parameters
• Replaced FeeEstimationConfigFile::default() with #[derive(Default)]

Checklist

• Test coverage for new or modified code paths
• Changelog is updated
• Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)
• New clarity functions have corresponding PR in clarity-benchmarking repo
• New integration test(s) added to bitcoin-tests.yml

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

Merging #3618 (04a504b) into develop (f6f4ea6) will decrease coverage by 1.04%.
The diff coverage is 47.82%.

@@             Coverage Diff             @@
##           develop    #3618      +/-   ##
===========================================
- Coverage    84.84%   83.80%   -1.04%     
===========================================
  Files          298      297       -1     
  Lines       276633   276103     -530     
===========================================
- Hits        234697   231397    -3300     
- Misses       41936    44706    +2770     

Impacted Files	Coverage Δ
testnet/stacks-node/src/tests/neon_integrations.rs	41.43% <0.00%> (-27.76%)	⬇️
testnet/stacks-node/src/config.rs	55.72% <32.14%> (-1.52%)	⬇️
src/net/atlas/mod.rs	74.11% <48.38%> (-15.72%)	⬇️
src/chainstate/coordinator/mod.rs	85.97% <100.00%> (-1.13%)	⬇️
src/chainstate/coordinator/tests.rs	90.55% <100.00%> (ø)
src/net/mod.rs	72.72% <100.00%> (-0.02%)	⬇️
src/net/p2p.rs	73.82% <100.00%> (+2.59%)	⬆️
testnet/stacks-node/src/neon_node.rs	81.96% <100.00%> (+0.31%)	⬆️
testnet/stacks-node/src/node.rs	84.42% <100.00%> (ø)
testnet/stacks-node/src/run_loop/neon.rs	83.35% <100.00%> (-0.55%)	⬇️

... and 55 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[jcnelson]

The maximum attachment size cannot be smaller than 1_048_576. This is what the current maximum size is in the network, so we have to continue to provide at least that degree of service. Nodes can agree to store larger attachments if they want, however.

[jbencin]

I'll add a constant ATTACHMENTS_MAX_SIZE_MIN and default to that if the user sets a lower value. Should we use the current defaults to enforce minimum values for the other parameters in the [atlas] config as well?

[jbencin]

Actually, it's probably better to crash with an error message on invalid config rather than applying hidden defaults. I'll do that instead

[jbencin]

I have a few more ideas for Atlas improvements that I'd like to get some feedback on. I will probably make a new issue(s) for this work, but I'm going to start the conversation here, so I have some idea of what to do before creating it.

• Add a zonefile-size parameter to name-register in the BNS contract. This could allow a fee to be charged based on size, which would reduce the opportunity for DoS attacks by flooding the network with large zonefiles
• Add a fetch_mempool_attachments parameter to the [atlas] config section. Currently, I am told, attachments are not replicated until a transaction is confirmed in a block, which creates a single point of failure until then. Even just one or two nodes with this parameter set would drastically reduce the possibility of attachments being lost.

I'd like some feedback from people more familiar with the Stacks ecosystem, and distributed systems in general, to let me know if these ideas are feasible, if my assumptions are correct, and if there are better solutions.

[jcnelson]

Add a zonefile-size parameter to name-register in the BNS contract. This could allow a fee to be charged based on size, which would reduce the opportunity for DoS attacks by flooding the network with large zonefiles

Sure; however, the BNS contract cannot be changed (generally goes for all contracts). You'll want to speak with the BNS X folks who are writing new wrappers around BNS to add new features.

Add a fetch_mempool_attachments parameter to the [atlas] config section. Currently, I am told, attachments are not replicated until a transaction is confirmed in a block, which creates a single point of failure until then. Even just one or two nodes with this parameter set would drastically reduce the possibility of attachments being lost.

Attachments are not stored in the mempool. They are only stored on the node that receives the transaction and attachment data from the client. The client could post this data to a few different public nodes and achieve the same end, however.

[kantai]

I think this should be opened against develop

[kantai]

This looks good to me, but it will have to be tested on some production nodes before this can merge.

[kantai]

LGTM -- please just open an issue for the genesis attachments point you raised.

[blockstack-devops]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.