doc: add folder and presentation

doc/CHECKLIST.md

@@ -0,0 +1,138 @@
+## Development Checklist
+
+#### Peers
+
+- [x] Bootstrap peer list with DNS
+  - [ ] Home brewed DNS resolver? (Too hard. Feature gated DNS instead)
+  - [x] Check for DNS flooding/poisoning? (Kind of: just limit to 256 peers per DNS query)
+- [x] Persist to storage
+  - [x] Organize by `/16`? (Just don't select peers from the same net group)
+  - [ ] Weight the priorities of high probability connections (DNS), service flags, and new peer discovery
+  - [x] Condense to single DB
+- [ ] Ban peers
+- [x] Add optional whitelist
+- [x] Add in-memory `PeerStore` implementor
+
+#### Headers
+
+- [x] Sync to known checkpoints with a designated "sync peer"
+- [ ] Validation
+  - [x] Median time past
+  - [x] All headers connect
+  - [x] No forks before last known checkpoint
+  - [x] Header pass their own PoW
+  - [ ] Difficulty retargeting audit:
+    - [x] [PR](https://github.com/rust-bitcoin/rust-bitcoin/pull/2740)
+  - [ ] Network adjusted time
+- [x] Handle forks (took the Neutrino approach and just disconnect peers if they send forks with less work)
+  - [ ] Manage orphaned header chains (Not necessary if we just follow the chain of most work and store the headers)
+  - [x] Extend valid forks
+  - [x] Create new forks
+  - [x] Try to reorg when encountering new forks
+  - [ ] Take the old best chain and make it a fork (Again, we just drop it and assume we will pick it back up if it gets more work)
+- [x] Persist to storage
+  - [x] Determine if the block hash or height should be the primary key
+  - [x] Speed up writes with pointers
+  - [x] Add method to write over heights in reorg
+  - [x] Move headers to DB when the `BTreeMap` is large
+- [x] Exponential backoff for locators
+
+#### Filters
+
+- [ ] API
+  - [ ] Compute block filter from block? (Non-trivial. The computed filter relies on previous outputs not contained in the block. We could estimate the `ScriptBuf` using the `Witness`?)
+  - [x] Check set inclusion given filter
+- [ ] Chain
+  - [x] Manage a queue of proposed header chains
+  - [x] Find disputes
+  - [x] Broadcast the next CF header message to all peers
+  - [ ] Resolve disputes by downloading blocks? (Again, hard to resolve the dispute if you can't compute the filter)
+  - [x] Add new filters to the chain, verifying with the `FilterHash`
+- [ ] Optimizations
+  - [x] Hashmap the `BlockHash` to `FilterHash` relationship in memory
+  - [ ] Persist SPKs that have already been proven to be in a filter? (Not necessary, the crate should remain mostly state-less)
+
+#### Main thread
+
+- [x] Respond to peers with next `getheader` message
+- [x] Manage the number of peers and disconnects
+- [x] Organize the peers in a `BTreeMap` or similar
+  - [x] Poll handles for progress
+  - [x] Designate a "sync" peer
+  - [x] Track "network adjusted time"
+- [x] Have some `State` to manage what messages to send out
+- [x] Seed with SPKs and wallet "birthday"
+  - [x] Add SPKs
+  - [x] Build from `HeaderCheckpoint`
+- [x] Rescan with new `ScriptBuf`
+
+#### Peer threads
+
+- [x] Reach out with v1 version message
+- [x] Respond to `Ping`
+- [x] Send `Verack` and eagerly send `GetAddr`
+  - [ ] May limit addresses if peer persistence is saturated
+- [x] Filter messages at the reader level
+  - [ ] Add back: `Inv`, `Block`, `TX`, ?
+    - [x] `Inv` (blocks)
+    - [x] `Block`
+  - [x] Update `Inv` of block headers to header chain
+- [ ] Set up "peer config"
+  - [x] TCP timeout
+  - [x] Should ask for IP addresses (More of a DB level thing. We need peers if we are below a certain threshold)
+  - [x] Should serve CPF
+- [ ] Set up "timer"
+  - [x] Check for DOS
+  - [x] Message counter
+  - [ ] `Ping` if peer has not been heard from (Probably better to just disconnect)
+- [x] `Disconnect` peers with high latency (If we send a critical message and a peer doesn't respond in 5 seconds, disconnect)
+- [ ] Add BIP-324 with V1 fallback
+
+#### Transaction Broadcaster
+
+- [ ] Rebroadcast for every TX not included in new blocks (Not possible without persistence, probably unnecessary)
+- [x] Add `ScriptBuf` to script set
+
+#### Meta
+
+- [x] Add more error cases for loading faulty headers from persistence
+- [x] Handle `Inv` during CF header download
+- [ ] Add local unconfirmed transaction DB? (Advised against by downstream)
+- [ ] Too many `clone`
+
+#### Testing
+
+- [ ] Chain
+  - [x] Usual extend
+  - [x] Fork with less work
+  - [x] Orphaned fork
+  - [x] Fork with equal work
+  - [x] Fork with more work
+- [ ] CF header chain
+  - [ ] Unexpected stop hash
+  - [ ] Unexpected filter hash
+  - [ ] Multiple peers expected filter hash
+  - [ ] Properly identify bad peers (Not testable without a way to compute the filters)
+- [ ] Filter chain
+  - [ ] Repeated filter
+  - [ ] Bad filter
+- [ ] Header Chain
+  - [x] Expected height
+  - [x] Expected height after fork
+  - [x] Expected hash at height
+  - [x] Expected work after height
+  - [x] Properly handles fork
+  - [x] `extend` handles forks, including down to the anchor
+- [ ] Bitcoin Core, Regtest
+  - [x] Normal sync
+  - [x] Valid transaction broadcast
+  - [x] Live fork
+  - [x] Fork with SQL
+  - [x] Fork with a stale anchor checkpoint start
+  - [x] Depth two fork, SQL
+- [ ] CI
+  - [x] MacOS, Windows, Linux
+  - [x] 1.63, stable, beta, nightly
+  - [x] Format and clippy
+  - [ ] Regtest sync with Bitcoin Core
+  - [x] On PR

doc/DETAILS.md

@@ -0,0 +1,80 @@
+This document outlines some miscellaneous information about the project and some recommendations for implementation. The goal for Kyoto is to run on mobile devices with a low-enough memory footprint to allow users to interact with Lightning Network applications as well. To that end, some assumptions and design goals were made to cater to low-resource devices. Neutrino and LND are the current standard of a server-style, SPV Lightning Node implementation, so Kyoto is generally compliment and not a substitution.
+
+# Scope
+
+### Functional Goals
+
+- [x] Provide an archival index for blocks and transactions related to a set of `scriptPubKey`, presumably because the user is interested in transactions with these scripts involved.
+- [x] Provide an interface to the P2P network, particularly to allow for new transaction broadcasting. Once BIP-324 is integrated, communication over the P2P network will also be encrypted.
+- [x] Provide a testing ground for experimentation and research into the Bitcoin P2P network.
+- [x] Provide rudimentary blockchain data, like the height of the chain, the "chainwork", the `CompactTarget` of the last block, etc.
+
+### Out of Scope
+
+- Any wallet functionality beyond indexing transactions. This includes balances, transaction construction, etc. Bitcoin wallets are complex for a number of reasons, and additional functionality within this scope would detract from other improvements.
+
+# Recommendations
+
+While Kyoto is configurable, there are tradeoffs to each configuration, and some may be better than others. The main advantage to using a light client is increased privacy, but the hope is Kyoto may even lead to a better user experience than standard server-client setups.
+
+### Privacy
+
+Under the assumption that only a few connections should be maintained, broadcasting transactions in a privacy-focused way presents a challenge. Reliable, seeded peers speed up the syncing process, but broadcasting transactions to seeded peers does not offer a significant privacy benefit compared to simply using a server. As such, it is recommended to use one or two seeded peers, but to allow connections for one or more random peers found by network gossip. When broadcasting a transaction, you may then broadcast your transaction to a random peer. When reaching out to nodes, the Kyoto version message does not contain your users' IP addresses, only _127.0.0.1_, so packet association by nodes you connect is made harder. When downloading blocks, requests are always made to random peers, so scanning for outputs associated with your scripts has a great anonymity set.
+
+### Runtime
+
+Kyoto does not require a large memory overhead like a modern video game for instance. From experience in experimentation, Kyoto is able to perform well with a single operating system thread which spawns a _multithreaded_ `tokio` runtime. The runtime must be multithreaded as Kyoto internally uses `tokio::task::spawn`, however, there is no limitation in first using `std::thread::spawn` and building a `tokio` multithreaded runtime within that handle.
+
+### Hosting a Full Archival Node
+
+From an analysis of a `peers.dat`, roughly 20% of nodes signaled for compact block filters support in their service flags. Oftentimes, Kyoto may churn through peers gossiped on the peer to peer network, and this may lead for a bad user experience. Seeded peers help Kyoto sync faster, but some seeds are better than others. Because compact block filters are a database index, they may be stored on an SSD, HD, or external drive. To host a node for end users, it is strongly recommended to use a machine with an SSD.
+
+# Usage Statistics
+
+Leveraging the [UniFFI](https://github.com/mozilla/uniffi-rs) project, as well as [Bitcoin Dev Kit's FFI project](https://github.com/bitcoindevkit/bdk-ffi), a Swift package was built and ran on an iPhone 15. The application recovered a wallet from block height 800,000 to roughly block height 860,000. The remote node stores compact filters on a hard disk, causing the filter retrieval to be slower than usual.
+
+#### Description
+
+The wallet required 12 block downloads, and took 5 minutes.
+
+#### Network
+
+- Received 1.1 GB of data from the remote node
+- Sent 34.4 KB
+- Averaged 5 MB/s downloads
+
+#### Energy Impact
+
+- Very high
+- 32.9% "overhead"
+- 30.4% CPU
+- 36.5% Network
+
+#### Memory
+
+- Maximum of 37.7 MB
+- Average of 35.5 MB
+
+#### CPU
+
+- Single CPU: 72% usage. Highest 100% during block download.
+
+# Implementation
+
+This section details what behavior to expect when using Kyoto, and why such decisions were made.
+
+### Peer Selection
+
+Kyoto will first connect to all of the configured peers to maintain the connection requirement, and will use peers gleaned from the peer-to-peer gossip thereafter. If no peers are configured when building the node, and no peers are in the database, Kyoto will resort to DNS. Kyoto will not select a peer of the same netgroup (/16) as a previously connected peer.
+
+### Block Headers and Storage
+
+Kyoto expects users to adopt some form of persistence between sessions when it comes to block header data. Reason being, Kyoto emits block headers that have been reorganized back to the client in such an event. To do so, in a rare but potential circumstance where the client has shut down on a stale tip, one that is reorganized in the future, Kyoto may use the header persistence to load the older chain into memory. Further, this allows the memory footprint of storing headers in a chain structure to remain small. Kyoto has a soft limit of 20,000 headers in memory at any given time, and if the chain representation exceeds that, Kyoto has a reliable backend to move the excess of block headers. To compensate for this, Kyoto only expects some generic datastore, and does not care about how persistence is implemented.
+
+### Filters
+
+Block filters for a full block may be 300-400 bytes, and may be needless overhead if scripts are revealed "into the future" for the underlying wallet. Full filters are checked for matches as they are downloaded, but are discarded thereafter. As a result, if the user adds scripts that are believe to be included in blocks _in the past_, Kyoto will have to redownload the filters. But if the wallet has up to date information, a revealing a new script is guaranteed to have not been used. This memory tradeoff was deemed worthwhile, as it is expected rescanning will only occur for recovery scenarios.
+
+### Structure
+
+![Layout](https://github.com/user-attachments/assets/21280bb4-aa88-4e11-9223-aed35a885e99)

doc/NASHVILLE24.md

@@ -0,0 +1,69 @@
+---
+theme:
+  name: terminal-light
+---
+
+# Road to compact filters
+
+- `rust-bitcoin` provides the robust types to parse P2P messages
+- `bdk_chain` has nice methods on `IndexedTxGraph` like `apply_block_relevant`
+- Missing link is a library to find and maintain TCP connections, interpret P2P messages, and relay relevant blocks to BDK
+- My project `https://github.com/rustaceanrob/kyoto` attempts to fill the gaps
+
+<!-- end_slide -->
+
+# Miscellaneous Details
+
+- Configurable with known peers or may be completely bootstrapped with DNS
+- Supports arbitrary number of outbound connections (although finding CBF peers can be challenging)
+- Scripts may be added up front when building or as the node is running
+- Calling `Node::run` will run the node continuously while the application is running
+  - Usable on the "client side" or an always-on "server side" application
+- `rust-bitcoin` handles serialization and deserialization of P2P messages, and `tokio` enables flexible concurrency
+
+```toml
+[dependencies]
+bitcoin_hashes = "0.14.0"
+bitcoin = { version = "0.32.0", features = [
+    "serde",
+    "rand-std",
+], default-features = false }
+tokio = { version = "1", default-features = false, features = [
+    "rt-multi-thread",
+    "sync",
+    "time",
+    "io-util",
+    "net",
+    "macros",
+] }
+
+# Optional dependencies
+rusqlite = { version = "0.31.0", features = ["bundled"], optional = true }
+```
+
+<!-- end_slide -->
+
+# Swift example
+
+- Called when the application starts
+
+```swift
+let spv = buildLightClient(wallet: wallet,
+    peers: peers,
+    connections: 1,
+    recoveryHeight: 170_000,
+    dataDir: path,
+    logger: messageHandler)
+let node = spv.node
+let client = spv.client
+runNode(node: node)
+Task {
+    while true {
+        let update = await client.update()
+        if update != nil {
+            try! wallet.applyUpdate(update: update!)
+            balance = wallet.balance().total.toSat()
+        }
+    }
+}
+```