Fix last seen unconfirmed #1385

ValuedMammal · 2024-03-22T17:45:25Z

A method update_last_seen_unconfirmed is added for TxGraph that allows inserting a seen_at time for all unanchored transactions.

fixes #1336

Changelog notice

Changed:

Methods into_tx_graph and into_confirmation_time_tx_graphfor RelevantTxids are changed to no longer accept a seen_at parameter.

Added:

Added method update_last_seen_unconfirmed for TxGraph.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

notmandatory · 2024-03-22T18:14:39Z

This looks fine to me, but my understanding is that a time value always should be provided, even if it's made up as long the the values are increasing. If we make this optional we should at least explain what the downside is of using None, which I believe was unneeded DB updates.

ValuedMammal · 2024-03-22T18:30:04Z

This looks fine to me, but my understanding is that a time value always should be provided, even if it's made up as long the the values are increasing. If we make this optional we should at least explain what the downside is of using None, which I believe was unneeded DB updates.

Ok, I think I agree it should be required.

rustaceanrob · 2024-03-22T18:39:45Z

I agree on required. It should also make for a better UX as the end user "refreshes" their wallet.

evanlinjin · 2024-03-23T10:48:44Z

Sorry about the conflicts! We have merged #1171

ValuedMammal · 2024-03-24T21:45:06Z

electrum: into_tx_graph for RelevantTxids takes an optional seen_at, that could be why I was undecided before whether it should always be provided.

@evanlinjin Can we assume in electrum_ext if a relevant txid has no associated anchors, it must be unconfirmed and therefore be an appropriate time to insert_seen_at?

evanlinjin · 2024-03-25T02:41:37Z

@ValuedMammal that sounds right to me! Let's add a couple of tests and I'll be happy to ACK and merge.

My proposed tests for both ElectrumExt and EsploraExt: When syncing, ensure that transactions that stay unconfirmed between rounds have their last_seen increased and transactions that stay confirmed have unchanging last_seens.

ValuedMammal · 2024-03-25T20:15:53Z

Update on test coverage:

For electrum, repeated syncing doesn't update the last seen timestamp, because if a tx is already present in graph, TxGraph won't report it as missing.

For esplora I did successfully check that timestamps increase for repeat syncs as long as a tx is unconfirmed. When the tx confirms, the timestamp is unchanged.

Note: I don't love the fact that every test needs to mine 100 regtest blocks, as that slows down CI. We should perhaps think about consolidating various assertions into more monolithic tests.

LLFourn

The changes here look exactly how I imagined they should. Thanks! But now that I see it, isn't it simpler just to have method like update_last_seen_unconfirmed on TxGraph which sets the last seen of all unanchored transactions? We can just call this on the update TxGraph and set it to now before applying it. This means we wouldn't need to pass it in, we can just mutate the TxGraph afterwards.

At least presently it doesn't look like we lose anything by doing this since the backends are both just setting whatever you tell them when you pass the time in. I was thinking that maybe esplora or electrum would tell you the last time it was seen in the API but I guess that's not the case. Mempool.space does have this api: https://mempool.space/docs/api/rest#get-transaction-times but that's "first seen" rather than last seen.

crates/electrum/tests/test_electrum.rs

ValuedMammal · 2024-03-27T23:47:13Z

The changes here look exactly how I imagined they should. Thanks! But now that I see it, isn't it simpler just to have method like update_last_seen_unconfirmed on TxGraph which sets the last seen of all unanchored transactions? We can just call this on the update TxGraph and set it to now before applying it. This means we wouldn't need to pass it in, we can just mutate the TxGraph afterwards.

I'll have another look at this.

ValuedMammal · 2024-03-30T16:02:21Z

Updated the PR description to reflect the approach using TxGraph, so the above comments are mostly outdated. This way has the advantage of not breaking the current API for electrum/esplora.

notmandatory · 2024-03-31T01:19:47Z

crates/electrum/src/electrum_ext.rs

-            if let Some(seen_at) = seen_at {
-                let _ = graph.insert_seen_at(txid, seen_at);
+            if anchors.is_empty() {
+                if let Some(seen_at) = seen_at {


❓ I know it breaks the current API but shouldn't seen_at be required and non-optional here too?

I don't know whether it makes more sense to make this seen_at parameter required, or to remove the param and use the new update_last_seen_unconfirmed method on the update TxGraph.

I think it's fine as it is. Will likely be radically changed soon.

[EDIT] but when you did:

l let mut graph_update = relevant_txids.into_confirmation_time_tx_graph(&client, None, missing)?; let now = std::time::UNIX_EPOCH.elapsed().unwrap().as_secs(); let _ = graph_update.update_last_seen_unconfirmed(now);

You are choosing to pass in None but then manually update it. If this is what you want to do in the examples then just remove the option from these functions. It's confusing to use both in the examples.

Fixed. I made seen_at a required parameter for into_tx_graph.

LLFourn

LGTM just need to make the electrum examples consistent (decide whether to pass it into the into_confirmation_time_tx_graph or just remove the Option and do it manually outside).

evanlinjin

This looks great! However, there may be a situation where the chain-source returns anchors that do not point to a block in the best chain. Those unconfirmed transactions will therefore have no "timestamp". Would it be better if TxGraph::update_last_seen_unconfirmed takes in a checkpoint tip so unconfirmed transactions are transactions that do not have an anchor that points to a checkpoint?

ValuedMammal · 2024-04-02T14:38:02Z

This looks great! However, there may be a situation where the chain-source returns anchors that do not point to a block in the best chain. Those unconfirmed transactions will therefore have no "timestamp". Would it be better if TxGraph::update_last_seen_unconfirmed takes in a checkpoint tip so unconfirmed transactions are transactions that do not have an anchor that points to a checkpoint?

I had not thought of that

LLFourn · 2024-04-03T00:41:59Z

This looks great! However, there may be a situation where the chain-source returns anchors that do not point to a block in the best chain. Those unconfirmed transactions will therefore have no "timestamp". Would it be better if TxGraph::update_last_seen_unconfirmed takes in a checkpoint tip so unconfirmed transactions are transactions that do not have an anchor that points to a checkpoint?

last_seen_unconfirmed is for transactions that we've actually seen as unconfirmed in a fully validating mempool. Why would we set it for transactions that we've only seen as confirmed? Or are you saying esplora will allow us to determine that a tx is both in the mempool but was also confirmed in some non-canonical block? If that's the case then we have to pass in the timestamp into the sync/scan function so the logic can set both. The suggestion that we add and use TxGraph::update_last_seen_unconfirmed was based on the assumption that esplora and electrum would not allow us to distinguish this case.

evanlinjin · 2024-04-03T02:34:40Z

@LLFourn the problem is we can't get blocks and transactions atomically. We either get blocks before getting transactions, or get transactions before getting blocks.

There might be a situation when a reorg happens between fetching txs and blocks. The tx might be sent back to mempool because it's not included in replacement block.

LLFourn · 2024-04-03T22:56:40Z

@LLFourn the problem is we can't get blocks and transactions atomically. We either get blocks before getting transactions, or get transactions before getting blocks.

There might be a situation when a reorg happens between fetching txs and blocks. The tx might be sent back to mempool because it's not included in replacement block.

...or it might not be if the re-org confirms a double spend. It might be in the mempool it might not be so we don't set last_seen because we don't know. Where's the problem?

LLFourn

ACK b21ea0f

From looking at it, the changes in electrum_ext.rs seem to do exactly the same thing that would be achieved if you just called update_last_seen_unconfirmed so it would make sense to just do that instead of passing in seen_at. I'm pretty sure @evanlinjin will change this soon so I don't think it matters that much.

crates/chain/src/tx_graph.rs

LLFourn

ACK 449dd9a

evanlinjin

LGTM! Thanks for the quality work. Let's drop the last_seen parameter from .into_tx_graph and I'll ACK/merge.

That accepts a `u64` as param representing the latest timestamp and internally calls `insert_seen_at` for all transactions in graph that aren't yet anchored in a confirmed block.

and `into_confirmation_time_tx_graph`, since now it makes sense to use `TxGraph::update_last_seen_unconfirmed`. Also, use `update_last_seen_unconfirmed` in examples for electrum/esplora. We show how to update the last seen time for transactions by calling `update_last_seen_unconfirmed` on the graph update returned from a blockchain source, passing in the current time, before applying it to another `TxGraph`.

ValuedMammal · 2024-04-06T16:30:53Z

LGTM! Thanks for the quality work. Let's drop the last_seen parameter from .into_tx_graph and I'll ACK/merge.

Done. Thank you to the reviewers @evanlinjin @LLFourn @notmandatory @rustaceanrob

evanlinjin

ACK a2a64ff

notmandatory assigned ValuedMammal Mar 22, 2024

notmandatory added module-blockchain api A breaking API change labels Mar 22, 2024

notmandatory added this to the 1.0.0-alpha milestone Mar 22, 2024

ValuedMammal force-pushed the fix/esplora-lastseen branch from ffce704 to 3162957 Compare March 23, 2024 02:05

ValuedMammal force-pushed the fix/esplora-lastseen branch from 3162957 to 84dc09e Compare March 24, 2024 21:38

ValuedMammal changed the title ~~[esplora] Change sync, full_scan to take a timestamp parameter~~ [electrum,esplora] Fix last seen unconfirmed Mar 25, 2024

ValuedMammal force-pushed the fix/esplora-lastseen branch from 5af0f99 to 5928143 Compare March 25, 2024 20:22

LLFourn reviewed Mar 27, 2024

View reviewed changes

crates/electrum/tests/test_electrum.rs Outdated Show resolved Hide resolved

ValuedMammal force-pushed the fix/esplora-lastseen branch from 5928143 to c94f27b Compare March 30, 2024 02:54

ValuedMammal changed the title ~~[electrum,esplora] Fix last seen unconfirmed~~ Fix last seen unconfirmed Mar 30, 2024

notmandatory reviewed Mar 31, 2024

View reviewed changes

LLFourn requested changes Apr 2, 2024

View reviewed changes

evanlinjin reviewed Apr 2, 2024

View reviewed changes

ValuedMammal force-pushed the fix/esplora-lastseen branch from c94f27b to b21ea0f Compare April 2, 2024 14:36

LLFourn approved these changes Apr 3, 2024

View reviewed changes

crates/chain/src/tx_graph.rs Outdated Show resolved Hide resolved

LLFourn approved these changes Apr 4, 2024

View reviewed changes

evanlinjin reviewed Apr 6, 2024

View reviewed changes

ValuedMammal added 2 commits April 6, 2024 11:50

feat(tx_graph): Add method update_last_seen_unconfirmed

37fca35

That accepts a `u64` as param representing the latest timestamp and internally calls `insert_seen_at` for all transactions in graph that aren't yet anchored in a confirmed block.

ValuedMammal force-pushed the fix/esplora-lastseen branch from 449dd9a to a2a64ff Compare April 6, 2024 16:21

evanlinjin approved these changes Apr 11, 2024

View reviewed changes

evanlinjin merged commit 62619d3 into bitcoindevkit:master Apr 11, 2024
12 checks passed

notmandatory mentioned this pull request Apr 12, 2024

Release 1.0.0-alpha.9 #1399

Closed

25 tasks

ValuedMammal deleted the fix/esplora-lastseen branch April 12, 2024 17:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix last seen unconfirmed #1385

Fix last seen unconfirmed #1385

ValuedMammal commented Mar 22, 2024 •

edited

Loading

notmandatory commented Mar 22, 2024

ValuedMammal commented Mar 22, 2024

rustaceanrob commented Mar 22, 2024

evanlinjin commented Mar 23, 2024

ValuedMammal commented Mar 24, 2024

evanlinjin commented Mar 25, 2024

ValuedMammal commented Mar 25, 2024 •

edited

Loading

LLFourn left a comment •

edited

Loading

ValuedMammal commented Mar 27, 2024

ValuedMammal commented Mar 30, 2024

notmandatory Mar 31, 2024

ValuedMammal Mar 31, 2024

LLFourn Apr 2, 2024 •

edited

Loading

ValuedMammal Apr 2, 2024

LLFourn left a comment

evanlinjin left a comment

ValuedMammal commented Apr 2, 2024

LLFourn commented Apr 3, 2024 •

edited

Loading

evanlinjin commented Apr 3, 2024

LLFourn commented Apr 3, 2024

LLFourn left a comment

LLFourn left a comment

evanlinjin left a comment

ValuedMammal commented Apr 6, 2024

evanlinjin left a comment

Fix last seen unconfirmed #1385

Fix last seen unconfirmed #1385

Conversation

ValuedMammal commented Mar 22, 2024 • edited Loading

Changelog notice

Checklists

All Submissions:

notmandatory commented Mar 22, 2024

ValuedMammal commented Mar 22, 2024

rustaceanrob commented Mar 22, 2024

evanlinjin commented Mar 23, 2024

ValuedMammal commented Mar 24, 2024

evanlinjin commented Mar 25, 2024

ValuedMammal commented Mar 25, 2024 • edited Loading

LLFourn left a comment • edited Loading

Choose a reason for hiding this comment

ValuedMammal commented Mar 27, 2024

ValuedMammal commented Mar 30, 2024

notmandatory Mar 31, 2024

Choose a reason for hiding this comment

ValuedMammal Mar 31, 2024

Choose a reason for hiding this comment

LLFourn Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

ValuedMammal Apr 2, 2024

Choose a reason for hiding this comment

LLFourn left a comment

Choose a reason for hiding this comment

evanlinjin left a comment

Choose a reason for hiding this comment

ValuedMammal commented Apr 2, 2024

LLFourn commented Apr 3, 2024 • edited Loading

evanlinjin commented Apr 3, 2024

LLFourn commented Apr 3, 2024

LLFourn left a comment

Choose a reason for hiding this comment

LLFourn left a comment

Choose a reason for hiding this comment

evanlinjin left a comment

Choose a reason for hiding this comment

ValuedMammal commented Apr 6, 2024

evanlinjin left a comment

Choose a reason for hiding this comment

ValuedMammal commented Mar 22, 2024 •

edited

Loading

ValuedMammal commented Mar 25, 2024 •

edited

Loading

LLFourn left a comment •

edited

Loading

LLFourn Apr 2, 2024 •

edited

Loading

LLFourn commented Apr 3, 2024 •

edited

Loading