Add API to delete spent UTXOs from database #725

vladimirfomene · 2022-08-17T08:41:38Z

Description

We currently store spent UTXOs in the database with an is_spent field and we don't provide users with a way to delete these UTXOs. The issue here is that these could easily bloat the database or memory. This PR provides methods that can allow users to delete spent UTXOs from the database following certain criteria like size of spent utxos and number of confirmations.
This PR fixes issue #573

Notes to the reviewers

I haven't added this change to the change log and written test for it. I'm thinking we can use this PR as a starting point to discuss how best we can solve this issue. I think having an API that users can call manually is a great start towards solving this issue but it might also be necessary to implement an automatic version of this.

Checklists

All Submissions:

I've signed all my commits
I followed the contribution guidelines
I ran cargo fmt and cargo clippy before committing

New Features:

I've added tests for the new feature
I've added docs for the new feature
I've updated CHANGELOG.md

rajarshimaitra

Concept ACK on we should have a utxo deleting API..

Below are few comments over the API and some internal codes..

src/database/keyvalue.rs

src/database/memory.rs

src/database/mod.rs

rajarshimaitra

Went over the code, and here are some more thoughts and comments.. Sorry for being too verbose on approach.. But I think we can do some cool stuffs here..

It seems to me the the basic operations of getting and deleting an utxos are already part of the Database trait.. So heres a wild thought.. Why not have a default deleting utxo implementation in Database trait itself??

I am not sure if its possible, but have you tried that route?? Could be really interesting and users won't have to define it for their custom type anymore if it can be default implemented.. Will also help us concentrate the deletion logic in just one place in the whole codebase, and not have codes for each impls..

It will work iff, all the basic method calls you need to make for the selection and deletion are part of the Database trait..

I also feel the delete_spent_utxo_by_criteria() should use the delete_spent_utxo() call internally and use that function to delete selected utxo upon given criteria.. Should not need its own separate code to make delete calls.. Only needs to pass on a correct vec of utxos to delete..

The two criteria condition confirmation and threshold_size should be asserted together.. For example
if confirmation = 5, threshold_size = 10, after deleting the database should have <10 utxos, all with confirmation<5.. I am not very if thats what happening currently..

I feel this will separate the logic a lot, and reuse it where possible.. And code will be much shorter in that way.. And if we can actually do the default way, that would be the best thing to do in terms of maintainability..

To verify that our DelCriteria is working as expected we should also include a test for it too..

rajarshimaitra · 2022-08-24T15:30:14Z

src/database/keyvalue.rs

+ //delete all spent utxos
+ return self.del_spent_utxos(None);


How about deleting only the excess utxos sorted by confirmations??

Will deleting just the excess not mean they will have to do frequent cleanups?

src/database/memory.rs

src/database/keyvalue.rs

vladimirfomene · 2022-09-01T18:05:46Z

Thanks @rajarshimaitra for the great feedback! I have pushed some changes with the following:

Made the code more ergonomic.
Implemented the functionality as defaults methods on the Database trait.
Use del_spent_utxos in delete_spent_utxos_by_criteria.

What is left:
I have not added test because I believe a lot will still change on the current implementation. I have also not amended the code so that the two criteria can work in conjunction.

rajarshimaitra

Review ACK c1cae75

Some nits and comments..

rajarshimaitra · 2022-09-03T13:05:14Z

src/database/mod.rs

+ /// UTXOs in the database.
+ fn del_spent_utxos(
+ &mut self,
+ spent_outpoints: Option<Vec<OutPoint>>,


probably better to name it something like to_delete? All utxos in this context are "spent".

@rajarshimaitra don't you think it is more descriptive to keep del_spent_utxos?

I think he means using to_delete instead of spent_outpoints

Yes that's what I meant..

rajarshimaitra · 2022-09-03T13:07:37Z

src/database/mod.rs

+ /// Delete UTXOs provided as params, if `spent_utxos` is empty it deletes all spent
+ /// UTXOs in the database.


Something like this can sound clearer?

Suggested change

/// Delete UTXOs provided as params, if `spent_utxos` is empty it deletes all spent

/// UTXOs in the database.

/// Delete a list of spent utxos from database. Delete all spent utxos if list is `None`.

Sure, will make the change

rajarshimaitra · 2022-09-03T13:30:54Z

src/database/mod.rs

+ if let Some(threshold_size) = criteria.threshold_size {
+ // if the threshold size is smaller than current size, delete everything
+ if threshold_size < spent_utxos.len() as u64 {
+ //delete all spent utxos
+ return self.del_spent_utxos(None);
+ }
+ }
+
+ let mut filtered_outpoints: Vec<OutPoint> = vec![];
+
+ // get utxos that matches criteria and delete them
+ match self.get_sync_time()? {
+ Some(sync_time) => {
+ for spent_utxo in spent_utxos {
+ let tx_details = self.get_tx(&spent_utxo.outpoint.txid, false)?;
+ let height = tx_details
+ .and_then(|details| details.confirmation_time)
+ .map(|conf| conf.height);
+ let confirmations = criteria.confirmations;
+ if height.is_some()
+ && confirmations.is_some()
+ && ((sync_time.block_time.height - height.unwrap())
+ > confirmations.unwrap())
+ {
+ filtered_outpoints.push(spent_utxo.outpoint);
+ }
+ }
+ Ok(self.del_spent_utxos(Some(filtered_outpoints))?)
+ }
+ None => Ok(vec![]),
+ }
+ }


I am not much sure on the criteria part.. If the wallet dev wants to delete utxos upon some criteria they should be able to do that on their own in their code. And such criteria in practice can be very diverse. It doesn't help for us to anticipate them and provide defaults. More options at such low level database API is more confusing..

But if it has to stay, I would expect a behavior from the structure something like below..

Both the criteria are defined as options. So they are both free to be set independent of each other. So when both is set I would expect an and of them, and when one is None it should not have any effect.

So there code here should do something like this, instead of deleting per each criteria.

Choose all utxos to delete by confirmation criteria.

Apply threshold criteria on remaining utxos and get how many more to delete, if any. If some, select them by decreasing confirmation time from utxo list.

Compile both of the two list together.

Delete those utxos..

danielabrozzoni

Concept ACK, but this definitely needs some tests, showing how to delete by confirmations and/or by threshold size, and checking that the deletions are correct. This is also useful for reviewers to check if the code is correct and/or if the API itself is easy to understand

I have not added test because I believe a lot will still change on the current implementation.

You're probably right, on the other hand we don't want to merge something broken, and tests are the easiest way to find out if it's all ok

danielabrozzoni · 2022-09-10T12:54:11Z

src/database/mod.rs

+ // if the threshold size is smaller than current size, delete everything
+ if threshold_size < spent_utxos.len() as u64 {
+ //delete all spent utxos
+ return self.del_spent_utxos(None);


I agree with what Raj is saying below: you can't really delete all the utxos if there are more than threshold_size, you probably just want to delete the oldest ones (=first spent ones)

Okay, I will go with Raj's suggestion

src/database/mod.rs

vladimirfomene · 2022-09-24T10:27:01Z

@rajarshimaitra and @danielabrozzoni , I have amended the PR based on your recommendations. I have also added tests for the methods.

danielabrozzoni · 2022-09-26T12:18:14Z

@vladimirfomene did you also fix #725 (comment) ? Looking at the code, it seems to me it hasn't been

vladimirfomene · 2022-09-27T12:56:17Z

Yes I did

danielabrozzoni · 2022-09-28T10:44:10Z

src/database/mod.rs

+ // call del_spent_utxos with None
+ let mut res = db.del_spent_utxos(None).unwrap();
+ res.sort_by(|a, b| a.txout.value.cmp(&b.txout.value));
+ // verify that the one spent utxo has been deleted.


nit: "verify that the spent ones has been deleted" is slightly easier to understand

danielabrozzoni · 2022-09-28T10:45:47Z

src/database/mod.rs

+ .del_spent_utxos(Some(vec![
+ first_utxo.outpoint,
+ third_utxo.outpoint,
+ fourth_utxo.outpoint,


Maybe here you should call del_spent_utxos with just 2 of the 3 utxos that have been spent, so that we can check that del_spent_utxos(Some(v)) deletes all the utxos in v and not all the spent ones

danielabrozzoni · 2022-09-28T10:55:07Z

src/database/mod.rs

+ if let Some(to_delete) = to_delete {
+ let deleted_utxos = to_delete
+ .iter()
+ .filter_map(|out| self.del_utxo(out).transpose())


I think we should also check that the to_delete txouts are effectively spent, otherwise we might have users accidentally deleting unspent txouts and then having problems with sync/balance

danielabrozzoni · 2022-09-28T10:58:00Z

src/database/mod.rs

+ let utxos_heights: Vec<(&LocalUtxo, &u32)> =
+ spent_utxos.iter().zip(txs_conf_heights.iter()).collect();


Can you really zip here? I think there are cases where txs_conf_heights is shorter than spent_utxos:

if self.get_tx(&utxo.outpoint.txid, false).transpose() is None

if details.confirmation_time.as_ref().map(|conf| conf.height) is None

danielabrozzoni · 2022-09-28T10:58:34Z

src/database/mod.rs

+ let mut failed_confirmation_criteria: Vec<(&LocalUtxo, &u32)> = vec![];
+
+ // Choose all utxos to delete by confirmation criteria.
+ if criteria.confirmations.is_some() {


nit: this could be an if let

danielabrozzoni · 2022-09-28T11:08:31Z

src/database/mod.rs

+ }
+
+ // only select on threshold if there are spent utxos to be deleted
+ if spent_utxos.len() - to_delete.len() > 0 {


Use spent_utxos.len() > to_delete.len() instead: if spent_utxos.len() - to_delete.len() is less than zero, I think Rust would panic (subtraction underflow)

danielabrozzoni · 2022-09-28T11:08:46Z

src/database/mod.rs

+ }
+
+ // apply threshold criteria on spent utxos
+ if criteria.threshold_size.is_some() {


same: I would use if let, so you don't have to unwrap below

danielabrozzoni · 2022-09-28T11:09:56Z

src/database/mod.rs

+ let qty_to_delete = spent_utxos.len() as u64
+ - criteria.threshold_size.unwrap()
+ - to_delete.len() as u64;


I think this could underflow as well... you might want to use https://doc.rust-lang.org/std/primitive.u32.html#method.saturating_sub to avoid problems

let qty_to_delete = spent_utxos.len() as u64 - to_delete.len() as u64 .saturanting_sub(criteria.threshold_size.unwrap());

(I didn't compile this code, it probably needs some more parenthesis to work)

danielabrozzoni · 2022-09-28T11:15:40Z

src/database/mod.rs

+ // split utxos_heights pair based on confirmation criteria
+ let (pass, fail): (Vec<_>, Vec<_>) =
+ utxos_heights.iter().partition(|(_utxo, height)| {
+ (current_block_height.unwrap() - **height) >= criteria.confirmations.unwrap()


I would use saturating_sub for current_block_height.unwrap() - **height, to avoid panics

We currently store spent UTXOs in the database with an `is_spent` field and we don't provide users with a way to delete these UTXOs. The issue here is that these could easily bloat the database or memory. This PR provides methods that can allow users to delete spent UTXOs from the database. This PR fixes issue bitcoindevkit#573 add verification

vladimirfomene · 2022-09-28T20:15:44Z

I have removed the del_spent_utxos_by_criteria method and its test. We decided to remove the del_spent_utxos_by_criteria because we don't currently store data that tells us how many blocks ago a utxo was spent. We only store data that tells us how many blocks ago, a utxo was received. This makes it hard to calculate the number of blocks passed since a utxo was spent and therefore difficult to use that as a criteria for deletion.

This commit adds test for `del_spent_utxos`.

notmandatory · 2022-09-29T17:22:31Z

Hey sorry but I need to take this out of the 0.23 release, and there's a chance that with the up-coming bdk_core work we may not need it. But this PR is a still a good bit of work so let's leave it open until we have more info.

rajarshimaitra · 2022-10-15T07:49:33Z

My 2 sats would be to get it into bdk anyway even if we might not need it after bdk_core integration.. A full featured bdk_core integration would take a bit of time, and this can be useful feature to add for wallet devs anyway who might wanna stay with the older versions of bdk..

danielabrozzoni · 2022-10-17T16:16:39Z

In my opinion getting this in might cause more disrupt than help:

It takes resources away from bdk-core (I can't work on bdk_core and review on bdk at the same time :))
We don't have proof that people need this right now
I agree that it might be useful to get this one in for the ones that want to stay on bdk 0.something instead of switching to 1.0, but IMHO we should wait for the bdk_core integration to be almost ready before deciding this

danielabrozzoni · 2023-03-16T17:15:59Z

Hey, we are in the process of releasing BDK 1.0, which will under the hood work quite differently from the current BDK. For this reason, I'm closing all the PRs that don't really apply anymore. If you think this is a mistake, feel free to rebase on master and re-open!

vladimirfomene force-pushed the manually-deleting-spent-utxos branch from 7bc3c0e to 4585ee6 Compare August 17, 2022 11:17

rajarshimaitra reviewed Aug 19, 2022

View reviewed changes

src/database/keyvalue.rs Outdated Show resolved Hide resolved

src/database/memory.rs Outdated Show resolved Hide resolved

src/database/memory.rs Outdated Show resolved Hide resolved

src/database/mod.rs Outdated Show resolved Hide resolved

src/database/mod.rs Outdated Show resolved Hide resolved

vladimirfomene force-pushed the manually-deleting-spent-utxos branch 3 times, most recently from 19fa758 to 0fd7110 Compare August 24, 2022 12:12

rajarshimaitra reviewed Aug 24, 2022

View reviewed changes

src/database/keyvalue.rs Outdated Show resolved Hide resolved

notmandatory assigned vladimirfomene Aug 25, 2022

vladimirfomene force-pushed the manually-deleting-spent-utxos branch from 5135064 to 3850be6 Compare September 1, 2022 17:55

vladimirfomene force-pushed the manually-deleting-spent-utxos branch from 3850be6 to c1cae75 Compare September 1, 2022 18:43

rajarshimaitra reviewed Sep 3, 2022

View reviewed changes

danielabrozzoni added this to the Release 0.23.0 Feature Freeze milestone Sep 5, 2022

danielabrozzoni reviewed Sep 10, 2022

View reviewed changes

notmandatory mentioned this pull request Sep 13, 2022

W37 BDK Library Team Call bitcoindevkit/.github#22

Closed

26 tasks

vladimirfomene force-pushed the manually-deleting-spent-utxos branch 3 times, most recently from 952fc77 to c957755 Compare September 24, 2022 10:23

notmandatory mentioned this pull request Sep 26, 2022

W39 BDK Library Team Call bitcoindevkit/.github#24

Closed

30 tasks

danielabrozzoni reviewed Sep 28, 2022

View reviewed changes

vladimirfomene force-pushed the manually-deleting-spent-utxos branch from c957755 to c1d1add Compare September 28, 2022 19:55

Test delete spent utxos functionality

3e4b2c9

This commit adds test for `del_spent_utxos`.

vladimirfomene force-pushed the manually-deleting-spent-utxos branch from c1d1add to 3e4b2c9 Compare September 28, 2022 20:24

notmandatory removed this from the Release 0.23.0 Feature Freeze milestone Sep 29, 2022

danielabrozzoni closed this Mar 16, 2023

		/// Delete UTXOs provided as params, if `spent_utxos` is empty it deletes all spent
		/// UTXOs in the database.

	/// Delete UTXOs provided as params, if `spent_utxos` is empty it deletes all spent
	/// UTXOs in the database.
	/// Delete a list of spent utxos from database. Delete all spent utxos if list is `None`.

		let utxos_heights: Vec<(&LocalUtxo, &u32)> =
		spent_utxos.iter().zip(txs_conf_heights.iter()).collect();

Add API to delete spent UTXOs from database #725

Add API to delete spent UTXOs from database #725

Conversation

vladimirfomene commented Aug 17, 2022 • edited Loading

Description

Notes to the reviewers

Checklists

All Submissions:

New Features:

rajarshimaitra left a comment

Choose a reason for hiding this comment

rajarshimaitra left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirfomene commented Sep 1, 2022 • edited Loading

rajarshimaitra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajarshimaitra Sep 3, 2022 • edited Loading

Choose a reason for hiding this comment

danielabrozzoni left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirfomene commented Sep 24, 2022

danielabrozzoni commented Sep 26, 2022

vladimirfomene commented Sep 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladimirfomene commented Sep 28, 2022

notmandatory commented Sep 29, 2022 • edited Loading

rajarshimaitra commented Oct 15, 2022 • edited Loading

danielabrozzoni commented Oct 17, 2022

danielabrozzoni commented Mar 16, 2023

vladimirfomene commented Aug 17, 2022 •

edited

Loading

rajarshimaitra left a comment •

edited

Loading

vladimirfomene commented Sep 1, 2022 •

edited

Loading

rajarshimaitra Sep 3, 2022 •

edited

Loading

danielabrozzoni left a comment •

edited

Loading

notmandatory commented Sep 29, 2022 •

edited

Loading

rajarshimaitra commented Oct 15, 2022 •

edited

Loading