Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace storer transactions with leveldb batches #4341

Closed
istae opened this issue Sep 23, 2023 · 3 comments · Fixed by #4555 or #4626
Closed

replace storer transactions with leveldb batches #4341

istae opened this issue Sep 23, 2023 · 3 comments · Fixed by #4555 or #4626
Assignees
Labels

Comments

@istae
Copy link
Member

istae commented Sep 23, 2023

the chunkstore operations do not currently accept the leveldb batches for writes.
this can cause issues especially in areas of the localstore where the chunkstore is used in tandem with other stores that does indeed utilizes batches for write (eg reserve.Put).

the chunkstore transaction implementation is too complex and bug prone.
transactions for sharky especially is not needed, if a db operation fails, the written data and used slot can simply remain as is and do not need to be recovered.
we have compact cmd now which can give the used sharky space back to the node.

also the reverseOP transaction idea for the indexstore should ideally be completely dropped and all writes must be done using the 'leveldb.batch`.

@bee-runner bee-runner bot added the issue label Sep 23, 2023
@istae istae changed the title chunkstore writes (put and delete) should use leveldb batches chunkstore writes (put and delete) should use leveldb batches and remove transactions Sep 27, 2023
@istae istae changed the title chunkstore writes (put and delete) should use leveldb batches and remove transactions replace storer transactions with leveldb batches Sep 27, 2023
@istae
Copy link
Member Author

istae commented Oct 27, 2023

The localstore should ideally looks like this.

The localstore exports core, parent functions like Reserve.Put.
The parent func initiates a Transaction.
Any internal storage calls the parent function makes must utilize the Transaction, and cannot independently make any Writes to the disk directly.

Any two independent parallel writes to the same resource must be mutex locked as a means of data race protection.

The transaction is internally a leveldb.Batch and exposes the storage calls Put, Delete, Get, Commit.
The transaction also exposes the Chunkstore.

Any write ops are accumulated in the batch of the transaction.

Any sharky writes are immediately written to disk.
Sharky releases are kept in memory until the batch is committed.

Here is a summary of the order of operations:
sharky_write -> write to disk, keep write locations in memory
sharky_release -> keep location in memory
batch_write -> write to batch
if batch_commit succeeds -> commit batch to disk, release sharky_release locations in disk
if batch_commit fails, release all sharky_write location from disk, do nothing for sharky_release

@istae
Copy link
Member Author

istae commented Oct 29, 2023

wrapSync should be removed with a multex in chunkstore ( see storer/reserve.go for what is a multex)

@nikipapadatou
Copy link
Collaborator

more discussions are needed between bee team engineers to proceed with this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment