Skip to content
This repository has been archived by the owner on Dec 8, 2021. It is now read-only.

backend/local: check and return iter.Error when pebble is not valid #497

Merged
merged 8 commits into from
Dec 8, 2020

Conversation

glorv
Copy link
Contributor

@glorv glorv commented Nov 30, 2020

What problem does this PR solve?

Check pebble iter.Error when iter is not valid so if iter returns an error (such as caused by exceeding max open file limit), lightning won't omit the actual error

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test

Side effects

Related changes

Copy link
Collaborator

@kennytm kennytm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

lightning/backend/local.go Outdated Show resolved Hide resolved
lightning/backend/local.go Outdated Show resolved Hide resolved
lightning/backend/local.go Outdated Show resolved Hide resolved
@kennytm
Copy link
Collaborator

kennytm commented Nov 30, 2020

LGTM

@kennytm
Copy link
Collaborator

kennytm commented Nov 30, 2020

/test

[2020-11-30T10:22:17.595Z] 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed connect to fileserver.pingcap.net:80; Connection refused

@kennytm kennytm added the status/LGT1 One reviewer already commented LGTM (LGTM1) label Nov 30, 2020
@glorv
Copy link
Contributor Author

glorv commented Nov 30, 2020

/run-all-tests

1 similar comment
@glorv
Copy link
Contributor Author

glorv commented Nov 30, 2020

/run-all-tests


if !iter.First() {
if !iter.Valid() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about iter.Last() at line 516?

@@ -851,6 +863,9 @@ func (local *local) writeAndIngestByRange(
defer iter.Close()
// Needs seek to first because NewIter returns an iterator that is unpositioned
hasKey := iter.First()
if iter.Error() != nil {
return errors.Annotate(iter.Error(), "failed to read the first key")
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about iter.Last() at line 877?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about this comment? @glorv

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to check Valid() after iter.Last(), as we did for iter.First()?

Copy link
Contributor Author

@glorv glorv Dec 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, see #880-883 @lance6716

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean iter.Error() here could cover iter.Valid()? For iter.First() why we need them both (maybe iterator will keep valid after normal operation on iterator 🤔 )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we only need to check iter.Error() != nil which indicates that something goes wrong. iter.Error() != nil always means iter.Valid() is false. if iter.Error() == nil and iter.Valid() is false, this means the iterator iters to the end, this is ok to our logic.

@glorv
Copy link
Contributor Author

glorv commented Dec 1, 2020

/run-all-tests

@overvenus
Copy link
Member

@glorv Please make sure this PR adds all necessary error checks for pebble usage.

@glorv
Copy link
Contributor Author

glorv commented Dec 1, 2020

@glorv Please make sure this PR adds all necessary error checks for pebble usage.

I have checked all the pebble iterators, there should be no remain.

@lance6716
Copy link
Contributor

LGTM

@lance6716 lance6716 added status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2) and removed status/LGT1 One reviewer already commented LGTM (LGTM1) labels Dec 8, 2020
@glorv glorv merged commit e363572 into master Dec 8, 2020
@glorv glorv deleted the iter-error branch December 8, 2020 09:13
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status/LGT2 Two reviewers already commented LGTM, ready for merge (LGTM2)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants