Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: only in neighborhood replication #2237

Merged
merged 3 commits into from
Jul 6, 2021
Merged

fix: only in neighborhood replication #2237

merged 3 commits into from
Jul 6, 2021

Conversation

metacertain
Copy link
Member

@metacertain metacertain commented Jun 29, 2021

This PR intends to improve pushsync behaviour by changing returned ErrWantSelf return ErrNoPush if chunk is not in our neighborhood (to avoid doing a failing replication and setting it as synced) from pushToClosest


This change is Reviewable

Copy link
Member

@acud acud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion (waiting on @metacertain)


pkg/pushsync/pushsync.go, line 302 at r1 (raw file):

				}

				if ps.topologyDriver.IsWithinDepth(ch.Address()) {

it would be better to reduce nesting by checking if !WithinDepth() { return ErrNoPush }, also, the extra error is not needed, and existing package-level errors could be reused.

@metacertain metacertain requested a review from acud June 29, 2021 17:10
Copy link
Member

@zelig zelig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r3, 1 of 1 files at r4.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @acud)

@metacertain metacertain added the ready for review The PR is ready to be reviewed label Jun 30, 2021
Copy link
Contributor

@mrekucci mrekucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r3, 1 of 1 files at r4.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @acud)

Copy link
Member

@acud acud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 1 files at r1, 2 of 2 files at r3, 1 of 1 files at r4.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @metacertain)


pkg/pushsync/pushsync.go, line 369 at r4 (raw file):

				// if the node has warmed up AND no other closer peer has been tried
				if time.Now().After(ps.warmupPeriod) && !ps.skipList.HasChunk(ch.Address()) && ps.topologyDriver.IsWithinDepth(ch.Address()) {

this change is outside of the scope of this PR, since it changes the skip list behavior. It would be at least nice to document this change in the PR description, about why it is needed and what it affects.


pkg/topology/mock/mock.go, line 167 at r4 (raw file):

		return m.isWithinFunc(addr)
	}
	return true

this is changing the test setup for another test that uses this mock and used to rely on a false here. how is it that this needs to change?

@metacertain metacertain force-pushed the push_repl branch 3 times, most recently from efa8966 to c4edf73 Compare July 5, 2021 12:57
@@ -164,7 +164,7 @@ func (m *mock) IsWithinDepth(addr swarm.Address) bool {
if m.isWithinFunc != nil {
return m.isWithinFunc(addr)
}
return false
return true
Copy link
Member

@istae istae Jul 6, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@acud The reason why the default is now true is because the chunk must be in the neighborhood of the storer peer with this change. The majority of the tests expect a valid receipt so the closest peers in the tests must be in the neighborhood. As of writing this, there is only one test that expects a peer to be out of depth.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In short, the previous default false causes every test to fail.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@istae then can we have an explicit change to the tests, in which a custom mocked IsWithinFunc is injected with a true? my concern is that with this change, also forwarder nodes (and there are forwarding tests) will see the chunk as within depth, leading to a change in the test cases' final state

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is also counter-intuitive that we would now test everything with the following assumptions by default:

  • originator node sees all chunks as within neighborhood
  • fowarder node sees all chunks as within neighborhood
  • storer node sees all chunks as within neighborhood
  • replicating node sees all chunks as within neighborhood

a bit too permissive IMO. so I would vote for reverting this back to false and have an explicit true injected with a mock wherever needed

@istae istae requested a review from acud July 6, 2021 10:27
@@ -164,7 +164,7 @@ func (m *mock) IsWithinDepth(addr swarm.Address) bool {
if m.isWithinFunc != nil {
return m.isWithinFunc(addr)
}
return false
return true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@istae then can we have an explicit change to the tests, in which a custom mocked IsWithinFunc is injected with a true? my concern is that with this change, also forwarder nodes (and there are forwarding tests) will see the chunk as within depth, leading to a change in the test cases' final state

@@ -164,7 +164,7 @@ func (m *mock) IsWithinDepth(addr swarm.Address) bool {
if m.isWithinFunc != nil {
return m.isWithinFunc(addr)
}
return false
return true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is also counter-intuitive that we would now test everything with the following assumptions by default:

  • originator node sees all chunks as within neighborhood
  • fowarder node sees all chunks as within neighborhood
  • storer node sees all chunks as within neighborhood
  • replicating node sees all chunks as within neighborhood

a bit too permissive IMO. so I would vote for reverting this back to false and have an explicit true injected with a mock wherever needed

@istae istae requested a review from acud July 6, 2021 10:59
Copy link
Member

@acud acud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks @istae 🙏

@istae istae merged commit a1cb4f0 into master Jul 6, 2021
@istae istae deleted the push_repl branch July 6, 2021 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pull-request ready for review The PR is ready to be reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants