Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove legacy NodeID #2404

Merged
merged 9 commits into from
Mar 18, 2021
Merged

Remove legacy NodeID #2404

merged 9 commits into from
Mar 18, 2021

Conversation

pav-kv
Copy link
Contributor

@pav-kv pav-kv commented Mar 17, 2021

This change eliminates the last cases of using NodeID, most notably in SubtreeCache,
and removes this type. Now compact.NodeID is used all the way down to storage, but
storage implementations still use []byte slices to address tiles, for backward compatibility
with the existing databases.

Review commit by commit is recommended.

Part of #2378

Checklist

@pav-kv pav-kv requested a review from a team as a code owner March 17, 2021 22:31
@pav-kv pav-kv force-pushed the rm_node_id branch 2 times, most recently from acf7b2f to fcdfb25 Compare March 17, 2021 22:48
@codecov
Copy link

codecov bot commented Mar 17, 2021

Codecov Report

Merging #2404 (9c50c27) into master (6080b90) will decrease coverage by 0.31%.
The diff coverage is 60.29%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2404      +/-   ##
==========================================
- Coverage   65.82%   65.50%   -0.32%     
==========================================
  Files         106      104       -2     
  Lines        7670     7515     -155     
==========================================
- Hits         5049     4923     -126     
+ Misses       2095     2071      -24     
+ Partials      526      521       -5     
Impacted Files Coverage Δ
storage/cloudspanner/tree_storage.go 36.31% <0.00%> (+1.39%) ⬆️
storage/memory/tree_debug.go 0.00% <0.00%> (ø)
storage/memory/tree_storage.go 2.04% <0.00%> (+0.04%) ⬆️
storage/cache/subtree_cache.go 58.53% <60.00%> (-0.33%) ⬇️
storage/mysql/tree_storage.go 53.51% <90.00%> (+0.66%) ⬆️
storage/tree/layout.go 75.00% <100.00%> (+23.14%) ⬆️
storage/tree/suffix.go 87.87% <100.00%> (+0.37%) ⬆️
storage/tree/node_id2.go 71.42% <0.00%> (-5.72%) ⬇️
log/operation_manager.go 87.30% <0.00%> (-1.06%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6080b90...9c50c27. Read the comment docs.

)

// NodeStorage provides an interface for storing and retrieving subtrees.
type NodeStorage interface {
GetSubtree(n tree.NodeID) (*storagepb.SubtreeProto, error)
GetSubtree(prefix []byte) (*storagepb.SubtreeProto, error)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately golang has no immutable byte string type. There could be value in using string instead. OTOH the original NodeID type was not immutable and I suspect it's not worth doing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Within the scope of this PR, the safety remains the same as it was. I might improve it in follow-ups.

storage/cache/subtree_cache.go Outdated Show resolved Hide resolved
storage/cache/subtree_cache.go Outdated Show resolved Hide resolved
// This should never happen - we should've already read all the data we
// need above, in Preload()
glog.Warningf("Unexpectedly reading from within getNodeHash(): %s", n.String())
ret, err := getSubtrees([]tree.NodeID{n})
glog.Warningf("Unexpectedly reading from within getNodeHash(): %x", id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we previously found the above comment to be incorrect. Might demote the log level. This condition is also not tested. Again, I suspect that was true before.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it's correct (or don't see why it's not). Also, I found 0 occurrences of this warning in our logs. I think we should rather return an error here, or don't have this callback in the first place. To be addressed in follow-ups.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it was maps again? I remember we tried to remove it before and it did happen.

storage/cache/subtree_cache.go Outdated Show resolved Hide resolved
storage/cache/subtree_cache.go Outdated Show resolved Hide resolved
storage/cache/subtree_cache.go Outdated Show resolved Hide resolved
storage/cache/subtree_cache.go Show resolved Hide resolved
pav-kv added 9 commits March 18, 2021 09:59
This reduces dependency on NodeID, but breaks SubtreeCache, because it
still uses NodeID. This is fixed in the next commit.
This migrates SubtreeCache from NodeID. Storage implementations still
pass callbacks using NodeID, this is fixed in the next commits.
@pav-kv
Copy link
Contributor Author

pav-kv commented Mar 18, 2021

Left pre-existing coverage/safety things for future addressing. For now I can see that SubtreeCache will almost go away, so fixing small aspects of it seems more work than a straight replacement with 100% test coverage.

@pav-kv pav-kv merged commit a8f7d5d into google:master Mar 18, 2021
@pav-kv pav-kv deleted the rm_node_id branch March 18, 2021 17:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants