[FAB-17795] Build channel metadata retroactively if not present at peer start #1369
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cendhu
reviewed
Jun 4, 2020
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_helper.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_helper.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_helper.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_helper.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_helper.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go
Outdated
Show resolved
Hide resolved
wenjianqiao
force-pushed
the
fab17795
branch
2 times, most recently
from
e3c09f6 to
e460c59
Compare
|
@cendhu Thank you for your comments. As discussed in our scrum, I have refactored the code and moved channelMetadata rebuild to newVersionedDB. I have addressed your comments unless the code that has been removed due to refactoring. |
cendhu
reviewed
Jun 6, 2020
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_provider.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_provider.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/statedb/statecouchdb/statecouchdb.go
Outdated
Show resolved
Hide resolved
manish-sethi
reviewed
Jun 8, 2020
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_provider.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_provider.go
Outdated
Show resolved
Hide resolved
core/ledger/kvledger/txmgmt/privacyenabledstate/namespace_provider.go
Outdated
Show resolved
Hide resolved
Channel metadata has been added to statecouchdb to store the mapping of db namespaces to db names. However, the channel metadata is not available in v2.0/v2.1 peers. This PR adds support to retroactively build channel metadata at peer start if such metadata is not present in statecouchdb. When a state CouchDB is opened (GetDBHandle), it retoactively builds channel metadata if the metadata is not present. A NamespaceProvider is implemented to build possible namespaces for the channel. The possible namespaces will be verified by the existing databases and only namespaces matching existing databases will be added to channel metadata. Signed-off-by: Wenjian Qiao <wenjianq@gmail.com>
manish-sethi
approved these changes
Jun 9, 2020
Comment on lines
+433
to
+434
| // channelInfoProvider interface enables the privateenabledstate package to retrieve all the config blocks | ||
| // and namespaces and collections. |
There was a problem hiding this comment.
forgot to update the comments?
I though that at this level, perhaps you would name this a bit more restrictive like nsCollNames or nsCollNamesProvider instead of a generic name channelInfoProvider
Comment on lines
+594
to
+597
| // define this interface to break circular dependency | ||
| type channelInfoProviderWrapper interface { | ||
| channelInfoProvider | ||
| } |
There was a problem hiding this comment.
not sure whether this wrapper is needed, as the actual interface itself is unexported.
| // and namespaces and collections. | ||
| type channelInfoProvider interface { | ||
| // NamespacesAndCollections returns namespaces and collections for the channel. | ||
| NamespacesAndCollections(vdb statedb.VersionedDB) (map[string][]string, error) |
There was a problem hiding this comment.
The earlier division was better where the query executor was still in this package and this function could the queryexector but it's fine for now.
| } | ||
|
|
||
| // test an existing DB with channelMetadata, namespace provider should not be called | ||
| require.NoError(t, vdb.initChannelMetadata(false, fakeNsProvider)) |
There was a problem hiding this comment.
a test with "true" flag could also be added.
xhens
added a commit
to xhens/fabric
that referenced
this pull request
Jun 12, 2020
* Add FullScanIterator to export channel's data present in the CouchDB (hyperledger#1348) * implement FullScanIterator for couchdb This commit implements statedb.FullScanIterator interface in the statecouchdb pkg for getting access to an iterator that can be used for scanning the entire state, including private data hashes for a particular channel FAB-17901 Signed-off-by: senthil <cendhu@gmail.com> * address review comments Signed-off-by: senthil <cendhu@gmail.com> * address review comments Signed-off-by: senthil <cendhu@gmail.com> * FAB-17912 Ch.Part.API: reject joins (hyperledger#1375) Orderer reject joins: 1. when system channel exists, or 2. if channel exists Signed-off-by: Yoav Tock <tock@il.ibm.com> Change-Id: Ia68bcc91fa7d9363f2320ad69439eedb7a7aeab2 * FAB-17967 Updates for new language contributors Makes it easier for new contributors to the documentation to get started, including international language translators. Signed-off-by: Anthony O'Dowd <a_o-dowd@uk.ibm.com> * Update fabric-config dep to v0.0.4 and associated int. test (hyperledger#1378) FAB-17956 #done Signed-off-by: Will Lahti <wtlahti@us.ibm.com> * Update private data tutorial for contract api Signed-off-by: NIKHIL E GUPTA <negupta@us.ibm.com> * [FAB-17795] Build channel metadata retroactively if not present (hyperledger#1369) Channel metadata has been added to statecouchdb to store the mapping of db namespaces to db names. However, the channel metadata is not available in v2.0/v2.1 peers. This PR adds support to retroactively build channel metadata at peer start if such metadata is not present in statecouchdb. When a state CouchDB is opened (GetDBHandle), it retoactively builds channel metadata if the metadata is not present. A NamespaceProvider is implemented to build possible namespaces for the channel. The possible namespaces will be verified by the existing databases and only namespaces matching existing databases will be added to channel metadata. Signed-off-by: Wenjian Qiao <wenjianq@gmail.com> * Regenerate proto to sync the go code The source file name in the generated code is somehow different and hence it shows the difference when running make protos. Signed-off-by: manish <manish.sethi@gmail.com> * Fix the constructor message in complex queries Signed-off-by: zhuzeyu <zhuzeyu0409@gmail.com> * add rlock to IsEmpty() in leveldb wrapper (hyperledger#1388) This commit adds rlock to IsEmpty() to synchronize between Close() and IsEmpty(). Signed-off-by: senthil <cendhu@gmail.com> * Bump Go and Alpine Version Bump Go to 1.14.4. This is also the first Go release to switch to alpine 3.12 with 1.14.3 being the last to support 3.11. Performing this update now will allow us to cross the alpine minor version barrier prior to our LTS release of 2.2 to give us time to test the stability of the release. Signed-off-by: Brett Logan <brett.t.logan@ibm.com> * Bump Go Version in Vagrant Signed-off-by: Brett Logan <brett.t.logan@ibm.com> * FAB-17951 fetch correct node id for validation (hyperledger#1367) * FAB-17951 fetch correct node id for validation Today, when a config update that modifies consenter set (add/remove node), there's validator performing basics checks, including dangerous configs that may result in quorum loss, e.g. removing active node from a network with 2/3 alive nodes. However, this validator always assumes node ID in consenter set to start from 1. This is benign in most cases, as nodes are normally online, and actual node ID doesn't really matter from validation pov. Although, in certain special cases, this is problematic: - we have [2, 3, 4] in consenter set - [2, 3] are alive, and [4] is inactive and subject to remove - if validator assume node starts from 1, then it would incorrectly conclude that [3] out of [1, 2, 3] is to be removed, while [2, 3] are the 2/3 alive nodes. Therefore, it would reject such request - instead, we need to take actual node IDs into account This commit fixes the problem and adds some UTs. IT will be added in a seperate commit. Signed-off-by: Jay Guo <guojiannan1101@gmail.com> * Amend IT to assert active node shrink An integration test is amended to assert that number of active nodes shrinks when a node is offline. Signed-off-by: Jay Guo <guojiannan1101@gmail.com> * Update Pipfile to support localization Signed-off-by: pama-ibm <pama@ibm.com> * add iter.Seek() in leveldb wrapper (hyperledger#1390) This commits add iter.Seek() API in the leveldb wrapper. Signed-off-by: senthil <cendhu@gmail.com> * Fixed a typo (that that) Signed-off-by: Vincent Déniel <denielvincent@gmail.com> Co-authored-by: Senthil Nathan N <cendhu@users.noreply.github.com> Co-authored-by: Yoav Tock <tock@il.ibm.com> Co-authored-by: Anthony O'Dowd <a_o-dowd@uk.ibm.com> Co-authored-by: Will Lahti <wtlahti@us.ibm.com> Co-authored-by: NIKHIL E GUPTA <negupta@us.ibm.com> Co-authored-by: Wenjian Qiao <wenjianq@gmail.com> Co-authored-by: manish <manish.sethi@gmail.com> Co-authored-by: zhuzeyu <zhuzeyu0409@gmail.com> Co-authored-by: Brett Logan <brett.t.logan@ibm.com> Co-authored-by: Jay Guo <guojiannan1101@gmail.com> Co-authored-by: pama-ibm <pama@ibm.com> Co-authored-by: Vincent DENIEL <vincentdnl@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Wenjian Qiao wenjianq@gmail.com
Type of change
Description
Channel metadata has been added to statecouchdb to store the mapping of db namespaces
to db names. However, the channel metadata is not available in v2.0/v2.1 peers. This PR
adds support to retroactively build channel metadata at peer start if such metadata
is not present in statecouchdb.
When a state CouchDB is opened (GetDBHandle), it retroactively builds channel metadata if
the metadata is not present. A NamespaceProvider is implemented to build possible namespaces for the channel. The possible namespaces will be verified by the existing databases and only namespaces matching existing databases will be added to channel metadata.
Additional details
The channel metadata will be used to remove per-channel databases and support checkpoint snapshots (e.g., create snapshot, clean up after bootstrap failure)
Related issues
https://jira.hyperledger.org/browse/FAB-17795
https://jira.hyperledger.org/browse/FAB-17787