[GSF] Syncing Fork (#32)

* Add FullScanIterator to export channel's data present in the CouchDB (hyperledger#1348)

* implement FullScanIterator for couchdb

This commit implements statedb.FullScanIterator interface
in the statecouchdb pkg for getting access to an iterator that
can be used for scanning the entire state, including private
data hashes for a particular channel

FAB-17901

Signed-off-by: senthil <cendhu@gmail.com>

* address review comments

Signed-off-by: senthil <cendhu@gmail.com>

* address review comments

Signed-off-by: senthil <cendhu@gmail.com>

* FAB-17912 Ch.Part.API: reject joins (hyperledger#1375)

Orderer reject joins:
1. when system channel exists, or
2. if channel exists

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: Ia68bcc91fa7d9363f2320ad69439eedb7a7aeab2

* FAB-17967 Updates for new language contributors

Makes it easier for new contributors to the documentation to get started,
including international language translators.

Signed-off-by: Anthony O'Dowd <a_o-dowd@uk.ibm.com>

* Update fabric-config dep to v0.0.4 and associated int. test (hyperledger#1378)

FAB-17956 #done

Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

* Update private data tutorial for contract api

Signed-off-by: NIKHIL E GUPTA <negupta@us.ibm.com>

* [FAB-17795] Build channel metadata retroactively if not present (hyperledger#1369)

Channel metadata has been added to statecouchdb to store the mapping of db namespaces
to db names. However, the channel metadata is not available in v2.0/v2.1 peers. This PR
adds support to retroactively build channel metadata at peer start if such metadata
is not present in statecouchdb.

When a state CouchDB is opened (GetDBHandle), it retoactively builds channel metadata if
the metadata is not present. A NamespaceProvider is implemented to build possible namespaces
for the channel. The possible namespaces will be verified by the existing databases and
only namespaces matching existing databases will be added to channel metadata.

Signed-off-by: Wenjian Qiao <wenjianq@gmail.com>

* Regenerate proto to sync the go code

The source file name in the generated code is somehow different
and hence it shows the difference when running make protos.

Signed-off-by: manish <manish.sethi@gmail.com>

* Fix the constructor message in complex queries

Signed-off-by: zhuzeyu <zhuzeyu0409@gmail.com>

* add rlock to IsEmpty() in leveldb wrapper (hyperledger#1388)

This commit adds rlock to IsEmpty() to synchronize
between Close() and IsEmpty().

Signed-off-by: senthil <cendhu@gmail.com>

* Bump Go and Alpine Version

Bump Go to 1.14.4. This is also the first Go release
to switch to alpine 3.12 with 1.14.3 being the last
to support 3.11. Performing this update now will allow
us to cross the alpine minor version barrier prior
to our LTS release of 2.2 to give us time to test
the stability of the release.

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* Bump Go Version in Vagrant

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* FAB-17951 fetch correct node id for validation (hyperledger#1367)

* FAB-17951 fetch correct node id for validation

Today, when a config update that modifies consenter set (add/remove
node), there's validator performing basics checks, including dangerous
configs that may result in quorum loss, e.g. removing active node from
a network with 2/3 alive nodes.

However, this validator always assumes node ID in consenter set to start
from 1. This is benign in most cases, as nodes are normally online, and
actual node ID doesn't really matter from validation pov. Although, in
certain special cases, this is problematic:
- we have [2, 3, 4] in consenter set
- [2, 3] are alive, and [4] is inactive and subject to remove
- if validator assume node starts from 1, then it would incorrectly conclude
  that [3] out of [1, 2, 3] is to be removed, while [2, 3] are the 2/3 alive
  nodes. Therefore, it would reject such request
- instead, we need to take actual node IDs into account

This commit fixes the problem and adds some UTs. IT will be added in a seperate
commit.

Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

* Amend IT to assert active node shrink

An integration test is amended to assert that number of
active nodes shrinks when a node is offline.

Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

* Update Pipfile to support localization

Signed-off-by: pama-ibm <pama@ibm.com>

* add iter.Seek() in leveldb wrapper (hyperledger#1390)

This commits add iter.Seek() API in the leveldb wrapper.

Signed-off-by: senthil <cendhu@gmail.com>

* Fixed a typo (that that)

Signed-off-by: Vincent Déniel <denielvincent@gmail.com>

Co-authored-by: Senthil Nathan N <cendhu@users.noreply.github.com>
Co-authored-by: Yoav Tock <tock@il.ibm.com>
Co-authored-by: Anthony O'Dowd <a_o-dowd@uk.ibm.com>
Co-authored-by: Will Lahti <wtlahti@us.ibm.com>
Co-authored-by: NIKHIL E GUPTA <negupta@us.ibm.com>
Co-authored-by: Wenjian Qiao <wenjianq@gmail.com>
Co-authored-by: manish <manish.sethi@gmail.com>
Co-authored-by: zhuzeyu <zhuzeyu0409@gmail.com>
Co-authored-by: Brett Logan <brett.t.logan@ibm.com>
Co-authored-by: Jay Guo <guojiannan1101@gmail.com>
Co-authored-by: pama-ibm <pama@ibm.com>
Co-authored-by: Vincent DENIEL <vincentdnl@users.noreply.github.com>

Makefile

@@ -44,7 +44,7 @@
 #   - unit-test - runs the go-test based unit tests
 #   - verify - runs unit tests for only the changed package tree
 
-ALPINE_VER ?= 3.11
+ALPINE_VER ?= 3.12
 BASE_VERSION = 2.2.0
 
 # 3rd party image version
@@ -76,7 +76,7 @@ METADATA_VAR += CommitSHA=$(EXTRA_VERSION)
 METADATA_VAR += BaseDockerLabel=$(BASE_DOCKER_LABEL)
 METADATA_VAR += DockerNamespace=$(DOCKER_NS)
 
-GO_VER = 1.14.1
+GO_VER = 1.14.4
 GO_TAGS ?=
 
 RELEASE_EXES = orderer $(TOOLS_EXES)

ci/azure-pipelines-merge.yml

@@ -11,7 +11,7 @@ pr: none
 variables:
   GOPATH: $(Agent.BuildDirectory)/go
   PATH: $(Agent.BuildDirectory)/go/bin:/usr/local/go/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
-  GOVER: 1.14.1
+  GOVER: 1.14.4
 
 jobs:
   - job: UnitTests

ci/azure-pipelines-release.yml

@@ -11,7 +11,7 @@ variables:
   - name: GOPATH
     value: $(Agent.BuildDirectory)/go
   - name: GOVER
-    value: 1.14.1
+    value: 1.14.4
 
 stages:
   - stage: BuildBinaries

ci/azure-pipelines.yml

@@ -11,7 +11,7 @@ pr:
 variables:
   GOPATH: $(Agent.BuildDirectory)/go
   PATH: $(Agent.BuildDirectory)/go/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin
-  GOVER: 1.14.1
+  GOVER: 1.14.4
 
 jobs:
   - job: VerifyBuild

common/ledger/util/leveldbhelper/leveldb_helper.go

@@ -79,6 +79,8 @@ func (dbInst *DB) Open() {
 
 // IsEmpty returns whether or not a database is empty
 func (dbInst *DB) IsEmpty() (bool, error) {
+	dbInst.mutex.RLock()
+	defer dbInst.mutex.RUnlock()
 	itr := dbInst.db.NewIterator(&goleveldbutil.Range{}, dbInst.readOpts)
 	defer itr.Release()
 	hasItems := itr.Next()

common/ledger/util/leveldbhelper/leveldb_provider.go

@@ -180,7 +180,7 @@ func (h *DBHandle) GetIterator(startKey []byte, endKey []byte) *Iterator {
 		eKey[len(eKey)-1] = lastKeyIndicator
 	}
 	logger.Debugf("Getting iterator for range [%#v] - [%#v]", sKey, eKey)
-	return &Iterator{h.db.GetIterator(sKey, eKey)}
+	return &Iterator{h.dbName, h.db.GetIterator(sKey, eKey)}
 }
 
 // UpdateBatch encloses the details of multiple `updates`
@@ -213,6 +213,7 @@ func (batch *UpdateBatch) Len() int {
 
 // Iterator extends actual leveldb iterator
 type Iterator struct {
+	dbName string
 	iterator.Iterator
 }
 
@@ -221,6 +222,14 @@ func (itr *Iterator) Key() []byte {
 	return retrieveAppKey(itr.Iterator.Key())
 }
 
+// Seek moves the iterator to the first key/value pair
+// whose key is greater than or equal to the given key.
+// It returns whether such pair exist.
+func (itr *Iterator) Seek(key []byte) bool {
+	levelKey := constructLevelKey(itr.dbName, key)
+	return itr.Iterator.Seek(levelKey)
+}
+
 func constructLevelKey(dbName string, key []byte) []byte {
 	return append(append([]byte(dbName), dbNameKeySep...), key...)
 }

common/ledger/util/leveldbhelper/leveldb_provider_test.go

@@ -15,6 +15,7 @@ import (
 
 	"github.com/hyperledger/fabric/common/flogging"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
 func TestMain(m *testing.M) {
@@ -40,17 +41,118 @@ func TestIterator(t *testing.T) {
 		db3.Put([]byte(createTestKey(i)), []byte(createTestValue("db3", i)), false)
 	}
 
-	itr1 := db2.GetIterator([]byte(createTestKey(2)), []byte(createTestKey(4)))
-	defer itr1.Release()
-	checkItrResults(t, itr1, createTestKeys(2, 3), createTestValues("db2", 2, 3))
+	rangeTestCases := []struct {
+		startKey       []byte
+		endKey         []byte
+		expectedKeys   []string
+		expectedValues []string
+	}{
+		{
+			startKey:       []byte(createTestKey(2)),
+			endKey:         []byte(createTestKey(4)),
+			expectedKeys:   createTestKeys(2, 3),
+			expectedValues: createTestValues("db2", 2, 3),
+		},
+		{
+			startKey:       []byte(createTestKey(2)),
+			endKey:         nil,
+			expectedKeys:   createTestKeys(2, 19),
+			expectedValues: createTestValues("db2", 2, 19),
+		},
+		{
+			startKey:       nil,
+			endKey:         nil,
+			expectedKeys:   createTestKeys(0, 19),
+			expectedValues: createTestValues("db2", 0, 19),
+		},
+	}
 
-	itr2 := db2.GetIterator([]byte(createTestKey(2)), nil)
-	defer itr2.Release()
-	checkItrResults(t, itr2, createTestKeys(2, 19), createTestValues("db2", 2, 19))
+	for i, testCase := range rangeTestCases {
+		t.Run(
+			fmt.Sprintf("range testCase %d", i),
+			func(t *testing.T) {
+				itr := db2.GetIterator(testCase.startKey, testCase.endKey)
+				defer itr.Release()
+				checkItrResults(t, itr, testCase.expectedKeys, testCase.expectedValues)
+			},
+		)
+	}
+
+	rangeWithSeekTestCases := []struct {
+		startKey          []byte
+		endKey            []byte
+		seekToKey         []byte
+		itrAtKeyAfterSeek []byte
+		expectedKeys      []string
+		expectedValues    []string
+	}{
+		{
+			startKey:          nil,
+			endKey:            nil,
+			seekToKey:         []byte(createTestKey(10)),
+			itrAtKeyAfterSeek: []byte(createTestKey(10)),
+			expectedKeys:      createTestKeys(11, 19),
+			expectedValues:    createTestValues("db1", 11, 19),
+		},
+		{
+			startKey:          []byte(createTestKey(11)),
+			endKey:            nil,
+			seekToKey:         []byte(createTestKey(5)),
+			itrAtKeyAfterSeek: []byte(createTestKey(11)),
+			expectedKeys:      createTestKeys(12, 19),
+			expectedValues:    createTestValues("db1", 12, 19),
+		},
+		{
+			startKey:          nil,
+			endKey:            nil,
+			seekToKey:         []byte(createTestKey(19)),
+			itrAtKeyAfterSeek: []byte(createTestKey(19)),
+			expectedKeys:      nil,
+			expectedValues:    nil,
+		},
+	}
+
+	for i, testCase := range rangeWithSeekTestCases {
+		t.Run(
+			fmt.Sprintf("range with seek testCase %d", i),
+			func(t *testing.T) {
+				itr := db1.GetIterator(testCase.startKey, testCase.endKey)
+				defer itr.Release()
+				require.True(t, itr.Seek(testCase.seekToKey))
+				require.Equal(t, testCase.itrAtKeyAfterSeek, itr.Key())
+				checkItrResults(t, itr, testCase.expectedKeys, testCase.expectedValues)
+			},
+		)
+	}
+
+	itr := db1.GetIterator(nil, nil)
+	defer itr.Release()
+	require.True(t, itr.Seek([]byte(createTestKey(10))))
+	require.Equal(t, []byte(createTestKey(10)), itr.Key())
+	checkItrResults(t, itr, createTestKeys(11, 19), createTestValues("db1", 11, 19))
+
+	require.True(t, itr.First())
+	require.True(t, itr.Seek([]byte(createTestKey(10))))
+	require.Equal(t, []byte(createTestKey(10)), itr.Key())
+	require.True(t, itr.Prev())
+	checkItrResults(t, itr, createTestKeys(10, 19), createTestValues("db1", 10, 19))
+
+	require.True(t, itr.First())
+	require.False(t, itr.Seek([]byte(createTestKey(20))))
+	require.True(t, itr.First())
+	checkItrResults(t, itr, createTestKeys(1, 19), createTestValues("db1", 1, 19))
+
+	require.True(t, itr.First())
+	require.False(t, itr.Prev())
+	checkItrResults(t, itr, createTestKeys(0, 19), createTestValues("db1", 0, 19))
+
+	require.True(t, itr.First())
+	require.True(t, itr.Last())
+	checkItrResults(t, itr, nil, nil)
+}
+
+func testRange(t *testing.T, start, end string, expectedKeys, expectedValues []string) {
 
-	itr3 := db2.GetIterator(nil, nil)
-	defer itr3.Release()
-	checkItrResults(t, itr3, createTestKeys(0, 19), createTestValues("db2", 0, 19))
 }
 
 func TestBatchedUpdates(t *testing.T) {
@@ -220,7 +322,6 @@ func testDBBasicWriteAndReads(t *testing.T, dbNames ...string) {
 }
 
 func checkItrResults(t *testing.T, itr *Iterator, expectedKeys []string, expectedValues []string) {
-	defer itr.Release()
 	var actualKeys []string
 	var actualValues []string
 	for itr.Next(); itr.Valid(); itr.Next() {