Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fsync and do not generate empty files in snapshots #1345

Merged
merged 1 commit into from
May 28, 2020
Merged

fsync and do not generate empty files in snapshots #1345

merged 1 commit into from
May 28, 2020

Conversation

manish-sethi
Copy link
Contributor

Signed-off-by: manish manish.sethi@gmail.com

Type of change

  • Improvement (improvement to code)

Description

This PR fixes the following issues

  • Perform sync on the snapshot files
  • Does not generate empty files (if there is no content in a snapshot-able component)
  • Use filepath package instead of path package

@manish-sethi manish-sethi requested a review from a team as a code owner May 28, 2020 05:17
cendhu
cendhu reviewed May 28, 2020
View reviewed changes
common/ledger/blkstorage/blockindex.go
Comment on lines +286 to +287
if numTxIDs == 0 { // first iteration, create the data file
dataFile, err = snapshot.CreateFile(filepath.Join(dir, snapshotDataFileName), snapshotFileFormat, newHashFunc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to have no TxId in the store? Even the genesis block has a txID right? 4eff624

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as possible, it's not good to drag the assumptions in the lower level code that how the consumer uses it. At this level of the code, you can open a store and invoke the Export function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see a useful use-case that would open an empty blockstore with no genesis block and then try to export txIDs. The same is true for the StateDB too. IMO, this disturbs the existing code flow unnecessarily. I understand your point too but I haven't got convinced.

IMO, empty files are okay as it conveys that the store is empty. As we have a difference of opinion, I leave this here.

core/ledger/confighistory/mgr.go
Comment on lines +198 to +199
if numCollectionConfigs == 0 { // first iteration, create the data file
dataFileWriter, err = snapshot.CreateFile(filepath.Join(dir, snapshotDataFileName), snapshotFileFormat, newHashFunc)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification Question

  • When the snapshot files are processed by the new peer, how would the new peer know all required files are present in a given directory? In other words, an admin can forget to copy a required file. I assume that in the snapshot.metadata file, there would be a mapping between file names (all non-empty) and its hash along with many other data such as height, hash. Then, the kvledger at the new peer would ensure all files are present as per the snapshot.metadata file before processing the snapshots. Am I correct?

Comment

  1. I agree that the store would be empty when there is no explicit collection config. I am worried that admin might say that a file is missing when it is missing intentionally. I assume snapshot.metadata is not in human readable format to find the list of files (a counter argument would be that admin can still look at each file and complain about empty content). A consistent set of files for all scenario looks natural to me and avoids confusion.
  2. If we allow empty files, the code path for both empty and non-empty files would be the same (does not look like a hack to me when we allow empty files for code simplicity). Further, the kvledger needs not to have logic like which files do not exist and which components should not be called, etc... Even if we pass this logic to individual components, it adds unnecessary complexity. Earlier code looked neat to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clarification Question

* When the snapshot files are processed by the new peer, how would the new peer know all required files are present in a given directory? In other words, an admin can forget to copy a required file. I assume that in the snapshot.metadata file, there would be a mapping between file names (all non-empty) and its hash along with many other data such as height, hash. Then, the kvledger at the new peer would ensure all files are present as per the snapshot.metadata file before processing the snapshots. Am I correct?

Yes, that how it would work.

Comment

I assume snapshot.metadata is not in human readable format to find the list of files (a counter argument would be that admin can still look at each file and complain about empty content). A consistent set of files for all scenario looks natural to me and avoids confusion.

It would be human readable JSON, so an admin can match the hashes manually. IMO, it much clearer to have a set of files and match the names and hashes in the metadata. If a channel does not use private data and we keep exporting hashes files and collection config files, that may cause more confusion.

2. If we allow empty files, the code path for both empty and non-empty files would be the same (does not look like a hack to me when we allow empty files for code simplicity). Further, the kvledger needs not to have logic like which files do not exist and which components should not be called, etc... Even if we pass this logic to individual components, it adds unnecessary complexity. Earlier code looked neat to me.

When it comes to exporting the stuff, we should not apply the same parameter as we do for fully in-memory data. The in-memory data is not seen by anyone. In the case of exported stuff, I prefer the neatness of the exported data dictating the requirement than the other way around.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More on human readable JSON, this works until we add compression. We have to think more for allowing for compression. In any case, there will be a final hash that would represent the snapshot and that hash along with the basic info about the snapshot would be present in a JSON file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I am saying is that when we have the same set of files all the time, it is easy for the consumer, i.e., the new peer processing these snapshot files. When the set of files can be different per snapshot, it adds unnecessary complication in the code at the consumer side.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said before, to me neatness of exported data should dictate the code than the other way. Also, I am not sure what code complication are you referring to. During import, if data files are not present for a particular component, it simply returns. That's rather far more simpler to reason about.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I was assuming that the kvledger would ask each component for the required file and only if it is present, kvledger would call the ImportXXX(dir). Now, I understand that the plan is to call ImportXXX(dir) and let the function return immediately if the respective store's file is not present.

core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go
Comment on lines +64 to +65
if pubStateWriter == nil { // encountered first time the pub state element
pubStateWriter, err = newSnapshotWriter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here also, the public state would be non-empty even at block height 1 as we store the genesis block. Earlier code structure -- file creation, dumping data to file, writing metadata, and closing it looked neater. If no data is dumped, it was reflected in the metadata.

If we still decide not to have empty files, having IsEmpty() at each store would make this more explicit and neat IMO.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same response as in the comment. Regarding IsEmpty, how do you handle for private state hashes?

cendhu
cendhu previously approved these changes May 28, 2020
View reviewed changes
common/ledger/blkstorage/blockindex.go Outdated
Comment on lines 299 to 301
if numTxIDs == 0 {
return nil, nil
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit.

Just to maintain consistency, similar to what is done in config history export, we can use if dataFile == nil here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

core/ledger/kvledger/txmgmt/privacyenabledstate/snapshot.go
pubStateDataHash, pubStateMetadataHash, err := pubStateWriter.done()
if err != nil {
return nil, err

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit.

Could return early here too if both writers are nil (just for the consistency with other export APIs).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave this as is. You typically return early to shorten an if block. Not applicable here.

@manish-sethi
fsync and do not generate empty files in snapshots
ec5343b 
This commit fixes following issues
- Perform sync on the snapshot files
- Does not generate empty files
- Use filepath instead of path package

Signed-off-by: manish <manish.sethi@gmail.com>
@manish-sethi manish-sethi force-pushed the donot_generate_empty_files branch from f22f730 to ec5343b Compare May 28, 2020 20:07
cendhu
cendhu approved these changes May 28, 2020
View reviewed changes
Copy link
Contributor

@cendhu cendhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

denyeart
denyeart approved these changes May 28, 2020
View reviewed changes
Copy link
Contributor

@denyeart denyeart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion either way on the points here, I'm good with it as-is.

@manish-sethi manish-sethi merged commit 251e989 into hyperledger:master May 28, 2020
@manish-sethi manish-sethi deleted the donot_generate_empty_files branch May 29, 2020 03:54
xhens added a commit to xhens/fabric that referenced this pull request Jun 1, 2020
@xhens @sykesm
@pratikspatil024 @tock-ibm @denyeart @christo4ferris @naonishijima @manish-sethi @wlahti @wenjianqiao @hello2mao @cendhu @guoger @caod123 @stephyee
[ASHF] Syncing Fork (#29)
4193c2b 
* Address more concerns highlighted by linters

These changes remove dead code, add error checks, and use assign unused
variables to the unnamed variable `_`.

Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

* Fixed write_first_app.rst typo

Signed-off-by: pratikpatil024 <pratikspatil024@gmail.com>

* FAB-17840 Ch.Part.API: Join channel REST handler (hyperledger#1305)

* FAB-17840 Ch.Part.API: Join channel REST handler

Implement a handler for a POST request to join a channel.

Here we support requests carrying application/json, support
for additional Content-Type will be added later.

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I8d09e3cba09842f2adc47fb60161eee814e33d31

* Review: simplify code

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: Iee1e8b66cb77f64b9762dee8f85304958081e0fe

* Review comments: simplify code

- extract error handling to separate method
- extract channelID extraction to separate method, reuse
- test Accept header allowed range
- better comments

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I11639a3f159694019e521e180e7fd5fadb42cb4f

* Review comments: support for multipart/form-data

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: Ic5a0307f56be5561f45910a02892dc7e7b9554d1

* Review comments: remove support for application/json

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I38bb3564a0e6482b7bf9dca25bd8424e6db2ac95

* Spelling: s/chainocde/chaincode/g

Signed-off-by: Matthew Sykes <matthew.sykes@gmail.com>

* Retire dormant Fabric maintainers

Retire maintainers that have been inactive for the last 3 months:
Jonathan Levi
Srinivasan Muralidharan

Note:
"A maintainer removed for inactivity should be restored following a
sustained resumption of contributions and reviews (a month or more)
demonstrating a renewed commitment to the project."

Signed-off-by: David Enyeart <enyeart@us.ibm.com>

* add NOTICE file

recommended per ASF and LF policy

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* Add NOTICE to license ignore list

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* Fix some typo in docs

This CR fixes the following some typo in docs.
- chainocode -> chaincode
- chainode -> chaincode
- lifecyle -> lifecycle
- Hyperlegder -> Hyperledger
- chanel -> channel
- scructured -> structured
- demostrate -> demonstrate
- certficates -> certificates
- how how -> how
- the the -> the
- a a -> a
- thefollowing -> the following

Signed-off-by: Nao Nishijima <nao.nishijima.xt@hitachi.com>

* Remove txmgr interface

This commit removes the interface TxMgr, as there is a single
implementation (i.e., LockbasedTxmgr). All the dependent packages
are able to use this single implementation directly.
The only exception is the validation package that causes a circular
dependency. the validation package uses only a single function from this
interface (NewTxSimulator). The validation package uses this function
for getting accessing to the TxSimulator, in order to allow for the
execution a post-order transaction (only channel config transaction in
the current code). For meeting this need, this commit now defines a
local interface with a single function in the validation package itself.

Signed-off-by: manish <manish.sethi@gmail.com>

* FAB-17841: Ch.Part.API: Remove channel REST handler (hyperledger#1330)

Handle DELETE request to remove a channel.

- Resolve the removeStorage flag from config & query.
- If system-channel exists - reject with StatusMethodNotAllowed and
  Allow header set to GET.
- If channel does not exist - reject with StatusNotFound.
- If successs - respond with StatusNoContent.

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I78581df3c7f0cb99007edddc83eb7a6dca5eba07

* Update commercial paper doc to use enrollUser.js

Signed-off-by: NIKHIL E GUPTA <negupta@us.ibm.com>

* Use protolator from fabric-config

Signed-off-by: Matthew Sykes <matthew.sykes@gmail.com>

* Revert "Bump viper version to the last working commit"

This reverts commit 5ad0a4f.

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* _lifecycle ignore previous build failure during install (hyperledger#1280)

When a chaincode build previously failed while installing the
chaincode, _lifecycle should ignore the cached error and attempt
to rebuild the chaincode. This is because we should assume the
end user knows something about why the build may succeed on retry
if they're reattempting an install.

Also, update integration tests to not care about exit status of
chaincode installs (since reinstalls now error).

FAB-17907

Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

* [FAB-17927] Add AllChaincodesInfo to DeployedChaincodeInfoProvider (hyperledger#1331)

Implement AllChaincodesInfo to query ledger and return chaincode info
for all the deployed chaincode (both new lifecycle and legacy chaincodes)
on a channel. This function is needed for ledger checkpointing
and deletion of channel-specific databases in statecouchdb.

Signed-off-by: Wenjian Qiao <wenjianq@gmail.com>

* [FAB-17471] Fix OrdererType key to correct line

SampleInsecureKafka profile tries to change consensus type to
kafka using OrdererType. But it was written in the line above
"<<: *OrdererDefaults". That causes the overwrite consensus
type again. This CR moves OrdererType to the correct line.
As a result, this CR can generate the correct ordererer type's
genesis block.

Signed-off-by: Nao Nishijima <nao.nishijima.xt@hitachi.com>

* [FAB-17059] Add missing mspID when init MembershipProvider.

Signed-off-by: Hongbin Mao <hello2mao@gmail.com>

* fsync and do not generate empty files in snapshots (hyperledger#1345)

This commit fixes following issues
- Perform sync on the snapshot files
- Does not generate empty files
- Use filepath instead of path package

Signed-off-by: manish <manish.sethi@gmail.com>

* fix infinite loop during full range query (hyperledger#1347)

We use a single scanner for achieving both paginated and
non-paginated range query.

We have internalQueryLimit and pageSize. For each
_all_docs?startKey="XXX"&endKey="YYY" REST API call to
CouchDB, we fetch atmost internalQueryLimit number of
records only by appending limit=internalQueryLimit.

When the requested pageSize is higher than the internalQueryLimit
or the total number of records present in the given range
is higher than the internalQueryLimit, iterator would execute the
query again once the records fetched in the first cycle is consumed
and so on. In order to do that, after an execution of the REST API in each
cycle, it updates the initially passed startKey to the nextStartKey.
If there is no nextStartKey, it is set to the endKey.

Currently, when the nextStartKey and the endKey are the same, we still
run one REST API call which is actually not needed as we always set
inclusive_end=false. However, this causes infinite loop in a particular
case. When we want to retrieve all the records, we would pass an empty
string as the startKey and the endKey. When the startKey is an empty
string, the REST API call would become _all_docs?endKey="YYY". When
both are empty, it would become _all_docs

Given that we set startKey to endKey when there is no nextStartKey and
still execute one REST API call, it gets into infinite loop by fetching
all records again and again.

We avovid this infinite loop by setting scanner.exhausted = true whe
the startKey and endKey become the same when there is no nextStartKey

Signed-off-by: senthil <cendhu@gmail.com>

* Remove unnecessary extension of osn (hyperledger#1351)

In the integration test touched by this patch, cert file
is read to remove and add node. It can be easily achieved
by reading file bytes directly, without extending network
to grab intermediate object.

Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

* [FAB-17935] Change unnecessary warning log line to debug in gossip (hyperledger#1350)

privdata

Signed-off-by: Danny Cao <dcao@us.ibm.com>

* [FAB-17900] Update BCCSP.PKCS11.Pin in examples

- The Pin value must be quoted when specified in the yaml

Signed-off-by: Tiffany Harris <tiffany.harris@ibm.com>

* [FAB-17900] Fixes numeric env variable override bug

Signed-off-by: Tiffany Harris <tiffany.harris@ibm.com>

* Remove s390x, powerpc64le from RELEASE_PLATFORMS

- Release automation only creates amd64 binaries for darwin, linux, and
  windows.
- Continuous integration no longer runs on powerpc64le or s390x

Also remove stale build tags related to plugins and race detection for
old versions of go, s390x, and ppc64le.

Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

* Backfill test for BCCSP environment overrides...

... and consistently use SW as the key for `SwOpts` in the configuration
structures. Right now tags for mapstructure and JSON do not match the
tags for YAML yet our sample configuration documents (in YAML) use `SW`.

Signed-off-by: Matthew Sykes <matthew.sykes@gmail.com>

* Updates in master for v2.1.1 release

Update master doc and bootstrap script for v2.1.1 release.

Signed-off-by: David Enyeart <enyeart@us.ibm.com>

Co-authored-by: Matthew Sykes <sykesmat@us.ibm.com>
Co-authored-by: pratikpatil024 <pratikspatil024@gmail.com>
Co-authored-by: Yoav Tock <tock@il.ibm.com>
Co-authored-by: Matthew Sykes <matthew.sykes@gmail.com>
Co-authored-by: David Enyeart <enyeart@us.ibm.com>
Co-authored-by: Christopher Ferris <chrisfer@us.ibm.com>
Co-authored-by: Brett Logan <brett.t.logan@ibm.com>
Co-authored-by: Nao Nishijima <nao.nishijima.xt@hitachi.com>
Co-authored-by: manish <manish.sethi@gmail.com>
Co-authored-by: NIKHIL E GUPTA <negupta@us.ibm.com>
Co-authored-by: Will Lahti <wtlahti@us.ibm.com>
Co-authored-by: Wenjian Qiao <wenjianq@gmail.com>
Co-authored-by: Hongbin Mao <hello2mao@gmail.com>
Co-authored-by: Senthil Nathan N <cendhu@users.noreply.github.com>
Co-authored-by: Jay Guo <guojiannan1101@gmail.com>
Co-authored-by: Danny Cao <dcao@us.ibm.com>
Co-authored-by: Tiffany Harris <tiffany.harris@ibm.com>
xhens added a commit to xhens/fabric that referenced this pull request Jun 1, 2020
@xhens @sykesm
@pratikspatil024 @tock-ibm @denyeart @christo4ferris @naonishijima @manish-sethi @wlahti @wenjianqiao @hello2mao @cendhu @guoger @caod123 @stephyee
Update base (#30)
da2990f 
* Address more concerns highlighted by linters

These changes remove dead code, add error checks, and use assign unused
variables to the unnamed variable `_`.

Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

* Fixed write_first_app.rst typo

Signed-off-by: pratikpatil024 <pratikspatil024@gmail.com>

* FAB-17840 Ch.Part.API: Join channel REST handler (hyperledger#1305)

* FAB-17840 Ch.Part.API: Join channel REST handler

Implement a handler for a POST request to join a channel.

Here we support requests carrying application/json, support
for additional Content-Type will be added later.

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I8d09e3cba09842f2adc47fb60161eee814e33d31

* Review: simplify code

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: Iee1e8b66cb77f64b9762dee8f85304958081e0fe

* Review comments: simplify code

- extract error handling to separate method
- extract channelID extraction to separate method, reuse
- test Accept header allowed range
- better comments

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I11639a3f159694019e521e180e7fd5fadb42cb4f

* Review comments: support for multipart/form-data

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: Ic5a0307f56be5561f45910a02892dc7e7b9554d1

* Review comments: remove support for application/json

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I38bb3564a0e6482b7bf9dca25bd8424e6db2ac95

* Spelling: s/chainocde/chaincode/g

Signed-off-by: Matthew Sykes <matthew.sykes@gmail.com>

* Retire dormant Fabric maintainers

Retire maintainers that have been inactive for the last 3 months:
Jonathan Levi
Srinivasan Muralidharan

Note:
"A maintainer removed for inactivity should be restored following a
sustained resumption of contributions and reviews (a month or more)
demonstrating a renewed commitment to the project."

Signed-off-by: David Enyeart <enyeart@us.ibm.com>

* add NOTICE file

recommended per ASF and LF policy

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* Add NOTICE to license ignore list

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* Fix some typo in docs

This CR fixes the following some typo in docs.
- chainocode -> chaincode
- chainode -> chaincode
- lifecyle -> lifecycle
- Hyperlegder -> Hyperledger
- chanel -> channel
- scructured -> structured
- demostrate -> demonstrate
- certficates -> certificates
- how how -> how
- the the -> the
- a a -> a
- thefollowing -> the following

Signed-off-by: Nao Nishijima <nao.nishijima.xt@hitachi.com>

* Remove txmgr interface

This commit removes the interface TxMgr, as there is a single
implementation (i.e., LockbasedTxmgr). All the dependent packages
are able to use this single implementation directly.
The only exception is the validation package that causes a circular
dependency. the validation package uses only a single function from this
interface (NewTxSimulator). The validation package uses this function
for getting accessing to the TxSimulator, in order to allow for the
execution a post-order transaction (only channel config transaction in
the current code). For meeting this need, this commit now defines a
local interface with a single function in the validation package itself.

Signed-off-by: manish <manish.sethi@gmail.com>

* FAB-17841: Ch.Part.API: Remove channel REST handler (hyperledger#1330)

Handle DELETE request to remove a channel.

- Resolve the removeStorage flag from config & query.
- If system-channel exists - reject with StatusMethodNotAllowed and
  Allow header set to GET.
- If channel does not exist - reject with StatusNotFound.
- If successs - respond with StatusNoContent.

Signed-off-by: Yoav Tock <tock@il.ibm.com>
Change-Id: I78581df3c7f0cb99007edddc83eb7a6dca5eba07

* Update commercial paper doc to use enrollUser.js

Signed-off-by: NIKHIL E GUPTA <negupta@us.ibm.com>

* Use protolator from fabric-config

Signed-off-by: Matthew Sykes <matthew.sykes@gmail.com>

* Revert "Bump viper version to the last working commit"

This reverts commit 5ad0a4f.

Signed-off-by: Brett Logan <brett.t.logan@ibm.com>

* _lifecycle ignore previous build failure during install (hyperledger#1280)

When a chaincode build previously failed while installing the
chaincode, _lifecycle should ignore the cached error and attempt
to rebuild the chaincode. This is because we should assume the
end user knows something about why the build may succeed on retry
if they're reattempting an install.

Also, update integration tests to not care about exit status of
chaincode installs (since reinstalls now error).

FAB-17907

Signed-off-by: Will Lahti <wtlahti@us.ibm.com>

* [FAB-17927] Add AllChaincodesInfo to DeployedChaincodeInfoProvider (hyperledger#1331)

Implement AllChaincodesInfo to query ledger and return chaincode info
for all the deployed chaincode (both new lifecycle and legacy chaincodes)
on a channel. This function is needed for ledger checkpointing
and deletion of channel-specific databases in statecouchdb.

Signed-off-by: Wenjian Qiao <wenjianq@gmail.com>

* [FAB-17471] Fix OrdererType key to correct line

SampleInsecureKafka profile tries to change consensus type to
kafka using OrdererType. But it was written in the line above
"<<: *OrdererDefaults". That causes the overwrite consensus
type again. This CR moves OrdererType to the correct line.
As a result, this CR can generate the correct ordererer type's
genesis block.

Signed-off-by: Nao Nishijima <nao.nishijima.xt@hitachi.com>

* [FAB-17059] Add missing mspID when init MembershipProvider.

Signed-off-by: Hongbin Mao <hello2mao@gmail.com>

* fsync and do not generate empty files in snapshots (hyperledger#1345)

This commit fixes following issues
- Perform sync on the snapshot files
- Does not generate empty files
- Use filepath instead of path package

Signed-off-by: manish <manish.sethi@gmail.com>

* fix infinite loop during full range query (hyperledger#1347)

We use a single scanner for achieving both paginated and
non-paginated range query.

We have internalQueryLimit and pageSize. For each
_all_docs?startKey="XXX"&endKey="YYY" REST API call to
CouchDB, we fetch atmost internalQueryLimit number of
records only by appending limit=internalQueryLimit.

When the requested pageSize is higher than the internalQueryLimit
or the total number of records present in the given range
is higher than the internalQueryLimit, iterator would execute the
query again once the records fetched in the first cycle is consumed
and so on. In order to do that, after an execution of the REST API in each
cycle, it updates the initially passed startKey to the nextStartKey.
If there is no nextStartKey, it is set to the endKey.

Currently, when the nextStartKey and the endKey are the same, we still
run one REST API call which is actually not needed as we always set
inclusive_end=false. However, this causes infinite loop in a particular
case. When we want to retrieve all the records, we would pass an empty
string as the startKey and the endKey. When the startKey is an empty
string, the REST API call would become _all_docs?endKey="YYY". When
both are empty, it would become _all_docs

Given that we set startKey to endKey when there is no nextStartKey and
still execute one REST API call, it gets into infinite loop by fetching
all records again and again.

We avovid this infinite loop by setting scanner.exhausted = true whe
the startKey and endKey become the same when there is no nextStartKey

Signed-off-by: senthil <cendhu@gmail.com>

* Remove unnecessary extension of osn (hyperledger#1351)

In the integration test touched by this patch, cert file
is read to remove and add node. It can be easily achieved
by reading file bytes directly, without extending network
to grab intermediate object.

Signed-off-by: Jay Guo <guojiannan1101@gmail.com>

* [FAB-17935] Change unnecessary warning log line to debug in gossip (hyperledger#1350)

privdata

Signed-off-by: Danny Cao <dcao@us.ibm.com>

* [FAB-17900] Update BCCSP.PKCS11.Pin in examples

- The Pin value must be quoted when specified in the yaml

Signed-off-by: Tiffany Harris <tiffany.harris@ibm.com>

* [FAB-17900] Fixes numeric env variable override bug

Signed-off-by: Tiffany Harris <tiffany.harris@ibm.com>

* Remove s390x, powerpc64le from RELEASE_PLATFORMS

- Release automation only creates amd64 binaries for darwin, linux, and
  windows.
- Continuous integration no longer runs on powerpc64le or s390x

Also remove stale build tags related to plugins and race detection for
old versions of go, s390x, and ppc64le.

Signed-off-by: Matthew Sykes <sykesmat@us.ibm.com>

* Backfill test for BCCSP environment overrides...

... and consistently use SW as the key for `SwOpts` in the configuration
structures. Right now tags for mapstructure and JSON do not match the
tags for YAML yet our sample configuration documents (in YAML) use `SW`.

Signed-off-by: Matthew Sykes <matthew.sykes@gmail.com>

* Updates in master for v2.1.1 release

Update master doc and bootstrap script for v2.1.1 release.

Signed-off-by: David Enyeart <enyeart@us.ibm.com>

Co-authored-by: Matthew Sykes <sykesmat@us.ibm.com>
Co-authored-by: pratikpatil024 <pratikspatil024@gmail.com>
Co-authored-by: Yoav Tock <tock@il.ibm.com>
Co-authored-by: Matthew Sykes <matthew.sykes@gmail.com>
Co-authored-by: David Enyeart <enyeart@us.ibm.com>
Co-authored-by: Christopher Ferris <chrisfer@us.ibm.com>
Co-authored-by: Brett Logan <brett.t.logan@ibm.com>
Co-authored-by: Nao Nishijima <nao.nishijima.xt@hitachi.com>
Co-authored-by: manish <manish.sethi@gmail.com>
Co-authored-by: NIKHIL E GUPTA <negupta@us.ibm.com>
Co-authored-by: Will Lahti <wtlahti@us.ibm.com>
Co-authored-by: Wenjian Qiao <wenjianq@gmail.com>
Co-authored-by: Hongbin Mao <hello2mao@gmail.com>
Co-authored-by: Senthil Nathan N <cendhu@users.noreply.github.com>
Co-authored-by: Jay Guo <guojiannan1101@gmail.com>
Co-authored-by: Danny Cao <dcao@us.ibm.com>
Co-authored-by: Tiffany Harris <tiffany.harris@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@denyeart denyeart denyeart approved these changes

@cendhu cendhu cendhu approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@manish-sethi @cendhu @denyeart