Skip to content

Commit

Permalink
[FAB-11796]high-throughput:Remove unnecessary prunesafe
Browse files Browse the repository at this point in the history 
fabric-samples/high-throughput is a sample Chaincode for
delta-based transaction model. When executing `update`, a new delta
for a particular variable is updated to the ledger. The sum of a
variable is updated to ledger by deleting all of its delta rows
while computing the sum. The current Chaincode has two types of
pruning functions. However, there is no difference in terms of
Fabric's transaction model.

This CR adds the following changes.
- Remove unnecessary `pruneSafe` function.
- Change the name of function from `pruneFast` to `prune`.
- Update or delete related scripts.
- Improve a related documentation.

FAB-11796 #done

Change-Id: I5daa21554e53d77b7b5081f02a2846a85ec06f9a
Signed-off-by: Yuki Kondo <yuki.kondo@hal.hitachi.com>
  • Loading branch information
@yuki-kon
yuki-kon committed Feb 15, 2019
1 parent 6007c09 commit f26477c
Show file tree
Hide file tree
Showing 5 changed files with 23 additions and 112 deletions.
19 changes: 9 additions & 10 deletions high-throughput/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -104,19 +104,19 @@ and run some invocations are provided below.
* In the `volumes` section of the `cli` container, edit the second line which refers to the chaincode folder to point to the chaincode folder
within the `high-throughput` folder, e.g.

`./../chaincode/:/opt/gopath/src/github.com/hyperledger/fabric/examples/chaincode/go` -->
`./../high-throughput/chaincode/:/opt/gopath/src/github.com/hyperledger/fabric/examples/chaincode/go`
`./../chaincode/:/opt/gopath/src/github.com/hyperledger/fabric-samples/chaincode` -->
`./../high-throughput/chaincode/:/opt/gopath/src/github.com/hyperledger/fabric-samples/chaincode`
* Again in the `volumes` section, edit the fourth line which refers to the scripts folder so it points to the scripts folder within the
`high-throughput` folder, e.g.

`./scripts:/opt/gopath/src/github.com/hyperledger/fabric/peer/scripts/` -->
`./scripts:/opt/gopath/src/github.com/hyperledger/fabric/peer/scripts/` -->
`./../high-throughput/scripts/:/opt/gopath/src/github.com/hyperledger/fabric/peer/scripts/`

* Finally, comment out the `docker exec cli scripts/script.sh` command from the `byfn.sh` script by placing a `#` before it so that the standard BYFN end to end script doesn't run, e.g.

`# docker exec cli scripts/script.sh $CHANNEL_NAME $CLI_DELAY $LANGUAGE $CLI_TIMEOUT $VERBOSE`

3. We can now bring our network up by typing in `./byfn.sh -m up -c mychannel`
3. We can now bring our network up by typing in `./byfn.sh up -c mychannel`
4. Open a new terminal window and enter the CLI container using `docker exec -it cli bash`, all operations on the network will happen within
this container from now on.

Expand Down Expand Up @@ -153,13 +153,11 @@ Example: `./delete-invoke.sh myvar`

#### Prune
Pruning takes all the deltas generated for a variable and combines them all into a single row, deleting all previous rows. This helps cleanup
the ledger when many updates have been performed. There are two types of pruning: `prunefast` and `prunesafe`. Prune fast performs the deletion
and aggregation simultaneously, so if an error happens along the way data integrity is not guaranteed. Prune safe performs the aggregation first,
backs up the results, then performs the deletion. This way, if an error occurs along the way, data integrity is maintained.
the ledger when many updates have been performed.

The format for pruning is: `./[prunesafe|prunefast]-invoke.sh name` where `name` is the name of the variable to prune.
The format for pruning is: `./prune-invoke.sh name` where `name` is the name of the variable to prune.

Example: `./prunefast-invoke.sh myvar` or `./prunesafe-invoke.sh myvar`
Example: `./prune-invoke.sh myvar`

### Test the Network
Two scripts are provided to show the advantage of using this system when running many parallel transactions at once: `many-updates.sh` and
Expand All @@ -175,5 +173,6 @@ errors in the peer and orderer logs.
There is one other script, `get-traditional.sh`, which simply gets the value of a row in the traditional way, with no deltas.

Examples:
`./many-updates.sh testvar 100 +` --> final value from `./get-invoke.sh` should be 100000
`./many-updates.sh testvar 100 +` --> final value from `./get-invoke.sh testvar` should be 100000

`./many-updates-traditional.sh testvar` --> final value from `./get-traditional.sh testvar` is undefined
98 changes: 9 additions & 89 deletions high-throughput/chaincode/high-throughput.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,7 @@ func (s *SmartContract) Init(APIstub shim.ChaincodeStubInterface) sc.Response {
// Current supported invocations are:
// - update, adds a delta to an aggregate variable in the ledger, all variables are assumed to start at 0
// - get, retrieves the aggregate value of a variable in the ledger
// - pruneFast, deletes all rows associated with the variable and replaces them with a single row containing the aggregate value
// - pruneSafe, same as pruneFast except it pre-computed the value and backs it up before performing any destructive operations
// - prune, deletes all rows associated with the variable and replaces them with a single row containing the aggregate value
// - delete, removes all rows associated with the variable
func (s *SmartContract) Invoke(APIstub shim.ChaincodeStubInterface) sc.Response {
// Retrieve the requested Smart Contract function and arguments
Expand All @@ -64,10 +63,8 @@ func (s *SmartContract) Invoke(APIstub shim.ChaincodeStubInterface) sc.Response
return s.update(APIstub, args)
} else if function == "get" {
return s.get(APIstub, args)
} else if function == "prunefast" {
return s.pruneFast(APIstub, args)
} else if function == "prunesafe" {
return s.pruneSafe(APIstub, args)
} else if function == "prune" {
return s.prune(APIstub, args)
} else if function == "delete" {
return s.delete(APIstub, args)
} else if function == "putstandard" {
Expand Down Expand Up @@ -202,17 +199,15 @@ func (s *SmartContract) get(APIstub shim.ChaincodeStubInterface, args []string)
/**
* Prunes a variable by deleting all of its delta rows while computing the final value. Once all rows
* have been processed and deleted, a single new row is added which defines a delta containing the final
* computed value of the variable. This function is NOT safe as any failures or errors during pruning
* will result in an undefined final value for the variable and loss of data. Use pruneSafe if data
* integrity is important. The args array contains the following argument:
* computed value of the variable. The args array contains the following argument:
* - args[0] -> The name of the variable to prune
*
* @param APIstub The chaincode shim
* @param args The args array for the pruneFast invocation
* @param args The args array for the prune invocation
*
* @return A response structure indicating success or failure with a message
*/
func (s *SmartContract) pruneFast(APIstub shim.ChaincodeStubInterface, args []string) sc.Response {
func (s *SmartContract) prune(APIstub shim.ChaincodeStubInterface, args []string) sc.Response {
// Check we have a valid number of ars
if len(args) != 1 {
return shim.Error("Incorrect number of arguments, expecting 1")
Expand Down Expand Up @@ -276,88 +271,13 @@ func (s *SmartContract) pruneFast(APIstub shim.ChaincodeStubInterface, args []st
}
}

// Update the ledger with the final value and return
// Update the ledger with the final value
updateResp := s.update(APIstub, []string{name, strconv.FormatFloat(finalVal, 'f', -1, 64), "+"})
if updateResp.Status == OK {
return shim.Success([]byte(fmt.Sprintf("Successfully pruned variable %s, final value is %f, %d rows pruned", args[0], finalVal, i)))
}

return shim.Error(fmt.Sprintf("Failed to prune variable: all rows deleted but could not update value to %f, variable no longer exists in ledger", finalVal))
}

/**
* This function performs the same function as pruneFast except it provides data backups in case the
* prune fails. The final aggregate value is computed before any deletion occurs and is backed up
* to a new row. This back-up row is deleted only after the new aggregate delta has been successfully
* written to the ledger. The args array contains the following argument:
* args[0] -> The name of the variable to prune
*
* @param APIstub The chaincode shim
* @param args The arguments array for the pruneSafe invocation
*
* @result A response structure indicating success or failure with a message
*/
func (s *SmartContract) pruneSafe(APIstub shim.ChaincodeStubInterface, args []string) sc.Response {
// Verify there are a correct number of arguments
if len(args) != 1 {
return shim.Error("Incorrect number of arguments, expecting 1 (the name of the variable to prune)")
}

// Get the var name
name := args[0]

// Get the var's value and process it
getResp := s.get(APIstub, args)
if getResp.Status == ERROR {
return shim.Error(fmt.Sprintf("Could not retrieve the value of %s before pruning, pruning aborted: %s", name, getResp.Message))
}

valueStr := string(getResp.Payload)
val, convErr := strconv.ParseFloat(valueStr, 64)
if convErr != nil {
return shim.Error(fmt.Sprintf("Could not convert the value of %s to a number before pruning, pruning aborted: %s", name, convErr.Error()))
}

// Store the var's value temporarily
backupPutErr := APIstub.PutState(fmt.Sprintf("%s_PRUNE_BACKUP", name), []byte(valueStr))
if backupPutErr != nil {
return shim.Error(fmt.Sprintf("Could not backup the value of %s before pruning, pruning aborted: %s", name, backupPutErr.Error()))
}

// Get all deltas for the variable
deltaResultsIterator, deltaErr := APIstub.GetStateByPartialCompositeKey("varName~op~value~txID", []string{name})
if deltaErr != nil {
return shim.Error(fmt.Sprintf("Could not retrieve value for %s: %s", name, deltaErr.Error()))
}
defer deltaResultsIterator.Close()

// Delete each row
var i int
for i = 0; deltaResultsIterator.HasNext(); i++ {
responseRange, nextErr := deltaResultsIterator.Next()
if nextErr != nil {
return shim.Error(fmt.Sprintf("Could not retrieve next row for pruning: %s", nextErr.Error()))
}

deltaRowDelErr := APIstub.DelState(responseRange.Key)
if deltaRowDelErr != nil {
return shim.Error(fmt.Sprintf("Could not delete delta row: %s", deltaRowDelErr.Error()))
}
}

// Insert new row for the final value
updateResp := s.update(APIstub, []string{name, valueStr, "+"})
if updateResp.Status == ERROR {
return shim.Error(fmt.Sprintf("Could not insert the final value of the variable after pruning, variable backup is stored in %s_PRUNE_BACKUP: %s", name, updateResp.Message))
}

// Delete the backup value
delErr := APIstub.DelState(fmt.Sprintf("%s_PRUNE_BACKUP", name))
if delErr != nil {
return shim.Error(fmt.Sprintf("Could not delete backup value %s_PRUNE_BACKUP, this does not affect the ledger but should be removed manually", name))
return shim.Error(fmt.Sprintf("Could not update the final value of the variable after pruning: %s", updateResp.Message))
}

return shim.Success([]byte(fmt.Sprintf("Successfully pruned variable %s, final value is %f, %d rows pruned", name, val, i)))
return shim.Success([]byte(fmt.Sprintf("Successfully pruned variable %s, final value is %f, %d rows pruned", args[0], finalVal, i)))
}

/**
Expand Down
8 changes: 4 additions & 4 deletions high-throughput/scripts/install-chaincode.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,25 @@ export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/pee
export CORE_PEER_ADDRESS=peer0.org1.example.com:7051
export CORE_PEER_LOCALMSPID="Org1MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric/examples/chaincode/go
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric-samples/chaincode

echo "========== Installing chaincode on peer1.org1 =========="
export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/Admin@org1.example.com/msp
export CORE_PEER_ADDRESS=peer1.org1.example.com:7051
export CORE_PEER_LOCALMSPID="Org1MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/tls/ca.crt
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric/examples/chaincode/go
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric-samples/chaincode

echo "========== Installing chaincode on peer0.org2 =========="
export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp
export CORE_PEER_ADDRESS=peer0.org2.example.com:7051
export CORE_PEER_LOCALMSPID="Org2MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/tls/ca.crt
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric/examples/chaincode/go
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric-samples/chaincode

echo "========== Installing chaincode on peer1.org2 =========="
export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/users/Admin@org2.example.com/msp
export CORE_PEER_ADDRESS=peer1.org2.example.com:7051
export CORE_PEER_LOCALMSPID="Org2MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org2.example.com/peers/peer1.org2.example.com/tls/ca.crt
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric/examples/chaincode/go
peer chaincode install -n $CC_NAME -v $1 -p github.com/hyperledger/fabric-samples/chaincode
2 changes: 1 addition & 1 deletion high-throughput/scripts/prunefast-invoke.sh → high-throughput/scripts/prune-invoke.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@
# SPDX-License-Identifier: Apache-2.0
#

peer chaincode invoke -o orderer.example.com:7050 --tls $CORE_PEER_TLS_ENABLED --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n $CC_NAME -c '{"Args":["prunefast","'$1'"]}'
peer chaincode invoke -o orderer.example.com:7050 --tls $CORE_PEER_TLS_ENABLED --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n $CC_NAME -c '{"Args":["prune","'$1'"]}'

8 changes: 0 additions & 8 deletions high-throughput/scripts/prunesafe-invoke.sh
View file

This file was deleted.

0 comments on commit f26477c

Please sign in to comment.