Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FAB-17774] support orderer restart without genesis block #1197

Merged
merged 1 commit into from
May 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 32 additions & 16 deletions orderer/common/server/etcdraft_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ func nextPort() int32 {
func TestSpawnEtcdRaft(t *testing.T) {
gt := NewGomegaWithT(t)

// Set the fabric root folder for easy navigation to sampleconfig folder
fabricRootDir, err := filepath.Abs(filepath.Join("..", "..", ".."))
gt.Expect(err).NotTo(HaveOccurred())

// Build the configtxgen binary
configtxgen, err := gexec.Build("github.com/hyperledger/fabric/cmd/configtxgen")
gt.Expect(err).NotTo(HaveOccurred())
Expand All @@ -56,23 +52,27 @@ func TestSpawnEtcdRaft(t *testing.T) {

t.Run("Bad", func(t *testing.T) {
t.Run("Invalid bootstrap block", func(t *testing.T) {
testEtcdRaftOSNFailureInvalidBootstrapBlock(NewGomegaWithT(t), tempDir, orderer, fabricRootDir, configtxgen, cryptoPath)
testEtcdRaftOSNFailureInvalidBootstrapBlock(NewGomegaWithT(t), tempDir, orderer, configtxgen, cryptoPath)
})

t.Run("TLS disabled single listener", func(t *testing.T) {
testEtcdRaftOSNNoTLSSingleListener(NewGomegaWithT(t), tempDir, orderer, fabricRootDir, configtxgen, cryptoPath)
testEtcdRaftOSNNoTLSSingleListener(NewGomegaWithT(t), tempDir, orderer, configtxgen, cryptoPath)
})
})

t.Run("Good", func(t *testing.T) {
// tests in this suite actually launch process with success, hence we need to avoid
// conflicts in listening port, opening files.
t.Run("TLS disabled dual listener", func(t *testing.T) {
testEtcdRaftOSNNoTLSDualListener(NewGomegaWithT(t), tempDir, orderer, fabricRootDir, configtxgen, cryptoPath)
testEtcdRaftOSNNoTLSDualListener(NewGomegaWithT(t), tempDir, orderer, configtxgen, cryptoPath)
})

t.Run("TLS enabled single listener", func(t *testing.T) {
testEtcdRaftOSNSuccess(NewGomegaWithT(t), tempDir, configtxgen, orderer, fabricRootDir, cryptoPath)
testEtcdRaftOSNSuccess(NewGomegaWithT(t), tempDir, configtxgen, orderer, cryptoPath)
})

t.Run("Restart orderer without Genesis Block", func(t *testing.T) {
testEtcdRaftOSNRestart(NewGomegaWithT(t), tempDir, configtxgen, orderer, cryptoPath)
})
})
}
Expand Down Expand Up @@ -120,11 +120,27 @@ func generateCryptoMaterials(gt *GomegaWithT, cryptogen, path string) string {
return cryptoPath
}

func testEtcdRaftOSNSuccess(gt *GomegaWithT, tempDir, configtxgen, orderer, fabricRootDir, cryptoPath string) {
func testEtcdRaftOSNRestart(gt *GomegaWithT, tempDir, configtxgen, orderer, cryptoPath string) {
genesisBlockPath := generateBootstrapBlock(gt, tempDir, configtxgen, "system", "SampleEtcdRaftSystemChannel")

// Launch the OSN
ordererProcess := launchOrderer(gt, orderer, tempDir, genesisBlockPath, cryptoPath, "file")
defer func() { gt.Eventually(ordererProcess.Kill(), time.Minute).Should(gexec.Exit()) }()
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say("Beginning to serve requests"))
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say("becomeLeader"))

// Restart orderer with ORDERER_GENERAL_BOOTSTRAPMETHOD = none
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure this is enough... shouldn't we also delete the file?

gt.Eventually(ordererProcess.Kill(), time.Minute).Should(gexec.Exit())
ordererProcess = launchOrderer(gt, orderer, tempDir, "", cryptoPath, "none")
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say("Beginning to serve requests"))
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say("becomeLeader"))
}

func testEtcdRaftOSNSuccess(gt *GomegaWithT, tempDir, configtxgen, orderer, cryptoPath string) {
genesisBlockPath := generateBootstrapBlock(gt, tempDir, configtxgen, "system", "SampleEtcdRaftSystemChannel")

// Launch the OSN
ordererProcess := launchOrderer(gt, orderer, tempDir, genesisBlockPath, fabricRootDir, cryptoPath)
ordererProcess := launchOrderer(gt, orderer, tempDir, genesisBlockPath, cryptoPath, "file")
defer func() { gt.Eventually(ordererProcess.Kill(), time.Minute).Should(gexec.Exit()) }()
// The following configuration parameters are not specified in the orderer.yaml, so let's ensure
// they are really configured autonomously via the localconfig code.
Expand All @@ -146,7 +162,7 @@ func testEtcdRaftOSNSuccess(gt *GomegaWithT, tempDir, configtxgen, orderer, fabr
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say("becomeLeader"))
}

func testEtcdRaftOSNFailureInvalidBootstrapBlock(gt *GomegaWithT, tempDir, orderer, fabricRootDir, configtxgen, cryptoPath string) {
func testEtcdRaftOSNFailureInvalidBootstrapBlock(gt *GomegaWithT, tempDir, orderer, configtxgen, cryptoPath string) {
// create an application channel genesis block
genesisBlockPath := generateBootstrapBlock(gt, tempDir, configtxgen, "mychannel", "SampleOrgChannel")
genesisBlockBytes, err := ioutil.ReadFile(genesisBlockPath)
Expand All @@ -158,14 +174,14 @@ func testEtcdRaftOSNFailureInvalidBootstrapBlock(gt *GomegaWithT, tempDir, order
gt.Expect(err).NotTo(HaveOccurred())

// Launch the OSN
ordererProcess := launchOrderer(gt, orderer, tempDir, genesisBlockPath, fabricRootDir, cryptoPath)
ordererProcess := launchOrderer(gt, orderer, tempDir, genesisBlockPath, cryptoPath, "")
defer func() { gt.Eventually(ordererProcess.Kill(), time.Minute).Should(gexec.Exit()) }()

expectedErr := "Failed validating bootstrap block: the block isn't a system channel block because it lacks ConsortiumsConfig"
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say(expectedErr))
}

func testEtcdRaftOSNNoTLSSingleListener(gt *GomegaWithT, tempDir, orderer, fabricRootDir string, configtxgen, cryptoPath string) {
func testEtcdRaftOSNNoTLSSingleListener(gt *GomegaWithT, tempDir, orderer string, configtxgen, cryptoPath string) {
genesisBlockPath := generateBootstrapBlock(gt, tempDir, configtxgen, "system", "SampleEtcdRaftSystemChannel")

cmd := exec.Command(orderer)
Expand All @@ -185,7 +201,7 @@ func testEtcdRaftOSNNoTLSSingleListener(gt *GomegaWithT, tempDir, orderer, fabri
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say(expectedErr))
}

func testEtcdRaftOSNNoTLSDualListener(gt *GomegaWithT, tempDir, orderer, fabricRootDir string, configtxgen, cryptoPath string) {
func testEtcdRaftOSNNoTLSDualListener(gt *GomegaWithT, tempDir, orderer string, configtxgen, cryptoPath string) {
ordererTLSPath := filepath.Join(cryptoPath, "ordererOrganizations", "example.com", "orderers", "127.0.0.1.example.com", "tls")
genesisBlockPath := generateBootstrapBlock(gt, tempDir, configtxgen, "system", "SampleEtcdRaftSystemChannel")

Expand Down Expand Up @@ -218,13 +234,13 @@ func testEtcdRaftOSNNoTLSDualListener(gt *GomegaWithT, tempDir, orderer, fabricR
gt.Eventually(ordererProcess.Err, time.Minute).Should(gbytes.Say("becomeLeader"))
}

func launchOrderer(gt *GomegaWithT, orderer, tempDir, genesisBlockPath, fabricRootDir, cryptoPath string) *gexec.Session {
func launchOrderer(gt *GomegaWithT, orderer, tempDir, genesisBlockPath, cryptoPath, bootstrapMethod string) *gexec.Session {
ordererTLSPath := filepath.Join(cryptoPath, "ordererOrganizations", "example.com", "orderers", "127.0.0.1.example.com", "tls")
// Launch the orderer process
cmd := exec.Command(orderer)
cmd.Env = []string{
fmt.Sprintf("ORDERER_GENERAL_LISTENPORT=%d", nextPort()),
"ORDERER_GENERAL_BOOTSTRAPMETHOD=file",
fmt.Sprintf("ORDERER_GENERAL_BOOTSTRAPMETHOD=%s", bootstrapMethod),
"ORDERER_GENERAL_SYSTEMCHANNEL=system",
"ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED=true",
"ORDERER_GENERAL_TLS_ENABLED=true",
Expand Down
95 changes: 60 additions & 35 deletions orderer/common/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -124,44 +124,49 @@ func Main() {
var clusterDialer *cluster.PredicateDialer
var clusterType, reuseGrpcListener bool
var serversToUpdate []*comm.GRPCServer
if conf.General.BootstrapMethod == "file" {
bootstrapMethod := conf.General.BootstrapMethod
if bootstrapMethod == "file" || bootstrapMethod == "none" {
bootstrapBlock := extractBootstrapBlock(conf)
if err := ValidateBootstrapBlock(bootstrapBlock, cryptoProvider); err != nil {
logger.Panicf("Failed validating bootstrap block: %v", err)
if bootstrapBlock == nil {
bootstrapBlock = extractSystemChannel(lf, cryptoProvider)
}
sysChanLastConfigBlock := extractSysChanLastConfig(lf, bootstrapBlock)
clusterBootBlock = selectClusterBootBlock(bootstrapBlock, sysChanLastConfigBlock)

typ := consensusType(bootstrapBlock, cryptoProvider)
clusterType = isClusterType(clusterBootBlock, cryptoProvider)
if clusterType {
logger.Infof("Setting up cluster for orderer type %s", typ)
clusterClientConfig = initializeClusterClientConfig(conf)
clusterDialer = &cluster.PredicateDialer{
Config: clusterClientConfig,
if bootstrapBlock != nil {
if err := ValidateBootstrapBlock(bootstrapBlock, cryptoProvider); err != nil {
logger.Panicf("Failed validating bootstrap block: %v", err)
}
sysChanLastConfigBlock := extractSysChanLastConfig(lf, bootstrapBlock)
clusterBootBlock = selectClusterBootBlock(bootstrapBlock, sysChanLastConfigBlock)

typ := consensusType(bootstrapBlock, cryptoProvider)
clusterType = isClusterType(clusterBootBlock, cryptoProvider)
if clusterType {
logger.Infof("Setting up cluster for orderer type %s", typ)
clusterClientConfig = initializeClusterClientConfig(conf)
clusterDialer = &cluster.PredicateDialer{
Config: clusterClientConfig,
}

r = createReplicator(lf, bootstrapBlock, conf, clusterClientConfig.SecOpts, signer, cryptoProvider)
// Only clusters that are equipped with a recent config block can replicate.
if conf.General.BootstrapMethod == "file" {
r.replicateIfNeeded(bootstrapBlock)
}
r = createReplicator(lf, bootstrapBlock, conf, clusterClientConfig.SecOpts, signer, cryptoProvider)
// Only clusters that are equipped with a recent config block can replicate.
if conf.General.BootstrapMethod == "file" {
r.replicateIfNeeded(bootstrapBlock)
}

if reuseGrpcListener = reuseListener(conf, typ); !reuseGrpcListener {
clusterServerConfig, clusterGRPCServer = configureClusterListener(conf, serverConfig, ioutil.ReadFile)
}
if reuseGrpcListener = reuseListener(conf, typ); !reuseGrpcListener {
clusterServerConfig, clusterGRPCServer = configureClusterListener(conf, serverConfig, ioutil.ReadFile)
}

// If we have a separate gRPC server for the cluster,
// we need to update its TLS CA certificate pool.
serversToUpdate = append(serversToUpdate, clusterGRPCServer)
}
// Are we bootstrapping?
if len(lf.ChannelIDs()) == 0 {
initializeBootstrapChannel(clusterBootBlock, lf)
} else {
logger.Info("Not bootstrapping because of existing channels")
// If we have a separate gRPC server for the cluster,
// we need to update its TLS CA certificate pool.
serversToUpdate = append(serversToUpdate, clusterGRPCServer)
}
// Are we bootstrapping?
if len(lf.ChannelIDs()) == 0 {
initializeBootstrapChannel(clusterBootBlock, lf)
} else {
logger.Info("Not bootstrapping because of existing channels")
}
}

}

identityBytes, err := signer.Serialize()
Expand Down Expand Up @@ -292,6 +297,25 @@ func extractSysChanLastConfig(lf blockledger.Factory, bootstrapBlock *cb.Block)
return lastConfigBlock
}

// extractSystemChannel loops through all channels, and return the last
// config block for the system channel. Returns nil if no system channel
// was found.
func extractSystemChannel(lf blockledger.Factory, bccsp bccsp.BCCSP) *cb.Block {
for _, cID := range lf.ChannelIDs() {
channelLedger, err := lf.GetOrCreate(cID)
if err != nil {
logger.Panicf("Failed getting channel %v's ledger: %v", cID, err)
}
channelConfigBlock := multichannel.ConfigBlock(channelLedger)

err = ValidateBootstrapBlock(channelConfigBlock, bccsp)
if err == nil {
return channelConfigBlock
}
}
return nil
}

// Select cluster boot block
func selectClusterBootBlock(bootstrapBlock, sysChanLastConfig *cb.Block) *cb.Block {
if sysChanLastConfig == nil {
Expand Down Expand Up @@ -703,15 +727,16 @@ func initializeMultichannelRegistrar(
bccsp bccsp.BCCSP,
callbacks ...channelconfig.BundleActor,
) *multichannel.Registrar {

registrar := multichannel.NewRegistrar(*conf, lf, signer, metricsProvider, bccsp, callbacks...)

consenters := map[string]consensus.Consenter{}

var icr etcdraft.InactiveChainRegistry
if conf.General.BootstrapMethod == "file" && isClusterType(bootstrapBlock, bccsp) {
etcdConsenter := initializeEtcdraftConsenter(consenters, conf, lf, clusterDialer, bootstrapBlock, ri, srvConf, srv, registrar, metricsProvider, bccsp)
icr = etcdConsenter.InactiveChainRegistry
if conf.General.BootstrapMethod == "file" || conf.General.BootstrapMethod == "none" {
if bootstrapBlock != nil && isClusterType(bootstrapBlock, bccsp) {
Copy link
Contributor

@yacovm yacovm Apr 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can the bootstrapBlock be nil here? We don't have the check on the left side, so how can it be nil? Wouldn't we have crashed because we didn't have this check in the past?

And let's say bootstrap block is nil, doesn't that mean that you don't initialize the etcdraft? Isn't that a problem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bootstrapBlock should not be nil here, however, in test TestInitializeMultichannelRegistrar/"registrar without a system channel", it directly calls initializeMultichannelRegistrar, and passed in bootstrapBlock as nil, and BootstrapMethod as "none". In this case, the test will pass the old code as it will only check cluster type if BootstrapMethod is "file", but not the new code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So why not not pass a nil bootstrap block then? we shouldn't change the production code just to fit tests...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's definitely good point, we should never change production code to just fit tests.

However, currently orderer supports an action to startup without system channel defined (FAB-15709), which sets BootstrapMethod to "none" and bootstrapBlock to be nil.

And in this PR, we are supporting restarting orderer without providing genesisblock, which was being requested to set BootstrapMethod to "none", and obtain bootstrapBlock from system channel (FAB-17774). Hence, the nil check here is to distinguish the two cases.

We can also adding a new bootstrapMethod phrase here, instead of using "none" for both use cases. I'm opening to any suggestions.

etcdConsenter := initializeEtcdraftConsenter(consenters, conf, lf, clusterDialer, bootstrapBlock, ri, srvConf, srv, registrar, metricsProvider, bccsp)
icr = etcdConsenter.InactiveChainRegistry
}
}

consenters["solo"] = solo.New()
Expand Down