Skip to content

Commit

Permalink
[FAB-1658] Fix Deliver failures on BDD tests
Browse files Browse the repository at this point in the history 
https://jira.hyperledger.org/browse/FAB-1658

In FAB-1382 [1] the Deliver API was modified so as to work with a range.
The BDD implementation of Deliver was also modified in that changeset so
as to take an end block of `NEWEST`.

Occasionally, the Kafka orderer will fail on one of these BDD tests
because the Deliver request comes before all the incoming (via
Broadcast) messages have been processed. In that case, the `NEWEST`
value is translated to a lower-numbered block than the expected one.

As an example, consider the case where we send 100 messages to the
orderer, and each block has 10 messages. When we send the Deliver call,
Kafka has processed 90 messages, so it translates NEWEST to 9. Our test
expects NEWEST to be 10 and thus fails.

Until we find an elegant fix for this, I am adding a 1-second sleep
before the Deliver call is invoked on the Kafka cases. This should add
14 seconds to the duration of these tests.

I stress that this is a temporary patch, and I am open to a more elegant
solution, but until then, the default should not be a setup where the
orderer fails occasionally.

[1] https://gerrit.hyperledger.org/r/#/c/3271/

Change-Id: I77a091546e77981838a4b8ee29019178ddf0d49b
Signed-off-by: Kostas Christidis <kostas@christidis.io>
  • Loading branch information
@kchristidis
kchristidis committed Jan 15, 2017
1 parent ba288e1 commit 2c6dbcc
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 30 deletions.
63 changes: 33 additions & 30 deletions bddtests/features/orderer.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,29 +15,30 @@ Feature: Orderer
And I wait "<BootTime>" seconds
And user "binhn" is an authorized user of the ordering service
When user "binhn" broadcasts "<NumMsgsToBroadcast>" unique messages on "orderer0"
And user "binhn" waits "<WaitTime>" seconds
And user "binhn" connects to deliver function on "orderer0"
And user "binhn" sends deliver a seek request on "orderer0" with properties:
| Start | End |
| 1 | Newest |
Then user "binhn" should get a delivery from "orderer0" of "<ExpectedBlocks>" blocks with "<NumMsgsToBroadcast>" messages within "<BatchTimeout>" seconds

Examples: Solo Orderer
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| docker-compose-orderer-solo.yml | 20 | 2 | 10 | .5 |
| docker-compose-orderer-solo.yml | 40 | 4 | 10 | .5 |
| docker-compose-orderer-solo.yml | 60 | 6 | 10 | .5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| docker-compose-orderer-solo.yml | 20 | 2 | 10 | .5 | 0 |
| docker-compose-orderer-solo.yml | 40 | 4 | 10 | .5 | 0 |
| docker-compose-orderer-solo.yml | 60 | 6 | 10 | .5 | 0 |

Examples: 1 Kafka Orderer and 1 Kafka Broker
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| environments/orderer-1-kafka-1 | 20 | 2 | 10 | 5 |
| environments/orderer-1-kafka-1 | 40 | 4 | 10 | 5 |
| environments/orderer-1-kafka-1 | 60 | 6 | 10 | 5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| environments/orderer-1-kafka-1 | 20 | 2 | 10 | 5 | 1 |
| environments/orderer-1-kafka-1 | 40 | 4 | 10 | 5 | 1 |
| environments/orderer-1-kafka-1 | 60 | 6 | 10 | 5 | 1 |

Examples: 1 Kafka Orderer and 3 Kafka Brokers
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| environments/orderer-1-kafka-3 | 20 | 2 | 10 | 5 |
| environments/orderer-1-kafka-3 | 40 | 4 | 10 | 5 |
| environments/orderer-1-kafka-3 | 60 | 6 | 10 | 5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| environments/orderer-1-kafka-3 | 20 | 2 | 10 | 5 | 1 |
| environments/orderer-1-kafka-3 | 40 | 4 | 10 | 5 | 1 |
| environments/orderer-1-kafka-3 | 60 | 6 | 10 | 5 | 1 |

# @doNotDecompose
Scenario Outline: Basic seek orderer function (Utilizing properties for atomic broadcast)
Expand All @@ -46,6 +47,7 @@ Feature: Orderer
And I wait "<BootTime>" seconds
And user "binhn" is an authorized user of the ordering service
When user "binhn" broadcasts "<NumMsgsToBroadcast>" unique messages on "orderer0"
And user "binhn" waits "<WaitTime>" seconds
And user "binhn" connects to deliver function on "orderer0"
And user "binhn" sends deliver a seek request on "orderer0" with properties:
| Start | End |
Expand All @@ -57,22 +59,22 @@ Feature: Orderer
Then user "binhn" should get a delivery from "orderer0" of "<ExpectedBlocks>" blocks with "<NumMsgsToBroadcast>" messages within "1" seconds

Examples: Solo Orderer
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| docker-compose-orderer-solo.yml | 20 | 2 | 10 | .5 |
| docker-compose-orderer-solo.yml | 40 | 4 | 10 | .5 |
| docker-compose-orderer-solo.yml | 60 | 6 | 10 | .5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| docker-compose-orderer-solo.yml | 20 | 2 | 10 | .5 | 0 |
| docker-compose-orderer-solo.yml | 40 | 4 | 10 | .5 | 0 |
| docker-compose-orderer-solo.yml | 60 | 6 | 10 | .5 | 0 |

Examples: 1 Kafka Orderer and 1 Kafka Broker
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| environments/orderer-1-kafka-1 | 20 | 2 | 10 | 5 |
| environments/orderer-1-kafka-1 | 40 | 4 | 10 | 5 |
| environments/orderer-1-kafka-1 | 60 | 6 | 10 | 5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| environments/orderer-1-kafka-1 | 20 | 2 | 10 | 5 | 1 |
| environments/orderer-1-kafka-1 | 40 | 4 | 10 | 5 | 1 |
| environments/orderer-1-kafka-1 | 60 | 6 | 10 | 5 | 1 |

Examples: 1 Kafka Orderer and 3 Kafka Brokers
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| environments/orderer-1-kafka-3 | 20 | 2 | 10 | 5 |
| environments/orderer-1-kafka-3 | 40 | 4 | 10 | 5 |
| environments/orderer-1-kafka-3 | 60 | 6 | 10 | 5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| environments/orderer-1-kafka-3 | 20 | 2 | 10 | 5 | 1 |
| environments/orderer-1-kafka-3 | 40 | 4 | 10 | 5 | 1 |
| environments/orderer-1-kafka-3 | 60 | 6 | 10 | 5 | 1 |


# @doNotDecompose
Expand All @@ -82,20 +84,21 @@ Feature: Orderer
And I wait "<BootTime>" seconds
And user "binhn" is an authorized user of the ordering service
When user "binhn" broadcasts "<NumMsgsToBroadcast>" unique messages on "orderer0"
And user "binhn" waits "<WaitTime>" seconds
And user "binhn" connects to deliver function on "orderer0"
And user "binhn" sends deliver a seek request on "orderer0" with properties:
| Start | End |
| Oldest | 2 |
Then user "binhn" should get a delivery from "orderer0" of "<ExpectedBlocks>" blocks with "<NumMsgsToBroadcast>" messages within "<BatchTimeout>" seconds

Examples: Solo Orderer
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| docker-compose-orderer-solo.yml | 20 | 3 | 10 | .5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| docker-compose-orderer-solo.yml | 20 | 3 | 10 | .5 | 0 |

Examples: 1 Kafka Orderer and 1 Kafka Broker
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| environments/orderer-1-kafka-1 | 20 | 3 | 10 | 5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| environments/orderer-1-kafka-1 | 20 | 3 | 10 | 5 | 1 |

Examples: 1 Kafka Orderer and 3 Kafka Brokers
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime |
| environments/orderer-1-kafka-3 | 20 | 3 | 10 | 5 |
| ComposeFile | NumMsgsToBroadcast | ExpectedBlocks | BatchTimeout | BootTime | WaitTime |
| environments/orderer-1-kafka-3 | 20 | 3 | 10 | 5 | 1 |
6 changes: 6 additions & 0 deletions bddtests/steps/orderer_impl.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@

import os
import re
import time
import subprocess
import devops_pb2
import fabric_pb2
Expand Down Expand Up @@ -50,6 +51,11 @@ def step_impl(context, enrollId, composeService):
streamHelper = userRegistration.connectToDeliverFunction(context, composeService)


@when(u'user "{enrollId}" waits "{waitTime}" seconds')
def step_impl(context, enrollId, waitTime):
time.sleep(float(waitTime))


@then(u'user "{enrollId}" should get a delivery from "{composeService}" of "{expectedBlocks}" blocks with "{numMsgsToBroadcast}" messages within "{batchTimeout}" seconds')
def step_impl(context, enrollId, expectedBlocks, numMsgsToBroadcast, batchTimeout, composeService):
userRegistration = orderer_util.getUserRegistration(context, enrollId)
Expand Down

0 comments on commit 2c6dbcc

Please sign in to comment.