several problems in hotstuff run

[SenjiMuramasa]

Hi, I recently started to learn about Hotstuff related content, and I tried to run your code. But I ran into a few problems running the code. I will show them in the pictures. I didn't make any changes when I ran the code, I just did the following steps:
1 make
2 ./hotstuff run
But the logs seem to indicate something unusual. Do these logs represent any problems in my operation? Or are these logs normal?
Thank you very much.

[johningve]

Hi!

None of these log messages indicate any serious issues. It's likely that they were all caused by messages arriving in a different order than expected. That can happen often, especially in a low-latency network.

I'll take a closer look at the sources of these log messages and consider if they should be reclassified as "debug" messages.

[SenjiMuramasa]

Thank you for your answer!
I am trying to do some experiments with your code, so I am wondering if you tested the performance of the code?
In addition, I see some content about fork in byzantine.go. Can fork be used directly? And how to use it?
I am new to hotstuff and Go, so my question may be low-level. I am very sorry to bother you.
Thank you very much.

[johningve]

I am trying to do some experiments with your code, so I am wondering if you tested the performance of the code?

We did some comparisons with libhotstuff a long time ago, but I haven't looked at the performance in a while.

In addition, I see some content about fork in byzantine.go. Can fork be used directly? And how to use it?

Uhh, I just noticed that it doesn't work on master. 😅I've pushed a commit to fix it!
You can enable it using the byzantine flag:

./hotstuff run --byzantine="fork:1"

The example above enables the "fork" behavior for a single replica. You could combine several byzantine behaviors also:

./hotstuff run --byzantine="fork:1,silence:2"

This would enable the "fork" behavior for one replica, and the "silence" behavior for two replicas.

[SenjiMuramasa]

Thank you very much!

I have tried using a fork attack but the code is stuck.

So I wonder if this is a sign of a success attack? Should this trigger the timeout system to go to the next view?

And, I want to try using Batch to improve the system throughput. I changed batch-size in the run.go file, but the result was 0.


Are there any other configuration changes that need to be made?

Again, thank you for your patient reply.

[johningve]

It looks like the default setting for max-concurrent, which controls the number of concurrent commands a client can issue, is too low for a fork scenario. This setting must be changed if you change the batch size. It should be at least 4 times the batch size if you are using a single client.

Going back to the fork attack, by increasing max-concurrent from 4 (the default) to 10, I get this result:

executed: 4900, failed: 1634

The failed commands shows that the fork attack was successful.

Now, you can compare with the fasthotstuff implementation:

./hotstuff run --consensus="fasthotstuff" --byzantine="fork:1" --max-concurrent=10 --timeout-multiplier=1

(setting --timeout-multiplier to 1 here stops the view-synchronizer from increasing the view duration when a timeout occurs)

This gives me the result: executed: 1510, failed: 0, showing that the fasthotstuff implementation is indeed resistant to forking attacks :)

[SenjiMuramasa]

Thank you! It helps a lot.
After increasing 'max-concurrent' from 4 to 10, the code started to run. But it seems the code can only run for a short time. I tried to continue increasing the 'max-concurrent' and it worked a little better. But it still doesn't run to the end. (I increased 'duration' to 60 seconds)

I'm using a virtual machine to run the code on Linux21. Is that the cause of the problem?

[SenjiMuramasa]

Hi, I tried to run the code many times, sometimes the code would run longer, and sometimes it would just get stuck.
According to the number of final executions and failures, the attack was successful.

These two images represent two typical cases (duration=10s, max-concurrent=10).
The first one stuck as soon as it started running.

The second one ran almost to the end of the duration.

Do you have this kind of problem? Or was it just me, for some other reason?

[johningve]

Sorry for the slow response. It looks to me like this is the client getting stuck due to some unacknowledged commands (or maybe unordered?). I am not quite sure why it happens. To remedy this, I have added a timeout to the client that is configurable through the --client-timeout flag (defaults to 500ms).

[SenjiMuramasa]

Thank you for your patience. It always takes time to deal with problems.
In my test, I found that the code kept returning from the consensus.go file as shown in the image below when it was stuck using a fork attack. In the Get() function, Select goes all the way to case <- ctx.Done().


My Go language level is not good enough to solve this problem. I just hope this little discovery will help.

Thanks again for your reply. You really helped me a lot.

In addition, I just started to use Github, so I did not deal with some contents that needed to be bold or marked properly. Please forgive me.

[SenjiMuramasa]

Hello, here I go again~~
I tried overwriting two blocks with a fork attack, but it seems the malicious block didn't get into the chain. Where can I modify so that the malicious blocks can be saved in the chain?
Thank you!

[johningve]

It's hard for me to tell exactly why the malicious block is not accepted without any more detail, but here is how it normally works:

1. One of the replicas creates the new block and proposes it to the other replicas. This is usually done by the Consensus.Propose method. However, the ProposeRuler interface allows the behavior of this to be changed, which is how the byzantine package works.
2. All replicas then process the block in the OnPropose function. This function verifies the block's quorum certificate, checks the identity of the proposer, and then checks the VoteRule of the consensus protocol. If all these checks pass, the command(s) stored in the block are inspected by the Acceptor module. If the Acceptor module accepts the command(s), then the block is accepted. Currently, the Acceptor module checks the serial numbers of commands to ensure that only new commands are accepted. This is implemented in the file replica/cmdcache.go
3. Depending on the consensus protocol, the block will be committed into the chain when its grandchild or great-grandchild is proposed (assuming that all blocks in between are accepted too).

PS: I'm converting this issue into a discussion, as I think that is more appropriate ;)

[SenjiMuramasa]

So, when I fork, does the number of failures in the picture below include malicious blocks?
Emmm, I mean, when I use the fork attack directly, without any modification, will the malicious block get into the chain successfully? Or will the malicious block be detected and classified into failure?

[johningve]

I'll take another look at it later, but if I recall correctly, the answer depends on which consensus protocol is used:

• The default, chainedhotstuff, as well as simplehotstuff, do not detect the malicious blocks. Thus, the failed commands come from the legitimate blocks that were not committed due to the fork.
• fasthotstuff does detect the malicious blocks and rejects them. Thus, there are no failed commands when using this protocol.

[SenjiMuramasa]

Hello~
As mentioned in the readme, a tool is provided to run experiments on multiple servers over SSH. What is this tool, please? How do I use it?
Thank you~

[johningve]

It comes as part of the main hotstuff binary. Please take a look at the documentation on how to set it up.

[SenjiMuramasa]

Hi~
I generated id_rsa file and known_hosts file by ssh, and I also modified ssh_config file.
Does the example_config.toml file needs to be modified? And are there any other files that need to be modified?
What's more, do I still run ./hotstuff run when I want to run experiments on multiple machines?
Thank you very much.

[SenjiMuramasa]

I encountered an error when I tried to run the experiment on a remote server as shown in the figure below. I tried many solutions I found on the Internet, but none of them solved the problem. Have you ever encountered this problem? If so, could you please tell me how to solve it? Thank you very much!

[johningve]

Maybe you can add a HostName field in the ssh config to specify the IP address?

For example:

Host test
    HostName 127.0.0.1
    Port 1234
    User foo
    ...

[SenjiMuramasa]

Hi, recently I have been trying to do multi-machine experiments, but I have experienced many problems and failed to run successfully. Have you ever run multi-machine experiments? Could you please tell me the steps in detail? I would really appreciate it if you could.

[meling]

Yes, of course we have and are currently doing multi-machine experiments. We are quite busy at the moment and we may not be able to assist with these things now. Hopefully after our deadline, we will be able to provide more assistance.

However, to give you some directions for you to try... Are you able to log into your configured machines via password-less ssh login? This is a requirement for running experiments. If you don't understand what this means, please consult the internet for information.

[SenjiMuramasa]

Thank you very much. You have given me a lot of help. Please feel free to focus on your experiments. I'll list the steps I tried below, and you can help me check what might go wrong when you have a spare moment. Thank you.

1. I used ssh-keygen -t rsa to generate an id_rsa file and an id_rsa.pub file.
2. Then I moved the id_rsa.pub file to the /home/user/.ssh folder on the other computer by scp.
3. Copy the id_rsa.pub file to the authorized_keys file. In this case, two computers can be connected to each other without passwords through ssh.
4. Modify the ssh_config file in the scripts folder. The format is as follows:

Host 192.168.xxx.xxx hotstuff_worker_1
           User abc
           Port 22
           IdentityFile ~/.ssh/id
           UserKnownHostsFile ~/.ssh/known_hosts

5. Change hosts in the run.go file to the IP address of another computer.

Finally, I tried to run it and get the result shown below.

[johningve]

Oh, that's an interesting problem! It looks like the machines you are deploying to have an older version of glibc than the one you are deploying from.

[johningve]

To work around this issue, I think you would have to compile hotstuff on one of the target machines, or find some way to build the binary with an older version of glibc. Either way, you can then tell hotstuff to upload a different binary to the remote machines by using the --exe flag.

[johningve]

You could also try to compile hotstuff without cgo. Try something like this

env CGO_ENABLED=0 go build -o ./hotstuff ./cmd/hotstuff

[SenjiMuramasa]

Thank you for taking time out of your busy schedule to reply. I have checked according to your suggestion, and it is true that the target computer has a lower version of the GLIBC that causes the problem. Then I tried two things.

1. I tried to compile hotstuff with env CGO_ENABLED=0 go build -o ./hotstuff ./cmd/hotstuff.
2. I tried to compile hotstuff on the target computer.
   Both attempts yielded the same results.
   

There appears to be no error, but the application freezes after replicas are being created without any hint.
Does this indicate that there are other problems besides the GLIBC version?

[SenjiMuramasa]

Maybe there are some problems in my ssh_config file?

The path of id_rsa file and known_hosts file have been checked, and there are no mistakes.

[SenjiMuramasa]

Hi, I found the following error when I tried to run experiments with more replicas (e.g. 36 replicas on one computer).

However, when I tried to run the experiment within ”vscode“, it worked.

What are the possible reasons for this? Is there a way to run this experiment directly?