Chan.Part.API: reduce flakiness after removing system channel

[wlahti]

Type of change

• Test update

Description

Perform the transaction submission for the application channel that existed before removing the system channel at the end of the test. This gives that channel more time to find a raft leader before we check that it's functioning properly. This seems to reduce the flakiness but it still needs work.

Related issues

FAB-18305

[wlahti]

This isn't a fix yet but it does reduce the frequency of the flake. The issue is we don't have a reliable way to ensure a Raft leader has been elected for the channel participation tests. The existing findLeader helper works with a single channel and when that channel stays around. With the ability to remove channels using the channel participation API, especially the system channel which results in stopping/restarting all of the channels and a new wave of leader elections, we can't reliably use that helper and need some other way to verify a leader has been elected to fully avoid this (and similar) flakes.

For the record, this has run successfully three times so far. Going to try one more and then see if we can get this in.

[wlahti]

Seems like part of the flakiness is around orderer3 sometimes not realizing its been evicted from the channel (removed from the consenter set) in time. Currently seeing if reducing the EvictionSuspicion from 10s to 5s helps there.

Rolling success count: 5

[wlahti]

Alright, the test itself worked 5 times in a row with the EvictionSuspicion config change. Going to clean this up and move it to review. This still isn't the total long term fix but it should greatly reduce the frequency of the flakes in this test for now.