No more heartbeat after grouping

[innocentliny]

Hi,

We deploy service to AWS VPC using:

• JGroups v5.3.1
• tcp-nio
• Docker compose
  • open 7800:7800 and 7900:7900

Below is the protocol stack:

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:org:jgroups"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd"
>
    <TCP_NIO2 bind_addr="${jgroups.bind_addr:site_local}"
              bind_port="${jgroups.bind_port:7800}"
              external_addr="${jgroups.external_addr}"
              external_port="${jgroups.external_port}"
              use_virtual_threads="true"
              thread_pool.min_threads="0"
              thread_pool.max_threads="200"
              thread_pool.keep_alive_time="30000"/>
    <RED/>
    <TCPPING async_discovery="${jgroups.async_discovery:true}"
             initial_hosts="${jgroups.tcpping.initial_hosts:localhost[7800]}"
             return_entire_cache="${jgroups.tcpping.return_entire_cache:true}"
             port_range="${jgroups.tcp.port_range:0}"/>
    <MERGE3/>
    <FD_SOCK2 bind_addr="${jgroups.bind_addr:site_local}"/>
    <FD_ALL3 timeout="30000" />
    <VERIFY_SUSPECT2 timeout="1500"  />
    <BARRIER />
    <pbcast.NAKACK2 use_mcast_xmit="false"/>
    <UNICAST3 />
    <pbcast.STABLE desired_avg_gossip="50000"
                   max_bytes="4M"/>
    <pbcast.GMS print_local_addr="true"
                log_collect_msgs="true"
                join_timeout="2000"/>
    <UFC max_credits="2M"
         min_threshold="0.4"/>
    <MFC max_credits="2M"
         min_threshold="0.4"/>
    <FRAG2 frag_size="60K"  />
    <!--RSVP resend_interval="2000" timeout="10000"/-->
    <pbcast.STATE_TRANSFER/>
</config>

At the beginning, 3 services can form a group, but after 30 seconds, member left one by one, we don't know why.

Below are logs from one of 3 services:

2024-01-23 13:30:56,698 DEBUG [main] TCP_NIO2 - thread pool min/max/keep-alive (ms): 0/200/30000
2024-01-23 13:30:56,700 DEBUG [main] Configurator - set property org.jgroups.stack....@466a5b80.mcast_addr to default value /[224.0.75.75](http://224.0.75.75/)
2024-01-23 13:30:56,708 INFO [main] JChannel - local_addr: d3140d62-8396-49d5-b4ed-0782c53d9d6f, name: my_backstage_api02-56702
2024-01-23 13:30:56,715 INFO [main] FD_SOCK2 - server listening on /[172.20.0.6:7900](http://172.20.0.6:7900/)
2024-01-23 13:30:56,751 DEBUG [main] GMS - my_backstage_api02-56702: sending JOIN(my_backstage_api02-56702) to my_backstage_api01-36270
2024-01-23 13:30:56,829 DEBUG [main] NAKACK2 -
[my_backstage_api02-56702 setDigest()]
existing digest:  []
new digest:       my_backstage_api01-36270: [59 (59)], my_backstage_api03-32603: [0 (0)], my_backstage_api02-56702: [0 (0)]
resulting digest: my_backstage_api01-36270: [59 (59)], my_backstage_api03-32603: [0 (0)], my_backstage_api02-56702: [0 (0)]
2024-01-23 13:30:56,830 DEBUG [main] GMS - my_backstage_api02-56702: installing view [my_backstage_api01-36270|56] (3) [my_backstage_api01-36270, my_backstage_api03-32603, my_backstage_api02-56702] ([my_backstage_api01-36270, my_backstage_api03-32603, my_backstage_api02-56702] joined)
2024-01-23 13:30:56,844 DEBUG [main] FD_SOCK2 - my_backstage_api02-56702: trying to connect to my_backstage_api01-36270
2024-01-23 13:30:56,852 DEBUG [main] FD_SOCK2 - my_backstage_api02-56702: connected successfully to my_backstage_api01-36270 ([10.88.1.233:7900](http://10.88.1.233:7900/)) in 8 ms
2024-01-23 13:30:56,856 INFO [main] MyService - I am member (myAddress=my_backstage_api02-56702; 3 members=[my_backstage_api01-36270, my_backstage_api03-32603, my_backstage_api02-56702]).
2024-01-23 13:31:36,854 DEBUG [Timer runner-2,my-backstage,my_backstage_api02-56702] FD_ALL3 - my_backstage_api02-56702: haven't received a heartbeat from my_backstage_api03-32603 in timeout period (30000 ms), adding it to suspect list
2024-01-23 13:31:44,855 DEBUG [Timer runner-2,my-backstage,my_backstage_api02-56702] FD_ALL3 - my_backstage_api02-56702: haven't received a heartbeat from my_backstage_api03-32603 in timeout period (30000 ms), adding it to suspect list
2024-01-23 13:31:46,078 DEBUG [NioConnection.Reader [10.88.1.233:7900]-23,my-backstage,my_backstage_api02-56702] FD_SOCK2 - my_backstage_api02-56702: connection to my_backstage_api01-36270 closed
2024-01-23 13:31:46,078 DEBUG [NioConnection.Reader [10.88.1.233:7900]-23,my-backstage,my_backstage_api02-56702] FD_SOCK2 - my_backstage_api02-56702: trying to connect to my_backstage_api03-32603
2024-01-23 13:31:46,080 DEBUG [NioConnection.Reader [10.88.1.233:7900]-23,my-backstage,my_backstage_api02-56702] FD_SOCK2 - my_backstage_api02-56702: broadcasting suspect([my_backstage_api01-36270, my_backstage_api03-32603])
2024-01-23 13:31:46,080 DEBUG [jgroups-12,my-backstage,my_backstage_api02-56702] FD_SOCK2 - my_backstage_api02-56702: suspecting [my_backstage_api01-36270, my_backstage_api03-32603]
2024-01-23 13:31:47,583 DEBUG [VERIFY_SUSPECT2.Runner-1] VERIFY_SUSPECT2 - my_backstage_api02-56702: sending up SUSPECT([my_backstage_api01-36270, my_backstage_api03-32603])
2024-01-23 13:31:47,584 DEBUG [VERIFY_SUSPECT2.Runner-1] GMS - my_backstage_api02-56702: members are (3) my_backstage_api01-36270,my_backstage_api03-32603,my_backstage_api02-56702, coord=my_backstage_api02-56702: I'm the new coordinator
2024-01-23 13:31:47,584 DEBUG [VERIFY_SUSPECT2.Runner-1] STABLE - my_backstage_api02-56702: resuming message garbage collection
2024-01-23 13:31:47,588 DEBUG [VERIFY_SUSPECT2.Runner-1] GMS - my_backstage_api02-56702: installing view [my_backstage_api02-56702|57] (1) [my_backstage_api02-56702] ([my_backstage_api01-36270, my_backstage_api03-32603] left)
2024-01-23 13:31:47,588 DEBUG [VERIFY_SUSPECT2.Runner-1] STABLE - my_backstage_api02-56702: resuming message garbage collection
2024-01-23 13:31:47,588 DEBUG [VERIFY_SUSPECT2.Runner-1] NAKACK2 - my_backstage_api02-56702: removed my_backstage_api01-36270 from xmit_table (not member anymore)
2024-01-23 13:31:47,588 DEBUG [VERIFY_SUSPECT2.Runner-1] NAKACK2 - my_backstage_api02-56702: removed my_backstage_api03-32603 from xmit_table (not member anymore)
2024-01-23 13:31:47,589 INFO [VERIFY_SUSPECT2.Runner-1] MyService - I am leader (myAddress=my_backstage_api02-56702; 1 members=[my_backstage_api02-56702]).

Should we open tcp ports 7800-7900:7800-7900 9000-9100:9000-9100 in docker compose file?
And "Configurator - set property org.jgroups.stack....@466a5b80.mcast_addr to default value /224.0.75.75" in log, should we do anything about 224.0.75.75?

Would you please give us some hints?
Let us know if any.

Thank you very much.

[innocentliny]

Hi:

We found that 2 of 3 services didn't get correct physical address via jgroups.bind_addr.
And we solved it by setting jgroups.bind_addr to a network interface, like -Djgroups.bind_addr=match-interface:eth0.

Thank you for your time.