dhcp_relay test is not stable

[lguohan]

Description

Steps to reproduce the issue:

1. check the vs image test.
   https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image-test/263/

Describe the results you received:

Describe the results you expected:

Additional information you deem important:

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[tahmed-dev]

It looks like Discover packets were not received by the ptf docker test script:

AssertionError: Failed: Discover count of 0 != 4

I also noticed in another build, that default route test cases were failing.

            # parse set_src
>           m = re.match(r"^default proto (bgp|186) src (\S+)", rt[0])
E           IndexError: list index out of range

common/devices.py:482: IndexError

[lguohan]

I think somehow, can we check the default route in the dhcp relay test, to make sure the default is there? otherwise, it is a little bit to troubleshoot the issue.

[lguohan]

for that build, it seems the dut is not receiving all the route from the neighbor, I do not know how this happens, need to check the bgp summary, but "show bgp summary" was not correct for that build, so I fixed the test to use "show ip bgp summary" to check.

[tahmed-dev]

I just updated my setup (6/4) and the test is passing on my setup. Will add fixture to verify routes exist from DUT for DHCP servers. Also, there is internal error where vlan membership was not reflected upon DUT running config. This caused the Discover packets to not be received. The sanity check should probably catch such testbed setup issues.

[lguohan]

here, it looks like the test case failed after switching the topology to two vlans.

https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image-pr/4173/consoleFull

[lguohan]

as you can see, the there is no default route advertise to the dut after switching the topology to 2vlan.

https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image-pr/4173/artifact/sonic-mgmt/tests/logs/2vlans/dhcp_relay/test_dhcp_relay.log

"Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd\tNeighborName", 
"10.0.0.57       4      64600      63      61        0    0    0 00:02:54            1\tARISTA01T1", 
"10.0.0.59       4      64600      61      60        0    0    0 00:02:49            1\tARISTA02T1", 
"10.0.0.61       4      64600      62      61        0    0    0 00:02:54            1\tARISTA03T1", 
"10.0.0.63       4      64600      62      61        0    0    0 00:02:54            1\tARISTA04T1", 

[lguohan]

somehow, the default routes are removed between these two tests. there is no topology change. I suspect the ptf docker got removed, maybe in the background?

https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image-pr/4173/artifact/sonic-mgmt/tests/logs/1vlan/snmp/test_snmp_cpu.log

https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image-pr/4173/artifact/sonic-mgmt/tests/logs/1vlan/snmp/test_snmp_interfaces.log

[tahmed-dev]

Yes, I am not sure if there was some infra work going behind the scene. I do now see last two build passing.

Just a quick note, that this error is different in nature from the error first reported as in the latter one, only the first DHCP relay test cases failed and subsequent ones did pass.

[lguohan]

I think it some infra work going behaind, but I haven't figure out. Likely the ptf docker got updated, or removed? but we have lock to prevent that.

[lguohan]

stabilized the infra part. now the test become stabler. close for now. will open issue if we see later.