Testing saivs using SAI Challanger On x86 with sonicvs(SONiC BuildImage from community SONiC) as well as sonic-vs enterprise sonic from Broadcom

[hasan-alkama]

I am trying to test saivs with sai challanger I followed this step

1. I have sonic-vs setup on my x86 server
2. running sonic with default account admin/YourPaSsWoRd
   followed the steps to build and sai challanger from clonning it on github
   https://github.com/opencomputeproject/SAI-Challenger/blob/main/docs/sonic_usecase.md#prepare-testbed-desciption-file
   setup the ip of eth0 as 10.11.12.13/24 and replaced into testbed getting this error in the both sonicvs
   `
   **70 skipped, 11 warnings, 1177 errors in 19.85s =======================================================================================
   ERROR: "docker exec ${TTY} ${CONTAINER} bash -c "${EXEC_CMD}"" command filed with exit code 1.

===========================================
SAI Challenger run options

Docker image type : client
Container name : sc-client-run
Command : pytest -v --testbed=saivs_standalone_vs

===========================================**
`
Is anyone used it earlier?

[andriy-kokhan]

Please provide your testbed description file, exact steps that were used and full output of the error that you see. I'd suggest to start with a single TC execution. E.g., run pytest with -k "access_to_access" option.

[hasan-alkama]

The testbed file that i am using
{
"npu": [
{
"alias": "sonic-1",
"asic": "generic",
"target": null,
"sku": null,
"client": {
"type": "redis",
"config": {
"mode": "sonic",
"username": "admin",
"password": "YourPaSsWoRd",
"ip": "10.11.12.13",
"port": "6379",
"loglevel": "NOTICE"
}
}
}
]
}
steps are
1.first i checked wheather the sonic-swss is running from last 3min and early by checking the status
2.configuring the eth0 management port with and ip addrss in my case it is 10.11.12.13/24
3.running the sai challanger (./run.sh -i client)
4.executing the testcase (./exec.sh -i client pytest -v --testbed=saivs_sonic)
this time i tried with -k"access_to_access" here is the output on console
https://gist.github.com/hasan-alkama/1e759527328e81b4f804860ed21a78a8

[andriy-kokhan]

From the logs I clearly see that SAI-C is not able to SSH to SONiC:

/usr/local/lib/python3.7/dist-packages/saichallenger/common/sai_dut.py:32: in _connect
    self.ssh.connect(self.server_ip, username=self.username, password=self.password)
/usr/lib/python3/dist-packages/paramiko/client.py:343: in connect
    retry_on_signal(lambda: sock.connect(addr))
/usr/lib/python3/dist-packages/paramiko/util.py:280: in retry_on_signal
    return function()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   retry_on_signal(lambda: sock.connect(addr))
E   TimeoutError: [Errno 110] Connection timed out

From the IP address 10.11.12.13 I make a conclusion that it was configured manually but not through DHCP. Also, I suspect that you are not able to ping this address either from the host where you run this SONiC VM or from SAI-C client docker. So, no chances SAI-C can do its job...

The simplest way to make it work is to run SONiC VS under GNS3. Connect standard GNS3 NAT appliance to eth0 of SONiC VS. As a result, SONiC VS will get mgmt IP dynamically which will be NAT'ed to your host PC. In such case you should be able to SSH to your SONiC VS from your host PC. This will make SAI-C happy as well ;)