Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ixia-c tests can fail if protobuf already installed with incompatible version #153

Closed
chrispsommers opened this issue Jul 16, 2022 · 0 comments · Fixed by #154
Closed

ixia-c tests can fail if protobuf already installed with incompatible version #153

chrispsommers opened this issue Jul 16, 2022 · 0 comments · Fixed by #154
Assignees

Comments

@chrispsommers
Copy link
Collaborator

Use of snappi package 0.7.37 as specified in https://github.com/Azure/DASH/blob/main/test/requirements.txt could cause snappi client errors if an incompatible version of protobuf python package is already installed.

Per ixia-c support Slack channel (https://ixia-c.slack.com/archives/C021DU5026R/p1657894469742489?thread_ts=1657847300.692199&cid=C021DU5026R), this can be fixed by upgrading to snappi 0.7.38. I will file a PR to do so.

An example error output is show below:

dash@chris-z4:~/chris-dash/DASH/dash-pipeline$ make run-all-tests 
# Ensure P4Runtime server is listening
t=5; \
while [ ${t} -ge 1 ]; do \
	if sudo lsof -i:9559 | grep LISTEN >/dev/null; then \
		break; \
	else \
		sleep 1; \
		t=`expr $t - 1`; \
	fi; \
done; \
docker exec -w /tests/vnet_out simple_switch-dash ./vnet_out
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
GRPC call Write::add_one_entry OK: GRPC call Write::add_one_entry OK: GRPC call Write::add_one_entry OK: Done.
python3 -m pip install -r ../test/requirements.txt
Requirement already satisfied: snappi==0.7.37 in /home/dash/.local/lib/python3.8/site-packages (from -r ../test/requirements.txt (line 1)) (0.7.37)
Requirement already satisfied: pytest==6.0.1 in /home/dash/.local/lib/python3.8/site-packages (from -r ../test/requirements.txt (line 2)) (6.0.1)
Requirement already satisfied: urllib3 in /usr/lib/python3/dist-packages (from snappi==0.7.37->-r ../test/requirements.txt (line 1)) (1.25.8)
Requirement already satisfied: grpcio-tools==1.44.0; python_version > "2.7" in /home/dash/.local/lib/python3.8/site-packages (from snappi==0.7.37->-r ../test/requirements.txt (line 1)) (1.44.0)
Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from snappi==0.7.37->-r ../test/requirements.txt (line 1)) (5.3.1)
Requirement already satisfied: grpcio==1.44.0; python_version > "2.7" in /home/dash/.local/lib/python3.8/site-packages (from snappi==0.7.37->-r ../test/requirements.txt (line 1)) (1.44.0)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from snappi==0.7.37->-r ../test/requirements.txt (line 1)) (2.22.0)
Requirement already satisfied: pluggy<1.0,>=0.12 in /home/dash/.local/lib/python3.8/site-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (0.13.1)
Requirement already satisfied: more-itertools>=4.0.0 in /usr/lib/python3/dist-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (4.2.0)
Requirement already satisfied: toml in /home/dash/.local/lib/python3.8/site-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (0.10.2)
Requirement already satisfied: attrs>=17.4.0 in /usr/lib/python3/dist-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (19.3.0)
Requirement already satisfied: py>=1.8.2 in /home/dash/.local/lib/python3.8/site-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (1.11.0)
Requirement already satisfied: packaging in /home/dash/.local/lib/python3.8/site-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (21.3)
Requirement already satisfied: iniconfig in /home/dash/.local/lib/python3.8/site-packages (from pytest==6.0.1->-r ../test/requirements.txt (line 2)) (1.1.1)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from grpcio-tools==1.44.0; python_version > "2.7"->snappi==0.7.37->-r ../test/requirements.txt (line 1)) (45.2.0)
Requirement already satisfied: protobuf<4.0dev,>=3.5.0.post1 in /usr/local/lib/python3.8/dist-packages (from grpcio-tools==1.44.0; python_version > "2.7"->snappi==0.7.37->-r ../test/requirements.txt (line 1)) (3.6.1)
Requirement already satisfied: six>=1.5.2 in /usr/lib/python3/dist-packages (from grpcio==1.44.0; python_version > "2.7"->snappi==0.7.37->-r ../test/requirements.txt (line 1)) (1.14.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/dash/.local/lib/python3.8/site-packages (from packaging->pytest==6.0.1->-r ../test/requirements.txt (line 2)) (3.0.9)
cd ../test/third-party/traffic_gen && ./deploy_ixiac.sh
.
deployment_traffic_engine_1_1 is up-to-date
deployment_controller_1 is up-to-date
deployment_traffic_engine_2_1 is up-to-date
# Ensure P4Runtime server is listening
t=5; \
while [ ${t} -ge 1 ]; do \
	if sudo lsof -i:9559 | grep LISTEN >/dev/null; then \
		break; \
	else \
		sleep 1; \
		t=`expr $t - 1`; \
	fi; \
done; \
docker exec -w /tests/init_switch simple_switch-dash ./init_switch
GRPC call SetForwardingPipelineConfig 0.0.0.0:9559 => /etc/dash/dash_pipeline.json, /etc/dash/dash_pipeline_p4rt.txt
Switch is initialized.
python3 -m pytest ../test/test-cases/bmv2_model/ -s
================================================================================================== test session starts ===================================================================================================
platform linux -- Python 3.8.10, pytest-6.0.1, py-1.11.0, pluggy-0.13.1
rootdir: /home/dash/chris-dash/DASH
collected 0 items / 1 error                                                                                                                                                                                              

========================================================================================================= ERRORS =========================================================================================================
____________________________________________________________________________ ERROR collecting test/test-cases/bmv2_model/test_hello_world.py _____________________________________________________________________________
../../../.local/lib/python3.8/site-packages/snappi/otg_pb2_grpc.py:7: in <module>
    import otg_pb2 as otg__pb2
E   ModuleNotFoundError: No module named 'otg_pb2'

During handling of the above exception, another exception occurred:
../test/test-cases/bmv2_model/test_hello_world.py:1: in <module>
    import snappi
../../../.local/lib/python3.8/site-packages/snappi/__init__.py:1: in <module>
    from .snappi import Config
../../../.local/lib/python3.8/site-packages/snappi/snappi.py:17: in <module>
    from snappi import otg_pb2_grpc as pb2_grpc
../../../.local/lib/python3.8/site-packages/snappi/otg_pb2_grpc.py:9: in <module>
    from snappi import otg_pb2 as otg__pb2
../../../.local/lib/python3.8/site-packages/snappi/otg_pb2.py:23: in <module>
    _CONFIG = DESCRIPTOR.message_types_by_name['Config']
E   AttributeError: 'NoneType' object has no attribute 'message_types_by_name'
==================================================================================================== warnings summary ====================================================================================================
/usr/local/lib/python3.8/dist-packages/google/protobuf/internal/containers.py:182
  /usr/local/lib/python3.8/dist-packages/google/protobuf/internal/containers.py:182: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    MutableMapping = collections.MutableMapping

/usr/local/lib/python3.8/dist-packages/google/protobuf/internal/containers.py:340
  /usr/local/lib/python3.8/dist-packages/google/protobuf/internal/containers.py:340: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
    collections.MutableSequence.register(BaseContainer)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
================================================================================================ short test summary info =================================================================================================
ERROR ../test/test-cases/bmv2_model/test_hello_world.py - AttributeError: 'NoneType' object has no attribute 'message_types_by_name'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================== 2 warnings, 1 error in 0.98s ==============================================================================================
make: *** [Makefile:355: run-ixiac-test] Error 2
@chrispsommers chrispsommers self-assigned this Jul 16, 2022
@chrispsommers chrispsommers changed the title ixia-c tests can fail if protobuf already installed with wrong version ixia-c tests can fail if protobuf already installed with incompatible version Jul 16, 2022
chrispsommers added a commit to chrispsommers/DASH that referenced this issue Jul 16, 2022
chrispsommers added a commit to chrispsommers/DASH that referenced this issue Jul 20, 2022
chrispsommers added a commit to chrispsommers/DASH that referenced this issue Jul 20, 2022
chrispsommers added a commit that referenced this issue Jul 20, 2022
* Update to snappi-0.7.38 for fix #153

* Add CI triggers for all .py, .sh, .yml and requirements.txt. under test/ dir

Co-authored-by: Chris Sommers <chrispsommers@gmail.com>
chrispsommers added a commit that referenced this issue Jul 21, 2022
* Build sai-thrift server (not working yet). New dockerfile.

* Removed extraneous step.

* README

* Move Dockerfile under dockerfiles/

* Pass flags to saithrift-build, expecting opencomputeproject/SAI#1514 to merge eventually. Meanwhile can locally modify SAI makefiles.

* Fix CI files for new location of Docekrfile.bmv2, add CI for dash-saithrift docker.

* Fix CI workflow for new dockerfiles.

* Add line to trigger CI.

* Restore CI filename, try to get it to trigger.

* Rename CI file; modify DOckerfile to trigger; add docker targets to Makefile.

* Modify dockerfiles to trigger CI.

* Fix make target names in CI and Makefile

* Update CI to trigger on CI file changes itself.

* Trigger on CI file change.

* Remove thrift source from dash-saithrift-bldr docker image. New tag 220625.

* Enhance saithrift docker to handle dash sai header generation and libsai compilation.

* Remove thrift source and source tarball from docker image; copy python dist tarball to /usr/lib inside container so can be extracted for deployment to a host.

* Experimenting with reduced Dockerfiles.

* Makefile: p4 source dependencies. dash_pipeline.p4: egress_spec=ingress_spec, comments.

* Fix stray merge conflict.

* Fix merge conflict in CI file.

* Optimized docker build and run images by creating a dedicated gRPC library image and selectively copying to BMV2 builder/runtime image based on available p4lang docker image.

* FIx CI, Makefile for new docker images.

* Optimize dockerfiles, reduced to 3 images totalling ~2.7G, down from ~10G.

* Fix CI files, add Make targets; cleanup.

* CI files, badge

* CI files.

* Split up READMEs into smaller files. Diagram.

* READMEs, diagram, SHA version of p4c

* Add missing CI workflow (p4c); update main CI workflow to trigger on any dockerfile change.

* Fix make command in CI file.

* Pin the SHA version of p4lang/behavioral-model:latest.

* Ping SHA versions of Ubuntu used in base images.

* READMEs - workflows etc.

* README typos, clarifications.

* README

* Use dev fork of SAI which passes flags to gensairpc.pl per opencomputeproject/SAI#1514.
Once merged, we can change back to upstream branch of SAI.

* Use fork of SAI with fixes for sairpcgen Make flags, and sairpcgen templates which add static_cast to avoid type mismatches due to sai extensions.

* Add placeholder code for SAI fixed functions, -fpermissive to avoid enum Long error. TODO: P4RT libs missing, otherwise sai-thrift server builds.

* Created init_switch test program and make target to initialize bmv2 model (P4RUntime SetForwardingPipelineRequest) to allow explicit switch intialization via P4Runtime. Removed sudo from most Make commands. READMEs.

* Remove dependency to avoid rebuilding files while running test in CI. Delete stray file.

* README

* Restore dash_pipeline.p4 to same as upstream. Had temporary changes (egress port = ingress port; comments).

* Add CI triggers for all .py, .sh, .yml and requirements.txt. under test/ dir

* Update to snappi-0.7.38 for fix #153

* Temporary workaround for container permissions issues per #143. Executes chmod as required. Permanent fix will require some Docker mods.

* Delete obxolete README, comment out WIP sai-thrift server make targets

Co-authored-by: Chris Sommers <chrispsommers@gmail.com>
KrisNey-MSFT pushed a commit that referenced this issue Aug 2, 2022
…164)

* Build sai-thrift server (not working yet). New dockerfile.

* Removed extraneous step.

* README

* Move Dockerfile under dockerfiles/

* Pass flags to saithrift-build, expecting opencomputeproject/SAI#1514 to merge eventually. Meanwhile can locally modify SAI makefiles.

* Fix CI files for new location of Docekrfile.bmv2, add CI for dash-saithrift docker.

* Fix CI workflow for new dockerfiles.

* Add line to trigger CI.

* Restore CI filename, try to get it to trigger.

* Rename CI file; modify DOckerfile to trigger; add docker targets to Makefile.

* Modify dockerfiles to trigger CI.

* Fix make target names in CI and Makefile

* Update CI to trigger on CI file changes itself.

* Trigger on CI file change.

* Remove thrift source from dash-saithrift-bldr docker image. New tag 220625.

* Enhance saithrift docker to handle dash sai header generation and libsai compilation.

* Remove thrift source and source tarball from docker image; copy python dist tarball to /usr/lib inside container so can be extracted for deployment to a host.

* Experimenting with reduced Dockerfiles.

* Makefile: p4 source dependencies. dash_pipeline.p4: egress_spec=ingress_spec, comments.

* Fix stray merge conflict.

* Fix merge conflict in CI file.

* Optimized docker build and run images by creating a dedicated gRPC library image and selectively copying to BMV2 builder/runtime image based on available p4lang docker image.

* FIx CI, Makefile for new docker images.

* Optimize dockerfiles, reduced to 3 images totalling ~2.7G, down from ~10G.

* Fix CI files, add Make targets; cleanup.

* CI files, badge

* CI files.

* Split up READMEs into smaller files. Diagram.

* READMEs, diagram, SHA version of p4c

* Add missing CI workflow (p4c); update main CI workflow to trigger on any dockerfile change.

* Fix make command in CI file.

* Pin the SHA version of p4lang/behavioral-model:latest.

* Ping SHA versions of Ubuntu used in base images.

* READMEs - workflows etc.

* README typos, clarifications.

* README

* Use dev fork of SAI which passes flags to gensairpc.pl per opencomputeproject/SAI#1514.
Once merged, we can change back to upstream branch of SAI.

* Use fork of SAI with fixes for sairpcgen Make flags, and sairpcgen templates which add static_cast to avoid type mismatches due to sai extensions.

* Add placeholder code for SAI fixed functions, -fpermissive to avoid enum Long error. TODO: P4RT libs missing, otherwise sai-thrift server builds.

* Progress on sai-thrift server, final saiserver linkage still has errors but was able to pass grpc/proto/pi libs into make.

* Created init_switch test program and make target to initialize bmv2 model (P4RUntime SetForwardingPipelineRequest) to allow explicit switch intialization via P4Runtime. Removed sudo from most Make commands. READMEs.

* Remove dependency to avoid rebuilding files while running test in CI. Delete stray file.

* README

* Build saithrift-bldr in Ubuntu-20.04 to avoid glibcc conflicts wth libgrpc etc. Modify sai-thrift Maefile to accomodate Python3 vs Python2.
TODO - saiserver builds but get errors in later stages of make saithrift-build target (ctypesgen...)

* Can build and run saithrift server. Fixed makefiles to call SAI makes with proper env vars etc., fixed install steps.

* Add sai-thrift-server to CI job.

* saithrift server starts up properly - added volume mounts to p4info

* Add DOCKER_FLAGS for sai targets

* Add sai-thrift-server CI test.

* Workaround for git repo ownership in CI.

* Correct trigger path for CI.

* Workaround for SAI submodule - fixed step name.

* Change dir into SAI/SAI to make safe directory

* Try safe directory from /dash

* Remove -u root from CI flags, remove git safe directory.

* ove lib dir creation to Makefile, remove from .pyu script - permission errors during CI.

* Add back -privileged flag

* ADd -u root to make sai in CI

* Change CI docker flags, use sudo for python script to overcome perms.

* Update SAI branch which fixes sairpcgen Makefiles (sai header include path).

* Remove experimental sudo from bash script, clean up overwrite options from python script (leave such things to makefiles.) Still struggling with perms issues in CI script.

* Try sudo in CI make sai step.

* Try -u root in CI file and git config --global --add safe.directory in Makefile

* Use "git config --global --add safe.directory /dash/dash-pipeline/SAI/SAI" per error message, even though it doesn't match actual path IMO.

* Try git config --global --add safe.directory '*' in Makefile.

* Temporarily skip make sai step, see if rest of pipeline runs.

* REstore CI script, try sudo workarounds in Makefile.

* fix chmod command

* Remove -u root, use sudo calling python script.

* fix chmod to a+rw and also commit bash script.

* chown and git safe dirs script attempt

* pass user to sudo

* export user

* FIx chown

* More wrappers and scripts to solve CI git ownership issues.

* REmove chown to try to fix "./checkenumlock.sh ...fatal: Unable to create '/dash/.git/modules/SAI/index.lock': Permission denied

* remove git safe dirs, getting lockfile error in CI

* REmove sudo calling python script; shotgun sudo for all CI steps.

* REstore sudo for py script; add -u root, --privileged to flags in CI

* Workaround, delete scripts in SAI/meta/Makefile which throw Git permission errors.

* Add sudo for sai-thrift-server target due to -u roots above. Got past git repo errors via previous commit, yay!

* Use sudo to avoid permissions problem in CI pipeline for SAI/rpc dir

* chown in sai-thrift Makefile to overcome CI perms issues.

* Change perms instead of chown in saithrift build.

* Add missing DOCKER_FLAGS to sai-thrift-server target

* Add DOCKER_FLAGS to CI step; remove obsolete wrapper scripts.

* Pass DOCKER_FLAGS to make target for sai-thrift-server test

* Use different SAI submodule branch (better name, synced to master). Add saithrift clean target to clean.

* Remove extraneous permissions workarounds.

* Update SAI submodule.

* Dockerfile for saithrift test clients.

* ADd CI for saithrift client docker

* Add prerequisite builds steps for sai-thrift-client. Update drawing.

* Build prerequisites first to make docker-saithrift-client

* Add missing requirements.txt, make targets.

* Progress with saithrift-client docker image, pytest, ptf test installation and skeletons.

* Refactor pytest to use common theift Client class; run sai client tests in CI. NOte - need sai_api_query impl to avoid sai thrift server crashes.

* Set docker flags for saithrift client test in CI

* Docs, diagrams. saithrift workflows.

* Use updated SAI dev branch to insert checks for NULL sai API funcs before calling them. Add simple test for switch attribute to prove thrift API working.

* Hand-modified sai_api_query to return SAI_API_DASH_VNET apis. Should be auto-generated.

* Use workaround for saithrift builder per opencomputeproject/SAI#1537. Reorganize nacent pytests to use fixtures to get saithrift client, put in directories, add custom markers, new README.

* mark vnet test x-fail for CI

* Update saithrift client docker image.

* Add empty README for saithrift/ptf

* Clean up make targets.

* Permissions fixes (manually, not merged). Add gdb to sai-thrift-server docker.

* Fix sai-thrift-server; got first RPC working (vnet). Slight refactoring of pytests (imports); fixed sai_api_query & made _impl structs non-static.

* Fix path for sai-thrift-server according to new -w path in Makefile

* Add CI triggers for all .py, .sh, .yml and requirements.txt. under test/ dir

* Update to snappi-0.7.38 for fix #153

* Temporary workaround for container permissions issues per #143. Executes chmod as required. Permanent fix will require some Docker mods.

* Update snappi to 0.7.38 in saithrift tests

* FIx dangling merge conflicts in README.

* s/sai-thrift/saithrift to for consistency.

* Delete sai_test which was accidentally copied from SAI repo.

* Fix #158 - libsai delete operation failure and log msg. THe test for error code was backwards and log message used same write updateType string for all operations. I fixed the compare and used a dynamic enum print method.

* Modify vnet pytest case to create/delete. README. Makefile polishing.

* Add more table create/deletes + cleanup to saithrift vnet test case

* Assign container names in ixia-c deployments (append username)

* Got preliminary PTF and Pytests including saithrift table accessors, packet echo tests. Removed ixia-c test as standalone test; now it runs as a pytest inside saithrift-client container so host tool installs are reduced (snappi, python, pip).

* Add back missing line to build saithrift server.

* Fix CI trigger, a prior merge messed this up.

* Add deploy-ixiac to Makefile deps and CI steps to support pytests.

* Refactor saithrift-client docker image into builder and runtime images; runtime client is always built locally, not pulled.
Change python requirements to hardlink to DASH/test/requirements.txt. Change DOckerfile name for consistency.

* Create tests/libsai dir and move "C++" tests there; fix typo in CI file.

* Defer deploying ixia-c until PTF tests complete to see if CI tests fail less often, suspect CI runner limits are being strained. Tests fail sometimes w/o reason.

* Moved saithrift tests under tests/ for consistency.

* FIx path, paste errors which run tests.

* Move requirements.txt to hardlink under tests (was under saithrift). Delete obs file.

* READMEs (saithrift test framework), Makefile (fix dev tests).

* READMEs - test workflows.

* Fix paste error - README

* READMEs - workflows. Diagram. Remove "make run-test," replaced by "make libsai-test."

* Fix CI action; READMEs polish.

* Minor README improvements; fix URL, diagrams (sirius->dash); improve Quick-start instructions.

* Add back URL.

* REmove python, pip install deps.

* READMEs; move saithrift test scripts into subdirs; change verify_packets() to verify_packet() to avoid failing due to junk entering veths from host network.

* Change CI job title Sirius->DASH

* Update SAI submodule to rescind changes to SAI include paths for sairpcgen, wasn't needed after all.

Co-authored-by: Chris Sommers <chrispsommers@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant