Pktgen for CI #148

kevindweb · 2019-06-24T02:17:07Z

Finally have Pktgen running for our Continuous Integration in the nimbus cluster!

Summary:

Long awaited, and tested recently in another PR, the default run mode is now to run Pktgen for our base performance testing. Right now here's the basic steps, ci sends our worker node to reboot, then fires the worker script. This script, depending on the worker-config, runs a certain mode (right now just pktgen or speed_test). If pktgen mode (MODE="0"), then we run a script that calls the Pktgen openNetVM-Scripts/run-pktgen.sh script in the other node through paramiko's SSHCLient. That uses our new lua script that sends packets for 30 seconds. We retrieve the data from basic monitor, send it back to CI for analysis. The new worker script also allows for multiple run modes, (MODE="0 1" for example). This way, we can potentially run pktgen then speed_tester and get all the results back to back after reboot.

Usage:

This PR includes
Resolves issues
Breaking API changes
Internal API changes
Usability improvements
Bug fixes
New functionality	👍
New NF/onvm_mgr args
Changes to starting NFs
Dependency updates
Web stats updates

Merging notes:

Dependencies: None
Please merge Ci busy no more #143 because I merged that code into this one, and all those commits are cluttering up this commit log as you see below (27 commits). There should only be like 3 commits for this PR so far.

TODO before merging :

PR is ready for review

Test Plan:

We have to figure out how we want to do different modes, from Github comment parsing or something. This way, we can stress test that this works. We should also test that this works on nn30 with nn33 (pktgen) so we know it's scalable to new nodes we want to use.

Review:

@koolzz @dennisafa

… develop

… ci_queue

… develop

kevindweb · 2019-06-24T02:21:10Z

@onvm you there now?

onvm · 2019-06-24T02:21:11Z

@onvm you there now?

CI Message

Your results will arrive shortly

kevindweb · 2019-08-03T16:48:52Z

@koolzz I added information about benchmarks to the README and changed the name of the helper script symlink

koolzz · 2019-08-04T19:19:44Z

@onvm
Olá

onvm · 2019-08-04T19:19:45Z

@onvm
Olá

CI Message

Your results will arrive shortly

koolzz

Just some small things, mostly everything lgtm

koolzz · 2019-08-04T18:05:33Z

ci/manager.sh

+    # put all files in one temporary folder for one scp
+    cp -r ./$worker_ip temp
+    cp -r ./repository temp/
+    cp -r ./worker_files/* temp/


Can we just copy all of these at once, will look cleaner cp -r ./$worker_ip ./repository temp/ ./worker_files/* temp/ k

koolzz · 2019-08-04T18:10:58Z

ci/worker_files/pktgen-worker.sh

+
+# run pktgen
+log "Collecting Pktgen Statistics"
+run_pktgen


run_pktgen doesn't tell us that it also fetches stats, also the function serves little purpose. Lets just extract the function here

koolzz · 2019-08-04T18:15:45Z

ci/worker_files/run-pktgen.py

+client.set_missing_host_key_policy(AutoAddPolicy())
+client.connect(worker_ip, timeout = 30, username=worker_user, pkey = key)
+
+(stdin, stdout, stderr) = client.exec_command("sudo ~/repository/tools/Pktgen/openNetVM-Scripts/run-pktgen.sh 1", get_pty=True)


Drop a comment saying get_pty is required so pktgen runs (I remember we had issues with this)

koolzz · 2019-08-04T19:03:11Z

ci/worker_files/worker-config

@@ -0,0 +1,8 @@
+WORKER_MODE="0"
+PKT_IFACE="p2p1"


Wait why do you have a different iface for pktgen and for mtcp? I'm pretty sure they can just share one

I had a hard time configuring the two interfaces separately. When one runs, it's ok, but configuring two interfaces is much easier than checking to see if the server was lying, before each check.

For now maybe we let it slide, but I'm not convinced that this is better. What's the topology you're using(which node is connected where), 2 NICs instead of one doesn't make much sense as it's using more resources

This is a tricky one, on the manager side (nimbnode17 for example) we only need 1 NIC, true, but on the client, we don't have to do anything if we can send Pktgen as one IP, and mTCP as another. This allows us to basically not touch the client server, because it's already configured. Your choice, but this makes it easy to we can run just mTCP, or just Pktgen, without any issues. This makes it faster, but I can definitely work a solution if necessary.

koolzz · 2019-08-04T19:12:12Z

ci/install_pktgen/install-pktgen.sh

+sudo rm -rf repository
+
+git clone https://github.com/sdnfv/openNetVM.git repository
+check_exit_code "ERROR: Failed installing onvm"


Failed cloning

koolzz · 2019-08-04T19:12:56Z

ci/install_pktgen/install-pktgen.sh

+
+cd repository
+
+print_header "Beginning Execution of Workload"


koolzz · 2019-08-04T19:15:26Z

ci/install_pktgen/pktgen-timed-config.lua

+    local idx = 0;
+    while( idx < waitTime ) do
+
+	-- Write port stats to output file separated by line


add 2 spaces so comment lines up and remove the newline above

onvm

@onvm
Olá

CI Message

Run successful see results:
✔️ PR submitted to develop branch
✔️ Pktgen performance check passed
✔️ Speed Test performance check passed
✔️ Linter passed

[Results from nimbnode17]

Median TX pps for Pktgen: 10205705
Performance rating - 102.06% (compared to 10000000 average)
Median TX pps for Speed Tester: 40185939
Performance rating - 100.46% (compared to 40000000 average)

…pktgen

kevindweb · 2019-08-06T00:04:34Z

@onvm did the changes mess things up?

onvm · 2019-08-06T00:04:35Z

@onvm did the changes mess things up?

CI Message

Your results will arrive shortly

onvm · 2019-08-06T00:19:00Z

@onvm did the changes mess things up?

CI Message

Error: ERROR: Failed to copy ONVM files to nimbnode17

onvm · 2019-08-06T00:24:22Z

Testing

CI Message

Your results will arrive shortly

kevindweb · 2019-08-06T00:32:50Z

@onvm that was my fault

onvm · 2019-08-06T00:32:51Z

@onvm that was my fault

CI Message

Your results will arrive shortly

onvm · 2019-08-06T00:33:19Z

@onvm that was my fault

CI Message

Error: ERROR: Failed to copy ONVM files to nimbnode17

onvm · 2019-08-06T01:51:44Z

Let us see

CI Message

Your results will arrive shortly

onvm

Let us see

CI Message

Run successful see results:
✔️ PR submitted to develop branch
✔️ Pktgen performance check passed
✔️ Speed Test performance check passed
✔️ Linter passed

[Results from nimbnode17]

Median TX pps for Pktgen: 10448860
Performance rating - 104.49% (compared to 10000000 average)
Median TX pps for Speed Tester: 40189750
Performance rating - 100.47% (compared to 40000000 average)

kevindweb · 2019-08-06T02:01:51Z

@koolzz I will figure out the mTCP side of it in that PR, but as of now, there is only one interface being used, identified by CLIENT_IFACE in the worker-config

koolzz · 2019-08-06T15:01:45Z

@onvm we good?

onvm · 2019-08-06T15:01:46Z

@onvm we good?

CI Message

Your results will arrive shortly

onvm

@onvm we good?

CI Message

Run successful see results:
✔️ PR submitted to develop branch
✔️ Pktgen performance check passed
✔️ Speed Test performance check passed
✔️ Linter passed

[Results from nimbnode17]

Median TX pps for Pktgen: 10432160
Performance rating - 104.32% (compared to 10000000 average)
Median TX pps for Speed Tester: 40192276
Performance rating - 100.48% (compared to 40000000 average)

koolzz

Great work man, happy to see Pktgen as a reliable CI check

kevindweb and others added 27 commits June 4, 2019 21:33

Save performance in ci

74676e5

Add flask to install

3a97ae4

Allowed for public linting of unauthorized users

ef255d0

Starting the queue

8ba65de

Start to fixing the queue problem

3a1874b

Merge branch 'develop' of https://github.com/kevindweb/openNetVM into…

bf5e2a8

… develop

Added run mode for future advancements

17882a6

Major fixes to ci list system

1c2f52d

Remove debug lines

29bbef6

Clean up definitions

a1749cb

Remove exit statement

0c3a90b

Merge branch 'develop' of https://github.com/sdnfv/openNetVM-dev into…

45fdf70

… ci_queue

Add error handling in run_ci and fix docs

c10dbb4

Create polling thread for requests

5b06b38

Delete new line

12935e3

Account for run mode from sdnfv#140

b771745

Fixes to merge run mode conflict

04380b1

Merge branch 'ci_performance_update' into ci_queue

e7639ef

Errors are much more difficult in threads :(

ed65879

Smarter than using tuples

3b01660

Merge branch 'develop' of https://github.com/kevindweb/openNetVM into…

58a13ff

… develop

Fix merge conflict

4bef1aa

Added event-based handling instead of polling

05a89f8

Let request handler clears event

24965be

Initialize ci for pktgen

7e71ec2

Huge update to pktgen configuration

ee2220c

Updated gitignore for log files

1c68a5f

kevindweb added the CI/Testing 🤖 Continuous Integration and Testing related label Jun 24, 2019

Added benchmark info to README and renamed file

2d305f7

kevindweb dismissed onvm’s stale review via 2d305f7 August 3, 2019 16:47

koolzz requested changes Aug 4, 2019

View reviewed changes

onvm previously approved these changes Aug 4, 2019

View reviewed changes

Fix Pktgen install requested changes

4f9928b

kevindweb dismissed onvm’s stale review via 4f9928b August 5, 2019 23:20

kevindweb added 2 commits August 5, 2019 19:21

Merge branch 'develop' of https://github.com/sdnfv/openNetVM into ci-…

fd6332d

…pktgen

Address Github change requests

a9168f9

Adjustments to interface configuration

0f3dfd2

onvm previously approved these changes Aug 6, 2019

View reviewed changes

Fixed manager debug

381f27d

kevindweb dismissed onvm’s stale review via 381f27d August 6, 2019 02:00

onvm approved these changes Aug 6, 2019

View reviewed changes

koolzz approved these changes Aug 6, 2019

View reviewed changes

koolzz merged commit ca145c6 into sdnfv:develop Aug 6, 2019

kevindweb deleted the ci-pktgen branch March 21, 2020 21:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pktgen for CI #148

Pktgen for CI #148

kevindweb commented Jun 24, 2019

kevindweb commented Jun 24, 2019

onvm commented Jun 24, 2019

kevindweb commented Aug 3, 2019

koolzz commented Aug 4, 2019

onvm commented Aug 4, 2019

koolzz left a comment

koolzz Aug 4, 2019

koolzz Aug 4, 2019

koolzz Aug 4, 2019

koolzz Aug 4, 2019

kevindweb Aug 4, 2019

koolzz Aug 4, 2019

kevindweb Aug 5, 2019

koolzz Aug 4, 2019

koolzz Aug 4, 2019

koolzz Aug 4, 2019

onvm left a comment

kevindweb commented Aug 6, 2019

onvm commented Aug 6, 2019

onvm commented Aug 6, 2019

onvm commented Aug 6, 2019

kevindweb commented Aug 6, 2019

onvm commented Aug 6, 2019

onvm commented Aug 6, 2019

onvm commented Aug 6, 2019

onvm left a comment

kevindweb commented Aug 6, 2019

koolzz commented Aug 6, 2019

onvm commented Aug 6, 2019

onvm left a comment

koolzz left a comment

		@@ -0,0 +1,8 @@
		WORKER_MODE="0"
		PKT_IFACE="p2p1"

Pktgen for CI #148

Pktgen for CI #148

Conversation

kevindweb commented Jun 24, 2019

Summary:

Merging notes:

Test Plan:

Review:

kevindweb commented Jun 24, 2019

onvm commented Jun 24, 2019

CI Message

kevindweb commented Aug 3, 2019

koolzz commented Aug 4, 2019

onvm commented Aug 4, 2019

CI Message

koolzz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onvm left a comment

Choose a reason for hiding this comment

CI Message

kevindweb commented Aug 6, 2019

onvm commented Aug 6, 2019

CI Message

onvm commented Aug 6, 2019

CI Message

onvm commented Aug 6, 2019

CI Message

kevindweb commented Aug 6, 2019

onvm commented Aug 6, 2019

CI Message

onvm commented Aug 6, 2019

CI Message

onvm commented Aug 6, 2019

CI Message

onvm left a comment

Choose a reason for hiding this comment

CI Message

kevindweb commented Aug 6, 2019

koolzz commented Aug 6, 2019

onvm commented Aug 6, 2019

CI Message

onvm left a comment

Choose a reason for hiding this comment

CI Message

koolzz left a comment

Choose a reason for hiding this comment