Ci busy no more #143

kevindweb · 2019-06-08T20:44:30Z

A huge improvement to testing pull requests throughout the day!

Summary:

Previously, CI would not run a request if another was already running. This is a bug annoyance (especially for @koolzz ) when trying to merge things and get the 'go ahead' from CI. Now, CI will send a message that if we are busy, it will run right after the last CI. This only happens if a duplicate request is not already there. For example, while CI is busy, we make a 2 requests in another PR, only one of them will be put in the list.

Usage:
Regular with CI comment arguments

This PR includes
Resolves issues
Breaking API changes
Internal API changes
Usability improvements
Bug fixes
New functionality	👍
New NF/onvm_mgr args
Changes to starting NFs
Dependency updates
Web stats updates

Merging notes:

Dependencies: Merge Ci performance update #141 , then Allow CI Lint and Branch Checks for Unauthorized Users #140 , then this branch
This branch has merged those two in that order

TODO before merging :

PR is ready for review

Test Plan:

Review:

@koolzz again please lmk if anything doesn't work

… develop

onvm · 2019-06-08T20:44:38Z

In response to PR creation

CI Message

Your results will arrive shortly

koolzz · 2019-06-08T20:49:39Z

Exiting stuff, will review soon!

onvm

In response to PR creation

CI Message

Run successful see results:
✔️ PR submitted to develop branch
✔️ Speed tester performance check passed
✔️ Linter passed

[Results from nimbnode30]

Median TX pps for Speed Tester: 38624205
Performance rating - 110.35% (compared to 35000000 average)

… ci_queue

koolzz

Not a fan of the main CI pipeline call both adding events to the queue and running all of them in a while loop, would be better if a dedicated thread was specifically waiting for those events to be added to the list, and then processing those.

Also the name ci_list isn't very descriptive, we should do better. Something like ci_job_list or ci_requests_list?

Overall cool stuff though looking forward to not waiting 10 minutes for ci requests 👍

koolzz · 2019-06-09T03:29:48Z

ci/webhook-receiver.py

+        duplicate_req = False
+        busy_msg = "Duplicate request already waiting, ignoring message"
+        with queue_lock:
+            # make sure we don't bypass thread safety


Should this comment be above one line? Before you grab the queue_lock

ci/webhook-receiver.py

koolzz · 2019-06-09T03:42:00Z

ci/webhook-receiver.py

+        run_manager(request_ctx)
+        while len(ci_list) > 0:
+            # new requests were made since our last call, run them in order
+            run_manager(ci_list.pop(0))


Even though I think this is safe (pop is atomic right?) the design isn't very elegant. Would it be possible for us to have a different thread poll the queue and run things when items appear there?

kevindweb · 2019-06-10T00:06:32Z

@onvm test thread

kevindweb · 2019-06-10T00:07:34Z

@onvm test me again

kevindweb · 2019-06-10T00:09:56Z

@onvm testing again

onvm · 2019-06-10T00:10:05Z

@onvm testing again

CI Message

Your results will arrive shortly

kevindweb · 2019-06-10T00:20:06Z

ci/webhook-receiver.py

+            run_manager(ci_request_list.pop(0))
+        else:
+            # no requests, sleep
+            time.sleep(1)


Figured we could just sleep for a second, but not sure how long is best

Might want a sleeping queue sort of implementation in the future, but good enough for now

what do you mean?

kevindweb · 2019-06-12T00:18:47Z

@onvm authorized stick in queue

kevindweb · 2019-06-12T00:36:29Z

@onvm test please run

koolzz · 2019-06-14T22:23:23Z

Discussing these in slack

kevindweb · 2019-06-14T23:04:58Z

@onvm test

kevindweb · 2019-06-14T23:08:58Z

@onvm test

kevindweb · 2019-06-14T23:12:46Z

@onvm test 1

onvm · 2019-06-15T18:21:30Z

New message here

onvm · 2019-06-15T18:30:30Z

New message here

onvm · 2019-06-15T18:30:47Z

New message here

onvm · 2019-06-15T18:41:44Z

New message here

kevindweb · 2019-06-17T03:25:22Z

@onvm first one

onvm · 2019-06-17T03:25:33Z

@onvm first one

CI Message

Your results will arrive shortly

kevindweb · 2019-06-17T03:39:00Z

@onvm run me

kevindweb · 2019-06-17T03:41:43Z

@onvm test

kevindweb · 2019-06-17T04:29:04Z

onvmstats "commited" the changes because on nn44, cronjob for stats is running

onvm · 2019-06-18T01:01:34Z

qqq

CI Message

Error: ERROR: Failed to copy ONVM files to nimbnode17

koolzz

Commented on the event flag, lmk what you think

koolzz · 2019-06-18T10:21:05Z

ci/webhook-receiver.py

+
+def add_request(request_ctx):
+    ci_request_list.append(request_ctx)
+    if not request_event.isSet():


Why not just set request_event.set() and then do equest_event.clear() in the request_handler?

koolzz · 2019-06-18T10:21:24Z

ci/webhook-receiver.py

+            manager_running = False
+        if not ci_request_list:
+            # list empty, go to sleep until signaled
+            request_event.wait()


request_event.wait() request_event.clear()

kevindweb · 2019-06-19T00:16:13Z

I liked that idea, makes it look cleaner in add_request

koolzz · 2019-06-20T09:24:33Z

onvmstats is now sentient and pushing commits? I'm impressed with our development program.
@onvm

onvm · 2019-06-23T22:50:18Z

qqq

CI Message

Error: ERROR: Failed to copy ONVM files to nimbnode17

onvm · 2019-06-23T22:51:36Z

qqq

CI Message

Error: ERROR: Failed to copy ONVM files to nimbnode17

onvm · 2019-06-23T22:51:39Z

qqq

CI Message

Error: ERROR: Failed to copy ONVM files to nimbnode17

koolzz · 2019-06-28T21:53:18Z

@onvm do your thing

onvm · 2019-06-28T21:53:20Z

@onvm do your thing

CI Message

Your results will arrive shortly

onvm

@onvm do your thing

CI Message

Run successful see results:
✔️ PR submitted to develop branch
✔️ Pktgen performance check passed
✔️ Speed Test performance check passed
✔️ Linter passed

[Results from nimbnode17]

Median TX pps for Pktgen: 6325632
Performance rating - 105.43% (compared to 6000000 average)
Median TX pps for Speed Tester: 38346945
Performance rating - 109.56% (compared to 35000000 average)

koolzz

Changes from previous review implemented, approving

kevindweb and others added 11 commits June 4, 2019 21:33

Save performance in ci

74676e5

Add flask to install

3a97ae4

Allowed for public linting of unauthorized users

ef255d0

Starting the queue

8ba65de

Start to fixing the queue problem

3a1874b

Merge branch 'develop' of https://github.com/kevindweb/openNetVM into…

bf5e2a8

… develop

Added run mode for future advancements

17882a6

Major fixes to ci list system

1c2f52d

Remove debug lines

29bbef6

Clean up definitions

a1749cb

Remove exit statement

0c3a90b

onvm approved these changes Jun 8, 2019

View reviewed changes

koolzz added the CI/Testing 🤖 Continuous Integration and Testing related label Jun 8, 2019

Merge branch 'develop' of https://github.com/sdnfv/openNetVM-dev into…

45fdf70

… ci_queue

koolzz requested changes Jun 9, 2019

View reviewed changes

Add error handling in run_ci and fix docs

c10dbb4

Create polling thread for requests

5b06b38

kevindweb commented Jun 10, 2019

View reviewed changes

kevindweb added 4 commits June 11, 2019 19:43

Delete new line

12935e3

Account for run mode from sdnfv#140

b771745

Fixes to merge run mode conflict

04380b1

Merge branch 'ci_performance_update' into ci_queue

e7639ef

Added event-based handling instead of polling

05a89f8

koolzz reviewed Jun 18, 2019

View reviewed changes

Let request handler clears event

24965be

kevindweb mentioned this pull request Jun 24, 2019

Pktgen for CI #148

Merged

1 task

onvm approved these changes Jun 28, 2019

View reviewed changes

koolzz approved these changes Jun 28, 2019

View reviewed changes

koolzz merged commit 5a40435 into sdnfv:develop Jun 28, 2019

kevindweb deleted the ci_queue branch June 29, 2019 21:21

koolzz added this to the ONVM v19.07 milestone Jul 31, 2019

Ci busy no more #143

Ci busy no more #143

Conversation

kevindweb commented Jun 8, 2019 • edited Loading

Summary:

Merging notes:

Test Plan:

Review:

onvm commented Jun 8, 2019

CI Message

koolzz commented Jun 8, 2019

onvm left a comment

Choose a reason for hiding this comment

CI Message

koolzz left a comment • edited Loading

Choose a reason for hiding this comment

koolzz Jun 9, 2019

Choose a reason for hiding this comment

koolzz Jun 9, 2019

Choose a reason for hiding this comment

kevindweb commented Jun 10, 2019

kevindweb commented Jun 10, 2019

kevindweb commented Jun 10, 2019

onvm commented Jun 10, 2019

CI Message

kevindweb Jun 10, 2019

Choose a reason for hiding this comment

koolzz Jun 14, 2019

Choose a reason for hiding this comment

kevindweb Jun 14, 2019

Choose a reason for hiding this comment

kevindweb commented Jun 12, 2019

kevindweb commented Jun 12, 2019

koolzz commented Jun 14, 2019

kevindweb commented Jun 14, 2019

kevindweb commented Jun 14, 2019

kevindweb commented Jun 14, 2019

onvm commented Jun 15, 2019 • edited Loading

onvm commented Jun 15, 2019 • edited Loading

onvm commented Jun 15, 2019 • edited Loading

onvm commented Jun 15, 2019 • edited Loading

kevindweb commented Jun 17, 2019

onvm commented Jun 17, 2019

CI Message

kevindweb commented Jun 17, 2019

kevindweb commented Jun 17, 2019

kevindweb commented Jun 17, 2019

onvm commented Jun 18, 2019

CI Message

koolzz left a comment

Choose a reason for hiding this comment

koolzz Jun 18, 2019

Choose a reason for hiding this comment

koolzz Jun 18, 2019

Choose a reason for hiding this comment

kevindweb commented Jun 19, 2019

koolzz commented Jun 20, 2019

onvm commented Jun 23, 2019

CI Message

onvm commented Jun 23, 2019

CI Message

onvm commented Jun 23, 2019

CI Message

koolzz commented Jun 28, 2019

onvm commented Jun 28, 2019

CI Message

onvm left a comment

Choose a reason for hiding this comment

CI Message

koolzz left a comment

Choose a reason for hiding this comment

kevindweb commented Jun 8, 2019 •

edited

Loading

koolzz left a comment •

edited

Loading

onvm commented Jun 15, 2019 •

edited

Loading

onvm commented Jun 15, 2019 •

edited

Loading

onvm commented Jun 15, 2019 •

edited

Loading

onvm commented Jun 15, 2019 •

edited

Loading