fix grpc flakiness in python unit tests #1816

RafalSkolasinski · 2020-05-11T11:25:02Z

Closes #1745.
It seems that problem is related to the fact that grpc server and and stub starts too close to each other (time-wise).

In principle this should be solved with wait_for_ready flag but this is still experimental and does not seem to cover this specific case - it didn't solve the issue for me but it did help me to reproduce the problem locally by adding wait_for_ready=True to stub.Predict and commenting out

         for q in range(10):
             s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
             r1 = s1.connect_ex(("127.0.0.1", 5000))
             s2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
             r2 = s2.connect_ex(("127.0.0.1", 6005))
             if r1 == 0 and r2 == 0:
                 break
             time.sleep(5)
         else:
             raise RuntimeError("Server did not bind to 127.0.0.1:5000")

The helper retry_method that I introduced to fix the problem repeats a given function n times using tenacity and sleeps with exponentially growing time in between attempts. If not successful it raises the last exception.

seldondev · 2020-05-11T11:26:35Z

Mon May 11 11:26:34 UTC 2020
The logs for [lint] [2] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/2.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=2

seldondev · 2020-05-11T11:26:36Z

Mon May 11 11:26:35 UTC 2020
The logs for [pr-build] [1] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/1.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=1

RafalSkolasinski · 2020-05-11T11:46:31Z

/retest

RafalSkolasinski · 2020-05-11T11:46:44Z

/test pr-build

RafalSkolasinski · 2020-05-11T11:47:11Z

/test

RafalSkolasinski · 2020-05-11T11:49:05Z

/test ?

RafalSkolasinski · 2020-05-11T11:49:36Z

/test all

edit... not exactly what I wanted...

seldondev · 2020-05-11T11:50:53Z

Mon May 11 11:50:52 UTC 2020
The logs for [pr-build] [3] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/3.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=3

seldondev · 2020-05-11T11:52:15Z

Mon May 11 11:52:14 UTC 2020
The logs for [lint] [6] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/6.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=6

seldondev · 2020-05-11T11:52:22Z

Mon May 11 11:52:20 UTC 2020
The logs for [integration] [5] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/5.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=5

seldondev · 2020-05-11T11:52:34Z

Mon May 11 11:52:33 UTC 2020
The logs for [notebooks] [4] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/4.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=4

RafalSkolasinski · 2020-05-11T12:42:08Z

/test this

seldondev · 2020-05-11T12:43:41Z

Mon May 11 12:43:40 UTC 2020
The logs for [pr-build] [7] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/7.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=7

RafalSkolasinski · 2020-05-11T13:02:37Z

/test this

seldondev · 2020-05-11T13:03:51Z

Mon May 11 13:03:49 UTC 2020
The logs for [pr-build] [8] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/8.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=8

seldondev · 2020-05-11T15:02:23Z

Mon May 11 15:02:22 UTC 2020
The logs for [pr-build] [9] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/9.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=9

seldondev · 2020-05-11T15:02:30Z

Mon May 11 15:02:26 UTC 2020
The logs for [lint] [10] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/10.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=10

python/tests/test_microservice.py

seldondev · 2020-05-11T16:38:41Z

Mon May 11 16:38:40 UTC 2020
The logs for [pr-build] [11] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/11.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=11

seldondev · 2020-05-11T16:40:16Z

Mon May 11 16:40:14 UTC 2020
The logs for [lint] [12] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/12.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=12

seldondev · 2020-05-11T16:50:19Z

Mon May 11 16:50:18 UTC 2020
The logs for [pr-build] [13] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/13.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=13

seldondev · 2020-05-11T16:52:43Z

Mon May 11 16:52:41 UTC 2020
The logs for [lint] [14] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/14.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=14

seldondev · 2020-05-11T16:59:33Z

Mon May 11 16:59:32 UTC 2020
The logs for [lint] [16] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/16.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=16

seldondev · 2020-05-11T16:59:34Z

Mon May 11 16:59:33 UTC 2020
The logs for [pr-build] [15] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/15.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=15

RafalSkolasinski · 2020-05-11T20:29:33Z

Merging as discussed with @adriangonz
/approve

seldondev · 2020-05-11T20:29:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RafalSkolasinski

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [RafalSkolasinski]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

seldondev · 2020-05-11T20:31:37Z

Mon May 11 20:31:36 UTC 2020
The logs for [lint] [18] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/18.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=18

seldondev · 2020-05-11T20:31:38Z

Mon May 11 20:31:37 UTC 2020
The logs for [pr-build] [17] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/17.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=17

seldondev · 2020-05-11T20:33:10Z

@RafalSkolasinski: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
notebooks	`9bf0bf3`	link	`/test notebooks`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

RafalSkolasinski · 2020-05-11T20:37:00Z

/retest

RafalSkolasinski · 2020-05-11T20:38:24Z

Interesting... had

[ERROR] PROTOC FAILED: [libprotobuf WARNING ../../../../../src/google/protobuf/compiler/parser.cc:646] No syntax specified for the proto file: tensorflow/core/framework/types.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)

in tests of Java Engine...

seldondev · 2020-05-11T20:38:37Z

Mon May 11 20:38:36 UTC 2020
The logs for [pr-build] [19] will show after the pipeline context has finished.
https://github.com/SeldonIO/seldon-core/blob/gh-pages/jenkins-x/logs/SeldonIO/seldon-core/PR-1816/19.log

impatient try
jx get build logs SeldonIO/seldon-core/PR-1816 --build=19

seldondev added do-not-merge/work-in-progress size/XS labels May 11, 2020

implement flakiness workaround for initial test

f23cc37

seldondev added size/S and removed size/XS labels May 11, 2020

RafalSkolasinski requested a review from adriangonz May 11, 2020 15:39

RafalSkolasinski changed the title ~~WIP: debug grpc test flakiness~~ debug grpc test flakiness May 11, 2020

seldondev removed the do-not-merge/work-in-progress label May 11, 2020

RafalSkolasinski changed the title ~~debug grpc test flakiness~~ fix grpc flakiness test in python unit tests May 11, 2020

RafalSkolasinski changed the title ~~fix grpc flakiness test in python unit tests~~ fix grpc flakiness in python unit tests May 11, 2020

adriangonz reviewed May 11, 2020

View reviewed changes

python/tests/test_microservice.py Outdated Show resolved Hide resolved

use tenacity in order to not reinvent the wheel

b6b4908

add tenacity to requirements-dev.txt

b43da05

seldondev added the approved label May 11, 2020

seldondev merged commit 3949a00 into SeldonIO:master May 11, 2020

RafalSkolasinski mentioned this pull request Aug 18, 2020

Investigate test_model_template_app_grpc_metrics flakiness #1745

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix grpc flakiness in python unit tests #1816

fix grpc flakiness in python unit tests #1816

RafalSkolasinski commented May 11, 2020 •

edited

Loading

seldondev commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020 •

edited

Loading

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020 •

edited

Loading

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

fix grpc flakiness in python unit tests #1816

fix grpc flakiness in python unit tests #1816

Conversation

RafalSkolasinski commented May 11, 2020 • edited Loading

seldondev commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020 • edited Loading

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020

seldondev commented May 11, 2020 • edited Loading

RafalSkolasinski commented May 11, 2020

RafalSkolasinski commented May 11, 2020

seldondev commented May 11, 2020

RafalSkolasinski commented May 11, 2020 •

edited

Loading

RafalSkolasinski commented May 11, 2020 •

edited

Loading

seldondev commented May 11, 2020 •

edited

Loading