This repository has been archived by the owner on Jul 11, 2023. It is now read-only.
kubernetes/test: fix slow tests #2019
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The tests had synchronization issues resulting from missing and blocking event notifications. The missing events were a result of the tests not waiting till the namespace used in the test is monitored, which in turn causes the caches for the other Kubernetes resources to be filtered out as unmonitored. The result is that the resource is only marked as monitored upon the next resync from k8s informers (currently every 30s). This is fixed by ensuring an async wait on the namespace to be monitored before waiting on k8s events for resources belonging to the namespace. Since unbuffered channels are used for announcements, it is better to read as soon as data is available rather than to batch reads, which results in writes being blocked. This is also addressed in the tests. This change speeds up by the test by over 60s to 90s when the events are missed which was happening often. Resolves openservicemesh#2014
eduser25
previously approved these changes
Nov 10, 2020
ksubrmnn
suggested changes
Nov 10, 2020
eduser25
approved these changes
Nov 10, 2020
Codecov Report
@@ Coverage Diff @@
## main #2019 +/- ##
==========================================
- Coverage 56.72% 56.51% -0.22%
==========================================
Files 141 142 +1
Lines 5706 5768 +62
==========================================
+ Hits 3237 3260 +23
- Misses 2466 2505 +39
Partials 3 3
Continue to review full report at Codecov.
|
ksubrmnn
approved these changes
Nov 10, 2020
draychev
pushed a commit
to draychev/osm
that referenced
this pull request
Nov 19, 2020
The tests had synchronization issues resulting from missing and blocking event notifications. The missing events were a result of the tests not waiting till the namespace used in the test is monitored, which in turn causes the caches for the other Kubernetes resources to be filtered out as unmonitored. The result is that the resource is only marked as monitored upon the next resync from k8s informers (currently every 30s). This is fixed by ensuring an async wait on the namespace to be monitored before waiting on k8s events for resources belonging to the namespace. Since unbuffered channels are used for announcements, it is better to read as soon as data is available rather than to batch reads, which results in writes being blocked. This is also addressed in the tests. This change speeds up by the test by over 60s to 90s when the events are missed which was happening often. Resolves openservicemesh#2014
draychev
pushed a commit
to draychev/osm
that referenced
this pull request
Nov 19, 2020
The tests had synchronization issues resulting from missing and blocking event notifications. The missing events were a result of the tests not waiting till the namespace used in the test is monitored, which in turn causes the caches for the other Kubernetes resources to be filtered out as unmonitored. The result is that the resource is only marked as monitored upon the next resync from k8s informers (currently every 30s). This is fixed by ensuring an async wait on the namespace to be monitored before waiting on k8s events for resources belonging to the namespace. Since unbuffered channels are used for announcements, it is better to read as soon as data is available rather than to batch reads, which results in writes being blocked. This is also addressed in the tests. This change speeds up by the test by over 60s to 90s when the events are missed which was happening often. Resolves openservicemesh#2014
draychev
pushed a commit
to draychev/osm
that referenced
this pull request
Dec 14, 2020
The tests had synchronization issues resulting from missing and blocking event notifications. The missing events were a result of the tests not waiting till the namespace used in the test is monitored, which in turn causes the caches for the other Kubernetes resources to be filtered out as unmonitored. The result is that the resource is only marked as monitored upon the next resync from k8s informers (currently every 30s). This is fixed by ensuring an async wait on the namespace to be monitored before waiting on k8s events for resources belonging to the namespace. Since unbuffered channels are used for announcements, it is better to read as soon as data is available rather than to batch reads, which results in writes being blocked. This is also addressed in the tests. This change speeds up by the test by over 60s to 90s when the events are missed which was happening often. Resolves openservicemesh#2014
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
The tests had synchronization issues resulting from missing and blocking
event notifications.
The missing events were a result of the tests
not waiting till the namespace used in the test is monitored, which
in turn causes the caches for the other Kubernetes resources to
be filtered out as unmonitored. The result is that the resource is
only marked as monitored upon the next resync from k8s informers
(currently every 30s). This is fixed by ensuring an async wait
on the namespace to be monitored before waiting on k8s events for
resources belonging to the namespace.
Since unbuffered channels are used for announcements, it is better
to read as soon as data is available rather than to batch reads,
which results in writes being blocked. This is also addressed in
the tests.
This change speeds up by the test by over 60s to 90s when the events
are missed which was happening often.
It also removes some unnecessary code in the tests.
Resolves #2014
Before this change:
After this change:
Affected area:
Please answer the following questions with yes/no.
No