Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time measure failover process #80

Merged
merged 1 commit into from
Apr 8, 2020
Merged

Conversation

ytsarev
Copy link
Member

@ytsarev ytsarev commented Apr 7, 2020

  • Move failover check retry functions to subtest
    to enable gathering of time metrics from the test suite
  • Current results look like:
--- PASS: TestOhmyglbBasicFailoverExample (289.48s)
    --- PASS: TestOhmyglbBasicFailoverExample/failover_happens_as_expected (157.23s)
PASS
ok  	ohmyterratest	291.032s
  • This way we objectively measured that the whole failover process
    (CRD->etcd->coredns) update takes more than 2.5 minutes

There is definitely something to improve.

* Move failover check retry functions to subtest
  to enable gathering of time metrics from the test suite
* Current results look like:
```
--- PASS: TestOhmyglbBasicFailoverExample (289.48s)
    --- PASS: TestOhmyglbBasicFailoverExample/failover_happens_as_expected (157.23s)
PASS
ok  	ohmyterratest	291.032s
```
* This way we objectively measured that the whole failover process
  (CRD->etcd->coredns) update takes more than 2.5 minutes

There is definitely something to improve.
@ytsarev ytsarev merged commit 3ab2abe into master Apr 8, 2020
@ytsarev ytsarev deleted the terratest-measure-failover branch April 8, 2020 08:44
ytsarev added a commit that referenced this pull request Apr 8, 2020
* Reduce external-dns ( CRD -> etcd ) sync interval to 20s
* Value is picked to be sligtly below default 30s DNS TTL
  we are currently working with
* Terratest results:
```
--- PASS: TestOhmyglbBasicFailoverExample (157.84s)
    --- PASS: TestOhmyglbBasicFailoverExample/failover_happens_as_expected (35.41s)
```
* This way we reduced failover(or any kind of load balancing
  reconfiguration) from ~2.5 min down to 35 seconds
  See #80 for initial values
ytsarev added a commit that referenced this pull request Apr 8, 2020
* Reduce external-dns ( CRD -> etcd ) sync interval to 20s
* Value is picked to be sligtly below default 30s DNS TTL
  we are currently working with
* Terratest results:
```
--- PASS: TestOhmyglbBasicFailoverExample (157.84s)
    --- PASS: TestOhmyglbBasicFailoverExample/failover_happens_as_expected (35.41s)
```
* This way we reduced failover(or any kind of load balancing
  reconfiguration) from ~2.5 min down to 35 seconds
  See #80 for initial values
kuritka pushed a commit that referenced this pull request Apr 15, 2020
* Reduce external-dns ( CRD -> etcd ) sync interval to 20s
* Value is picked to be sligtly below default 30s DNS TTL
  we are currently working with
* Terratest results:
```
--- PASS: TestOhmyglbBasicFailoverExample (157.84s)
    --- PASS: TestOhmyglbBasicFailoverExample/failover_happens_as_expected (35.41s)
```
* This way we reduced failover(or any kind of load balancing
  reconfiguration) from ~2.5 min down to 35 seconds
  See #80 for initial values
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants