Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

slack-vitess-r14.0.4-dsdefense branch: initial cherry-picks, part 1 #54

Merged
merged 25 commits into from
Mar 23, 2023
Merged

slack-vitess-r14.0.4-dsdefense branch: initial cherry-picks, part 1 #54

merged 25 commits into from
Mar 23, 2023

Commits on Mar 1, 2023

  1. Allow override of build git env in docker/base builds (vitessio#11968) 

    * Pass BUILD_GIT_BRANCH and BUILD_GIT_REV env into docker build

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix bootstrap version

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix bootstrap version, again

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix ws

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>
Co-authored-by: Florent Poinsard <florent.poinsard@outlook.fr>
    @timvaillancourt @frouioui
    timvaillancourt and frouioui committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    2cbae9a View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Add basic metrics to vttablet transaction throttler (vitessio#12418) 

    * Add basic stats to vttablet tx throttler

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* test new metrics

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* reorder

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* short names

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Add max rate

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Move NewGaugeFunc to under conditional

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Use env

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Remove env from TxThrottler struct

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix tests

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* PR suggestion

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* Fix unit test

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

* reorder test vars

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

---------

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    af87fe8 View commit details
    Browse the repository at this point in the history

  2. Add CpuUsage to RealtimeStats 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    f691b24 View commit details
    Browse the repository at this point in the history

  3. Cleanup 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    6e43da7 View commit details
    Browse the repository at this point in the history

  4. Update flag help message 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    b334d22 View commit details
    Browse the repository at this point in the history

  5. Update flag help message, pt 2 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    a74daa7 View commit details
    Browse the repository at this point in the history

  6. Use github.com/shirou/gopsutil/v3 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    73a67ec View commit details
    Browse the repository at this point in the history

  7. Prevent double start/.Open() 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    8f4b49c View commit details
    Browse the repository at this point in the history

  8. Move started bool inside .Open 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    d74587e View commit details
    Browse the repository at this point in the history

  9. Consolidate .stopCollection() -> .Close() 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    1565250 View commit details
    Browse the repository at this point in the history

  10. Remove startErr 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    1f83989 View commit details
    Browse the repository at this point in the history

  11. Add test comment, pass env to newSystemHealthMonitor() 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    47a6ab8 View commit details
    Browse the repository at this point in the history

  12. Fix ordering in unserveCommon() 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    647ee16 View commit details
    Browse the repository at this point in the history

  13. Use sync/atomic, reorder .Open()/.Close() 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    0c4a4b3 View commit details
    Browse the repository at this point in the history

  14. A bit more cleanup, var rename 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    1bf487d View commit details
    Browse the repository at this point in the history

  15. Disabled test 

    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    487e85b View commit details
    Browse the repository at this point in the history

  16. Use vitess.io/vitess/go/sync2 because go1.18 has no Uint64 sync/atomic

    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    c90905c View commit details
    Browse the repository at this point in the history

  17. go mod tidy

    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    8fa7042 View commit details
    Browse the repository at this point in the history

  18. Fix whitespace

    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    f5327df View commit details
    Browse the repository at this point in the history

  19. reorder e2e vttablet flag

    @timvaillancourt
    timvaillancourt committed Mar 14, 2023
    Configuration menu
    Copy the full SHA
    34f24da View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2023

  1. Emit per workload labels for existing per table vttablet metrics (vit… 

    …essio#12394)

* Emit per workload labels for existing per table vttablet metrics

This adds the possibility to configure vttablet (via CLI flag) to also have a
workload label for existing per table metrics (query counts, query times, query
errors, query rows affected, query rows returned, query error counts). Workload
can be any string that makes sense for the client application. For example, API
endpoint name, controller, batch job name, application name or something else.

This is usefult to be able to gain observability about how the query load is
distributed across different workloads.

This is achieved with two new CLI flags, namely:

* `enable-per-workload-table-metrics`: whether to enable or disable per
  workload metric collection - disabled by default to preserve the current
  behavior, thus making the new feature opt-in only.
* `workload-label`: a string to look for in query comments to identify the
  workload running the current query.

The workload is obtained by parsing query comments of the form:

/* ... <workload_label>=<workload_name>; ... */

For example, if vttablet is started with

`--enable-per-workload-table-metrics --workload-label app_name`

anda query is issued with a comment like

/* ... app_name=shop; ... */

then metrics will look like

```
vttablet_query_counts{plan="Select",table="dual", workload="shop"} 15479
```

instead of

```
vttablet_query_counts{plan="Select",table="dual"} 15479
```

Query comment parsing only takes place if `--enable-per-workload-table-metrics`
is used, as to not incur parsing performance impact if the user does not want
per workload metrics.

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* make linter happy

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* fix flags e2e test

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Address PR comments:

* Obtain workload information on the vtgate instead of the vttablet, avoiding
  double parsing.
* Treat workload name as a query directive.
* Send workload name from vtgate to vttablet as ExecuteOptions.

Additionally, annotate tabletserver's execution span with the workload name
to also enrich traces with workload name data, in addition to metrics.

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* A few fixes:

1. Rebuild some files with `make proto`.
2. Protect against nil ExecuteOptions on the tabletserver.

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fix flags e2e test

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Address PR comments

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fixes

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fix a comment

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fix e2e flag test

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Update JS code for protobuf changes.

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fix QueryEngine unit test

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fix e2e flag test

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Fix spurious tab in comment

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

* Address PR comment

Don't use dual format flag for new flags - stick with - separated ones.

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>

---------

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    @ejortegau @timvaillancourt
    ejortegau authored and timvaillancourt committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    dfb623a View commit details
    Browse the repository at this point in the history

  2. Add support for criticality query directive, and have TxThrottler res… 

    …pect that

Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    @ejortegau @timvaillancourt
    ejortegau authored and timvaillancourt committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    7483a22 View commit details
    Browse the repository at this point in the history

  3. Remove unused variable 

    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    @ejortegau @timvaillancourt
    ejortegau authored and timvaillancourt committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    3b15553 View commit details
    Browse the repository at this point in the history

  4. Fix CI pipeline 

    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    @ejortegau @timvaillancourt
    ejortegau authored and timvaillancourt committed Mar 22, 2023
    Configuration menu
    Copy the full SHA
    dfef214 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2023

  1. Fix qe.queryRowCounts

    @timvaillancourt
    timvaillancourt committed Mar 23, 2023
    Configuration menu
    Copy the full SHA
    0f5c18f View commit details
    Browse the repository at this point in the history