perf(plugins): use batch queue in datadog & statsd plugin to reduce timer usage #9521

windmgc · 2022-10-10T14:01:37Z

Summary

This PR adds batch queue into the Datadog plugin and the StatsD plugin, to reduce the number of timers used by these two plugins.

Whenever a request is processed, during the log phase a new instantly running timer will be created. This may cause a shortage of timers under heavy traffic and lead to unpredictable consequences (internal timers are killed randomly and some of them cannot recover automatically).

Adding batch queues in these two logging plugins will mitigate this problem to some extent.

Full changelog

Add batch queue into the Datadog and StatsD plugin to reduce timer usage

Issue reference

Mitigate FTI-4367 FTI-4269

windmgc · 2022-10-10T14:03:15Z

It seems hard to add test cases to this fix, but exists test cases should get passed without any problem.

hbagdi · 2022-10-10T19:08:20Z

The right commit type for this change is 'perf' as per our contribution policy: https://github.com/Kong/kong/blob/master/CONTRIBUTING.md#type.
Please update your PR and commit accordingly.

windmgc · 2022-10-11T01:40:57Z

PR type changed to perf

kong/plugins/datadog/handler.lua

windmgc · 2022-10-11T07:42:35Z

@chronolaw please review again, thanks!

Also @bungle could you please take a look at this PR?

kong/plugins/datadog/handler.lua

mayocream · 2022-10-26T07:23:05Z

kong/plugins/datadog/schema.lua

@@ -79,6 +79,9 @@ return {
          { service_name_tag = { type = "string", default = "name" }, },
          { status_tag = { type = "string", default = "status" }, },
          { consumer_tag = { type = "string", default = "consumer" }, },
+          { retry_count = { type = "integer", default = 10 }, },


I was about to ask where the default values come from, but I found them here:

kong/kong/plugins/http-log/schema.lua

Lines 17 to 19 in 7397d6f

{ retry_count = { type = "integer", default = 10 }, },

{ queue_size = { type = "integer", default = 1 }, },

{ flush_timeout = { type = "number", default = 2 }, },

: )

dndx · 2022-10-27T17:30:33Z

Ping @bungle for a review as well.

windmgc · 2022-10-28T07:16:51Z

Hi, @bungle do you need the performance test job running? It seems that it requires some cloud provider credentials to be properly configured, and in my fork repo, these configs do not exist. Maybe I need to push this branch to kong:kong and open a new PR so that we can get the job running

ADD-SP · 2022-11-09T05:33:39Z

kong/plugins/datadog/schema.lua

@@ -79,6 +79,9 @@ return {
          { service_name_tag = { type = "string", default = "name" }, },
          { status_tag = { type = "string", default = "status" }, },
          { consumer_tag = { type = "string", default = "consumer" }, },
+          { retry_count = { type = "integer", default = 10 }, },


We need to update kong/clustering/compat/removed_fields.lua.

removed_fields added.

ADD-SP

If a field is not required, it can be set to ngx.null or cjson.null by set it to null in Admin API.

ADD-SP · 2022-11-14T05:33:46Z

kong/plugins/datadog/handler.lua

+    end
+
+    local opts = {
+      retry_count    = conf.retry_count or 10,


retry_count maybe ngx.null, because this field is not required, as well as queue_size, flush_timeout.

Thanks for reminding this! I tested it and it is true that a field that is not mandatory can be set to null without any error.
So I made these fields required = true now, although I feel it is more like a nitpick because many of these fields in the OSS plugin do not have this limitation...

ADD-SP

LGTM

hanshuebner · 2022-11-22T09:51:14Z

@windmgc can you rebase so that we can merge this, please?

…imer usage

windmgc · 2022-11-22T14:45:46Z

@windmgc can you rebase so that we can merge this, please?

Rebase done.

gszr · 2022-11-22T15:33:29Z

@windmgc please add a changelog entry for this. Nice work!

windmgc · 2022-11-22T16:33:41Z

The missing changelog is added in #9804

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version.

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version. (cherry picked from commit b5baf94)

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version.

windmgc requested a review from a team as a code owner October 10, 2022 14:01

pull-request-size bot added the size/L label Oct 10, 2022

github-actions bot added plugins/datadog plugins/statsd labels Oct 10, 2022

windmgc force-pushed the batch-queue-in-plugins branch from cb53478 to 4b8bc6b Compare October 11, 2022 01:40

windmgc changed the title ~~fix(plugins): use batch queue in datadog & statsd plugin to reduce timer usage~~ perf(plugins): use batch queue in datadog & statsd plugin to reduce timer usage Oct 11, 2022

chronolaw reviewed Oct 11, 2022

View reviewed changes

kong/plugins/datadog/handler.lua Show resolved Hide resolved

kong/plugins/datadog/handler.lua Outdated Show resolved Hide resolved

kong/plugins/datadog/handler.lua Outdated Show resolved Hide resolved

chronolaw reviewed Oct 11, 2022

View reviewed changes

kong/plugins/datadog/handler.lua Show resolved Hide resolved

windmgc force-pushed the batch-queue-in-plugins branch 2 times, most recently from 8b85c98 to 341189b Compare October 11, 2022 07:11

chronolaw reviewed Oct 11, 2022

View reviewed changes

kong/plugins/datadog/handler.lua Outdated Show resolved Hide resolved

dndx requested a review from bungle October 17, 2022 02:42

mayocream approved these changes Oct 21, 2022

View reviewed changes

mayocream reviewed Oct 26, 2022

View reviewed changes

dndx force-pushed the batch-queue-in-plugins branch from 6b2e00d to df2c12c Compare October 27, 2022 17:30

dndx force-pushed the batch-queue-in-plugins branch from f9ce1c4 to a1005ce Compare November 3, 2022 16:55

dndx assigned ADD-SP Nov 8, 2022

ADD-SP requested changes Nov 9, 2022

View reviewed changes

ADD-SP requested changes Nov 14, 2022

View reviewed changes

ADD-SP added the pr/changes requested Changes were requested to this PR by a maintainer. Please address them and ping back once done. label Nov 15, 2022

windmgc force-pushed the batch-queue-in-plugins branch from a1005ce to 56371b1 Compare November 15, 2022 08:16

github-actions bot added the core/clustering label Nov 15, 2022

ADD-SP approved these changes Nov 16, 2022

View reviewed changes

windmgc added 13 commits November 22, 2022 22:24

perf(plugins): use batch queue in datadog & statsd plugin to reduce t…

7f1075a

…imer usage

lint

c064cbf

style

827b707

split queue by metric configs as well

322f318

use conf.__key__ directly

4928629

fix tests

a793c81

goto shortcut

6f14f5e

style

d3b76f9

add batch test

635401f

fix goto position

6cc500f

let field required

5c228d3

add removed_fields

14102e6

fix ut

ff725ab

windmgc force-pushed the batch-queue-in-plugins branch from aee6967 to ff725ab Compare November 22, 2022 14:29

gszr approved these changes Nov 22, 2022

View reviewed changes

gszr merged commit e2df4e6 into Kong:master Nov 22, 2022

windmgc added a commit to windmgc/kong that referenced this pull request Nov 22, 2022

docs(changelog): add changelog for Kong#9521

487899b

windmgc mentioned this pull request Nov 22, 2022

docs(changelog): add changelog for #9521 #9804

Merged

3 tasks

fffonion pushed a commit that referenced this pull request Nov 23, 2022

docs(changelog): add changelog for #9521 (#9804)

f62356b

dcarley mentioned this pull request Dec 14, 2022

Plugin and timer performance on Kong 3.x #9964

Closed

1 task

windmgc mentioned this pull request Jan 3, 2023

add datadog & statsd plugin missing doc Kong/docs.konghq.com#4979

Merged

outsinre added a commit that referenced this pull request Jun 26, 2024

feat(ci): automatically add release version to Pongo (#9521)

3255407

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version.

bungle pushed a commit that referenced this pull request Jun 26, 2024

feat(ci): automatically add release version to Pongo (#9521)

b5baf94

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version.

hanshuebner pushed a commit that referenced this pull request Jun 27, 2024

feat(ci): automatically add release version to Pongo (#9521)

2583760

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version. (cherry picked from commit b5baf94)

oowl pushed a commit that referenced this pull request Jul 15, 2024

feat(ci): automatically add release version to Pongo (#9521)

30c9872

When a tag is created, this workflow would automatically create a PR on 'kong/kong-pongo' to add that version.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(plugins): use batch queue in datadog & statsd plugin to reduce timer usage #9521

perf(plugins): use batch queue in datadog & statsd plugin to reduce timer usage #9521

windmgc commented Oct 10, 2022 •

edited by ms2008

Loading

windmgc commented Oct 10, 2022

hbagdi commented Oct 10, 2022

windmgc commented Oct 11, 2022

windmgc commented Oct 11, 2022

mayocream Oct 26, 2022

dndx commented Oct 27, 2022

windmgc commented Oct 28, 2022

ADD-SP Nov 9, 2022

windmgc Nov 15, 2022

ADD-SP left a comment

ADD-SP Nov 14, 2022

windmgc Nov 15, 2022

ADD-SP left a comment

hanshuebner commented Nov 22, 2022

windmgc commented Nov 22, 2022

gszr commented Nov 22, 2022

windmgc commented Nov 22, 2022

	{ retry_count = { type = "integer", default = 10 }, },
	{ queue_size = { type = "integer", default = 1 }, },
	{ flush_timeout = { type = "number", default = 2 }, },

perf(plugins): use batch queue in datadog & statsd plugin to reduce timer usage #9521

perf(plugins): use batch queue in datadog & statsd plugin to reduce timer usage #9521

Conversation

windmgc commented Oct 10, 2022 • edited by ms2008 Loading

Summary

Full changelog

Issue reference

windmgc commented Oct 10, 2022

hbagdi commented Oct 10, 2022

windmgc commented Oct 11, 2022

windmgc commented Oct 11, 2022

mayocream Oct 26, 2022

Choose a reason for hiding this comment

dndx commented Oct 27, 2022

windmgc commented Oct 28, 2022

ADD-SP Nov 9, 2022

Choose a reason for hiding this comment

windmgc Nov 15, 2022

Choose a reason for hiding this comment

ADD-SP left a comment

Choose a reason for hiding this comment

ADD-SP Nov 14, 2022

Choose a reason for hiding this comment

windmgc Nov 15, 2022

Choose a reason for hiding this comment

ADD-SP left a comment

Choose a reason for hiding this comment

hanshuebner commented Nov 22, 2022

windmgc commented Nov 22, 2022

gszr commented Nov 22, 2022

windmgc commented Nov 22, 2022

windmgc commented Oct 10, 2022 •

edited by ms2008

Loading