Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(clustering) improve calculate hash performance by utilizing string buffer and tablepool #9073

Merged
merged 1 commit into from
Jul 11, 2022

Conversation

bungle
Copy link
Member

@bungle bungle commented Jul 8, 2022

Summary

Improve calculate hash performance by utilizing string buffer and tablepool.

@bungle bungle force-pushed the perf/calculate-hash-string-buffer branch 2 times, most recently from b0689f0 to e55b07b Compare July 8, 2022 11:52
@bungle bungle marked this pull request as ready for review July 8, 2022 11:53
@bungle bungle requested a review from a team as a code owner July 8, 2022 11:53

else
local t = type(value)
if t == "string" or t == "number" then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are using string buffer, i think we can also just skip type assertions and let it do the job. previously we do this because {} could be from a table or from a string "{}". but they are already encoded differently in string.buffer.

Copy link
Member Author

@bungle bungle Jul 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fffonion only string, number and object with __tostring are allowed with :put

Alternatively we can use buffer.encode but not sure it is worth it here as it is only really usable (in this case) with string, boolean and number.

@bungle bungle force-pushed the perf/calculate-hash-string-buffer branch from e55b07b to 66543c3 Compare July 8, 2022 13:22
@bungle
Copy link
Member Author

bungle commented Jul 8, 2022

A small improvement (so far):

-rw-r--r--  1 bungle staff 395M Dec 15  2021 kong.json

Calculation:

reading file...
reading file done in 0.16999983787537 secs
decoding json...
decoding json done in 2.0450000762939 secs
calculating hash...
c5c725a9b2776f947c198379a54a65d6
calculating hash done in 2.8819999694824 secs
calculating hash new...
57bfa7fc4fa1ec577f3b0a464dd6e96a
calculating hash new done in 2.2009999752045 secs

@bungle
Copy link
Member Author

bungle commented Jul 8, 2022

Small test.

Data loaded:

Inserted 200000 services in 4 seconds
Inserted 200000 routes in 5 seconds
Inserted 200000 consumers in 4 seconds
Inserted 200000 upstreams in 4 seconds
Inserted 200000 targets in 5 seconds
Inserted 200000 plugins in 5 seconds
Inserted 200000 services in 4 seconds
Inserted 200000 upstreams in 4 seconds

Export data:

2022/07/08 16:50:09 [error] 99950#0: *7 [lua] wrpc_control_plane.lua:111: took 18718 ms to export config, context: ngx.timer

Calculate hash:

2022/07/08 16:50:19 [error] 99950#0: *7 [lua] wrpc_control_plane.lua:125: took 10302 ms to calculate hash, context: ngx.timer

@bungle bungle requested a review from mayocream July 8, 2022 13:55
@fffonion fffonion merged commit dd05eb7 into master Jul 11, 2022
@fffonion fffonion deleted the perf/calculate-hash-string-buffer branch July 11, 2022 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants