feat(pdk): increase the precision of JSON number encoding from 14 to 16 decimals

[ms2008]

Summary

Currently, when Kong is serializing JSON, it can only encode numbers with a precision of up to 14 decimals. For numbers with a precision exceeding 14 decimals, kong will output them using scientific notation, resulting in a loss of decimal precision. This also causes an issue where KIC cannot use integers exceeding 2^46 bits to configure the priority attribute of route.

Now, we are increasing this limitation to 16 decimals. With this change, we can safely store integers up to 2^53 bits without losing precision.

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix FTI-5515

[tzssangglass]

as a remember: if some else that use require("cjson.safe").new() or require("cjson").new(), the encode_number_precision(16) would be invalid

releated: #11389

[zhongweiy]

How are we going to solve or mitigate the concern from @tzssangglass ? Is it a bug in cjson package?

[ms2008]

Is it a bug in cjson package?

I don't think so. cjson.new is actually an instantiated copy of a separate Lua CJSON module. The new module has a separate persistent encoding buffer and default settings.

I think we can wrap the cjson.new method in the global patch and set cjson.encode_number_precision(16) by default for instances initialized through it, and that should solve the problem.

[tzssangglass]

I think we can wrap the cjson.new method in the global patch and set cjson.encode_number_precision(16) by default for instances initialized through it, and that should solve the problem.

work for me, but it is a break change.

[Tieske]

Didn't read up on the specific case here, but we ran into something similar before, see Tieske/lua-resty-ljsonschema#21 by @vm-001

Just leaving it here as info for whomever wants to double check.

[vm-001]

Just leaving it here as info for whomever wants to double check.

The PR Tieske/lua-resty-ljsonschema#21 fixed loss precision by string.format, and does not related to cjson lib.

[chronolaw]

Could we replace it with kong.tools.cjson?

[Tieske]

I doubt you can, since this is in globalpatches, which gets loaded very early on.

[ms2008]

You can't simply replace it, because currently, our global patch doesn't set cjson.decode_array_with_array_mt(true), whereas kong.tools.cjson does.

kong/kong/tools/cjson.lua

Line 4 in b7a8361

cjson.decode_array_with_array_mt(true)

[chronolaw]

Got. The code makes me a bit confusing.

[team-gateway-bot]

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-12019-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-12019-to-master-to-upstream
git checkout -b cherry-pick-12019-to-master-to-upstream
ancref=$(git merge-base 2a3a013766d99887f6a0416c2a6abcf0ecae9b27 9deee708395ba714a2813bcdba35638b9bc86bea)
git cherry-pick -x $ancref..9deee708395ba714a2813bcdba35638b9bc86bea