OpenAPI/Swagger parsing is (?) performance bottleneck

[justinsb]

I was wondering why running kustomize (& kpt) is not effectively instant, and I tracked down a large performance bottleneck - initializing the OpenAPI / parsing the swagger.

I believe this is the most significant constant-overhead for kustomize - obviously on very large kustomizations the data manipulation will eventually dominate.

I sent #3669 which adds a benchmark to quantify the impact; based on that it takes about 900ms to json parse the swagger itself; it takes about 20ms to un-gzip from the data embedded into the binary. These roughly tally with the performance overhead I see in the real world.

This particularly matters for e.g. kpt, where a kpt setter which uses kyaml also takes at least 1 second to run.

I think it would be interesting to cache the deserialized form.

But there's also some low-hanging fruit, e.g. the swagger UnmarshalJSON code calls json.Unmarshal twice over the same data (once for SwaggerProps, once for VendorExtensible). But that only gets us 2x, whereas caching (I'd SWAG) could be 100-1000x.

[Shell32-Natsu]

@natasha41575

[natasha41575]

Thank you for filing the issue, will look into it

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[natasha41575]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[natasha41575]

/remove-lifecycle rotten

[natasha41575]

Discussed offline, one option would be to try to switch to using kube-openapi, which stores the the openapi data in proto form. This could be worth looking into.

[mengqiy]

/assign

[natasha41575]

Only took a year, but I believe this is fixed now by #4568 :)