Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib-cove performance opportunities #114

Open
1 of 3 tasks
jpmckinney opened this issue Jul 10, 2023 · 0 comments
Open
1 of 3 tasks

lib-cove performance opportunities #114

jpmckinney opened this issue Jul 10, 2023 · 0 comments

Comments

@jpmckinney
Copy link
Member

jpmckinney commented Jul 10, 2023

  • See Performance option: Reduce traversals of entire data OpenDataServices/lib-cove#65 which refactors get_json_data_generic_paths, get_fields_present_with_examples, get_additional_fields_info, fields_present_generator
  • The other hotspot (outside jsonschema, which is the majority of the time) is get_additional_codelist_values. It calls _generate_data_path and then rejects/ignores a bunch of its output. It is the only function to call that function, so there's an opportunity to merge the path_no_num and not in schema_obj.extended_codelist_schema_paths logic into _generate_data_path.
  • We can cache the our_validator = validator( calls. For example: Allow lib-cove-* libraries to provide their own function for returning a validator. Then, have it cache similarly to how SchemaOCDS is cached in Kingfisher Process.

curl -Lo 1__1__5.zip https://github.com/open-contracting/standard/archive/1__1__5.zip

python -m cProfile -o code.prof -m libcoveocds --skip-aggregates --additional-checks none --standard-zip 1__1__5.zip 2017_0.json; gprof2dot -f pstats code.prof | dot -Tpng -o output.png; open output.png

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant