Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AsyncValidatorService Update #1068

Merged
merged 13 commits into from
Sep 17, 2024
Merged

AsyncValidatorService Update #1068

merged 13 commits into from
Sep 17, 2024

Conversation

CalebCourier
Copy link
Collaborator

@CalebCourier CalebCourier commented Sep 12, 2024

PR Summary

  • Refactors validator_service.py into a directory with different files for the different implementations of ValidatorService
  • Updates AsyncValidatorService to always run validators concurrently
  • Uses the default executor on the obtained event loop so that the user can specify threads vs processes if they want/need to.
  • Performs corrections on individual ValidationResults instead of many
  • Includes uv extra which installs uvloop for use within validator service
  • Increases Unit and Integration test coverage of AsynvValidatorService
  • Introduces Validator.async_validate
    • This allows users to create validators with async validation logic
    • This allows validation to be async all the way down to enable better concurrency
    • Backfills existing validators by running the synchronous Validator.validate method in the event loop's default executor
  • Adds a ser/de wrapped merge function to allow fix_value merging for builtin types plus Pydantic models and Dataclasses

TODO

  • Profile updated AsyncValidatorService vs previous implementation with and without run_in_separate_process on validators
  • Better type custom onfail function
  • Document on fail behaviour

@CalebCourier CalebCourier requested review from zsimjee, nichwch and a team September 12, 2024 17:43
@CalebCourier CalebCourier marked this pull request as ready for review September 12, 2024 22:10
docs/concepts/concurrency.md Outdated Show resolved Hide resolved
docs/concepts/concurrency.md Outdated Show resolved Hide resolved
docs/concepts/concurrency.md Outdated Show resolved Hide resolved
dtam
dtam previously approved these changes Sep 17, 2024
Copy link
Contributor

@dtam dtam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some spelling updates in the doc updates but otherwise lgtm

Co-authored-by: dtam <david.cheung.tam@gmail.com>
CalebCourier and others added 2 commits September 17, 2024 13:12
Co-authored-by: dtam <david.cheung.tam@gmail.com>
Co-authored-by: dtam <david.cheung.tam@gmail.com>
@dtam dtam merged commit 2ec1ee5 into main Sep 17, 2024
12 checks passed
@dtam dtam deleted the async-validator-service branch September 17, 2024 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants