-
-
Notifications
You must be signed in to change notification settings - Fork 310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi index dataframes with datetime level cannot drop invalid rows #1469
Comments
I have a fix for this, Will raise a PR today |
It would also be useful to have the SchemaErrors object in the case that there are invalid rows, and we opt to drop them. I have a workflow whereby I run a validation, dropping invalid rows, and persist the error report of the invalid observations. This way I can track data quality over time. It would be useful to have that. I can add after or as part of the fix to this. |
Sounds good! Were you thinking the errors object would be in the |
…ai-oss#1469 Signed-off-by: Rory <rory@rorymcstay.com> Signed-off-by: Rory McStay <rory.mcstay@fulcrumasset.com>
…ai-oss#1469 Signed-off-by: Rory <rory@rorymcstay.com> Signed-off-by: Rory McStay <rory.mcstay@fulcrumasset.com>
Describe the bug
A clear and concise description of what the bug is.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
We use raw
eval
on interpretting string tuples. This fails under datetime types serialised as Timestamp(...) in a string.Easy work around would be to pass
{'Timestamp': pd.Timestamp}
in globals to evalExpected behavior
Invalid rows are dropped
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: