Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use default play to predefine column types in update_db() #325

Merged
merged 5 commits into from
May 17, 2022

Conversation

mrcaseb
Copy link
Member

@mrcaseb mrcaseb commented May 17, 2022

This resolves #324

Added a new object to sysdata which is a single tibble row that uses random strings for character types with maximum length across the complete dataset (as of 2021).
This should make db drivers to correctly set datatypes in update_db() esp. for long string columns.

It is definitely possible that this needs an update at some point, if one or more of the defined string columns isn't big enough.

Thanks to @friscojosh for the idea. I hope this works.

@mrcaseb mrcaseb requested review from guga31bb and tanho63 May 17, 2022 17:52
Copy link
Member

@tanho63 tanho63 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

guga31bb
guga31bb previously approved these changes May 17, 2022
Copy link
Member

@tanho63 tanho63 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also just needs to hit redocument to bring in the redact_path arg

Copy link
Member

@tanho63 tanho63 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, maybe josh would be willing to test new methodology

@mrcaseb
Copy link
Member Author

mrcaseb commented May 17, 2022

nflreadr sitrep is redocumented. I had to update nflreadr first smh.
I hope the any:: part fixes the tidyverse problem in pkgdown

@mrcaseb mrcaseb enabled auto-merge May 17, 2022 18:06
@tanho63
Copy link
Member

tanho63 commented May 17, 2022

encouraging people Seb to always update their packages

@mrcaseb mrcaseb merged commit 80d2450 into master May 17, 2022
@mrcaseb mrcaseb deleted the dbdatatypes branch May 17, 2022 18:11
@friscojosh
Copy link

Testing now!

@mrcaseb
Copy link
Member Author

mrcaseb commented May 17, 2022

There is a good chance that non character datatypes are broken now because of the single row principle now. Let's hope it works

That's actually wrong. The first season written to the table is combined with the default play. So we have a complete season for the datatype method to set the types.

@mrcaseb
Copy link
Member Author

mrcaseb commented May 17, 2022

Ok, it's definitely broken because I did something stupid, hold on

@friscojosh
Copy link

friscojosh commented May 17, 2022

from 4.3.0.9011

── Update nflfastR Play-by-Play Database ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── nflfastR version 4.3.0.9011 ──
• 11:13:53 | Purging the complete data table "nflfastR_pbp_test" in your connected database...
• 11:13:54 | Starting download of 23 seasons between 1999 and 2021...
Error: Data too long for column 'side_of_field' at row 1 [1406]

@mrcaseb
Copy link
Member Author

mrcaseb commented May 17, 2022

from 4.3.0.9011

── Update nflfastR Play-by-Play Database ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── nflfastR version 4.3.0.9011 ──
• 11:13:53 | Purging the complete data table "nflfastR_pbp_test" in your connected database...
• 11:13:54 | Starting download of 23 seasons between 1999 and 2021...
Error: Data too long for column 'side_of_field' at row 1 [1406]

That's interesting. So nflfastR calls DBI::dbRemoveTable internally when a complete rebuild is forced. I assume that RMariaDB keeps the field list even when the table is removed.

@friscojosh
Copy link

friscojosh commented May 17, 2022

OK, I tried a fresh update to a new table and got the same error.

── Update nflfastR Play-by-Play Database ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── nflfastR version 4.3.0.9011 ──
ℹ 12:17:28 | Can't find the data table "nflfastR_pbp_test2" in your database. Will load the play by play data from scratch.
• 12:17:28 | Starting download of 23 seasons between 1999 and 2021...
Error: Data too long for column 'side_of_field' at row 1 [1406]

However, update_db() works as expected when the table is built manually after binding rows for all available data into one dataframe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use dummy play in update_db() to help predefine column types in some db drivers
4 participants