-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sqlite direct download #37
Conversation
To test: remotes::install_github("bcgov/rems@sqlite-path")
rems:::burn_it_down()
### Restart R session before proceeding ###
library(rems)
download_historic_data()
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
to_date = as.Date("1991-05-11")) |
this is a great idea @ateucher ! I have followed the instructions above. I am able to initiate downloading the data but have now failed on 5 attempts (all at different stages of completion). I do feel that my poor Haida Gwaii may be the culprit. Hopefully someone else can test as well to confirm. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto, great idea. I got a couple function errors when I ran it. Am I missing something?
` > download_historic_data()
Error in .remsCache$exists("cache_dates") :
attempt to apply non-function
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
to_date = as.Date("1991-05-11"))
Error in .remsCache$exists("cache_dates") :
attempt to apply non-function`
@HeatherGranger can you try restarting your R session prior to running |
Ah yes, I forgot to mention that. Instructions updated |
That worked. It's currently downloading. I'll report back on it's success! |
Success! downloaded, stored and read in the data. |
Thanks for testing @HeatherGranger - that's great! |
@sebdalgarno I've just pushed a commit to allow passing download_historic_data(ask = FALSE, force = TRUE, httr_config = timeout(5000)) |
@ateucher I'm trying now with |
Similar success over here 🎉 |
Thanks @boshek - that's great! |
Well I set up a better cell signal booster in my tree this morning and can now say I have successfuly downloaded the historic dataset! thanks @ateucher for the changes. |
Excellent! I have a bit more cleanup to do then I will merge this to master |
One thing I haven't tested is updating if you had an old version without using remotes::install_github("bcgov/rems@sqlite-path")
library(rems)
download_historic_data()
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
to_date = as.Date("1991-05-11")) |
@ateucher done, all seems to be working! |
This PR does two major things:
Provides the historic sqlite database as a direct download from a rems release. This means that
download_historic_data()
does not involve reading the csv from the BCDC and iteratively loading it into sqlite. That process is now done by the package maintainer anddownload_historic_data()
simply downloads the sqlite file and stores it in the rems data dir.Allow a user to specify/customize the location of the
sqlite
database via an optionrems.historic.path
- (getOption("rems.historic.path", default = rems_data_dir())
)