Sqlite direct download

[ateucher]

This PR does two major things:

1. Provides the historic sqlite database as a direct download from a rems release. This means that download_historic_data() does not involve reading the csv from the BCDC and iteratively loading it into sqlite. That process is now done by the package maintainer and download_historic_data() simply downloads the sqlite file and stores it in the rems data dir.

2. Allow a user to specify/customize the location of the sqlite database via an option rems.historic.path - (getOption("rems.historic.path", default = rems_data_dir()))

[ateucher]

To test:

remotes::install_github("bcgov/rems@sqlite-path")
rems:::burn_it_down()

### Restart R session before proceeding ###

library(rems)
download_historic_data()
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
                   to_date = as.Date("1991-05-11"))

[ateucher]

@HeatherGranger @sebdalgarno

[sebdalgarno]

this is a great idea @ateucher ! I have followed the instructions above. I am able to initiate downloading the data but have now failed on 5 attempts (all at different stages of completion). I do feel that my poor Haida Gwaii may be the culprit. Hopefully someone else can test as well to confirm.

[HeatherGranger]

Ditto, great idea. I got a couple function errors when I ran it. Am I missing something?

` > download_historic_data()
Error in .remsCache$exists("cache_dates") :
attempt to apply non-function

read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
to_date = as.Date("1991-05-11"))
Error in .remsCache$exists("cache_dates") :
attempt to apply non-function`

[sebdalgarno]

@HeatherGranger can you try restarting your R session prior to running library(rems) and all that follows?

[ateucher]

Ah yes, I forgot to mention that. Instructions updated

[HeatherGranger]

That worked. It's currently downloading. I'll report back on it's success!

[HeatherGranger]

Success! downloaded, stored and read in the data.

[ateucher]

Thanks for testing @HeatherGranger - that's great!

[ateucher]

@sebdalgarno I've just pushed a commit to allow passing httr::GET configurations. Can you try with a really long timeout threshold (timeout() value is in seconds)? eg.

download_historic_data(ask = FALSE, force = TRUE, httr_config = timeout(5000))

[sebdalgarno]

@ateucher I'm trying now with timeout(5000) as suggested. Will report back soon!

[boshek]

Similar success over here 🎉

[ateucher]

Thanks @boshek - that's great!

[sebdalgarno]

Well I set up a better cell signal booster in my tree this morning and can now say I have successfuly downloaded the historic dataset! thanks @ateucher for the changes.

[ateucher]

Excellent! I have a bit more cleanup to do then I will merge this to master

[ateucher]

One thing I haven't tested is updating if you had an old version without using burn_it_down(). @j-krogh can you please try the above without using burn_it_down()? i.e.:

remotes::install_github("bcgov/rems@sqlite-path")
library(rems)
download_historic_data()
read_historic_data(emsid = "0400203", from_date = as.Date("1984-11-20"),
                   to_date = as.Date("1991-05-11"))

[j-krogh]

@ateucher done, all seems to be working!