-
Notifications
You must be signed in to change notification settings - Fork 129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decorated cache and specialized data storage #971
Comments
A better use: increase the cache performance beyond StrategyFrom @richfitz: richfitz/storr#77 (comment)
Command APIUsers will have special
Example: plan <- drake_plan(
data = return_fst(get_big_data_frame()),
model = return_hdf5(fit_big_keras_model(data))
)
StorageLet's use a
Retrieval
Garbage collection
ImplementationThe implementation will be a decorated cache: an outer Benefits
Related |
We should keep schnorr/starvz#6 in mind as well. |
Let's implement a decorated driver too so the API is completely the same. |
Nope. We can just use the existing driver. No need to decorate that. We just have to make sure it's there (and public). |
Change of directionDisadvantages of
ProposalLet the data format be a custom column in the plan. plan <- drake_plan(
data = target(get_big_data_frame(), format = "fst"),
model = target(fit_big_keras_model(data), format = "keras")
) The internals in https://github.com/ropensci/drake/tree/971 will not need much adjustment. All we need is an |
Remaining for this issue: use |
Now, most of the hash tables are in the decorated cache where they belong. We might put some of these in a dedicated encoder, but that's for #968. |
Prework
drake
's code of conduct.Description
I am thinking about a decorated
storr
to contain things likeconfig$progress_hashmap
,config$cache_path
(a guarantee on the path), and the encoders. Maybe it could also help history somehow.The text was updated successfully, but these errors were encountered: