-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't use @load inside a package #613
Comments
Thanks for reporting! This is not expected. Can you instead do load("RandomForestClassifier", pkg="DecisionTree", modl=TestMLJ) and report any output (this version is verbose by default)? |
Sure, the output with julia> using TestMLJ
[ Info: Precompiling TestMLJ [4191ff91-1a08-48a9-a49b-84d00a67d6cb]
[ Info: Loading into module "TestMLJ":
import MLJModels ✔
import DecisionTree ✔
import MLJModels.DecisionTree_ WARNING: could not import MLJModels.DecisionTree_ into TestMLJ
ERROR: LoadError: UndefVarError: DecisionTree_ not defined
Stacktrace:
[1] getproperty(::Module, ::Symbol) at ./Base.jl:26
[2] top-level scope at /Users/bieganek/.julia/packages/MLJModels/BQAzu/src/loading.jl:124
[3] eval at ./boot.jl:331 [inlined]
[4] eval(::Expr) at /Users/bieganek/projects/TestMLJ/src/TestMLJ.jl:1
[5] load(::NamedTuple{(:name, :package_name, :is_supervised, :docstring, :hyperparameter_ranges, :hyperparameter_types, :hyperparameters, :implemented_methods, :is_pure_julia, :is_wrapper, :load_path, :package_license, :package_url, :package_uuid, :prediction_type, :supports_online, :supports_weights, :input_scitype, :target_scitype, :output_scitype),Tuple{String,String,Bool,String,NTuple{8,Nothing},NTuple{8,String},NTuple{8,Symbol},Array{Symbol,1},Bool,Bool,String,String,String,String,Symbol,Bool,Bool,UnionAll,UnionAll,DataType}}; modl::Module, verbosity::Int64, name::Nothing) at /Users/bieganek/.julia/packages/MLJModels/BQAzu/src/loading.jl:128
[6] load(::String; pkg::String, kwargs::Base.Iterators.Pairs{Symbol,Module,Tuple{Symbol},NamedTuple{(:modl,),Tuple{Module}}}) at /Users/bieganek/.julia/packages/MLJModels/BQAzu/src/loading.jl:135
[7] top-level scope at /Users/bieganek/projects/TestMLJ/src/TestMLJ.jl:6
[8] include(::Module, ::String) at ./Base.jl:377
[9] top-level scope at none:2
[10] eval at ./boot.jl:331 [inlined]
[11] eval(::Expr) at ./client.jl:449
[12] top-level scope at ./none:3
in expression starting at /Users/bieganek/projects/TestMLJ/src/TestMLJ.jl:6
ERROR: Failed to precompile TestMLJ [4191ff91-1a08-48a9-a49b-84d00a67d6cb] to /Users/bieganek/.julia/compiled/v1.4/TestMLJ/JCWpN_nW22B.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
[3] _require(::Base.PkgId) at ./loading.jl:1029
[4] require(::Base.PkgId) at ./loading.jl:927
[5] require(::Module, ::Symbol) at ./loading.jl:922 |
Oddly enough, the following does not reproduce the error: julia> module A
using MLJ
@load RandomForestClassifier pkg=DecisionTree
end
Main.A
julia> using .A |
EDIT: Added Having the following in the TestMLJ.jl file also produces a similar error: module TestMLJ
using MLJ
import DecisionTree
import MLJModels
const RandomForestClassifier = MLJModels.DecisionTree_.RandomForestClassifier
end julia> using TestMLJ
[ Info: Precompiling TestMLJ [4191ff91-1a08-48a9-a49b-84d00a67d6cb]
ERROR: LoadError: UndefVarError: DecisionTree_ not defined
Stacktrace:
[1] getproperty(::Module, ::Symbol) at ./Base.jl:26
[2] top-level scope at /Users/bieganek/projects/TestMLJ/src/TestMLJ.jl:8
[3] include(::Module, ::String) at ./Base.jl:377
[4] top-level scope at none:2
[5] eval at ./boot.jl:331 [inlined]
[6] eval(::Expr) at ./client.jl:449
[7] top-level scope at ./none:3
in expression starting at /Users/bieganek/projects/TestMLJ/src/TestMLJ.jl:8
ERROR: Failed to precompile TestMLJ [4191ff91-1a08-48a9-a49b-84d00a67d6cb] to /Users/bieganek/.julia/compiled/v1.4/TestMLJ/JCWpN_nW22B.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
[3] _require(::Base.PkgId) at ./loading.jl:1029
[4] require(::Base.PkgId) at ./loading.jl:927
[5] require(::Module, ::Symbol) at ./loading.jl:922 |
Do you happen to know a temporary workaround? I wanted to package up my code so that I could run it on the university cluster. |
Okay, this is a known issue which I suspect is somehow related to Requires. JuliaAI/MLJModels.jl#22 . Is it just the one package DecisionTree you want to use or are there are bunch of model-providing packages you want? |
One way to confirm my suspicion is for you to load something not provided by MLJModels. Can you try I'll try to get you a workaround by the end of my day. |
Yes, it's a Requires issue, it seems to me. I can load ImageClassifier (not provided by MLJModels) fine. |
Yeah, my code also uses XGBoost and LIBSVM (and MLJLinearModels). Not on my work machine right now. I'll double check tomorrow that I can do |
Requires is not working as expected unless you put the loading code in the TestMLJ.jlmodule TestMLJ
using MLJ
function do_stuff()
model = RandomForestClassifier()
X, y = @load_iris;
e = evaluate(model, X, y, measure=cross_entropy)
return e
end
function __init__()
eval(quote
load("RandomForestClassifier",
pkg="DecisionTree",
modl=TestMLJ)
end)
end
end module Then |
Thanks for the workaround! Unfortunately, I do use
Note that this also requires adding some dependencies to the Project.toml file. Statistics, Tables, and MLJModelInterface have to be added since they're used in different places in the
However, sometimes (although not always) when I do
|
Did you by any chance redefine any of the above methods. |
No, this is just the warning messages for TestMLJ. The TestMLJ module shown above is a MWE with no methods defined.
When I try that I get this error message:
The import ..DecisionTree As I'm guessing you've noticed, the various types in those warning messages are defined in DecisionTree.jl, but for the ScikitLearn.jl interface, not the MLJ interface. Obviously the sorts of code loading shenanigans that I have in TestMLJ are not ideal. 😂 Hopefully the various package authors will natively implement the MLJ interface soon. |
Yeah |
I think i may have a solution. module TestMLJ
using MLJ
import MLJModels
import DecisionTree
import LIBSVM
import XGBoost
#import MLJModels
#include(joinpath(MLJModels.srcdir, "DecisionTree.jl"))
using MLJModels.DecisionTree_: RandomForestClassifier
include(joinpath(MLJModels.srcdir, "XGBoost.jl"))
using .XGBoost_: XGBoostClassifier
include(joinpath(MLJModels.srcdir, "LIBSVM.jl"))
using .LIBSVM_: LinearSVC, SVC
end # module |
Hmm, that doesn't seem to work for me. I get this: julia> using TestMLJ
[ Info: Precompiling TestMLJ [4191ff91-1a08-48a9-a49b-84d00a67d6cb]
ERROR: LoadError: UndefVarError: DecisionTree_ not defined
Stacktrace:
[1] include(::Module, ::String) at ./Base.jl:377
[2] top-level scope at none:2
[3] eval at ./boot.jl:331 [inlined]
[4] eval(::Expr) at ./client.jl:449
[5] top-level scope at ./none:3
in expression starting at /Users/bieganek/projects/TestMLJ/src/TestMLJ.jl:12
ERROR: Failed to precompile TestMLJ [4191ff91-1a08-48a9-a49b-84d00a67d6cb] to /Users/bieganek/.julia/compiled/v1.4/TestMLJ/JCWpN_nW22B.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
[3] _require(::Base.PkgId) at ./loading.jl:1029
[4] require(::Base.PkgId) at ./loading.jl:927
[5] require(::Module, ::Symbol) at ./loading.jl:922 |
It seems like the cause of those warnings might actually be the Both of those functions have a line like this that might be causing problems: parentmodule(T).eval(ex) The reason I noticed this is because I made a fork of MLJModels where I got rid of EDIT: Maybe not. I commented out that part of the code and still get the warnings. 😂 |
@CameronBieganek Hurray. I have finally found what's causing the warnings. (c::TreePrinter)(depth) = DT.print_tree(c.tree, depth)
(c::TreePrinter)() = DT.print_tree(c.tree, 5) The above methods were somehow being redefined due to a bug Here is the module definition i used module TestMLJ
using MLJ
import DecisionTree
import LIBSVM
import XGBoost
import MLJModels
include(joinpath(MLJModels.srcdir, "DecisionTree.jl"))
using .DecisionTree_: RandomForestClassifier
include(joinpath(MLJModels.srcdir, "XGBoost.jl"))
using .XGBoost_: XGBoostClassifier
include(joinpath(MLJModels.srcdir, "LIBSVM.jl"))
using .LIBSVM_: LinearSVC, SVC
end Tested it 3 times already an i haven't seen any Warnings |
Thanks @OkonSamuel for jumping in here. Just so it is clear. I can confirm the original issue is only an issue with packages whose glue code is lazy loaded from MLJModels. The issue will ultimately be resolved with the planned "�disintegration" of MLJModels. See here for the progress on this: JuliaAI/MLJModels.jl#244 (comment) |
@OkonSamuel Awesome, thanks for looking into those warnings! @ablaom Yeah, the warnings that @OkonSamuel and I were discussing were a bit tangential to the issue in the original post. They just made me a little worried about my workaround. However, my code seems to be working fine despite the warnings, so it's all good. 😃 |
@CameronBieganek hope the warning are gone now? if so can we close this? |
Well, the original issue of not being able to load models inside a package is not completely resolved, though we do have a workaround. It might be worth keeping this issue open so that other people who encounter the same problem can find this issue. Although the issue title might not be conducive to discoverability... Of course there's also JuliaAI/MLJModels.jl#22 tracking this, so it's up to you guys. EDIT: Yes, the warnings are gone now. :) |
Ok, I changed the name to make it more discoverable, although now it just looks like a duplicate of #321. 😂 |
Thanks @CameronBieganek . I think i'll just let @ablaom decide here |
Closing in favour of #321. |
Reproducible Example
Create a package as follows.
Package structure
Project.toml
TestMLJ.jl
Project environment
Error
The text was updated successfully, but these errors were encountered: