Tweaks to the model API for supporting machine serialization #429

ablaom · 2020-10-01T01:17:48Z

This PR makes tweaks to the (experimental and unpublicised) API for models with special serizialization requirements. It is not breaking for users. I have made it in response to failures of the existing API to handle the XGBoost serialization issue.

The proposed API is outlined in the draft documentation given below, for adding to "Adding New Models for General Use". As a POC I have implemented and locally tested the new API at a forthcoming external glue code package for XGBoost.

Very roughly, the PR enables: (i) automatic generation of model-specific serialization formats in addition to a self-contained MLJ/JLSO format; (ii) a workaround for "non-persistent" representations of learned parameters, such as pointers generated by wrapped C code, provided the model-providing package already implements some kind of serialization.

@yalwan-iqvia @OkonSamuel Be great to get your feedback.

I will leave this open for comment until Monday 11th.

Draft documentation

Serialization

The MLJ user can serialize and deserialize a machine, which means
serializing/deserializing:

the associated Model object (storing hyperparameters)
the fitresult (learned parameters)
the report generating during training

These are bundled into a single file or IO stream specified by the
user using the package JLSO. There are two scenarios in which a new
MLJ model API implementation will want to overload two additional
methods save and restore to support serialization:

The algorithm-providing package already has it's own serialization
format for learned parameters and/or hyper-parameters, which users
may want to access. In that case the implementation overloads save.
The fitresult is not a sufficiently persistent object; for
example, it is a pointer passed from wrapped C code. In that case
the implementation overloads save and restore.

In case 2, 1 presumably holds also, for otherwise MLJ serialization is
probably not going to be possible without changes to the
algorithm-providing package. An example is given below.

Note that in case 1, MLJ will continue to create it's own
self-contained serialization of the machine. Below filename refers
to the corresponding serialization file name, as specified by the
user, but with any final extension (e.g., ".jlso", ".gz") removed. If
the user has alternatively specified an IO object for serialization,
then filename is a randomly generated numeric string.

The save method

MMI.save(filename, model::SomeModel, fitresult; kwargs...) -> serializable_fitresult

Implement this method to serialize using a format specific to models
of type SomeModel. The fitresult is the first return value of
MMI.fit for such model types; kwargs is a list of keyword
arguments specified by the user and understood to relate to a some
model-specific serialization (cannot be format=... or
compression=...). The value of serializable_fitresult should be a
persistent representation of fitresult, from which a correct and
valid fitresult can be reconstructed using restore (see
below).

The fallback of save performs no action and returns fitresult.

The restore method

MMI.restore(filename, model::SomeModel, serializable_fitresult) -> fitresult

Implement this method to reconstruct a fitresult (as returned by
MMI.fit) from a persistent representation constructed using
MMI.save as described above.

The fallback of restore returns serializable_fitresult.

Example

Below is an example drawn from MLJ's XGBoost wrapper. In this example
the fitresult returned by MMI.fit is a tuple (booster, a_target_element) where booster is the XGBoost.jl object storing
the learned parameters (essentially a pointer to some object created
by C code) and a_target_element is an ordinary CategoricalValue
used to track the target classes (a persistent object, requiring no
special treatment).

function MLJModelInterface.save(filename,
                                ::XGBoostClassifier,
                                fitresult;
                                kwargs...)
    booster, a_target_element = fitresult

    xgb_filename = string(filename, ".xgboost.model")
    XGBoost.save(booster, xgb_filename)
    persistent_booster = read(xgb_filename)
    @info "Additional XGBoost serialization file \"$xgb_filename\" generated. "
    return (persistent_booster, a_target_element)
end

function MLJModelInterface.restore(filename,
                                   ::XGBoostClassifier,
                                   serializable_fitresult)
    persistent_booster, a_target_element = serializable_fitresult

    xgb_filename = string(filename, ".tmp")
    open(xgb_filename, "w") do file
        write(file, persistent_booster)
    end
    booster = XGBoost.Booster(model_file=xgb_filename)
    rm(xgb_filename)
    fitresult = (booster, a_target_element)
    return fitresult
end

Add a minor check to the arguments of range

codecov-commenter · 2020-10-01T01:29:36Z

Codecov Report

Merging #429 into master will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #429      +/-   ##
==========================================
+ Coverage   81.83%   81.86%   +0.03%     
==========================================
  Files          38       38              
  Lines        2703     2713      +10     
==========================================
+ Hits         2212     2221       +9     
- Misses        491      492       +1

Impacted Files	Coverage Δ
src/hyperparam/one_dimensional_ranges.jl	`80.76% <100.00%> (-1.59%)`	⬇️
src/interface/model_api.jl	`92.30% <100.00%> (-0.55%)`	⬇️
src/machines.jl	`84.35% <100.00%> (+1.14%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a4920c...29a2797. Read the comment docs.

src/machines.jl

yalwan-iqvia · 2020-10-02T14:19:05Z

src/interface/model_api.jl

-    dict = JLSO.load(file)
-    return dict[:model], dict[:fitresult], dict[:report]
-end
+MLJModelInterface.save(filename, model, fitresult; kwargs...) = fitresult


So just to be clear, this is fallback to an asumed serialisable function and if this doesnt work the method implementer overrides this to produce a serialisable fitresult?

Yes, although I would say seriasable object not function. It may not be a function (and usually isn't).

Of course, thanks for clarification

OkonSamuel

Clever.
LGTM

codecov-io · 2020-10-12T23:44:23Z

Codecov Report

Merging #429 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #429      +/-   ##
==========================================
+ Coverage   81.83%   81.84%   +0.01%     
==========================================
  Files          38       38              
  Lines        2703     2710       +7     
==========================================
+ Hits         2212     2218       +6     
- Misses        491      492       +1

Impacted Files	Coverage Δ
src/hyperparam/one_dimensional_ranges.jl	`80.76% <100.00%> (-1.59%)`	⬇️
src/interface/model_api.jl	`92.30% <100.00%> (-0.55%)`	⬇️
src/machines.jl	`84.02% <100.00%> (+0.81%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7a4920c...0e368ae. Read the comment docs.

ablaom · 2020-10-13T00:32:54Z

@OkonSamuel @yalwan-iqvia Thanks for this feedback.

ablaom and others added 5 commits September 11, 2020 12:00

add a minor check to the arguments of range

95b3c32

Merge pull request #426 from alan-turing-institute/range-checks

2cb4eac

Add a minor check to the arguments of range

Implement new save/restore API

839cda8

truncate .jlso from filename

29b1fa2

no, truncate any final extension

29a2797

ablaom added the API label Oct 1, 2020

ablaom marked this pull request as draft October 1, 2020 01:42

yalwan-iqvia reviewed Oct 2, 2020

View reviewed changes

src/machines.jl Outdated Show resolved Hide resolved

yalwan-iqvia reviewed Oct 2, 2020

View reviewed changes

OkonSamuel requested changes Oct 5, 2020

View reviewed changes

OkonSamuel approved these changes Oct 5, 2020

View reviewed changes

ablaom marked this pull request as ready for review October 12, 2020 23:30

use splitext

0e368ae

ablaom merged commit 2b01dae into master Oct 13, 2020

This was referenced Oct 13, 2020

For a 0.15.3 release #434

Merged

Update manual for new serialization API. No release JuliaAI/MLJ.jl#671

Merged

no method matching _process_model_def(::Expr) when importing JuliaAI/MLJScikitLearnInterface.jl#21

Closed

DilumAluthge deleted the restore branch January 23, 2022 20:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tweaks to the model API for supporting machine serialization #429

Tweaks to the model API for supporting machine serialization #429

ablaom commented Oct 1, 2020 •

edited

Loading

codecov-commenter commented Oct 1, 2020 •

edited

Loading

yalwan-iqvia Oct 2, 2020

ablaom Oct 12, 2020

yalwan-iqvia Oct 13, 2020

OkonSamuel left a comment

codecov-io commented Oct 12, 2020 •

edited

Loading

ablaom commented Oct 13, 2020

Tweaks to the model API for supporting machine serialization #429

Tweaks to the model API for supporting machine serialization #429

Conversation

ablaom commented Oct 1, 2020 • edited Loading

Draft documentation

Serialization

The save method

The restore method

Example

codecov-commenter commented Oct 1, 2020 • edited Loading

Codecov Report

yalwan-iqvia Oct 2, 2020

Choose a reason for hiding this comment

ablaom Oct 12, 2020

Choose a reason for hiding this comment

yalwan-iqvia Oct 13, 2020

Choose a reason for hiding this comment

OkonSamuel left a comment

Choose a reason for hiding this comment

codecov-io commented Oct 12, 2020 • edited Loading

Codecov Report

ablaom commented Oct 13, 2020

ablaom commented Oct 1, 2020 •

edited

Loading

codecov-commenter commented Oct 1, 2020 •

edited

Loading

codecov-io commented Oct 12, 2020 •

edited

Loading