Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MetaCreator and MetaEphemeral for better pickling #76

Closed
wants to merge 6 commits into from

Conversation

cmd-ntrf
Copy link
Member

@cmd-ntrf cmd-ntrf commented May 5, 2015

This pull-request solves two problems we have in DEAP using Python metaclass.

  1. Calling creator.create outside the global scope would prevent instances of the created classes to be pickled. This complicated things when trying to communicate the objects in a context of distributed computation.
  2. Genetic programming (GP) ephemeral constant could not be pickled unless the PrimitiveSet is instantiated in the main module global scope. The reason is that when adding ephemeral constant to a PrimitiveSet, a new class is generated with type and it is added to the gp module scope. This is hackish, but allowed to a certain extend the pickling of PrimitiveTree that would contain Ephemerals.

The solutions to each problem are similar and are inspired by Python issue 7689 Pickling of classes with a metaclass and copy_reg. The idea is to replace calls to type by our own metaclass and to register in copy_reg how classes were created using this metaclass. This means that when we pickle an instance of a class created dynamically, we also pickled how this class can be built from scratch.

This pull request introduces two metaclasses, one for each problem.

  1. creator.MetaCreator
  2. gp.MetaEphemeral

The idea should be exploited further. For example, we could have a MetaFitness that would make sure the weights attribute is set when creating the class. The behavior of the Fitness class could also be adjusted, so for example the evaluation function would not have to return a tuple for single objective problem.

N.B.: This is a repost of PR #58. I have closed the former as it was not in a proper feature branch.

@cmd-ntrf cmd-ntrf added this to the 2.0 milestone May 5, 2015
cmd-ntrf and others added 6 commits May 4, 2015 21:57
The usage of metaclass allow us to stop
registering Ephemeral classes in the gp
module global scope. We also register the
__reduce__ function of MetaEphemeral so
Ephemeral could easily be pickled, as long
as the ephemeral function is picklable
(i.e.: no lambda function). The metaclass
also has a cache based on class id to
avoid creating new object for the same
ephemeral when unpickling.
lambda can't pickled and will never be picklable.
The test was failing because of Python incapicity
to pickle lambdas, not because of MetaEphemeral
logic. Therefore, I am replacing lambda by a partial
and plan to do it for every examples in GP.
@cmd-ntrf cmd-ntrf assigned cmd-ntrf and fmder and unassigned cmd-ntrf and fmder May 5, 2015
@cmd-ntrf cmd-ntrf mentioned this pull request Dec 1, 2015
jbrant pushed a commit to jbrant/deap that referenced this pull request May 16, 2018
tommmlij added a commit to tommmlij/deap that referenced this pull request Apr 18, 2019

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
Merged DEAP pull request DEAP#76.
@Nitrooo
Copy link

Nitrooo commented Jul 4, 2019

@fmder Regarding problem 1 and relative PR, you may want to consider making creator compatible with dill too, a popular alternative to pickle, which is part of pathos (by the same author), a package that enables "parallel graph management and execution in heterogeneous computing". It's very useful in those scenarios where multiprocessing/pickle fail. This is the code you may want to add to the patched creator.py:

try:
    import dill
except ImportError:
    pass
else:
    def recreate_mc(name, base, dct):
        class_ = MetaCreator(name, base, dct)
        globals()[name] = class_
        return class_


    @dill.register(MetaCreator)
    def save_mc(pickler, obj):
        pickler.save_reduce(recreate_mc, (obj.reduce_args[0], obj.reduce_args[1], obj.reduce_args[2],), obj=obj)

Something similar could be done for problem 2, I guess.

Sources:
https://github.com/uqfoundation/dill
https://github.com/uqfoundation/pathos
https://stackoverflow.com/questions/27351980/how-to-add-a-custom-type-to-dills-pickleable-types

@berndtlindner
Copy link

Any further feedback on this pull request?
This feature would be of great use to get spark working with DEAP (#268)

@gfviegas
Copy link

... nothing yet? Multiple multiprocessing libs depends on this to work non-trivial GP settings

@ChristopherMayes
Copy link

Is there a chance to merge this? I still have to use some patched code: https://github.com/ChristopherMayes/Xopt/blob/master/xopt/creator.py

@bje-
Copy link
Contributor

bje- commented Jul 15, 2023

Is it possible that we might be able to get this merged after 8 years, please? This one is really important to make DEAP work under Ray.

@fmder fmder mentioned this pull request Jul 19, 2023
@fmder fmder closed this in #711 Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants