-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MetaCreator and MetaEphemeral for better pickling #76
Conversation
The usage of metaclass allow us to stop registering Ephemeral classes in the gp module global scope. We also register the __reduce__ function of MetaEphemeral so Ephemeral could easily be pickled, as long as the ephemeral function is picklable (i.e.: no lambda function). The metaclass also has a cache based on class id to avoid creating new object for the same ephemeral when unpickling.
lambda can't pickled and will never be picklable. The test was failing because of Python incapicity to pickle lambdas, not because of MetaEphemeral logic. Therefore, I am replacing lambda by a partial and plan to do it for every examples in GP.
Merged DEAP pull request DEAP#76.
@fmder Regarding problem 1 and relative PR, you may want to consider making
Something similar could be done for problem 2, I guess. Sources: |
Any further feedback on this pull request? |
... nothing yet? Multiple multiprocessing libs depends on this to work non-trivial GP settings |
Is there a chance to merge this? I still have to use some patched code: https://github.com/ChristopherMayes/Xopt/blob/master/xopt/creator.py |
Is it possible that we might be able to get this merged after 8 years, please? This one is really important to make DEAP work under Ray. |
This pull-request solves two problems we have in DEAP using Python metaclass.
creator.create
outside the global scope would prevent instances of the created classes to be pickled. This complicated things when trying to communicate the objects in a context of distributed computation.PrimitiveSet
is instantiated in the main module global scope. The reason is that when adding ephemeral constant to a PrimitiveSet, a new class is generated withtype
and it is added to thegp
module scope. This is hackish, but allowed to a certain extend the pickling ofPrimitiveTree
that would containEphemerals
.The solutions to each problem are similar and are inspired by Python issue 7689 Pickling of classes with a metaclass and copy_reg. The idea is to replace calls to
type
by our own metaclass and to register incopy_reg
how classes were created using this metaclass. This means that when we pickle an instance of a class created dynamically, we also pickled how this class can be built from scratch.This pull request introduces two metaclasses, one for each problem.
creator.MetaCreator
gp.MetaEphemeral
The idea should be exploited further. For example, we could have a
MetaFitness
that would make sure theweights
attribute is set when creating the class. The behavior of theFitness
class could also be adjusted, so for example the evaluation function would not have to return a tuple for single objective problem.N.B.: This is a repost of PR #58. I have closed the former as it was not in a proper feature branch.