Skip to content

Commit

Permalink
use dill to store the spawn function as bytes
Browse files Browse the repository at this point in the history
when creating a worker, we pass in a spawn function. Dill is an improved version of pickle so dill can serialize many
more functions than pickle can. Thus we use dill to serialize the spawn function to bytes, which can then be serialized/deserialized by pickle when
spawning subprocesses. Then when the worker needs to run the spawn function, we can use dill to deserialize the bytes into the desired function.
  • Loading branch information
pattonw committed May 11, 2024
1 parent 27ad340 commit aabda6e
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions daisy/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import multiprocessing
import os
import queue
import dill

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -50,6 +51,14 @@ def __init__(self, spawn_function, context=None, error_queue=None):

self.start()

@property
def spawn_function(self):
return dill.loads(self._spawn_function)

@spawn_function.setter
def spawn_function(self, value):
self._spawn_function = dill.dumps(value)

def start(self):
"""Start this worker. Note that workers are automatically started when
created. Use this function to re-start a stopped worker."""
Expand Down

0 comments on commit aabda6e

Please sign in to comment.