How to handle definitions in the same file?

[liamhuber]

I'm still playing around integrating pympipool with pyiron_workflow in this PR. I had been struggling with a couple of tests that hang local, and on the CI gave errors like ModuleNotFoundError: no module named 'unit' or no module named 'test_macro'.

I managed to whittle things down to this minimal(ish) working example:

Suppose I run a test file, e.g. test_pympipool.py, with the following contents:

from unittest import TestCase

from pympipool import PyMPIExecutor


def add_two(x):
    return x + 2


class TestPyMPIExecutor(TestCase):

    def test_local(self):
        # Works perfectly
        def add_one(x):
            return x + 1

        class Foo:
            def add(self, x):
                return add_one(x)

        foo = Foo()
        executor = PyMPIExecutor()
        fs = executor.submit(foo.add, 1)
        self.assertEqual(2, fs.result())

    def test_semi_local(self):
        # Hangs
        # CI logs make it look like cloudpickle is trying to import from this module,
        # but can't find it
        class Foo:
            def add(self, x):
                return add_two(x)

        foo = Foo()
        executor = PyMPIExecutor()
        fs = executor.submit(foo.add, 1)
        self.assertEqual(3, fs.result())

When the function and class are defined in exactly the same scope, everything works perfectly. Not shown but also working perfectly is if I imported the add_x function from some other accessible location. The only thing that seems to be breaking is when I make a local definition that relies on something else defined in a different scope but the same file.

In a very hand-wavey way this makes sense to me -- the executor sees that add_two is not defined in this scope and so decides its something that can be pickled by reference; the only problem is that the file it's defined in is not in my import path! In contrast, when add_one is defined in exactly the same scope as the object being pickled, it's clear to the code that this object also needs to be pickled by value.

My questions are:

• Should this count as a bug that can be fixed here in pympipool, or is what I'm asking for unreasonable? In the latter case, what is the recommended syntax for same-file-different-scope definitions like this, i.e. how to succinctly make sure everything is added to path, or can I flag particular stuff for by-value treatment?
• On the CI it's super helpful to get the hard crash and the import error message -- why is it only hanging on my local machine, and can we force a crash under these conditions there too?

[jan-janssen]

I tired the test in #236 - somehow it works fine.

[liamhuber]

I tired the test in #236 - somehow it works fine.

Aha, indeed. In the pympipool/tests directory it's working fine for me too. Still hangs in the test dir of other projects though. That smells to me like it's either an out-of-date version or a path issue; I'm updating pympipool on my conda env to check the former. Would be lovely if that's all it is.

[liamhuber]

Ok, super!!! I guess I have a hiccup in my pycharm setup that it was not actually using the repo version of the code -- updating my conda environment thankfully did the trick.