Don't import luigi.worker from __init__.py #438
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
lgtm! |
There's some really fishy bug going on with multiprocessing that we are investigating. It *seems* like (but still to be confirmed) that luigi.hadoop ends up importing luigi.worker which in turn imports multiprocessing. multiprocessing *seems* to have a bug causing an exception in the global scope in case the temporary directory is too long (bind() fails if the temp path is > 107 characters, this is a known UNIX issue)
Exception from a Hadoop job:
__import__('pkg_resources').declare_namespace(__name__)
Process SyncManager-1:
Traceback (most recent call last):
File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run()
File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 517, in
_run_server
server = cls._Server(registry, address, authkey, serializer)
File "/usr/lib/python2.6/multiprocessing/managers.py", line 136, in
__init__
self.listener = Listener(address=address, backlog=5)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 106, in
__init__
self._listener = SocketListener(address, family, backlog)
File "/usr/lib/python2.6/multiprocessing/connection.py", line 227, in
__init__
self._socket.bind(address)
File "<string>", line 1, in bind
error: AF_UNIX path too long
Even if the theory above isn't correct, this still cleans up some import dependencies, by moving Event to its own module
erikbern
force-pushed
the
erikbern/import-worker-event
branch
from
b8a060b
to
bb5d81d
Compare
|
Thanks for the proper commit message! :) LGTM |
|
I've always suspected there's an inverse correlation between commit size and commit message length :) |
erikbern
pushed a commit
that referenced
this pull request
Sep 9, 2014
Don't import luigi.worker from __init__.py
Tarrasch
added a commit
that referenced
this pull request
Aug 1, 2016
It was introduced in #438. Would that bug be reintroduced I'm think that (1) we have tests nowadays for it and (2) this comment doesn't help any more.
Tarrasch
added a commit
that referenced
this pull request
Aug 24, 2016
It was introduced in #438. Would that bug be reintroduced I'm think that (1) we have tests nowadays for it and (2) this comment doesn't help any more.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There's some really fishy bug going on with multiprocessing that we are investigating. It seems like (but still to be confirmed) that luigi.hadoop ends up importing luigi.worker which in turn imports multiprocessing. multiprocessing seems to have a bug causing an exception in the global scope in case the temporary directory is too long (bind() fails if the temp path is > 107 characters, this is a known UNIX issue)
Even if the theory above isn't correct, this still cleans some import dependencies, by moving Event to its own module