Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error running on ddp (can't pickle local object 'SummaryTopic) with comet logger #1704

Closed
dvirginz opened this issue May 2, 2020 · 1 comment · Fixed by #2029
Closed
Labels
bug Something isn't working help wanted Open to be worked on

Comments

@dvirginz
Copy link

dvirginz commented May 2, 2020

I have the following problem running on ddp mode with cometlogger.
When I detach the logger from the trainer (i.e deletinglogger=comet_logger) the code runs.

Exception has occurred: AttributeError
Can't pickle local object 'SummaryTopic.__init__.<locals>.default'
  File "/path/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/path/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/path/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/path/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/path/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/path/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/path/site-packages/torch/multiprocessing/spawn.py", line 162, in spawn
    process.start()
  File "/path/site-packages/pytorch_lightning/trainer/trainer.py", line 751, in fit
    mp.spawn(self.ddp_train, nprocs=self.num_processes, args=(model,))
  File "/repo_path/train.py", line 158, in main_train
    trainer.fit(model)
  File "/repo_path/train.py", line 72, in main
    main_train(model_class_pointer, hyperparams, logger)
  File "/repo_path/train.py", line 167, in <module>
    main()
  File "/path/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/path/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/path/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/path/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/path/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
@dvirginz dvirginz added bug Something isn't working help wanted Open to be worked on labels May 2, 2020
@Borda
Copy link
Member

Borda commented May 3, 2020

@ceyzaguirre4 pls ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants