TypeError on run_training_teardown method on the master branch #1999

SubodhDahal · 2020-05-29T15:09:08Z

🐛 Bug

I switched to the master branch in order to test the bugfix for #1919 but the same code that was running on the stable version 0.7.6 is not running anymore.

To Reproduce

Maybe just switching to the recent master branch would reproduce the issue.

GPU available: True, used: True
INFO:lightning:GPU available: True, used: True
No environment variable for node rank defined. Set as 0.
WARNING:lightning:No environment variable for node rank defined. Set as 0.
CUDA_VISIBLE_DEVICES: [0]
INFO:lightning:CUDA_VISIBLE_DEVICES: [0]
Traceback (most recent call last):
  File "/home/mycode/src/vae.py", line 457, in <module>
    main()
  File "/home/mycode/src/vae.py", line 450, in main
    run_model(hparams)
  File "/home/mycode/src/vae.py", line 386, in run_model
    trainer.fit(vae)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 894, in fit
    self.single_gpu_train(model)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 502, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 980, in run_pretrain_routine
    self.logger.save()
  File "/home/mycode/src/venv/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py", line 10, in wrapped_fn
    return fn(*args, **kwargs)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/pytorch_lightning/loggers/tensorboard.py", line 161, in save
    save_hparams_to_yaml(hparams_file, self.hparams)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 151, in save_hparams_to_yaml
    yaml.dump(hparams, fp)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 342, in represent_object
    return self.represent_mapping(
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 52, in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
  File "/home/mycode/src/venv/lib/python3.8/site-packages/yaml/representer.py", line 317, in represent_object
    reduce = data.__reduce_ex__(2)
TypeError: cannot pickle '_thread.lock' object
Error in atexit._run_exitfuncs:
TypeError: run_training_teardown() missing 1 required positional argument: 'self'

Expected behavior

I would've expected it to run exactly the same way as it does while using the 0.7.6 version

Environment

* CUDA:
	- GPU:
		- GeForce GTX 1050 Ti with Max-Q Design
	- available:         True
	- version:           10.2
* Packages:
	- numpy:             1.18.3
	- pyTorch_debug:     False
	- pyTorch_version:   1.5.0
	- pytorch-lightning: 0.7.7-dev
	- tensorboard:       2.2.1
	- tqdm:              4.46.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- ELF
	- processor:         x86_64
	- python:            3.8.2
Error in atexit._run_exitfuncs:
TypeError: run_training_teardown() missing 1 required positional argument: 'self'

How you installed PyTorch (conda, pip, source): pip

Additional context

The same error is present while running the collect_env_details.py script.

The text was updated successfully, but these errors were encountered:

Borda · 2020-05-29T20:52:41Z

@justusschock ^^

awaelchli · 2020-05-30T20:46:56Z

Probably this line here
https://github.com/PyTorchLightning/pytorch-lightning/blob/ceecf1cea92dc2d8c29b1364237ac9467abf2f9b/pytorch_lightning/trainer/training_loop.py#L309
should be
self.run_training_teardown()

Not sure

awaelchli · 2020-05-31T01:19:47Z

Investigated.
It is because the atexit.register decorator can only be applied to functions, not methods.
The self argument is not passed in.

This error should have shown up in the tests.

justusschock · 2020-06-01T10:46:52Z

@awaelchli so changing this to self.run_training_teardown would fix it?

awaelchli · 2020-06-01T16:24:03Z

No, I tested it, the problem is not there. The problem is that the atexit.register is applied to a method (which has self as an argument) but the decorator is meant for functions which don't get self as input.
It seems this is causing the problem.

justusschock · 2020-06-01T17:13:07Z

Yes. I think, that's why I explicitly passed the self argument :) I'll try to come up with another solution for this :)

awaelchli · 2020-06-01T17:14:44Z

Maybe we can wrap the cleanup code into a closure that binds self and then the decorator can be applied to this closure function. Not sure though, have not looked at the details

justusschock · 2020-06-01T17:15:53Z

Do you want to take this over? Otherwise I'd try to make some time for it later/tomorrow :)

williamFalcon · 2020-06-01T17:45:54Z

more broadly, why is that function needed? we already have teardown that works on ctrl+c no?
is this to teardown with a USSIG1 as well?

And we already have that signal registered...

https://github.com/PyTorchLightning/pytorch-lightning/blob/8b9b923ca8ad9fdb0ae22928de0029e7c2e7a782/pytorch_lightning/trainer/training_io.py#L206

justusschock · 2020-06-01T17:50:02Z

This would start teardown for all kills except SIGKILL (like SIGTERM etc.).

There are clusters (like non-SLURM) that also need this kind of signal handling. And I think we should do cleanup whenever a job ends if possible (either exit after program end or exit on error). Otherwise you may get issues with checkpointing etc.

Also this would maybe enable proper hparam logging with metrics (not sure about that though).

awaelchli · 2020-06-01T20:18:10Z

Do you want to take this over? Otherwise I'd try to make some time for it later/tomorrow :)

I think I better keep my hands away from it. I can't test the SLURM signals anyway due to lack of this setup.

Borda · 2020-06-01T20:59:35Z

Do you want to take this over? Otherwise I'd try to make some time for it later/tomorrow :)

I think I better keep my hands away from it. I can't test the SLURM signals anyway due to lack of this setup.

SLURM shall be also running on CPU :]

williamFalcon · 2020-10-05T01:10:08Z

@awaelchli please see comments. i'm not sure we should have this handler thing. i think it'll also break ddp

SubodhDahal added the help wanted Open to be worked on label May 29, 2020

Borda added the bug Something isn't working label May 29, 2020

Borda mentioned this issue May 30, 2020

Graceful shutdown on python interpreter exit #1631

Merged

5 tasks

awaelchli mentioned this issue Jun 12, 2020

[blocked by refactor] [WIP] graceful shutdown signal handling #2165

Closed

6 tasks

edenlightning assigned edenlightning and unassigned edenlightning Jul 29, 2020

edenlightning assigned awaelchli Sep 23, 2020

edenlightning added this to the 0.9.x milestone Sep 23, 2020

awaelchli mentioned this issue Sep 23, 2020

signal handling and teardown #3632

Closed

7 tasks

edenlightning modified the milestones: 0.9.x, 1.0 Oct 4, 2020

williamFalcon closed this as completed Oct 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeError on run_training_teardown method on the master branch #1999

TypeError on run_training_teardown method on the master branch #1999

SubodhDahal commented May 29, 2020

Borda commented May 29, 2020

awaelchli commented May 30, 2020 •

edited

Loading

awaelchli commented May 31, 2020 •

edited

Loading

justusschock commented Jun 1, 2020

awaelchli commented Jun 1, 2020

justusschock commented Jun 1, 2020

awaelchli commented Jun 1, 2020

justusschock commented Jun 1, 2020

williamFalcon commented Jun 1, 2020 •

edited

Loading

justusschock commented Jun 1, 2020

awaelchli commented Jun 1, 2020

Borda commented Jun 1, 2020

williamFalcon commented Oct 5, 2020

TypeError on run_training_teardown method on the master branch #1999

TypeError on run_training_teardown method on the master branch #1999

Comments

SubodhDahal commented May 29, 2020

To Reproduce

Expected behavior

Environment

Additional context

Borda commented May 29, 2020

awaelchli commented May 30, 2020 • edited Loading

awaelchli commented May 31, 2020 • edited Loading

justusschock commented Jun 1, 2020

awaelchli commented Jun 1, 2020

justusschock commented Jun 1, 2020

awaelchli commented Jun 1, 2020

justusschock commented Jun 1, 2020

williamFalcon commented Jun 1, 2020 • edited Loading

justusschock commented Jun 1, 2020

awaelchli commented Jun 1, 2020

Borda commented Jun 1, 2020

williamFalcon commented Oct 5, 2020

awaelchli commented May 30, 2020 •

edited

Loading

awaelchli commented May 31, 2020 •

edited

Loading

williamFalcon commented Jun 1, 2020 •

edited

Loading