(fgdal) dongzengao@DONG:/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg$ bash train_with_sd.sh /home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** Traceback (most recent call last): File "train_src.py", line 13, in Traceback (most recent call last): File "train_src.py", line 13, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in bootstrap() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in return _bootstrap._gcd_import(name[level:], package, level) ImportError : bootstrap()libGL.so.1: cannot open shared object file: No such file or directory File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2880) of binary: /home/dongzengao/anaconda3/envs/fgdal/bin/python Traceback (most recent call last): File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/runpy.py", line 192, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run elastic_launch( File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ train_src.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-04-21_23:16:17 host : DONG. rank : 1 (local_rank: 1) exitcode : 1 (pid: 2881) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-04-21_23:16:17 host : DONG. rank : 0 (local_rank: 0) exitcode : 1 (pid: 2880) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ /home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** Traceback (most recent call last): File "train_SEmsf_dnt_adv_BD+FD_FreezeBackbone.py", line 15, in Traceback (most recent call last): File "train_SEmsf_dnt_adv_BD+FD_FreezeBackbone.py", line 15, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in bootstrap() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in bootstrap() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2927) of binary: /home/dongzengao/anaconda3/envs/fgdal/bin/python Traceback (most recent call last): File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/runpy.py", line 192, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run elastic_launch( File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ train_SEmsf_dnt_adv_BD+FD_FreezeBackbone.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-04-21_23:16:23 host : DONG. rank : 1 (local_rank: 1) exitcode : 1 (pid: 2928) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-04-21_23:16:23 host : DONG. rank : 0 (local_rank: 0) exitcode : 1 (pid: 2927) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ Optionally, required to modify the network in the code to conduct self distill Traceback (most recent call last): File "test.py", line 16, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in bootstrap() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory /home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects `--local_rank` argument to be set, please change it to read from `os.environ['LOCAL_RANK']` instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions warnings.warn( WARNING:torch.distributed.run: ***************************************** Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. ***************************************** Traceback (most recent call last): File "train_self_distill.py", line 14, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in Traceback (most recent call last): File "train_self_distill.py", line 14, in from core.datasets import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/__init__.py", line 1, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from .build import build_dataset File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/build.py", line 3, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in from . import transform File "/mnt/d/AAA_DATA/Test_data/3DBIE-SolarPV-main/3DBIE-SolarPV-main/UDA-Seg/core/datasets/transform.py", line 12, in import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in bootstrap() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory import cv2 File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 181, in bootstrap() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/cv2/__init__.py", line 153, in bootstrap native_module = importlib.import_module("cv2") File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) ImportError: libGL.so.1: cannot open shared object file: No such file or directory ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2997) of binary: /home/dongzengao/anaconda3/envs/fgdal/bin/python Traceback (most recent call last): File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/runpy.py", line 192, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run elastic_launch( File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) File "/home/dongzengao/anaconda3/envs/fgdal/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ train_self_distill.py FAILED ------------------------------------------------------------ Failures: [1]: time : 2024-04-21_23:16:30 host : DONG. rank : 1 (local_rank: 1) exitcode : 1 (pid: 2998) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-04-21_23:16:30 host : DONG. rank : 0 (local_rank: 0) exitcode : 1 (pid: 2997) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html