Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SEGFAULT on GPU version compiled with Clang #3475

Open
Tracked by #5153
StrikerRUS opened this issue Oct 19, 2020 · 4 comments
Open
Tracked by #5153

SEGFAULT on GPU version compiled with Clang #3475

StrikerRUS opened this issue Oct 19, 2020 · 4 comments
Labels
bug

Comments

@StrikerRUS
Copy link
Collaborator

Compilation goes OK, but then the first test fails

2020-10-19T16:00:21.0290034Z Preparing transaction: ...working... done
2020-10-19T16:00:25.6329190Z Verifying transaction: ...working... done
2020-10-19T16:01:13.2790426Z Executing transaction: ...working... done
2020-10-19T16:01:15.1459209Z -- The C compiler identification is Clang 9.0.0
2020-10-19T16:01:15.2257066Z -- The CXX compiler identification is Clang 9.0.0
2020-10-19T16:01:15.2466799Z -- Detecting C compiler ABI info
2020-10-19T16:01:15.3236586Z -- Detecting C compiler ABI info - done
2020-10-19T16:01:15.3379594Z -- Check for working C compiler: /usr/local/bin/clang - skipped
2020-10-19T16:01:15.3387622Z -- Detecting C compile features
2020-10-19T16:01:15.3398848Z -- Detecting C compile features - done
2020-10-19T16:01:15.3483944Z -- Detecting CXX compiler ABI info
2020-10-19T16:01:15.4399097Z -- Detecting CXX compiler ABI info - done
2020-10-19T16:01:15.4552660Z -- Check for working CXX compiler: /usr/local/bin/clang++ - skipped
2020-10-19T16:01:15.4557587Z -- Detecting CXX compile features
2020-10-19T16:01:15.4568283Z -- Detecting CXX compile features - done
2020-10-19T16:01:15.7952728Z -- Found OpenMP_C: -fopenmp=libomp (found version "3.1") 
2020-10-19T16:01:15.9055523Z -- Found OpenMP_CXX: -fopenmp=libomp (found version "3.1") 
2020-10-19T16:01:15.9062893Z -- Found OpenMP: TRUE (found version "3.1")  
2020-10-19T16:01:15.9097000Z -- Looking for CL_VERSION_2_2
2020-10-19T16:01:15.9984384Z -- Looking for CL_VERSION_2_2 - not found
2020-10-19T16:01:15.9996623Z -- Looking for CL_VERSION_2_1
2020-10-19T16:01:16.0746097Z -- Looking for CL_VERSION_2_1 - not found
2020-10-19T16:01:16.0752948Z -- Looking for CL_VERSION_2_0
2020-10-19T16:01:16.1804345Z -- Looking for CL_VERSION_2_0 - found
2020-10-19T16:01:16.1868030Z -- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.0") 
2020-10-19T16:01:16.1881427Z -- OpenCL include directory: /__w/1/s/AMDAPPSDK/include
2020-10-19T16:01:16.2275133Z -- Found Boost: /usr/include (found suitable version "1.67.0", minimum required is "1.56.0") found components: filesystem system 
2020-10-19T16:01:16.2310558Z -- Performing Test MM_PREFETCH
2020-10-19T16:01:16.3892016Z -- Performing Test MM_PREFETCH - Success
2020-10-19T16:01:16.3894597Z -- Using _mm_prefetch
2020-10-19T16:01:16.3903990Z -- Performing Test MM_MALLOC
2020-10-19T16:01:16.4881015Z -- Performing Test MM_MALLOC - Success
2020-10-19T16:01:16.4885937Z -- Using _mm_malloc
2020-10-19T16:01:16.4906436Z -- Configuring done
2020-10-19T16:01:16.5154130Z -- Generating done
2020-10-19T16:01:16.5168671Z -- Build files have been written to: /__w/1/s/build
2020-10-19T16:01:16.5789308Z Scanning dependencies of target _lightgbm
2020-10-19T16:01:16.5892509Z [  3%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
2020-10-19T16:01:16.5897056Z [  6%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
2020-10-19T16:01:16.5919428Z [  9%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
2020-10-19T16:01:16.5937943Z [ 12%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
2020-10-19T16:01:24.0812103Z [ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
2020-10-19T16:01:25.3568301Z [ 18%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
2020-10-19T16:01:26.5125529Z [ 21%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
2020-10-19T16:01:26.9643809Z [ 24%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
2020-10-19T16:01:27.6007137Z [ 27%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
2020-10-19T16:01:27.8897768Z [ 30%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
2020-10-19T16:01:33.8843991Z [ 33%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
2020-10-19T16:01:35.9924448Z [ 36%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
2020-10-19T16:01:38.2650450Z [ 39%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
2020-10-19T16:01:41.0695067Z [ 42%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
2020-10-19T16:01:42.4647852Z [ 45%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
2020-10-19T16:01:45.3122917Z [ 48%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
2020-10-19T16:01:46.8321955Z [ 51%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
2020-10-19T16:01:47.0291797Z [ 54%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
2020-10-19T16:01:51.5333934Z [ 57%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/ifaddrs_patch.cpp.o
2020-10-19T16:01:51.6012531Z [ 60%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
2020-10-19T16:01:54.4605799Z [ 63%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
2020-10-19T16:01:54.5284496Z [ 66%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
2020-10-19T16:01:54.6243936Z [ 69%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
2020-10-19T16:01:58.0885683Z [ 72%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
2020-10-19T16:01:59.0015910Z [ 75%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/cuda_tree_learner.cpp.o
2020-10-19T16:01:59.0696168Z [ 78%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
2020-10-19T16:01:59.7729994Z [ 81%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
2020-10-19T16:02:00.1212542Z [ 84%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
2020-10-19T16:02:11.3016151Z [ 87%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
2020-10-19T16:02:15.5645866Z [ 90%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
2020-10-19T16:02:17.8285361Z [ 93%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
2020-10-19T16:02:29.7002845Z [ 96%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
2020-10-19T16:02:57.1828229Z [100%] Linking CXX shared library ../lib_lightgbm.so
2020-10-19T16:02:57.4969939Z [100%] Built target _lightgbm
2020-10-19T16:02:57.7241666Z running install
2020-10-19T16:02:57.7474751Z running build
2020-10-19T16:02:57.7477326Z running build_py
2020-10-19T16:02:57.7649671Z creating build
2020-10-19T16:02:57.7654861Z creating build/lib
2020-10-19T16:02:57.7656368Z creating build/lib/lightgbm
2020-10-19T16:02:57.7659889Z copying lightgbm/__init__.py -> build/lib/lightgbm
2020-10-19T16:02:57.7667287Z copying lightgbm/engine.py -> build/lib/lightgbm
2020-10-19T16:02:57.7668184Z copying lightgbm/basic.py -> build/lib/lightgbm
2020-10-19T16:02:57.7678269Z copying lightgbm/plotting.py -> build/lib/lightgbm
2020-10-19T16:02:57.7684139Z copying lightgbm/callback.py -> build/lib/lightgbm
2020-10-19T16:02:57.7689636Z copying lightgbm/compat.py -> build/lib/lightgbm
2020-10-19T16:02:57.7694749Z copying lightgbm/sklearn.py -> build/lib/lightgbm
2020-10-19T16:02:57.7705597Z copying lightgbm/libpath.py -> build/lib/lightgbm
2020-10-19T16:02:57.7710396Z running egg_info
2020-10-19T16:02:57.7774396Z creating lightgbm.egg-info
2020-10-19T16:02:57.7775220Z writing lightgbm.egg-info/PKG-INFO
2020-10-19T16:02:57.7781511Z writing dependency_links to lightgbm.egg-info/dependency_links.txt
2020-10-19T16:02:57.7787258Z writing requirements to lightgbm.egg-info/requires.txt
2020-10-19T16:02:57.7788123Z writing top-level names to lightgbm.egg-info/top_level.txt
2020-10-19T16:02:57.7791652Z writing manifest file 'lightgbm.egg-info/SOURCES.txt'
2020-10-19T16:02:57.7806374Z reading manifest file 'lightgbm.egg-info/SOURCES.txt'
2020-10-19T16:02:57.7807261Z reading manifest template 'MANIFEST.in'
2020-10-19T16:02:57.7906209Z no previously-included directories found matching 'build'
2020-10-19T16:02:57.7907227Z warning: no files found matching 'LICENSE'
2020-10-19T16:02:57.7907974Z warning: no files found matching '*.txt'
2020-10-19T16:02:57.7909325Z warning: no files found matching '*.so' under directory 'lightgbm'
2020-10-19T16:02:57.7910157Z warning: no files found matching '*.txt' under directory 'compile'
2020-10-19T16:02:57.7910999Z warning: no files found matching '*.so' under directory 'compile'
2020-10-19T16:02:57.7911861Z warning: no files found matching '*.dll' under directory 'compile/Release'
2020-10-19T16:02:57.7912699Z warning: no files found matching '*' under directory 'compile/compute'
2020-10-19T16:02:57.7913550Z warning: no files found matching '*' under directory 'compile/include'
2020-10-19T16:02:57.7914676Z warning: no files found matching '*' under directory 'compile/src'
2020-10-19T16:02:57.7916270Z warning: no files found matching 'LightGBM.sln' under directory 'compile/windows'
2020-10-19T16:02:57.7917522Z warning: no files found matching 'LightGBM.vcxproj' under directory 'compile/windows'
2020-10-19T16:02:57.7918418Z warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
2020-10-19T16:02:57.7919358Z warning: no previously-included files matching '*.py[co]' found anywhere in distribution
2020-10-19T16:02:57.7920174Z writing manifest file 'lightgbm.egg-info/SOURCES.txt'
2020-10-19T16:02:57.7920947Z copying lightgbm/VERSION.txt -> build/lib/lightgbm
2020-10-19T16:02:57.7921455Z running install_lib
2020-10-19T16:02:57.7922189Z creating /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7923141Z copying build/lib/lightgbm/VERSION.txt -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7924330Z copying build/lib/lightgbm/__init__.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7925419Z copying build/lib/lightgbm/engine.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7926327Z copying build/lib/lightgbm/basic.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7927277Z copying build/lib/lightgbm/plotting.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7928215Z copying build/lib/lightgbm/callback.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7929314Z copying build/lib/lightgbm/compat.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7930596Z copying build/lib/lightgbm/sklearn.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7931601Z copying build/lib/lightgbm/libpath.py -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7932537Z INFO:LightGBM:Installing lib_lightgbm from: ['../lib_lightgbm.so']
2020-10-19T16:02:57.7933476Z copying ../lib_lightgbm.so -> /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm
2020-10-19T16:02:57.7948842Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/__init__.py to __init__.cpython-38.pyc
2020-10-19T16:02:57.7955746Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/engine.py to engine.cpython-38.pyc
2020-10-19T16:02:57.8006452Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/basic.py to basic.cpython-38.pyc
2020-10-19T16:02:57.8232341Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/plotting.py to plotting.cpython-38.pyc
2020-10-19T16:02:57.8263903Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/callback.py to callback.cpython-38.pyc
2020-10-19T16:02:57.8280872Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/compat.py to compat.cpython-38.pyc
2020-10-19T16:02:57.8315630Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/sklearn.py to sklearn.cpython-38.pyc
2020-10-19T16:02:57.8345412Z byte-compiling /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm/libpath.py to libpath.cpython-38.pyc
2020-10-19T16:02:57.8352098Z running install_egg_info
2020-10-19T16:02:57.8391324Z Copying lightgbm.egg-info to /home/vsts_azpcontainer/.local/lib/python3.8/site-packages/lightgbm-3.0.0.99-py3.8.egg-info
2020-10-19T16:02:57.8402913Z running install_scripts
2020-10-19T16:02:59.4058495Z ============================= test session starts ==============================
2020-10-19T16:02:59.4060562Z platform linux -- Python 3.8.2, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
2020-10-19T16:02:59.4061087Z rootdir: /__w/1/s
2020-10-19T16:02:59.4061613Z collected 121 items
2020-10-19T16:02:59.4061847Z 
2020-10-19T16:02:59.7174266Z ../tests/c_api_test/test_.py .Fatal Python error: Segmentation fault
2020-10-19T16:02:59.7175096Z 
2020-10-19T16:02:59.7175415Z Current thread 0x00007fe702ef7740 (most recent call first):
2020-10-19T16:02:59.7182342Z   File "/__w/1/s/tests/c_api_test/test_.py", line 226 in test_booster
2020-10-19T16:02:59.7183867Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/python.py", line 184 in pytest_pyfunc_call
2020-10-19T16:02:59.7185853Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
2020-10-19T16:02:59.7186877Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-10-19T16:02:59.7187999Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
2020-10-19T16:02:59.7225484Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
2020-10-19T16:02:59.7282807Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/python.py", line 1627 in runtest
2020-10-19T16:02:59.7284117Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 163 in pytest_runtest_call
2020-10-19T16:02:59.7285165Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
2020-10-19T16:02:59.7286255Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-10-19T16:02:59.7287209Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
2020-10-19T16:02:59.7288103Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
2020-10-19T16:02:59.7289022Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 256 in <lambda>
2020-10-19T16:02:59.7297754Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 310 in from_call
2020-10-19T16:02:59.7298899Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
2020-10-19T16:02:59.7306175Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 216 in call_and_report
2020-10-19T16:02:59.7311684Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 127 in runtestprotocol
2020-10-19T16:02:59.7320276Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/runner.py", line 110 in pytest_runtest_protocol
2020-10-19T16:02:59.7330585Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
2020-10-19T16:02:59.7337096Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-10-19T16:02:59.7338167Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
2020-10-19T16:02:59.7344519Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
2020-10-19T16:02:59.7351770Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/main.py", line 338 in pytest_runtestloop
2020-10-19T16:02:59.7358910Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
2020-10-19T16:02:59.7364945Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-10-19T16:02:59.7372141Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
2020-10-19T16:02:59.7379013Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
2020-10-19T16:02:59.7387584Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/main.py", line 313 in _main
2020-10-19T16:02:59.7393114Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/main.py", line 257 in wrap_session
2020-10-19T16:02:59.7400836Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/main.py", line 306 in pytest_cmdline_main
2020-10-19T16:02:59.7408221Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
2020-10-19T16:02:59.7419131Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-10-19T16:02:59.7420383Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
2020-10-19T16:02:59.7428034Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
2020-10-19T16:02:59.7436779Z   File "/opt/conda/envs/test-env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 164 in main
2020-10-19T16:02:59.7439012Z   File "/opt/conda/envs/test-env/bin/pytest", line 11 in <module>
2020-10-19T16:03:00.3346040Z /__w/1/s/.ci/test.sh: line 156:  1285 Segmentation fault      (core dumped) pytest $BUILD_DIRECTORY/tests
2020-10-19T16:03:00.3391719Z 
2020-10-19T16:03:00.3478114Z ##[error]Bash exited with code '255'.
2020-10-19T16:03:00.4423827Z ##[section]Finishing: Test

What I have already tried with no luck:

How to reproduce: simply change gcc to clang here

LightGBM/.vsts-ci.yml

Line 22 in 7e2f353

COMPILER: gcc

@StrikerRUS StrikerRUS added the bug label Oct 19, 2020
@guolinke
Copy link
Collaborator

It seems we have several CUDA bugs to fix.
Maybe we can summarize them into one issue, for easily tracking?

@StrikerRUS
Copy link
Collaborator Author

It is not CUDA bug, but bug with our original OpenCL GPU version.

@StrikerRUS StrikerRUS mentioned this issue Oct 26, 2020
Merged
@StrikerRUS StrikerRUS mentioned this issue Jan 12, 2021
Merged
@jameslamb
Copy link
Collaborator

I want to add that the gpu_source job is working with Clang 7.0.0 on Ubuntu 20.04 on Travis (https://travis-ci.org/github/microsoft/LightGBM/jobs/753978660#L7). So maybe the issue is limited to a subset of clang versions and not all of clang.

-- The C compiler identification is Clang 7.0.0
-- The CXX compiler identification is Clang 7.0.0
-- Check for working C compiler: /usr/local/clang-7.0.0/bin/clang
-- Check for working C compiler: /usr/local/clang-7.0.0/bin/clang -- works
...
-- Check for working CXX compiler: /usr/local/clang-7.0.0/bin/clang++
-- Check for working CXX compiler: /usr/local/clang-7.0.0/bin/clang++ -- works
...
====== 235 passed, 1 skipped, 2 xfailed, 74 warnings in 481.00s (0:08:01) ======

@StrikerRUS
Copy link
Collaborator Author

StrikerRUS commented Jan 13, 2021 •
edited
Loading

Allowing tests to fail and running our CI script further, we can see that all examples work OK but example Notebook fails with LightGBMError: No OpenCL device found error.

image

Log: 
2021-01-13T01:09:58.8874142Z -- The C compiler identification is Clang 9.0.0
2021-01-13T01:09:58.9448400Z -- The CXX compiler identification is Clang 9.0.0
2021-01-13T01:09:58.9539401Z -- Detecting C compiler ABI info
2021-01-13T01:09:59.0074270Z -- Detecting C compiler ABI info - done
2021-01-13T01:09:59.0161178Z -- Check for working C compiler: /usr/local/bin/clang - skipped
2021-01-13T01:09:59.0163817Z -- Detecting C compile features
2021-01-13T01:09:59.0171907Z -- Detecting C compile features - done
2021-01-13T01:09:59.0216720Z -- Detecting CXX compiler ABI info
2021-01-13T01:09:59.2082686Z -- Detecting CXX compiler ABI info - done
2021-01-13T01:09:59.2178889Z -- Check for working CXX compiler: /usr/local/bin/clang++ - skipped
2021-01-13T01:09:59.2180515Z -- Detecting CXX compile features
2021-01-13T01:09:59.2188022Z -- Detecting CXX compile features - done
2021-01-13T01:09:59.4446368Z -- Found OpenMP_C: -fopenmp=libomp (found version "3.1") 
2021-01-13T01:09:59.5219691Z -- Found OpenMP_CXX: -fopenmp=libomp (found version "3.1") 
2021-01-13T01:09:59.5223377Z -- Found OpenMP: TRUE (found version "3.1")  
2021-01-13T01:09:59.5248696Z -- Looking for CL_VERSION_2_2
2021-01-13T01:09:59.5766162Z -- Looking for CL_VERSION_2_2 - not found
2021-01-13T01:09:59.5771610Z -- Looking for CL_VERSION_2_1
2021-01-13T01:09:59.6265511Z -- Looking for CL_VERSION_2_1 - not found
2021-01-13T01:09:59.6270558Z -- Looking for CL_VERSION_2_0
2021-01-13T01:09:59.6981780Z -- Looking for CL_VERSION_2_0 - found
2021-01-13T01:09:59.7015569Z -- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.0") 
2021-01-13T01:09:59.7016500Z -- OpenCL include directory: /__w/1/s/AMDAPPSDK/include
2021-01-13T01:09:59.7243444Z -- Found Boost: /usr/include (found suitable version "1.74.0", minimum required is "1.56.0") found components: filesystem system 
2021-01-13T01:09:59.7263403Z -- Performing Test MM_PREFETCH
2021-01-13T01:09:59.8255895Z -- Performing Test MM_PREFETCH - Success
2021-01-13T01:09:59.8256872Z -- Using _mm_prefetch
2021-01-13T01:09:59.8259554Z -- Performing Test MM_MALLOC
2021-01-13T01:09:59.8988264Z -- Performing Test MM_MALLOC - Success
2021-01-13T01:09:59.8989241Z -- Using _mm_malloc
2021-01-13T01:09:59.8999984Z -- Configuring done
2021-01-13T01:09:59.9149557Z -- Generating done
2021-01-13T01:09:59.9158789Z -- Build files have been written to: /__w/1/s/build
2021-01-13T01:10:00.0005307Z Scanning dependencies of target _lightgbm
2021-01-13T01:10:00.0103544Z [  2%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
2021-01-13T01:10:00.0105556Z [  5%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
2021-01-13T01:10:00.0111816Z [  8%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
2021-01-13T01:10:00.0170301Z [ 11%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
2021-01-13T01:10:02.3940410Z [ 14%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
2021-01-13T01:10:03.4158101Z [ 17%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
2021-01-13T01:10:04.3219307Z [ 20%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
2021-01-13T01:10:05.0608677Z [ 23%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
2021-01-13T01:10:05.6498963Z [ 26%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
2021-01-13T01:10:08.2745632Z [ 29%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
2021-01-13T01:10:09.7904035Z [ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
2021-01-13T01:10:10.8408284Z [ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
2021-01-13T01:10:12.8366853Z [ 38%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
2021-01-13T01:10:12.8493681Z [ 41%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
2021-01-13T01:10:14.7820708Z [ 44%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/train_share_states.cpp.o
2021-01-13T01:10:15.4593951Z [ 47%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
2021-01-13T01:10:16.0610760Z [ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
2021-01-13T01:10:17.2764073Z [ 52%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
2021-01-13T01:10:18.7122353Z [ 55%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/ifaddrs_patch.cpp.o
2021-01-13T01:10:18.7381356Z [ 58%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
2021-01-13T01:10:19.6905229Z [ 61%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
2021-01-13T01:10:19.7170014Z [ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
2021-01-13T01:10:20.6421625Z [ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
2021-01-13T01:10:22.9360423Z [ 70%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
2021-01-13T01:10:23.2221666Z [ 73%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/cuda_tree_learner.cpp.o
2021-01-13T01:10:23.2481817Z [ 76%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
2021-01-13T01:10:24.0054284Z [ 79%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
2021-01-13T01:10:25.0109541Z [ 82%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
2021-01-13T01:10:30.2199978Z [ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/linear_tree_learner.cpp.o
2021-01-13T01:10:32.2194405Z [ 88%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
2021-01-13T01:10:33.0176892Z [ 91%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
2021-01-13T01:10:40.0864371Z [ 94%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
2021-01-13T01:10:41.0098112Z [ 97%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
2021-01-13T01:11:01.5954873Z [100%] Linking CXX shared library ../lib_lightgbm.so
2021-01-13T01:11:01.8352333Z [100%] Built target _lightgbm
2021-01-13T01:11:02.0431852Z running install
2021-01-13T01:11:02.0594969Z running build
2021-01-13T01:11:02.0595635Z running build_py
2021-01-13T01:11:02.0731303Z creating build
2021-01-13T01:11:02.0731907Z creating build/lib
2021-01-13T01:11:02.0732293Z creating build/lib/lightgbm
2021-01-13T01:11:02.0733437Z copying lightgbm/basic.py -> build/lib/lightgbm
2021-01-13T01:11:02.0743831Z copying lightgbm/__init__.py -> build/lib/lightgbm
2021-01-13T01:11:02.0745621Z copying lightgbm/plotting.py -> build/lib/lightgbm
2021-01-13T01:11:02.0750407Z copying lightgbm/compat.py -> build/lib/lightgbm
2021-01-13T01:11:02.0755496Z copying lightgbm/sklearn.py -> build/lib/lightgbm
2021-01-13T01:11:02.0759647Z copying lightgbm/libpath.py -> build/lib/lightgbm
2021-01-13T01:11:02.0761781Z copying lightgbm/dask.py -> build/lib/lightgbm
2021-01-13T01:11:02.0764967Z copying lightgbm/callback.py -> build/lib/lightgbm
2021-01-13T01:11:02.0766665Z copying lightgbm/engine.py -> build/lib/lightgbm
2021-01-13T01:11:02.0769626Z running egg_info
2021-01-13T01:11:02.0800103Z creating lightgbm.egg-info
2021-01-13T01:11:02.0802070Z writing lightgbm.egg-info/PKG-INFO
2021-01-13T01:11:02.0807288Z writing dependency_links to lightgbm.egg-info/dependency_links.txt
2021-01-13T01:11:02.0810258Z writing requirements to lightgbm.egg-info/requires.txt
2021-01-13T01:11:02.0811216Z writing top-level names to lightgbm.egg-info/top_level.txt
2021-01-13T01:11:02.0813796Z writing manifest file 'lightgbm.egg-info/SOURCES.txt'
2021-01-13T01:11:02.0819821Z reading manifest file 'lightgbm.egg-info/SOURCES.txt'
2021-01-13T01:11:02.0822357Z reading manifest template 'MANIFEST.in'
2021-01-13T01:11:02.0830202Z no previously-included directories found matching 'build'
2021-01-13T01:11:02.0831182Z warning: no files found matching 'LICENSE'
2021-01-13T01:11:02.0831980Z warning: no files found matching '*.txt'
2021-01-13T01:11:02.0845449Z warning: no files found matching '*.so' under directory 'lightgbm'
2021-01-13T01:11:02.0846628Z warning: no files found matching 'compile/CMakeLists.txt'
2021-01-13T01:11:02.0847650Z warning: no files found matching 'compile/CMakeIntegratedOpenCL.cmake'
2021-01-13T01:11:02.0848671Z warning: no files found matching '*.so' under directory 'compile'
2021-01-13T01:11:02.0849687Z warning: no files found matching '*.dll' under directory 'compile/Release'
2021-01-13T01:11:02.0850677Z warning: no files found matching 'compile/compute/CMakeLists.txt'
2021-01-13T01:11:02.0851702Z warning: no files found matching '*' under directory 'compile/compute/cmake'
2021-01-13T01:11:02.0852738Z warning: no files found matching '*' under directory 'compile/compute/include'
2021-01-13T01:11:02.0853791Z warning: no files found matching '*' under directory 'compile/compute/meta'
2021-01-13T01:11:02.0854766Z warning: no files found matching 'compile/eigen/CMakeLists.txt'
2021-01-13T01:11:02.0855751Z warning: no files found matching 'compile/eigen/Eigen/Cholesky'
2021-01-13T01:11:02.0856669Z warning: no files found matching 'compile/eigen/Eigen/Core'
2021-01-13T01:11:02.0857542Z warning: no files found matching 'compile/eigen/Eigen/Dense'
2021-01-13T01:11:02.0858403Z warning: no files found matching 'compile/eigen/Eigen/Eigenvalues'
2021-01-13T01:11:02.0859301Z warning: no files found matching 'compile/eigen/Eigen/Geometry'
2021-01-13T01:11:02.0860221Z warning: no files found matching 'compile/eigen/Eigen/Householder'
2021-01-13T01:11:02.0861086Z warning: no files found matching 'compile/eigen/Eigen/Jacobi'
2021-01-13T01:11:02.0861915Z warning: no files found matching 'compile/eigen/Eigen/LU'
2021-01-13T01:11:02.0862753Z warning: no files found matching 'compile/eigen/Eigen/QR'
2021-01-13T01:11:02.0863593Z warning: no files found matching 'compile/eigen/Eigen/SVD'
2021-01-13T01:11:02.0864610Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/Cholesky'
2021-01-13T01:11:02.0865601Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/Core'
2021-01-13T01:11:02.0866648Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/Eigenvalues'
2021-01-13T01:11:02.0867668Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/Geometry'
2021-01-13T01:11:02.0868708Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/Householder'
2021-01-13T01:11:02.0869732Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/Jacobi'
2021-01-13T01:11:02.0870667Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/LU'
2021-01-13T01:11:02.0871593Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/misc'
2021-01-13T01:11:02.0872618Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/plugins'
2021-01-13T01:11:02.0873595Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/QR'
2021-01-13T01:11:02.0874552Z warning: no files found matching '*' under directory 'compile/eigen/Eigen/src/SVD'
2021-01-13T01:11:02.0877495Z warning: no files found matching 'compile/external_libs/fast_double_parser/CMakeLists.txt'
2021-01-13T01:11:02.0878860Z warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE'
2021-01-13T01:11:02.0879803Z warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE.BSL'
2021-01-13T01:11:02.0880983Z warning: no files found matching '*' under directory 'compile/external_libs/fast_double_parser/include'
2021-01-13T01:11:02.0882088Z warning: no files found matching 'compile/external_libs/fmt/CMakeLists.txt'
2021-01-13T01:11:02.0883039Z warning: no files found matching 'compile/external_libs/fmt/LICENSE.rst'
2021-01-13T01:11:02.0884060Z warning: no files found matching '*' under directory 'compile/external_libs/fmt/include'
2021-01-13T01:11:02.0885007Z warning: no files found matching '*' under directory 'compile/include'
2021-01-13T01:11:02.0886267Z warning: no files found matching '*' under directory 'compile/src'
2021-01-13T01:11:02.0887219Z warning: no files found matching 'LightGBM.sln' under directory 'compile/windows'
2021-01-13T01:11:02.0888496Z warning: no files found matching 'LightGBM.vcxproj' under directory 'compile/windows'
2021-01-13T01:11:02.0889568Z warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
2021-01-13T01:11:02.0890721Z warning: no previously-included files matching '*.py[co]' found anywhere in distribution
2021-01-13T01:11:02.0891793Z warning: no previously-included files found matching 'compile/compute/.git'
2021-01-13T01:11:02.0892767Z writing manifest file 'lightgbm.egg-info/SOURCES.txt'
2021-01-13T01:11:02.0893660Z copying lightgbm/VERSION.txt -> build/lib/lightgbm
2021-01-13T01:11:02.0894189Z running install_lib
2021-01-13T01:11:02.0895068Z creating /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0896237Z copying build/lib/lightgbm/basic.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0897489Z copying build/lib/lightgbm/__init__.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0898725Z copying build/lib/lightgbm/plotting.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0900039Z copying build/lib/lightgbm/compat.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0901299Z copying build/lib/lightgbm/sklearn.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0902504Z copying build/lib/lightgbm/libpath.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0903781Z copying build/lib/lightgbm/dask.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0904927Z copying build/lib/lightgbm/callback.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0906049Z copying build/lib/lightgbm/VERSION.txt -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0907144Z copying build/lib/lightgbm/engine.py -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0914623Z INFO:LightGBM:Installing lib_lightgbm from: ['../lib_lightgbm.so']
2021-01-13T01:11:02.0915766Z copying ../lib_lightgbm.so -> /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm
2021-01-13T01:11:02.0968283Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py to basic.cpython-36.pyc
2021-01-13T01:11:02.1136867Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/__init__.py to __init__.cpython-36.pyc
2021-01-13T01:11:02.1140746Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/plotting.py to plotting.cpython-36.pyc
2021-01-13T01:11:02.1166533Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/compat.py to compat.cpython-36.pyc
2021-01-13T01:11:02.1172355Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/sklearn.py to sklearn.cpython-36.pyc
2021-01-13T01:11:02.1210648Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/libpath.py to libpath.cpython-36.pyc
2021-01-13T01:11:02.1216334Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/dask.py to dask.cpython-36.pyc
2021-01-13T01:11:02.1233073Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/callback.py to callback.cpython-36.pyc
2021-01-13T01:11:02.1245428Z byte-compiling /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/engine.py to engine.cpython-36.pyc
2021-01-13T01:11:02.1285523Z running install_egg_info
2021-01-13T01:11:02.1296896Z Copying lightgbm.egg-info to /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm-3.1.1.99-py3.6.egg-info
2021-01-13T01:11:02.1305503Z running install_scripts
2021-01-13T01:11:03.7883887Z ============================= test session starts ==============================
2021-01-13T01:11:03.7885474Z platform linux -- Python 3.6.10, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
2021-01-13T01:11:03.7885967Z rootdir: /__w/1/s
2021-01-13T01:11:03.7886251Z collected 238 items
2021-01-13T01:11:03.7886468Z 
2021-01-13T01:11:04.1194605Z ../tests/c_api_test/test_.py .Fatal Python error: Segmentation fault
2021-01-13T01:11:04.1195850Z 
2021-01-13T01:11:04.1198448Z Thread 0x00007f85d9554700 (most recent call first):
2021-01-13T01:11:04.1200373Z   File "/opt/conda/envs/test-env/lib/python3.6/threading.py", line 295 in wait
2021-01-13T01:11:04.1201552Z   File "/opt/conda/envs/test-env/lib/python3.6/queue.py", line 164 in get
2021-01-13T01:11:04.1202841Z   File "/opt/conda/envs/test-env/lib/python3.6/concurrent/futures/thread.py", line 67 in _worker
2021-01-13T01:11:04.1203970Z   File "/opt/conda/envs/test-env/lib/python3.6/threading.py", line 864 in run
2021-01-13T01:11:04.1205105Z   File "/opt/conda/envs/test-env/lib/python3.6/threading.py", line 916 in _bootstrap_inner
2021-01-13T01:11:04.1206258Z   File "/opt/conda/envs/test-env/lib/python3.6/threading.py", line 884 in _bootstrap
2021-01-13T01:11:04.1206869Z 
2021-01-13T01:11:04.1207465Z Current thread 0x00007f85f162d740 (most recent call first):
2021-01-13T01:11:04.1208105Z   File "/__w/1/s/tests/c_api_test/test_.py", line 226 in test_booster
2021-01-13T01:11:04.1222451Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
2021-01-13T01:11:04.1227516Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
2021-01-13T01:11:04.1228597Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
2021-01-13T01:11:04.1229612Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
2021-01-13T01:11:04.1230640Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
2021-01-13T01:11:04.1231677Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/python.py", line 1641 in runtest
2021-01-13T01:11:04.1232742Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
2021-01-13T01:11:04.1233928Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
2021-01-13T01:11:04.1234938Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
2021-01-13T01:11:04.1235965Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
2021-01-13T01:11:04.1237108Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
2021-01-13T01:11:04.1238397Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 255 in <lambda>
2021-01-13T01:11:04.1287635Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 311 in from_call
2021-01-13T01:11:04.1332974Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
2021-01-13T01:11:04.1341427Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 215 in call_and_report
2021-01-13T01:11:04.1348718Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 126 in runtestprotocol
2021-01-13T01:11:04.1357285Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
2021-01-13T01:11:04.1364617Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
2021-01-13T01:11:04.1371836Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
2021-01-13T01:11:04.1379036Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
2021-01-13T01:11:04.1386262Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
2021-01-13T01:11:04.1393581Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
2021-01-13T01:11:04.1401209Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
2021-01-13T01:11:04.1409464Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
2021-01-13T01:11:04.1417914Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
2021-01-13T01:11:04.1424112Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
2021-01-13T01:11:04.1431222Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 323 in _main
2021-01-13T01:11:04.1439718Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 269 in wrap_session
2021-01-13T01:11:04.1446951Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
2021-01-13T01:11:04.1455023Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
2021-01-13T01:11:04.1462192Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
2021-01-13T01:11:04.1469344Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
2021-01-13T01:11:04.1476528Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
2021-01-13T01:11:04.1483884Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/config/__init__.py", line 163 in main
2021-01-13T01:11:04.1491418Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/_pytest/config/__init__.py", line 185 in console_main
2021-01-13T01:11:04.1496420Z   File "/opt/conda/envs/test-env/bin/pytest", line 11 in <module>
2021-01-13T01:11:04.4676748Z /__w/1/s/.ci/test.sh: line 172:  1431 Segmentation fault      (core dumped) pytest $BUILD_DIRECTORY/tests
2021-01-13T01:11:10.9571314Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1570: UserWarning: categorical_feature in Dataset is overridden.
2021-01-13T01:11:10.9572709Z New categorical_feature is [21]
2021-01-13T01:11:10.9573554Z   'New categorical_feature is {}'.format(sorted(list(categorical_feature))))
2021-01-13T01:11:10.9574712Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1566: UserWarning: Using categorical_feature in Dataset.
2021-01-13T01:11:10.9575679Z   warnings.warn('Using categorical_feature in Dataset.')
2021-01-13T01:11:10.9576793Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1570: UserWarning: categorical_feature in Dataset is overridden.
2021-01-13T01:11:10.9577965Z New categorical_feature is [21]
2021-01-13T01:11:10.9578868Z   'New categorical_feature is {}'.format(sorted(list(categorical_feature))))
2021-01-13T01:11:10.9580051Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1301: UserWarning: Overriding the parameters from Reference Dataset.
2021-01-13T01:11:10.9581232Z   warnings.warn('Overriding the parameters from Reference Dataset.')
2021-01-13T01:11:10.9582395Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1113: UserWarning: categorical_column in param dict is overridden.
2021-01-13T01:11:10.9583440Z   warnings.warn('{} in param dict is overridden.'.format(cat_alias))
2021-01-13T01:11:10.9584521Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1566: UserWarning: Using categorical_feature in Dataset.
2021-01-13T01:11:10.9585485Z   warnings.warn('Using categorical_feature in Dataset.')
2021-01-13T01:11:10.9586589Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1301: UserWarning: Overriding the parameters from Reference Dataset.
2021-01-13T01:11:10.9587638Z   warnings.warn('Overriding the parameters from Reference Dataset.')
2021-01-13T01:11:10.9588779Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1113: UserWarning: categorical_column in param dict is overridden.
2021-01-13T01:11:10.9589802Z   warnings.warn('{} in param dict is overridden.'.format(cat_alias))
2021-01-13T01:11:10.9590284Z Loading data...
2021-01-13T01:11:10.9590632Z Starting training...
2021-01-13T01:11:10.9591288Z [1]	training's binary_logloss: 0.682311
2021-01-13T01:11:10.9592007Z [2]	training's binary_logloss: 0.674204
2021-01-13T01:11:10.9592746Z [3]	training's binary_logloss: 0.664662
2021-01-13T01:11:10.9593455Z [4]	training's binary_logloss: 0.655524
2021-01-13T01:11:10.9594187Z [5]	training's binary_logloss: 0.648385
2021-01-13T01:11:10.9594892Z [6]	training's binary_logloss: 0.64227
2021-01-13T01:11:10.9595853Z [7]	training's binary_logloss: 0.63507
2021-01-13T01:11:10.9596586Z [8]	training's binary_logloss: 0.628378
2021-01-13T01:11:10.9597471Z [9]	training's binary_logloss: 0.621862
2021-01-13T01:11:10.9598237Z [10]	training's binary_logloss: 0.615907
2021-01-13T01:11:10.9598701Z Finished first 10 rounds...
2021-01-13T01:11:10.9599128Z 7th feature name is: feature_6
2021-01-13T01:11:10.9599508Z Saving model...
2021-01-13T01:11:10.9599879Z Dumping model to JSON...
2021-01-13T01:11:10.9601713Z Feature names: ['feature_0', 'feature_1', 'feature_2', 'feature_3', 'feature_4', 'feature_5', 'feature_6', 'feature_7', 'feature_8', 'feature_9', 'feature_10', 'feature_11', 'feature_12', 'feature_13', 'feature_14', 'feature_15', 'feature_16', 'feature_17', 'feature_18', 'feature_19', 'feature_20', 'feature_21', 'feature_22', 'feature_23', 'feature_24', 'feature_25', 'feature_26', 'feature_27']
2021-01-13T01:11:10.9603144Z Feature importances: [7, 1, 1, 21, 5, 30, 2, 1, 1, 18, 7, 0, 1, 8, 2, 1, 0, 7, 3, 3, 0, 0, 36, 5, 32, 52, 28, 28]
2021-01-13T01:11:10.9603768Z Loading model to predict...
2021-01-13T01:11:10.9604534Z The rmse of loaded model's prediction is: 0.4624446333730955
2021-01-13T01:11:10.9605074Z Dumping and loading model with pickle...
2021-01-13T01:11:10.9605864Z The rmse of pickled model's prediction is: 0.4717432479774246
2021-01-13T01:11:10.9606640Z [11]	valid_0's binary_logloss: 0.615668
2021-01-13T01:11:10.9607366Z [12]	valid_0's binary_logloss: 0.61131
2021-01-13T01:11:10.9608074Z [13]	valid_0's binary_logloss: 0.607092
2021-01-13T01:11:10.9608800Z [14]	valid_0's binary_logloss: 0.603532
2021-01-13T01:11:10.9609520Z [15]	valid_0's binary_logloss: 0.600335
2021-01-13T01:11:10.9610250Z [16]	valid_0's binary_logloss: 0.596209
2021-01-13T01:11:10.9610983Z [17]	valid_0's binary_logloss: 0.592408
2021-01-13T01:11:10.9611702Z [18]	valid_0's binary_logloss: 0.588936
2021-01-13T01:11:10.9612434Z [19]	valid_0's binary_logloss: 0.585849
2021-01-13T01:11:10.9613225Z [20]	valid_0's binary_logloss: 0.582769
2021-01-13T01:11:10.9614087Z Finished 10 - 20 rounds with model file...
2021-01-13T01:11:10.9614878Z [21]	valid_0's binary_logloss: 0.580061
2021-01-13T01:11:10.9615602Z [22]	valid_0's binary_logloss: 0.576959
2021-01-13T01:11:10.9616316Z [23]	valid_0's binary_logloss: 0.575347
2021-01-13T01:11:10.9617040Z [24]	valid_0's binary_logloss: 0.573232
2021-01-13T01:11:10.9617746Z [25]	valid_0's binary_logloss: 0.571052
2021-01-13T01:11:10.9618476Z [26]	valid_0's binary_logloss: 0.568534
2021-01-13T01:11:10.9619185Z [27]	valid_0's binary_logloss: 0.567642
2021-01-13T01:11:10.9619908Z [28]	valid_0's binary_logloss: 0.565333
2021-01-13T01:11:10.9620634Z [29]	valid_0's binary_logloss: 0.562915
2021-01-13T01:11:10.9621353Z [30]	valid_0's binary_logloss: 0.560932
2021-01-13T01:11:10.9622098Z Finished 20 - 30 rounds with decay learning rates...
2021-01-13T01:11:10.9622822Z [31]	valid_0's binary_logloss: 0.559653
2021-01-13T01:11:10.9623552Z [32]	valid_0's binary_logloss: 0.557064
2021-01-13T01:11:10.9624268Z [33]	valid_0's binary_logloss: 0.556072
2021-01-13T01:11:10.9625007Z [34]	valid_0's binary_logloss: 0.554828
2021-01-13T01:11:10.9625712Z [35]	valid_0's binary_logloss: 0.552499
2021-01-13T01:11:10.9626439Z [36]	valid_0's binary_logloss: 0.550979
2021-01-13T01:11:10.9627146Z [37]	valid_0's binary_logloss: 0.550261
2021-01-13T01:11:10.9627886Z [38]	valid_0's binary_logloss: 0.549716
2021-01-13T01:11:10.9628604Z [39]	valid_0's binary_logloss: 0.549701
2021-01-13T01:11:10.9629354Z [40]	valid_0's binary_logloss: 0.547916
2021-01-13T01:11:10.9630129Z Finished 30 - 40 rounds with changing bagging_fraction...
2021-01-13T01:11:10.9630901Z [LightGBM] [Warning] Using self-defined objective function
2021-01-13T01:11:10.9631776Z [LightGBM] [Warning] Using self-defined objective function
2021-01-13T01:11:10.9632617Z [41]	valid_0's binary_logloss: 5.66638	valid_0's error: 0.266
2021-01-13T01:11:10.9633514Z [42]	valid_0's binary_logloss: 5.65638	valid_0's error: 0.266
2021-01-13T01:11:10.9634401Z [43]	valid_0's binary_logloss: 5.59421	valid_0's error: 0.266
2021-01-13T01:11:10.9635300Z [44]	valid_0's binary_logloss: 5.64496	valid_0's error: 0.266
2021-01-13T01:11:10.9636189Z [45]	valid_0's binary_logloss: 5.70971	valid_0's error: 0.266
2021-01-13T01:11:10.9637264Z [46]	valid_0's binary_logloss: 5.70913	valid_0's error: 0.266
2021-01-13T01:11:10.9638214Z [47]	valid_0's binary_logloss: 5.69903	valid_0's error: 0.266
2021-01-13T01:11:10.9639094Z [48]	valid_0's binary_logloss: 5.56476	valid_0's error: 0.26
2021-01-13T01:11:10.9639992Z [49]	valid_0's binary_logloss: 5.55436	valid_0's error: 0.26
2021-01-13T01:11:10.9640858Z [50]	valid_0's binary_logloss: 5.55083	valid_0's error: 0.26
2021-01-13T01:11:10.9641769Z Finished 40 - 50 rounds with self-defined objective function and eval metric...
2021-01-13T01:11:10.9642595Z [LightGBM] [Warning] Using self-defined objective function
2021-01-13T01:11:10.9643397Z [LightGBM] [Warning] Using self-defined objective function
2021-01-13T01:11:10.9644337Z [51]	valid_0's binary_logloss: 5.60944	valid_0's error: 0.26	valid_0's accuracy: 0.74
2021-01-13T01:11:10.9645415Z [52]	valid_0's binary_logloss: 5.60465	valid_0's error: 0.26	valid_0's accuracy: 0.74
2021-01-13T01:11:10.9646478Z [53]	valid_0's binary_logloss: 5.60458	valid_0's error: 0.26	valid_0's accuracy: 0.74
2021-01-13T01:11:10.9647551Z [54]	valid_0's binary_logloss: 5.59957	valid_0's error: 0.254	valid_0's accuracy: 0.746
2021-01-13T01:11:10.9648637Z [55]	valid_0's binary_logloss: 5.6058	valid_0's error: 0.256	valid_0's accuracy: 0.744
2021-01-13T01:11:10.9649697Z [56]	valid_0's binary_logloss: 5.65331	valid_0's error: 0.256	valid_0's accuracy: 0.744
2021-01-13T01:11:10.9650787Z [57]	valid_0's binary_logloss: 5.70403	valid_0's error: 0.256	valid_0's accuracy: 0.744
2021-01-13T01:11:10.9651857Z [58]	valid_0's binary_logloss: 5.70323	valid_0's error: 0.26	valid_0's accuracy: 0.74
2021-01-13T01:11:10.9652994Z [59]	valid_0's binary_logloss: 5.63756	valid_0's error: 0.258	valid_0's accuracy: 0.742
2021-01-13T01:11:10.9654078Z [60]	valid_0's binary_logloss: 5.63795	valid_0's error: 0.262	valid_0's accuracy: 0.738
2021-01-13T01:11:10.9655361Z Finished 50 - 60 rounds with self-defined objective function and multiple self-defined eval metrics...
2021-01-13T01:11:10.9655939Z Starting a new training job...
2021-01-13T01:11:10.9656603Z [1]	training's binary_logloss: 0.68406
2021-01-13T01:11:10.9657338Z [2]	training's binary_logloss: 0.675795
2021-01-13T01:11:10.9658083Z [3]	training's binary_logloss: 0.666132
2021-01-13T01:11:10.9658796Z [4]	training's binary_logloss: 0.656879
2021-01-13T01:11:10.9659524Z [5]	training's binary_logloss: 0.649635
2021-01-13T01:11:10.9660014Z Add a new valid dataset at iteration 5...
2021-01-13T01:11:10.9660522Z [LightGBM] [Info] Size of histogram bin entry: 8
2021-01-13T01:11:10.9661162Z [LightGBM] [Info] 27 dense feature groups (0.15 MB) transferred to GPU in 0.001736 secs. 1 sparse feature groups
2021-01-13T01:11:10.9662174Z [6]	training's binary_logloss: 0.643418	new_valid's binary_logloss: 0.647428
2021-01-13T01:11:10.9663147Z [7]	training's binary_logloss: 0.63615	new_valid's binary_logloss: 0.64008
2021-01-13T01:11:10.9664151Z [8]	training's binary_logloss: 0.629407	new_valid's binary_logloss: 0.633584
2021-01-13T01:11:10.9665154Z [9]	training's binary_logloss: 0.622842	new_valid's binary_logloss: 0.627219
2021-01-13T01:11:10.9666132Z [10]	training's binary_logloss: 0.616898	new_valid's binary_logloss: 0.621758
2021-01-13T01:11:10.9666754Z Finished first 10 rounds with callback function...
2021-01-13T01:11:13.0395204Z Performance of `binary` objective with binary labels:
2021-01-13T01:11:13.0397613Z {'time': 0.13200116157531738, 'correlation': 0.6012584923188865, 'logloss': 0.15545640417973186}
2021-01-13T01:11:13.0398447Z Performance of `xentropy` objective with binary labels:
2021-01-13T01:11:13.0399493Z {'time': 0.04804587364196777, 'correlation': 0.6012584923188865, 'logloss': 0.15545640417973186}
2021-01-13T01:11:13.0400358Z Performance of `xentropy` objective with probability labels:
2021-01-13T01:11:13.0401402Z {'time': 0.04898548126220703, 'correlation': 0.8841891507148136, 'logloss': 0.15514485221667457}
2021-01-13T01:11:13.0402094Z Best `binary` time: 0.04302620887756348
2021-01-13T01:11:13.0402589Z Best `xentropy` time: 0.04200100898742676
2021-01-13T01:11:18.1492870Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1570: UserWarning: categorical_feature in Dataset is overridden.
2021-01-13T01:11:18.1493655Z New categorical_feature is [21]
2021-01-13T01:11:18.1494307Z   'New categorical_feature is {}'.format(sorted(list(categorical_feature))))
2021-01-13T01:11:18.1495063Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1301: UserWarning: Overriding the parameters from Reference Dataset.
2021-01-13T01:11:18.1495732Z   warnings.warn('Overriding the parameters from Reference Dataset.')
2021-01-13T01:11:18.1496464Z /home/AzDevOps_azpcontainer/.local/lib/python3.6/site-packages/lightgbm/basic.py:1113: UserWarning: categorical_column in param dict is overridden.
2021-01-13T01:11:18.1497136Z   warnings.warn('{} in param dict is overridden.'.format(cat_alias))
2021-01-13T01:11:18.1497478Z Loading data...
2021-01-13T01:11:18.1497748Z Starting training...
2021-01-13T01:11:18.1498377Z [10]	training's l1: 0.457448	training's l2: 0.217995	valid_1's l1: 0.456464	valid_1's l2: 0.21641
2021-01-13T01:11:18.1499119Z [20]	training's l1: 0.436869	training's l2: 0.205099	valid_1's l1: 0.434057	valid_1's l2: 0.201616
2021-01-13T01:11:18.1499851Z [30]	training's l1: 0.421302	training's l2: 0.197421	valid_1's l1: 0.417019	valid_1's l2: 0.192514
2021-01-13T01:11:18.1500592Z [40]	training's l1: 0.411107	training's l2: 0.192856	valid_1's l1: 0.406303	valid_1's l2: 0.187258
2021-01-13T01:11:18.1501316Z [50]	training's l1: 0.403695	training's l2: 0.189593	valid_1's l1: 0.398997	valid_1's l2: 0.183688
2021-01-13T01:11:18.1502049Z [60]	training's l1: 0.398704	training's l2: 0.187043	valid_1's l1: 0.393977	valid_1's l2: 0.181009
2021-01-13T01:11:18.1502785Z [70]	training's l1: 0.394876	training's l2: 0.184982	valid_1's l1: 0.389805	valid_1's l2: 0.178803
2021-01-13T01:11:18.1503830Z [80]	training's l1: 0.391147	training's l2: 0.1828	valid_1's l1: 0.386476	valid_1's l2: 0.176799
2021-01-13T01:11:18.1504572Z [90]	training's l1: 0.388101	training's l2: 0.180817	valid_1's l1: 0.384404	valid_1's l2: 0.175775
2021-01-13T01:11:18.1505296Z [100]	training's l1: 0.385174	training's l2: 0.179171	valid_1's l1: 0.382929	valid_1's l2: 0.175321
2021-01-13T01:11:18.1505742Z Plotting metrics recorded during training...
2021-01-13T01:11:18.1506060Z Plotting feature importances...
2021-01-13T01:11:18.1506346Z Plotting split value histogram...
2021-01-13T01:11:18.1506630Z Plotting 54th tree...
2021-01-13T01:11:18.1506906Z Plotting 54th tree with graphviz...
2021-01-13T01:11:19.7152857Z Loading data...
2021-01-13T01:11:19.7153859Z Starting training...
2021-01-13T01:11:19.7155138Z [1]	valid_0's l1: 0.493018	valid_0's l2: 0.244076
2021-01-13T01:11:19.7155985Z Training until validation scores don't improve for 5 rounds
2021-01-13T01:11:19.7156808Z [2]	valid_0's l1: 0.489056	valid_0's l2: 0.240297
2021-01-13T01:11:19.7157776Z [3]	valid_0's l1: 0.484089	valid_0's l2: 0.235733
2021-01-13T01:11:19.7158586Z [4]	valid_0's l1: 0.479088	valid_0's l2: 0.231352
2021-01-13T01:11:19.7159385Z [5]	valid_0's l1: 0.476159	valid_0's l2: 0.228939
2021-01-13T01:11:19.7160187Z [6]	valid_0's l1: 0.472664	valid_0's l2: 0.22593
2021-01-13T01:11:19.7160975Z [7]	valid_0's l1: 0.468425	valid_0's l2: 0.222515
2021-01-13T01:11:19.7161789Z [8]	valid_0's l1: 0.464594	valid_0's l2: 0.219569
2021-01-13T01:11:19.7162569Z [9]	valid_0's l1: 0.460795	valid_0's l2: 0.2168
2021-01-13T01:11:19.7163390Z [10]	valid_0's l1: 0.457276	valid_0's l2: 0.214371
2021-01-13T01:11:19.7164197Z [11]	valid_0's l1: 0.453923	valid_0's l2: 0.211988
2021-01-13T01:11:19.7165018Z [12]	valid_0's l1: 0.451235	valid_0's l2: 0.210264
2021-01-13T01:11:19.7165978Z [13]	valid_0's l1: 0.448992	valid_0's l2: 0.208926
2021-01-13T01:11:19.7166779Z [14]	valid_0's l1: 0.44634	valid_0's l2: 0.207403
2021-01-13T01:11:19.7167599Z [15]	valid_0's l1: 0.444016	valid_0's l2: 0.20601
2021-01-13T01:11:19.7168408Z [16]	valid_0's l1: 0.441362	valid_0's l2: 0.204447
2021-01-13T01:11:19.7169226Z [17]	valid_0's l1: 0.43891	valid_0's l2: 0.202712
2021-01-13T01:11:19.7170016Z [18]	valid_0's l1: 0.436192	valid_0's l2: 0.201066
2021-01-13T01:11:19.7170839Z [19]	valid_0's l1: 0.433884	valid_0's l2: 0.1998
2021-01-13T01:11:19.7171634Z [20]	valid_0's l1: 0.431129	valid_0's l2: 0.198063
2021-01-13T01:11:19.7172209Z Did not meet early stopping. Best iteration is:
2021-01-13T01:11:19.7173013Z [20]	valid_0's l1: 0.431129	valid_0's l2: 0.198063
2021-01-13T01:11:19.7173484Z Saving model...
2021-01-13T01:11:19.7173852Z Starting predicting...
2021-01-13T01:11:19.7174406Z The rmse of prediction is: 0.44504264484579004
2021-01-13T01:11:37.8084270Z Loading data...
2021-01-13T01:11:37.8085015Z Starting training...
2021-01-13T01:11:37.8086188Z [1]	valid_0's l1: 0.491735	valid_0's l2: 0.242765
2021-01-13T01:11:37.8086968Z Training until validation scores don't improve for 5 rounds
2021-01-13T01:11:37.8090973Z [2]	valid_0's l1: 0.486563	valid_0's l2: 0.237898
2021-01-13T01:11:37.8092520Z [3]	valid_0's l1: 0.48149	valid_0's l2: 0.233279
2021-01-13T01:11:37.8093530Z [4]	valid_0's l1: 0.476849	valid_0's l2: 0.229253
2021-01-13T01:11:37.8094464Z [5]	valid_0's l1: 0.473051	valid_0's l2: 0.226158
2021-01-13T01:11:37.8095319Z [6]	valid_0's l1: 0.469178	valid_0's l2: 0.22308
2021-01-13T01:11:37.8096144Z [7]	valid_0's l1: 0.465809	valid_0's l2: 0.220584
2021-01-13T01:11:37.8096977Z [8]	valid_0's l1: 0.46241	valid_0's l2: 0.218042
2021-01-13T01:11:37.8103926Z [9]	valid_0's l1: 0.458543	valid_0's l2: 0.215247
2021-01-13T01:11:37.8158196Z [10]	valid_0's l1: 0.455358	valid_0's l2: 0.213072
2021-01-13T01:11:37.8159547Z [11]	valid_0's l1: 0.452153	valid_0's l2: 0.210952
2021-01-13T01:11:37.8160588Z [12]	valid_0's l1: 0.448965	valid_0's l2: 0.20882
2021-01-13T01:11:37.8161862Z [13]	valid_0's l1: 0.445868	valid_0's l2: 0.206819
2021-01-13T01:11:37.8163225Z [14]	valid_0's l1: 0.443026	valid_0's l2: 0.205014
2021-01-13T01:11:37.8164316Z [15]	valid_0's l1: 0.440732	valid_0's l2: 0.203766
2021-01-13T01:11:37.8165249Z [16]	valid_0's l1: 0.438111	valid_0's l2: 0.20219
2021-01-13T01:11:37.8166204Z [17]	valid_0's l1: 0.435348	valid_0's l2: 0.200555
2021-01-13T01:11:37.8173322Z [18]	valid_0's l1: 0.433066	valid_0's l2: 0.199331
2021-01-13T01:11:37.8174739Z [19]	valid_0's l1: 0.430563	valid_0's l2: 0.197837
2021-01-13T01:11:37.8175765Z [20]	valid_0's l1: 0.428386	valid_0's l2: 0.196621
2021-01-13T01:11:37.8176471Z Did not meet early stopping. Best iteration is:
2021-01-13T01:11:37.8177418Z [20]	valid_0's l1: 0.428386	valid_0's l2: 0.196621
2021-01-13T01:11:37.8178146Z Starting predicting...
2021-01-13T01:11:37.8179007Z The rmse of prediction is: 0.44341943175774834
2021-01-13T01:11:37.8179712Z Feature importances: [27, 4, 1, 34, 3, 63, 14, 2, 2, 24, 2, 4, 1, 18, 7, 4, 1, 9, 4, 6, 0, 32, 56, 3, 52, 97, 47, 83]
2021-01-13T01:11:37.8180386Z Starting training with custom eval function...
2021-01-13T01:11:37.8181364Z [1]	valid_0's l2: 0.242765	valid_0's RMSLE: 0.344965
2021-01-13T01:11:37.8182239Z Training until validation scores don't improve for 5 rounds
2021-01-13T01:11:37.8183075Z [2]	valid_0's l2: 0.237898	valid_0's RMSLE: 0.341702
2021-01-13T01:11:37.8183924Z [3]	valid_0's l2: 0.233279	valid_0's RMSLE: 0.33847
2021-01-13T01:11:37.8184774Z [4]	valid_0's l2: 0.229253	valid_0's RMSLE: 0.335665
2021-01-13T01:11:37.8185606Z [5]	valid_0's l2: 0.226158	valid_0's RMSLE: 0.33344
2021-01-13T01:11:37.8186619Z [6]	valid_0's l2: 0.22308	valid_0's RMSLE: 0.331188
2021-01-13T01:11:37.8187450Z [7]	valid_0's l2: 0.220584	valid_0's RMSLE: 0.329332
2021-01-13T01:11:37.8188290Z [8]	valid_0's l2: 0.218042	valid_0's RMSLE: 0.327426
2021-01-13T01:11:37.8189214Z [9]	valid_0's l2: 0.215247	valid_0's RMSLE: 0.325385
2021-01-13T01:11:37.8190029Z [10]	valid_0's l2: 0.213072	valid_0's RMSLE: 0.32372
2021-01-13T01:11:37.8190831Z [11]	valid_0's l2: 0.210952	valid_0's RMSLE: 0.32213
2021-01-13T01:11:37.8191661Z [12]	valid_0's l2: 0.20882	valid_0's RMSLE: 0.320486
2021-01-13T01:11:37.8192494Z [13]	valid_0's l2: 0.206819	valid_0's RMSLE: 0.318913
2021-01-13T01:11:37.8193310Z [14]	valid_0's l2: 0.205014	valid_0's RMSLE: 0.317429
2021-01-13T01:11:37.8194277Z [15]	valid_0's l2: 0.203766	valid_0's RMSLE: 0.31641
2021-01-13T01:11:37.8195097Z [16]	valid_0's l2: 0.20219	valid_0's RMSLE: 0.315169
2021-01-13T01:11:37.8195943Z [17]	valid_0's l2: 0.200555	valid_0's RMSLE: 0.313858
2021-01-13T01:11:37.8196782Z [18]	valid_0's l2: 0.199331	valid_0's RMSLE: 0.312869
2021-01-13T01:11:37.8197789Z [19]	valid_0's l2: 0.197837	valid_0's RMSLE: 0.311625
2021-01-13T01:11:37.8198624Z [20]	valid_0's l2: 0.196621	valid_0's RMSLE: 0.310598
2021-01-13T01:11:37.8199205Z Did not meet early stopping. Best iteration is:
2021-01-13T01:11:37.8200021Z [20]	valid_0's l2: 0.196621	valid_0's RMSLE: 0.310598
2021-01-13T01:11:37.8200584Z Starting training with multiple custom eval functions...
2021-01-13T01:11:37.8201489Z [1]	valid_0's l2: 0.242765	valid_0's RMSLE: 0.344965	valid_0's RAE: 0.991146
2021-01-13T01:11:37.8202379Z Training until validation scores don't improve for 5 rounds
2021-01-13T01:11:37.8203302Z [2]	valid_0's l2: 0.237898	valid_0's RMSLE: 0.341702	valid_0's RAE: 0.980721
2021-01-13T01:11:37.8204341Z [3]	valid_0's l2: 0.233279	valid_0's RMSLE: 0.33847	valid_0's RAE: 0.970495
2021-01-13T01:11:37.8205371Z [4]	valid_0's l2: 0.229253	valid_0's RMSLE: 0.335665	valid_0's RAE: 0.961141
2021-01-13T01:11:37.8206545Z [5]	valid_0's l2: 0.226158	valid_0's RMSLE: 0.33344	valid_0's RAE: 0.953486
2021-01-13T01:11:37.8207512Z [6]	valid_0's l2: 0.22308	valid_0's RMSLE: 0.331188	valid_0's RAE: 0.945679
2021-01-13T01:11:37.8208486Z [7]	valid_0's l2: 0.220584	valid_0's RMSLE: 0.329332	valid_0's RAE: 0.938889
2021-01-13T01:11:37.8209459Z [8]	valid_0's l2: 0.218042	valid_0's RMSLE: 0.327426	valid_0's RAE: 0.932037
2021-01-13T01:11:37.8210897Z [9]	valid_0's l2: 0.215247	valid_0's RMSLE: 0.325385	valid_0's RAE: 0.924243
2021-01-13T01:11:37.8214027Z [10]	valid_0's l2: 0.213072	valid_0's RMSLE: 0.32372	valid_0's RAE: 0.917824
2021-01-13T01:11:37.8218121Z [11]	valid_0's l2: 0.210952	valid_0's RMSLE: 0.32213	valid_0's RAE: 0.911365
2021-01-13T01:11:37.8221239Z [12]	valid_0's l2: 0.20882	valid_0's RMSLE: 0.320486	valid_0's RAE: 0.904939
2021-01-13T01:11:37.8224486Z [13]	valid_0's l2: 0.206819	valid_0's RMSLE: 0.318913	valid_0's RAE: 0.898696
2021-01-13T01:11:37.8227622Z [14]	valid_0's l2: 0.205014	valid_0's RMSLE: 0.317429	valid_0's RAE: 0.892967
2021-01-13T01:11:37.8230814Z [15]	valid_0's l2: 0.203766	valid_0's RMSLE: 0.31641	valid_0's RAE: 0.888344
2021-01-13T01:11:37.8233934Z [16]	valid_0's l2: 0.20219	valid_0's RMSLE: 0.315169	valid_0's RAE: 0.88306
2021-01-13T01:11:37.8237067Z [17]	valid_0's l2: 0.200555	valid_0's RMSLE: 0.313858	valid_0's RAE: 0.877491
2021-01-13T01:11:37.8240500Z [18]	valid_0's l2: 0.199331	valid_0's RMSLE: 0.312869	valid_0's RAE: 0.872892
2021-01-13T01:11:37.8243713Z [19]	valid_0's l2: 0.197837	valid_0's RMSLE: 0.311625	valid_0's RAE: 0.867847
2021-01-13T01:11:37.8246964Z [20]	valid_0's l2: 0.196621	valid_0's RMSLE: 0.310598	valid_0's RAE: 0.863459
2021-01-13T01:11:37.8249734Z Did not meet early stopping. Best iteration is:
2021-01-13T01:11:37.8251886Z [20]	valid_0's l2: 0.196621	valid_0's RMSLE: 0.310598	valid_0's RAE: 0.863459
2021-01-13T01:11:37.8254660Z Starting predicting...
2021-01-13T01:11:37.8256129Z The rmsle of prediction is: 0.3105980469191399
2021-01-13T01:11:37.8257786Z The rae of prediction is: 0.8634589616622911
2021-01-13T01:11:37.8260027Z Best parameters found by grid search are: {'learning_rate': 0.1, 'n_estimators': 40}
2021-01-13T01:11:39.3085398Z Collecting package metadata (current_repodata.json): ...working... done
2021-01-13T01:11:42.6493240Z Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
2021-01-13T01:11:45.3936517Z Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
2021-01-13T01:11:46.8317799Z Collecting package metadata (repodata.json): ...working... done
2021-01-13T01:11:53.6858051Z Solving environment: ...working... done
2021-01-13T01:11:56.1233673Z 
2021-01-13T01:11:56.1242147Z ## Package Plan ##
2021-01-13T01:11:56.1242549Z 
2021-01-13T01:11:56.1243713Z   environment location: /opt/conda/envs/test-env
2021-01-13T01:11:56.1244143Z 
2021-01-13T01:11:56.1244521Z   added / updated specs:
2021-01-13T01:11:56.1245171Z     - ipywidgets
2021-01-13T01:11:56.1245752Z     - notebook
2021-01-13T01:11:56.1246050Z 
2021-01-13T01:11:56.1246311Z 
2021-01-13T01:11:56.1246704Z The following packages will be downloaded:
2021-01-13T01:11:56.1247069Z 
2021-01-13T01:11:56.1247443Z     package                    |            build
2021-01-13T01:11:56.1248260Z     ---------------------------|-----------------
2021-01-13T01:11:56.1249117Z     argon2-cffi-20.1.0         |   py36h7b6447c_1          46 KB
2021-01-13T01:11:56.1249984Z     async_generator-1.10       |   py36h28b3542_0          39 KB
2021-01-13T01:11:56.1250864Z     backcall-0.2.0             |             py_0          15 KB
2021-01-13T01:11:56.1251718Z     bleach-3.2.1               |             py_0         112 KB
2021-01-13T01:11:56.1252589Z     cffi-1.14.0                |   py36h2e261b9_0         223 KB
2021-01-13T01:11:56.1253450Z     decorator-4.4.2            |             py_0          14 KB
2021-01-13T01:11:56.1254272Z     defusedxml-0.6.0           |             py_0          23 KB
2021-01-13T01:11:56.1255161Z     entrypoints-0.3            |           py36_0          12 KB
2021-01-13T01:11:56.1255982Z     ipykernel-5.3.4            |   py36h5ca1d4c_0         181 KB
2021-01-13T01:11:56.1256886Z     ipython-7.16.1             |   py36h5ca1d4c_0         999 KB
2021-01-13T01:11:56.1257722Z     ipython_genutils-0.2.0     |     pyhd3eb1b0_1          27 KB
2021-01-13T01:11:56.1258607Z     ipywidgets-7.6.3           |     pyhd3eb1b0_1         105 KB
2021-01-13T01:11:56.1259445Z     jedi-0.18.0                |   py36h06a4308_0         905 KB
2021-01-13T01:11:56.1260593Z     jsonschema-3.2.0           |             py_2          47 KB
2021-01-13T01:11:56.1261570Z     jupyter_client-6.1.7       |             py_0          77 KB
2021-01-13T01:11:56.1262448Z     jupyter_core-4.7.0         |   py36h06a4308_0          68 KB
2021-01-13T01:11:56.1263308Z     jupyterlab_pygments-0.1.2  |             py_0           8 KB
2021-01-13T01:11:56.1264199Z     jupyterlab_widgets-1.0.0   |     pyhd3eb1b0_1         109 KB
2021-01-13T01:11:56.1265014Z     libsodium-1.0.18           |       h7b6447c_0         244 KB
2021-01-13T01:11:56.1265906Z     mistune-0.8.4              |   py36h7b6447c_0          55 KB
2021-01-13T01:11:56.1266731Z     nbclient-0.5.1             |             py_0          58 KB
2021-01-13T01:11:56.1267592Z     nbconvert-6.0.7            |           py36_0         480 KB
2021-01-13T01:11:56.1268444Z     nbformat-5.0.8             |             py_0          88 KB
2021-01-13T01:11:56.1269311Z     nest-asyncio-1.4.3         |     pyhd3eb1b0_0          11 KB
2021-01-13T01:11:56.1270182Z     notebook-6.1.6             |   py36h06a4308_0         4.1 MB
2021-01-13T01:11:56.1271045Z     pandoc-2.11                |       hb0f4dca_0         9.6 MB
2021-01-13T01:11:56.1271920Z     pandocfilters-1.4.3        |   py36h06a4308_1          14 KB
2021-01-13T01:11:56.1272748Z     parso-0.7.0                |             py_0          72 KB
2021-01-13T01:11:56.1273643Z     pexpect-4.8.0              |     pyhd3eb1b0_3          53 KB
2021-01-13T01:11:56.1274462Z     pickleshare-0.7.5          |  pyhd3eb1b0_1003          13 KB
2021-01-13T01:11:56.1275356Z     prometheus_client-0.9.0    |     pyhd3eb1b0_0          45 KB
2021-01-13T01:11:56.1276190Z     prompt-toolkit-3.0.8       |             py_0         248 KB
2021-01-13T01:11:56.1277191Z     ptyprocess-0.7.0           |     pyhd3eb1b0_2          17 KB
2021-01-13T01:11:56.1278077Z     pygments-2.7.3             |     pyhd3eb1b0_0         676 KB
2021-01-13T01:11:56.1278913Z     pyrsistent-0.17.3          |   py36h7b6447c_0          89 KB
2021-01-13T01:11:56.1279799Z     pyzmq-20.0.0               |   py36h2531618_1         438 KB
2021-01-13T01:11:56.1280621Z     send2trash-1.5.0           |     pyhd3eb1b0_1          14 KB
2021-01-13T01:11:56.1281519Z     terminado-0.9.2            |   py36h06a4308_0          25 KB
2021-01-13T01:11:56.1282328Z     testpath-0.4.4             |             py_0          82 KB
2021-01-13T01:11:56.1283234Z     traitlets-4.3.3            |           py36_0         140 KB
2021-01-13T01:11:56.1284050Z     wcwidth-0.2.5              |             py_0          29 KB
2021-01-13T01:11:56.1284926Z     webencodings-0.5.1         |           py36_1          19 KB
2021-01-13T01:11:56.1285803Z     widgetsnbextension-3.5.1   |           py36_0         862 KB
2021-01-13T01:11:56.1286631Z     zeromq-4.3.3               |       he6710b0_3         500 KB
2021-01-13T01:11:56.1287495Z     ------------------------------------------------------------
2021-01-13T01:11:56.1288084Z                                            Total:        20.8 MB
2021-01-13T01:11:56.1288513Z 
2021-01-13T01:11:56.1288926Z The following NEW packages will be INSTALLED:
2021-01-13T01:11:56.1289304Z 
2021-01-13T01:11:56.1290096Z   argon2-cffi        pkgs/main/linux-64::argon2-cffi-20.1.0-py36h7b6447c_1
2021-01-13T01:11:56.1291062Z   async_generator    pkgs/main/linux-64::async_generator-1.10-py36h28b3542_0
2021-01-13T01:11:56.1292084Z   backcall           pkgs/main/noarch::backcall-0.2.0-py_0
2021-01-13T01:11:56.1292948Z   bleach             pkgs/main/noarch::bleach-3.2.1-py_0
2021-01-13T01:11:56.1293807Z   cffi               pkgs/main/linux-64::cffi-1.14.0-py36h2e261b9_0
2021-01-13T01:11:56.1294668Z   decorator          pkgs/main/noarch::decorator-4.4.2-py_0
2021-01-13T01:11:56.1295523Z   defusedxml         pkgs/main/noarch::defusedxml-0.6.0-py_0
2021-01-13T01:11:56.1296386Z   entrypoints        pkgs/main/linux-64::entrypoints-0.3-py36_0
2021-01-13T01:11:56.1297420Z   ipykernel          pkgs/main/linux-64::ipykernel-5.3.4-py36h5ca1d4c_0
2021-01-13T01:11:56.1298539Z   ipython            pkgs/main/linux-64::ipython-7.16.1-py36h5ca1d4c_0
2021-01-13T01:11:56.1299626Z   ipython_genutils   pkgs/main/noarch::ipython_genutils-0.2.0-pyhd3eb1b0_1
2021-01-13T01:11:56.1300517Z   ipywidgets         pkgs/main/noarch::ipywidgets-7.6.3-pyhd3eb1b0_1
2021-01-13T01:11:56.1301446Z   jedi               pkgs/main/linux-64::jedi-0.18.0-py36h06a4308_0
2021-01-13T01:11:56.1302287Z   jsonschema         pkgs/main/noarch::jsonschema-3.2.0-py_2
2021-01-13T01:11:56.1303187Z   jupyter_client     pkgs/main/noarch::jupyter_client-6.1.7-py_0
2021-01-13T01:11:56.1304087Z   jupyter_core       pkgs/main/linux-64::jupyter_core-4.7.0-py36h06a4308_0
2021-01-13T01:11:56.1305113Z   jupyterlab_pygmen~ pkgs/main/noarch::jupyterlab_pygments-0.1.2-py_0
2021-01-13T01:11:56.1306046Z   jupyterlab_widgets pkgs/main/noarch::jupyterlab_widgets-1.0.0-pyhd3eb1b0_1
2021-01-13T01:11:56.1306985Z   libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
2021-01-13T01:11:56.1307894Z   mistune            pkgs/main/linux-64::mistune-0.8.4-py36h7b6447c_0
2021-01-13T01:11:56.1308778Z   nbclient           pkgs/main/noarch::nbclient-0.5.1-py_0
2021-01-13T01:11:56.1309607Z   nbconvert          pkgs/main/linux-64::nbconvert-6.0.7-py36_0
2021-01-13T01:11:56.1310461Z   nbformat           pkgs/main/noarch::nbformat-5.0.8-py_0
2021-01-13T01:11:56.1311358Z   nest-asyncio       pkgs/main/noarch::nest-asyncio-1.4.3-pyhd3eb1b0_0
2021-01-13T01:11:56.1312263Z   notebook           pkgs/main/linux-64::notebook-6.1.6-py36h06a4308_0
2021-01-13T01:11:56.1313157Z   pandoc             pkgs/main/linux-64::pandoc-2.11-hb0f4dca_0
2021-01-13T01:11:56.1314082Z   pandocfilters      pkgs/main/linux-64::pandocfilters-1.4.3-py36h06a4308_1
2021-01-13T01:11:56.1314961Z   parso              pkgs/main/noarch::parso-0.7.0-py_0
2021-01-13T01:11:56.1315791Z   pexpect            pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
2021-01-13T01:11:56.1316650Z   pickleshare        pkgs/main/noarch::pickleshare-0.7.5-pyhd3eb1b0_1003
2021-01-13T01:11:56.1317713Z   prometheus_client  pkgs/main/noarch::prometheus_client-0.9.0-pyhd3eb1b0_0
2021-01-13T01:11:56.1318547Z   prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
2021-01-13T01:11:56.1319369Z   ptyprocess         pkgs/main/noarch::ptyprocess-0.7.0-pyhd3eb1b0_2
2021-01-13T01:11:56.1320210Z   pycparser          pkgs/main/noarch::pycparser-2.20-py_2
2021-01-13T01:11:56.1321047Z   pygments           pkgs/main/noarch::pygments-2.7.3-pyhd3eb1b0_0
2021-01-13T01:11:56.1321980Z   pyrsistent         pkgs/main/linux-64::pyrsistent-0.17.3-py36h7b6447c_0
2021-01-13T01:11:56.1322829Z   pyzmq              pkgs/main/linux-64::pyzmq-20.0.0-py36h2531618_1
2021-01-13T01:11:56.1323697Z   send2trash         pkgs/main/noarch::send2trash-1.5.0-pyhd3eb1b0_1
2021-01-13T01:11:56.1324616Z   terminado          pkgs/main/linux-64::terminado-0.9.2-py36h06a4308_0
2021-01-13T01:11:56.1325521Z   testpath           pkgs/main/noarch::testpath-0.4.4-py_0
2021-01-13T01:11:56.1326419Z   traitlets          pkgs/main/linux-64::traitlets-4.3.3-py36_0
2021-01-13T01:11:56.1327294Z   wcwidth            pkgs/main/noarch::wcwidth-0.2.5-py_0
2021-01-13T01:11:56.1328197Z   webencodings       pkgs/main/linux-64::webencodings-0.5.1-py36_1
2021-01-13T01:11:56.1329108Z   widgetsnbextension pkgs/main/linux-64::widgetsnbextension-3.5.1-py36_0
2021-01-13T01:11:56.1330023Z   zeromq             pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
2021-01-13T01:11:56.1330437Z 
2021-01-13T01:11:56.1330697Z 
2021-01-13T01:11:57.3866325Z Preparing transaction: ...working... done
2021-01-13T01:11:57.8747012Z Verifying transaction: ...working... done
2021-01-13T01:12:00.5605899Z Executing transaction: ...working... done
2021-01-13T01:12:01.7435203Z [NbConvertApp] Converting notebook interactive_plot_example.ipynb to notebook
2021-01-13T01:12:02.6216058Z [NbConvertApp] Executing notebook with kernel: python3
2021-01-13T01:12:04.6427881Z Traceback (most recent call last):
2021-01-13T01:12:04.6429898Z   File "/opt/conda/envs/test-env/bin/jupyter-nbconvert", line 11, in <module>
2021-01-13T01:12:04.6430647Z     sys.exit(main())
2021-01-13T01:12:04.6431484Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/jupyter_core/application.py", line 254, in launch_instance
2021-01-13T01:12:04.6432103Z     return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
2021-01-13T01:12:04.6432719Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
2021-01-13T01:12:04.6433086Z     app.start()
2021-01-13T01:12:04.6433624Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 350, in start
2021-01-13T01:12:04.6433971Z     self.convert_notebooks()
2021-01-13T01:12:04.6434538Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
2021-01-13T01:12:04.6434927Z     self.convert_single_notebook(notebook_filename)
2021-01-13T01:12:04.6435527Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
2021-01-13T01:12:04.6436003Z     output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
2021-01-13T01:12:04.6436681Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
2021-01-13T01:12:04.6437321Z     output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
2021-01-13T01:12:04.6437969Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
2021-01-13T01:12:04.6438384Z     return self.from_file(f, resources=resources, **kw)
2021-01-13T01:12:04.6438963Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
2021-01-13T01:12:04.6439433Z     return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
2021-01-13T01:12:04.6440099Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
2021-01-13T01:12:04.6440531Z     nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
2021-01-13T01:12:04.6441174Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
2021-01-13T01:12:04.6441589Z     nb_copy, resources = self._preprocess(nb_copy, resources)
2021-01-13T01:12:04.6442189Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
2021-01-13T01:12:04.6442573Z     nbc, resc = preprocessor(nbc, resc)
2021-01-13T01:12:04.6443125Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
2021-01-13T01:12:04.6443509Z     return self.preprocess(nb, resources)
2021-01-13T01:12:04.6444071Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
2021-01-13T01:12:04.6444441Z     self.execute()
2021-01-13T01:12:04.6444943Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/util.py", line 74, in wrapped
2021-01-13T01:12:04.6445313Z     return just_run(coro(*args, **kwargs))
2021-01-13T01:12:04.6445855Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/util.py", line 53, in just_run
2021-01-13T01:12:04.6446204Z     return loop.run_until_complete(coro)
2021-01-13T01:12:04.6446752Z   File "/opt/conda/envs/test-env/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
2021-01-13T01:12:04.6447088Z     return future.result()
2021-01-13T01:12:04.6447627Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/client.py", line 541, in async_execute
2021-01-13T01:12:04.6448013Z     cell, index, execution_count=self.code_cells_executed + 1
2021-01-13T01:12:04.6448634Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 123, in async_execute_cell
2021-01-13T01:12:04.6449188Z     cell, resources = self.preprocess_cell(cell, self.resources, cell_index)
2021-01-13T01:12:04.6449880Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 146, in preprocess_cell
2021-01-13T01:12:04.6450365Z     cell = run_sync(NotebookClient.async_execute_cell)(self, cell, index, store_history=self.store_history)
2021-01-13T01:12:04.6450969Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/util.py", line 74, in wrapped
2021-01-13T01:12:04.6451336Z     return just_run(coro(*args, **kwargs))
2021-01-13T01:12:04.6451880Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/util.py", line 53, in just_run
2021-01-13T01:12:04.6452231Z     return loop.run_until_complete(coro)
2021-01-13T01:12:04.6452788Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nest_asyncio.py", line 98, in run_until_complete
2021-01-13T01:12:04.6453125Z     return f.result()
2021-01-13T01:12:04.6453619Z   File "/opt/conda/envs/test-env/lib/python3.6/asyncio/tasks.py", line 180, in _step
2021-01-13T01:12:04.6454066Z     result = coro.send(None)
2021-01-13T01:12:04.6454984Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/client.py", line 832, in async_execute_cell
2021-01-13T01:12:04.6455369Z     self._check_raise_for_error(cell, exec_reply)
2021-01-13T01:12:04.6455927Z   File "/opt/conda/envs/test-env/lib/python3.6/site-packages/nbclient/client.py", line 740, in _check_raise_for_error
2021-01-13T01:12:04.6456508Z     raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
2021-01-13T01:12:04.6456908Z nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
2021-01-13T01:12:04.6457377Z ------------------

evals_result = {}  # to record eval results for plotting
gbm = lgb.train(params,
                lgb_train,
                num_boost_round=100,
                valid_sets=[lgb_train, lgb_test],
                feature_name=['f' + str(i + 1) for i in range(X_train.shape[-1])],
                categorical_feature=[21],
                evals_result=evals_result,
                verbose_eval=10)
------------------

---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
<ipython-input-1-fb9e9ad1250c> in <module>
      7                 categorical_feature=[21],
      8                 evals_result=evals_result,
----> 9                 verbose_eval=10)

~/.local/lib/python3.6/site-packages/lightgbm/engine.py in train(params, train_set, num_boost_round, valid_sets, valid_names, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, evals_result, verbose_eval, learning_rates, keep_training_booster, callbacks)
    226     # construct booster
    227     try:
--> 228         booster = Booster(params=params, train_set=train_set)
    229         if is_valid_contain_train:
    230             booster.set_train_data_name(train_data_name)

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in __init__(self, params, train_set, model_file, model_str, silent)
   2074                 train_set.handle,
   2075                 c_str(params_str),
-> 2076                 ctypes.byref(self.handle)))
   2077             # save reference to data
   2078             self.train_set = train_set

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in _safe_call(ret)
     50     """
     51     if ret != 0:
---> 52         raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
     53 
     54 

LightGBMError: No OpenCL device found
LightGBMError: No OpenCL device found

##[error]Bash exited with code '255'.
Finishing: Test

@StrikerRUS StrikerRUS mentioned this issue Jan 28, 2021
Merged
@StrikerRUS StrikerRUS mentioned this issue Jan 20, 2022
Merged
@StrikerRUS StrikerRUS mentioned this issue Feb 12, 2022
Merged
@jameslamb jameslamb mentioned this issue Apr 14, 2022
Closed
60 tasks
@StrikerRUS StrikerRUS mentioned this issue Jun 13, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jameslamb @guolinke @StrikerRUS