clean library and header dependencies #930

njzjz · 2021-08-06T10:23:56Z

codecov-commenter · 2021-08-06T10:27:44Z

Codecov Report

Merging #930 (fa73dc9) into devel (4ced020) will decrease coverage by 11.13%.
The diff coverage is n/a.

❗ Current head fa73dc9 differs from pull request most recent head 56959ca. Consider uploading reports for the commit 56959ca to get more accurate results

@@             Coverage Diff             @@
##            devel     #930       +/-   ##
===========================================
- Coverage   75.41%   64.28%   -11.14%     
===========================================
  Files          85        5       -80     
  Lines        6729       14     -6715     
===========================================
- Hits         5075        9     -5066     
+ Misses       1654        5     -1649

Impacted Files	Coverage Δ
deepmd/infer/model_devi.py
source/op/_prod_virial_se_r_grad.py
deepmd/infer/__init__.py
deepmd/model/model_stat.py
deepmd/__init__.py
deepmd/fit/dipole.py
deepmd/utils/compat.py
deepmd/descriptor/se_a_ebd.py
deepmd/model/__init__.py
deepmd/__about__.py
... and 62 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ced020...56959ca. Read the comment docs.

njzjz

Current dependencies:

source/lib/src/rocm/CMakeLists.txt

denghuilu

An error accured when I tried to compile and run the deepmd-kit python interface:

root deepmd-kit $ pip install .
Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/
Processing /root/dp-devel/deepmd-kit
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: python-hostlist>=1.21 in /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages (from deepmd-kit==2.0.0b4.dev28+g2684f11) (1.21)
Requirement already satisfied: typing-extensions in /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages (from deepmd-kit==2.0.0b4.dev28+g2684f11) (3.7.4.3)
Requirement already satisfied: scipy in /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages (from deepmd-kit==2.0.0b4.dev28+g2684f11) (1.5.4)
Requirement already satisfied: dargs>=0.2.6 in /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/dargs-0.2.6-py3.6.egg (from deepmd-kit==2.0.0b4.dev28+g2684f11) (0.2.6)
Requirement already satisfied: numpy in /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages (from deepmd-kit==2.0.0b4.dev28+g2684f11) (1.19.5)
Requirement already satisfied: pyyaml in /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/PyYAML-5.4.1-py3.6-linux-x86_64.egg (from deepmd-kit==2.0.0b4.dev28+g2684f11) (5.4.1)
Building wheels for collected packages: deepmd-kit
  Building wheel for deepmd-kit (PEP 517) ... done
  Created wheel for deepmd-kit: filename=deepmd_kit-2.0.0b4.dev28+g2684f11-cp36-cp36m-linux_x86_64.whl size=1624618 sha256=d480952a7cbb436be83f76e2825688d2955accde84723c7e5b4ac44733c16bc1
  Stored in directory: /root/.cache/pip/wheels/d7/0d/2a/b6f1653be91ad8dd465c1560d36da90e510c5d488bc4fd6563
Successfully built deepmd-kit
Installing collected packages: deepmd-kit
Successfully installed deepmd-kit-2.0.0b4.dev28+g2684f11
root deepmd-kit $ dp -h
WARNING:tensorflow:From /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
Traceback (most recent call last):
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/env.py", line 176, in get_module
    module = tf.load_op_library(str(module_file))
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 57, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: libdeepmd_op_cuda.so: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/dp-devel/tensorflow_venv/bin/dp", line 5, in <module>
    from deepmd.entrypoints.main import main
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/__init__.py", line 3, in <module>
    import deepmd.utils.network as network
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/utils/__init__.py", line 2, in <module>
    from .data import DeepmdData
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/utils/data.py", line 10, in <module>
    from deepmd.env import GLOBAL_NP_FLOAT_PRECISION
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/env.py", line 253, in <module>
    op_module = get_module("libop_abi")
  File "/root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/env.py", line 224, in get_module
    )) from e
RuntimeError: This deepmd-kit package is inconsitent with TensorFlowRuntime, thus an error is raised when loading libop_abi.You need to rebuild deepmd-kit against this TensorFlowruntime.

njzjz · 2021-08-09T00:06:55Z

@denghuilu I cannot reproduce the error. Can you check with ldd?

(dpdev) [jz748@localhost deepmd-kit]$ ldd /home/jz748/anaconda3/envs/dpdev/lib/python3.8/site-packages/deepmd/op/libop_abi.so
        linux-vdso.so.1 (0x00007ffd9b1ef000)
        libdeepmd.so => /home/jz748/anaconda3/envs/dpdev/lib/python3.8/site-packages/deepmd/op/libdeepmd.so (0x00007f80e4561000)
        libtensorflow_framework.so.2 => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f80e4329000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f80e41e5000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f80e419f000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f80e4184000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f80e4161000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f80e3f92000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f80e3f8b000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f80e3f80000)
        libdeepmd_op_cuda.so => /home/jz748/anaconda3/envs/dpdev/lib/python3.8/site-packages/deepmd/op/libdeepmd_op_cuda.so (0x00007f80e3c5f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f80e46e2000)

dp -h
2021-08-08 20:06:38.747579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /home/jz748/anaconda3/envs/dpdev/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
usage: dp [-h] {config,transfer,train,freeze,test,compress,doc-train-input,model-devi,convert-from} ...

DeePMD-kit: A deep learning package for many-body potential energy representation and molecular dynamics

optional arguments:
  -h, --help            show this help message and exit

Valid subcommands:
  {config,transfer,train,freeze,test,compress,doc-train-input,model-devi,convert-from}
    config              fast configuration of parameter file for smooth model
    transfer            pass parameters to another model
    train               train a model
    freeze              freeze the model
    test                test the model
    compress            compress a model
    doc-train-input     print the documentation (in rst format) of input training parameters.
    model-devi          calculate model deviation
    convert-from        convert lower model version to supported version

denghuilu · 2021-08-09T01:40:39Z

@njzjz

root deepmd-kit $ ls /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/op/
_gelu.py              libdeepmd.so    _prod_force_grad.py       _prod_virial_grad.py       __pycache__               _tabulate_grad.py
__init__.py           libop_abi.so    _prod_force_se_a_grad.py  _prod_virial_se_a_grad.py  _soft_min_force_grad.py
libdeepmd_op_cuda.so  libop_grads.so  _prod_force_se_r_grad.py  _prod_virial_se_r_grad.py  _soft_min_virial_grad.py

root deepmd-kit $ ldd /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/op/libop_abi.so
        linux-vdso.so.1 (0x00007ffec3ed7000)
        libdeepmd.so => /root/dp-devel/tensorflow_venv/lib/python3.6/site-packages/deepmd/op/libdeepmd.so (0x00007f7630ef1000)
        libtensorflow_framework.so.2 => not found
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7630b68000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f76307ca000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f763059b000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7630383000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7630164000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f762fd73000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f762fb6f000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f762f967000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f76314a8000)
        libdeepmd_op_cuda.so => not found

njzjz · 2021-08-09T01:55:12Z

Maybe related to https://stackoverflow.com/q/61479487/9567349, but it's unclear to me why I don't have this error...

denghuilu · 2021-08-09T02:29:18Z

Maybe related to https://stackoverflow.com/q/61479487/9567349, but it's unclear to me why I don't have this error...

I can only get this problem when using DP_VARIANT=cuda, which may be helpful to reproduce the problem.

njzjz · 2021-08-09T02:56:35Z

@denghuilu what is your cmake version?

njzjz · 2021-08-09T03:22:36Z

@denghuilu Please check if 56959ca works.

denghuilu · 2021-08-09T06:43:04Z

@denghuilu what is your cmake version?

cmake version 3.10.2

denghuilu

everything works well on my local workstation now.

* clean library and header dependencies Fix deepmodeling#926. * fix typo in rocm * set INSTALL_RPATH for libraries

…#930) - skip cell_type if from_poscar is True - use from_poscar = jdata.get('from_poscar', False) to get from_poscar

njzjz linked an issue Aug 6, 2021 that may be closed by this pull request

[Feature Request] rebuild library dependencies #926

Closed

njzjz assigned amcadmus Aug 6, 2021

clean library and header dependencies

791e5a9

Fix deepmodeling#926.

njzjz force-pushed the dep branch from d8b8864 to 791e5a9 Compare August 6, 2021 10:25

njzjz commented Aug 6, 2021

View reviewed changes

amcadmus requested review from galeselee and denghuilu August 6, 2021 13:00

galeselee reviewed Aug 7, 2021

View reviewed changes

source/lib/src/rocm/CMakeLists.txt Outdated Show resolved Hide resolved

galeselee reviewed Aug 7, 2021

View reviewed changes

source/lib/src/rocm/CMakeLists.txt Show resolved Hide resolved

fix typo in rocm

2684f11

amcadmus requested a review from galeselee August 8, 2021 00:17

galeselee approved these changes Aug 8, 2021

View reviewed changes

denghuilu reviewed Aug 8, 2021

View reviewed changes

set INSTALL_RPATH for libraries

56959ca

amcadmus requested a review from denghuilu August 9, 2021 05:42

denghuilu approved these changes Aug 9, 2021

View reviewed changes

amcadmus approved these changes Aug 10, 2021

View reviewed changes

amcadmus merged commit 06d591d into deepmodeling:devel Aug 10, 2021

njzjz deleted the dep branch August 10, 2021 01:03

njzjz mentioned this pull request Aug 22, 2021

give a clear message if model.get_ntypes()<data.get_ntypes() #1016

Merged

gzq942560379 pushed a commit to HPC-AI-Team/deepmd-kit that referenced this pull request Sep 2, 2021

clean library and header dependencies (deepmodeling#930)

e01e7d7

* clean library and header dependencies Fix deepmodeling#926. * fix typo in rocm * set INSTALL_RPATH for libraries

njzjz pushed a commit to njzjz/deepmd-kit that referenced this pull request Sep 21, 2023

fix: skip cell_type if from_poscar is True in init_bulk (deepmodeling…

1f4f1e3

…#930) - skip cell_type if from_poscar is True - use from_poscar = jdata.get('from_poscar', False) to get from_poscar

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clean library and header dependencies #930

clean library and header dependencies #930

njzjz commented Aug 6, 2021

codecov-commenter commented Aug 6, 2021 •

edited

Loading

njzjz left a comment

denghuilu left a comment

njzjz commented Aug 9, 2021

denghuilu commented Aug 9, 2021

njzjz commented Aug 9, 2021

denghuilu commented Aug 9, 2021 •

edited

Loading

njzjz commented Aug 9, 2021

njzjz commented Aug 9, 2021

denghuilu commented Aug 9, 2021

denghuilu left a comment

clean library and header dependencies #930

clean library and header dependencies #930

Conversation

njzjz commented Aug 6, 2021

codecov-commenter commented Aug 6, 2021 • edited Loading

Codecov Report

njzjz left a comment

Choose a reason for hiding this comment

denghuilu left a comment

Choose a reason for hiding this comment

njzjz commented Aug 9, 2021

denghuilu commented Aug 9, 2021

njzjz commented Aug 9, 2021

denghuilu commented Aug 9, 2021 • edited Loading

njzjz commented Aug 9, 2021

njzjz commented Aug 9, 2021

denghuilu commented Aug 9, 2021

denghuilu left a comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 6, 2021 •

edited

Loading

denghuilu commented Aug 9, 2021 •

edited

Loading