Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError: No module named paddle.trainer.PyDataProvider2 #32

Closed
ershersh opened this issue Sep 3, 2016 · 10 comments
Closed

ImportError: No module named paddle.trainer.PyDataProvider2 #32

ershersh opened this issue Sep 3, 2016 · 10 comments
Labels

Comments

@ershersh
Copy link

ershersh commented Sep 3, 2016

First time to use paddle and not very familiar, so I try to do as the quickstart doc step by step.
http://www.paddlepaddle.org/doc/demo/quick_start/index_en.html
http://www.paddlepaddle.org/doc_cn/index.html
Environment: ubuntu 12.04 and install cpu-deb.
I have finished data preparation step and when i try to use dataprovider_bow.py script, it throw the error as my title,here is the log:

cloud@cloud-virtual-machine:~/paddle-git/demo/quick_start$ python dataprovider_bow.py
Traceback (most recent call last):
File "dataprovider_bow.py", line 15, in
from paddle.trainer.PyDataProvider2 import *
ImportError: No module named paddle.trainer.PyDataProvider2

What's the problem and how can i get that package imported correctly?? And i doubt if i install paddlepaddle correctly, how can i conform that??thx~

@gangliao
Copy link
Contributor

gangliao commented Sep 3, 2016

Hi, @ershersh

http://www.paddlepaddle.org/doc_cn/demo/quick_start/index.html
Most of python snippets in this page are to describe the details of our system how it works.
All python files in quick_start, including dataprovider_bow.py already in our script.
To make demo/quick_start runs, just typing following commands:

  1. ./preprocess.sh
  2. ./train.sh
    you can open and modify train.sh, choose different models to train data.
  3. ./predict.sh

@stoneyang
Copy link
Contributor

stoneyang commented Sep 3, 2016

@ershersh

It is an import issue of a python librarary named paddle.trainer.PyDataProvider2.
How did you build and install PaddlePaddle? If you didn't install it and python can't find a clue from the import commands in the script. As a fix, you may choose from the following alternatives:

  1. Install PaddlePaddle: after successfully executed make and install it using make install, sudo may be necessary considering your local machines or consult your system administrator to properly install it.
  2. Environment ablation: just add the directory of PaddlePaddle's python utilities before you could run any py script in PaddlePaddle, or simply copy the following line to your preprocess.sh, just before the lines that call any PaddlePaddle's py script.
export PYTHONPATH=/path/to/Paddle/python:$PYTHONPATH

P.S.: The link of PaddlePaddle's quick start in Chinese you provided in the original post is wrong and maybe you could fix it?

@ershersh
Copy link
Author

ershersh commented Sep 3, 2016

First i have fix the link and add the English website. ^_^
@gangliao ./preprocess.sh run ok. and next I try this:
cloud@cloud-virtual-machine:~/paddle-git/demo/quick_start$ ./train.sh
/usr/bin/../opt/paddle/bin/paddle_trainer: error while loading shared libraries: libprotobuf.so.8: cannot open shared object file: No such file or directory
it shows that error about that .so library...

@stoneyang I use vmware11+ubuntu12.04 and install paddlepaddle by deb
root@cloud-virtual-machine:/home/cloud# dpkg -i paddle-0.8.0b0-Linux-cpu.deb
Selecting previously unselected package paddle.
(正在读取数据库 ... 系统当前共安装有 147560 个文件和目录。)
正在解压缩 paddle (从 paddle-0.8.0b0-Linux-cpu.deb) ...
dpkg:依赖关系问题使得 paddle 的配置工作不能继续:
paddle 依赖于 libatlas3-base;然而:
未安装软件包 libatlas3-base。
paddle 依赖于 libgflags2;然而:
未安装软件包 libgflags2。
paddle 依赖于 libgoogle-glog0;然而:
未安装软件包 libgoogle-glog0。
paddle 依赖于 libprotobuf8;然而:
未安装软件包 libprotobuf8。
paddle 依赖于 python-numpy;然而:
未安装软件包 python-numpy。
paddle 依赖于 python-pip;然而:
未安装软件包 python-pip。
paddle 依赖于 python-pip-whl;然而:
未安装软件包 python-pip-whl。
dpkg:处理 paddle (--install)时出错:
依赖关系问题 - 仍未被配置
在处理时有错误发生:
paddle
root@cloud-virtual-machine:/home/cloud# apt-get install -f
正在读取软件包列表... 完成
正在分析软件包的依赖关系树
正在读取状态信息... 完成
正在更正依赖关系... 完成
下列软件包将被【卸载】:
paddle
升级了 0 个软件包,新安装了 0 个软件包,要卸载 1 个软件包,有 686 个软件包未被升级。
有 1 个软件包没有被完全安装或卸载。
解压缩后将会空出 320 MB 的空间。
您希望继续执行吗?[Y/n]n
中止执行。

After command dpkg -i paddle-0.8.0b0-Linux-cpu.deb ,it shows that i still have error--some dependences not install, the guide doc tell me then apt-get install -f will continue install paddle, and install dependences, but the fact is it try to remove paddle so i choose canceled.Maybe there's something wrong here??

So I may try @stoneyang your ways to install later , see whether it can fix and update the result in time.thx all

@lqniunjunlper
Copy link

meybe you should enter the paddle/python dir and run "python setup.py install"

@ershersh
Copy link
Author

ershersh commented Sep 3, 2016

@NIULQfromNJU i also try your method and did ont work
root@cloud-virtual-machine:/home/cloud/paddle-git/python# python setup.py.in install
Traceback (most recent call last):
File "setup.py.in", line 1, in
from setuptools import setup
ImportError: No module named setuptools

@stoneyang
Copy link
Contributor

stoneyang commented Sep 3, 2016

@ershersh

From your rely, you installed PaddlePaddle on your virtual machine using the shipped debian package according to the official instructions. Since you try to install CPU version of PaddlePaddle, maybe you should pay attention to the Note about the instruction set of your CPU listed at the bottom of the instructions. And make sure you have the proper access permission during installation.

As a reminder, the two solutions I provided is about building PaddlePaddle from the source code on your local machine. You can try it out if needed.

@reyoung
Copy link
Collaborator

reyoung commented Sep 5, 2016

@ershersh You should use apt-get update before apt-get install -f to make apt can find dependencies.

@reyoung
Copy link
Collaborator

reyoung commented Sep 5, 2016

@ershersh Then you can follow the install instructions.

@ershersh
Copy link
Author

ershersh commented Sep 6, 2016

@reyoung I also try your method but it comes to the same result: after"apt-get install -f",it tells me to delete paddle. Now I use my spare time to install paddle by docker
thanks all the same!

@backyes
Copy link
Contributor

backyes commented Oct 11, 2016

more than 1 months passed, so close this issue.

@backyes backyes closed this as completed Oct 11, 2016
sabreshao pushed a commit to sabreshao/Paddle that referenced this issue Oct 10, 2018
Fix functions in cuda_device_function.h
qingqing01 pushed a commit to qingqing01/Paddle that referenced this issue Apr 30, 2020
CtfGo pushed a commit to CtfGo/Paddle that referenced this issue Aug 25, 2021
* cmake for llvm

* fix compilation error

* part 2 of primitive ir emitter

* part 2 of primitive ir emitter

* generate llvm ir for add instruction

* fix compilation error

* add primitive instruction(For) and ElementTypeToLLVMType

* support different dtypes for Add and Mul

* merge develop and fix conflict

* polish the code

* polish code
KPatr1ck pushed a commit to KPatr1ck/Paddle that referenced this issue Sep 10, 2021
XiaoguangHu01 pushed a commit that referenced this issue Sep 18, 2021
* 1. add interface for fft;
2. add data type predicate;
3. fix paddle.roll.

* add fft c2c cufft kernel

* implement argument checking & op calling parts for fft_c2c and fftn_c2c

* add operator and opmaker definitions

* only register float and double for cpu.

* add common code for implementing FFT, add pocketfft as a dependency

* add fft c2c cufft kernel function

* fix bugs in python interface

* add support for c2r, r2c operators, op makers, kernels and kernel functors.

* test and fix bugs

* 1. fft_c2c function: add support for onesided=False;
2. add complex<float>, complex<double> support for concat and flip.

* 1. fft: fix python api bugs;
2. shape_op: add support for complex data types.

* fft c2c cufft kernel done with complie and link

* fix shape_op, add mkl placeholder

* remove mkl

* complete fft c2c in gpu

* 1. implement mkl-based fft, FFTC2CFunctor and common function exec_fft;
2. change the design, add input and output typename as template parameter for all FFTFunctors, update pocketfft-based implementation.

* complete fft c2c on gpu in ND

* complete fft c2c on gpu in ND

* complete fft c2c backward in ND

* fix MKL-based implementation

* Add frame op and CPU/GPU kernels.

* Add frame op forward unittest.

* Add frame op forward unittest.

* Remove axis parameter in FrameFunctor.

* Add frame op grad CPU/GPU kernels and unittest.

* Add frame op grad CPU/GPU kernels and unittest.

* Update doc string.

* Update after review and remove librosa requirement in unittest.

* Update grad kernel.

* add fft_c2r op

* Remove data allocation in TransCompute function.

* add fft r2c onesided with cpu(pocketfft/mkl) and gpu

* last fft c2r functor

* fix C2R and R2C for cufft, becase the direction is not an option in these cases.

* add fft r2c onesided with cpu(pocketfft/mkl) and gpu

* fix bugs in python APIs

* fix fft_c2r grad kernal

* fix bugs in python APIs

* add cuda fft c2r grad kernal functor

* clean code

* fix fft_c2r python API

* fill fft r2c result with conjugate symmetry (#19)

fill fft r2c result with conjugate symmetry

* add placeholder for unittests (#24)

* simple parameterize test function by auto generate test case from parm list (#25)

* miscellaneous fixes for python APIs (#26)

* add placeholder for unittests

* resize fft inputs before computation is n or s is provided.

* add complex kernels for pad and pad_grad

* simplify argument checking.

* add type promotion

* add int to float or complex promotion

* fix output data type for static mode

* fix fft's input dtype dispatch, import fft to paddle

* fix typos in axes checking (#27)

* fix typos in axes checking

* fix argument checking (#28)

* fix argument checking

* Add C2R Python layer normal and abnormal use cases (#29)

* documents and single case

* test c2r case

* New C2R Python layer normal and exception use cases

* complete rfft,rfft2,rfftn,ihfft,ihfft2,ihfftn unittest and doc string (#30)

* Documentation of the common interfaces of c2r and c2c (#31)

* Documentation of the common interfaces of c2r and c2c

* clean c++ code  (#32)

* clean code

* Add numpy-based implementation of spectral ops (#33)

* add numpy reference implementation of spectral ops

* Add fft_c2r numpy based implementation for unittest. (#34)

* add fft_c2r numpy implementation

* Add deframe op and stft/istft api. (#23)

* Add frame api

* Add deframe op and kernels.

* Add stft and istft apis.

* Add deframe api. Update stft and istft apis.

* Fix bug in frame_from_librosa function when input dims >= 3

* Rename deframe to overlap_add.

* Update istft.

* Update after code review.

* Add overlap_add op and stft/istft api unittest (#35)

* Add overlap_add op unittest.

* Register complex kernels of squeeze/unsquuze op.

* Add stft/istft api unittest.

* Add unittest for fft helper functions (#36)

* add unittests for fft helper functions. add complex kernel for roll op.

* complete static graph unittest for all public api (#37)

* Unittest of op with FFT C2C, C2R and r2c added (#38)

* documents and single case

* test c2r case

* New C2R Python layer normal and exception use cases

* Documentation of the common interfaces of c2r and c2c

* Unittest of op with FFT C2C, C2R and r2c added

Co-authored-by: lijiaqi <lijiaqi0612@163.com>

* add fft related options to CMakeLists.txt

* fix typos and clean code (#39)

* fix invisible character in mkl branch and fix error in error message

* clean code: remove docstring from unittest for signal.py.

* always convert numpy array to paddle.Tensor to avoid comparing numpy dtype with paddle dtype. (#40)

* always convert numpy array to paddle.Tensor to avoid comparing numpy dtype with paddle dtype.

* fix CI Errors: numpy dtype comparison, thrust when cuda is not available (#41)

1. always convert numpy array to paddle.Tensor to avoid comparing numpy dtype with paddle dtype.
2. promote floating point tensor to complex tensor ior fft_c2c and fft_c2r;
3. fix unittest to catch UnImplementedError and RuntimeError;
4. fix compile error by avoid using thrust when cuda is not available.
5.  fix sample code, use paddle.fft instead of paddle.tensor.fft

* remove inclusion of thrust, add __all__ list for fft (#42)

* Add api doc and update unittest. (#43)

* Add doc strings.
* Update overlap_add op unittest

* fix MKL-based FFT implementation (#44)

* fix MKL-based FFT implementation, MKL CDFT's FORWARD DOMAIN is always REAL for R2C and C2R

* remove code for debug (#45)

* use dynload for cufft (#46)

* use std::ptrdiff_t as datatype of stride (instead of int64_t) to avoid argument mismatch on some platforms.

* add complex support for fill_zeros_like

* use dynload for cufft

* Update doc and unittest. (#47)

* Add doc of frame op and overlap_add op.

* Update unittest.

* use dynload for cufft (#48)

1. use dynload for cufft
2. fix unittest;
3. temporarily disable Rocm.

* fix conflicts and merge upstream (#49)

fix conflicts and merge upstream

* fix compile error: only link dyload_cuda when cuda is available (#50)

* fix compile error: only link dyload_cuda when cuda is available

* fix dynload for cufft on windows (#51)

1. fix dynload for cufft on windows;
2. fix unittests.

* add NOMINMAX to compile on windows (#52)

 add NOMINMAX to compile on windows

* explicitly specify capture mode for lambdas (#55)

 explicitly specify capture mode for lambdas

* fix fft sample (#53)

* fix fft sample

* update scipy and numpy version for unittests of fft (#56)

update scipy and numpy version for unittests of fft

* Add static graph unittests of frame and overlap_add api. (#57)

* Remove cache of cuFFT & Disable ONEMKL (#59)

1. replace numpy.fft with scipy.fft as numpy<1.20 not support ortho norm
2. remove cache of cufft plans;
3. enhance error checking.
4. default WITH_ONEMKL to OFF

Co-authored-by: jeff41404 <jeff41404@gmail.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming9.bjyz.baidu.com>
Co-authored-by: KP <109694228@qq.com>
Co-authored-by: lijiaqi <lijiaqi0612@163.com>
Co-authored-by: Xiaoxu Chen <chenxx_id@163.com>
Co-authored-by: lijiaqi0612 <33169170+lijiaqi0612@users.noreply.github.com>
AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this issue Sep 29, 2021
* 1. add interface for fft;
2. add data type predicate;
3. fix paddle.roll.

* add fft c2c cufft kernel

* implement argument checking & op calling parts for fft_c2c and fftn_c2c

* add operator and opmaker definitions

* only register float and double for cpu.

* add common code for implementing FFT, add pocketfft as a dependency

* add fft c2c cufft kernel function

* fix bugs in python interface

* add support for c2r, r2c operators, op makers, kernels and kernel functors.

* test and fix bugs

* 1. fft_c2c function: add support for onesided=False;
2. add complex<float>, complex<double> support for concat and flip.

* 1. fft: fix python api bugs;
2. shape_op: add support for complex data types.

* fft c2c cufft kernel done with complie and link

* fix shape_op, add mkl placeholder

* remove mkl

* complete fft c2c in gpu

* 1. implement mkl-based fft, FFTC2CFunctor and common function exec_fft;
2. change the design, add input and output typename as template parameter for all FFTFunctors, update pocketfft-based implementation.

* complete fft c2c on gpu in ND

* complete fft c2c on gpu in ND

* complete fft c2c backward in ND

* fix MKL-based implementation

* Add frame op and CPU/GPU kernels.

* Add frame op forward unittest.

* Add frame op forward unittest.

* Remove axis parameter in FrameFunctor.

* Add frame op grad CPU/GPU kernels and unittest.

* Add frame op grad CPU/GPU kernels and unittest.

* Update doc string.

* Update after review and remove librosa requirement in unittest.

* Update grad kernel.

* add fft_c2r op

* Remove data allocation in TransCompute function.

* add fft r2c onesided with cpu(pocketfft/mkl) and gpu

* last fft c2r functor

* fix C2R and R2C for cufft, becase the direction is not an option in these cases.

* add fft r2c onesided with cpu(pocketfft/mkl) and gpu

* fix bugs in python APIs

* fix fft_c2r grad kernal

* fix bugs in python APIs

* add cuda fft c2r grad kernal functor

* clean code

* fix fft_c2r python API

* fill fft r2c result with conjugate symmetry (#19)

fill fft r2c result with conjugate symmetry

* add placeholder for unittests (#24)

* simple parameterize test function by auto generate test case from parm list (#25)

* miscellaneous fixes for python APIs (#26)

* add placeholder for unittests

* resize fft inputs before computation is n or s is provided.

* add complex kernels for pad and pad_grad

* simplify argument checking.

* add type promotion

* add int to float or complex promotion

* fix output data type for static mode

* fix fft's input dtype dispatch, import fft to paddle

* fix typos in axes checking (#27)

* fix typos in axes checking

* fix argument checking (#28)

* fix argument checking

* Add C2R Python layer normal and abnormal use cases (#29)

* documents and single case

* test c2r case

* New C2R Python layer normal and exception use cases

* complete rfft,rfft2,rfftn,ihfft,ihfft2,ihfftn unittest and doc string (PaddlePaddle#30)

* Documentation of the common interfaces of c2r and c2c (PaddlePaddle#31)

* Documentation of the common interfaces of c2r and c2c

* clean c++ code  (PaddlePaddle#32)

* clean code

* Add numpy-based implementation of spectral ops (PaddlePaddle#33)

* add numpy reference implementation of spectral ops

* Add fft_c2r numpy based implementation for unittest. (PaddlePaddle#34)

* add fft_c2r numpy implementation

* Add deframe op and stft/istft api. (#23)

* Add frame api

* Add deframe op and kernels.

* Add stft and istft apis.

* Add deframe api. Update stft and istft apis.

* Fix bug in frame_from_librosa function when input dims >= 3

* Rename deframe to overlap_add.

* Update istft.

* Update after code review.

* Add overlap_add op and stft/istft api unittest (PaddlePaddle#35)

* Add overlap_add op unittest.

* Register complex kernels of squeeze/unsquuze op.

* Add stft/istft api unittest.

* Add unittest for fft helper functions (PaddlePaddle#36)

* add unittests for fft helper functions. add complex kernel for roll op.

* complete static graph unittest for all public api (PaddlePaddle#37)

* Unittest of op with FFT C2C, C2R and r2c added (PaddlePaddle#38)

* documents and single case

* test c2r case

* New C2R Python layer normal and exception use cases

* Documentation of the common interfaces of c2r and c2c

* Unittest of op with FFT C2C, C2R and r2c added

Co-authored-by: lijiaqi <lijiaqi0612@163.com>

* add fft related options to CMakeLists.txt

* fix typos and clean code (PaddlePaddle#39)

* fix invisible character in mkl branch and fix error in error message

* clean code: remove docstring from unittest for signal.py.

* always convert numpy array to paddle.Tensor to avoid comparing numpy dtype with paddle dtype. (PaddlePaddle#40)

* always convert numpy array to paddle.Tensor to avoid comparing numpy dtype with paddle dtype.

* fix CI Errors: numpy dtype comparison, thrust when cuda is not available (PaddlePaddle#41)

1. always convert numpy array to paddle.Tensor to avoid comparing numpy dtype with paddle dtype.
2. promote floating point tensor to complex tensor ior fft_c2c and fft_c2r;
3. fix unittest to catch UnImplementedError and RuntimeError;
4. fix compile error by avoid using thrust when cuda is not available.
5.  fix sample code, use paddle.fft instead of paddle.tensor.fft

* remove inclusion of thrust, add __all__ list for fft (PaddlePaddle#42)

* Add api doc and update unittest. (PaddlePaddle#43)

* Add doc strings.
* Update overlap_add op unittest

* fix MKL-based FFT implementation (PaddlePaddle#44)

* fix MKL-based FFT implementation, MKL CDFT's FORWARD DOMAIN is always REAL for R2C and C2R

* remove code for debug (PaddlePaddle#45)

* use dynload for cufft (PaddlePaddle#46)

* use std::ptrdiff_t as datatype of stride (instead of int64_t) to avoid argument mismatch on some platforms.

* add complex support for fill_zeros_like

* use dynload for cufft

* Update doc and unittest. (PaddlePaddle#47)

* Add doc of frame op and overlap_add op.

* Update unittest.

* use dynload for cufft (PaddlePaddle#48)

1. use dynload for cufft
2. fix unittest;
3. temporarily disable Rocm.

* fix conflicts and merge upstream (PaddlePaddle#49)

fix conflicts and merge upstream

* fix compile error: only link dyload_cuda when cuda is available (PaddlePaddle#50)

* fix compile error: only link dyload_cuda when cuda is available

* fix dynload for cufft on windows (PaddlePaddle#51)

1. fix dynload for cufft on windows;
2. fix unittests.

* add NOMINMAX to compile on windows (PaddlePaddle#52)

 add NOMINMAX to compile on windows

* explicitly specify capture mode for lambdas (PaddlePaddle#55)

 explicitly specify capture mode for lambdas

* fix fft sample (PaddlePaddle#53)

* fix fft sample

* update scipy and numpy version for unittests of fft (PaddlePaddle#56)

update scipy and numpy version for unittests of fft

* Add static graph unittests of frame and overlap_add api. (PaddlePaddle#57)

* Remove cache of cuFFT & Disable ONEMKL (PaddlePaddle#59)

1. replace numpy.fft with scipy.fft as numpy<1.20 not support ortho norm
2. remove cache of cufft plans;
3. enhance error checking.
4. default WITH_ONEMKL to OFF

Co-authored-by: jeff41404 <jeff41404@gmail.com>
Co-authored-by: root <root@bjyz-sys-gpu-kongming9.bjyz.baidu.com>
Co-authored-by: KP <109694228@qq.com>
Co-authored-by: lijiaqi <lijiaqi0612@163.com>
Co-authored-by: Xiaoxu Chen <chenxx_id@163.com>
Co-authored-by: lijiaqi0612 <33169170+lijiaqi0612@users.noreply.github.com>
thisjiang pushed a commit to thisjiang/Paddle that referenced this issue Oct 28, 2021
danleifeng added a commit to danleifeng/Paddle that referenced this issue Jun 20, 2022
* add adam/sharedadam optimzier for gpups;edit optimizer struct;test=develop

* remove useless code;test=develop

* remove useless code;test=develop

* remove useless code;test=develop

* remove useless code;test=develop

* remove useless code;test=develop
AnnaTrainingG pushed a commit to AnnaTrainingG/Paddle that referenced this issue Sep 19, 2022
jack603047588 referenced this issue in jack603047588/Paddle Nov 9, 2022
jack603047588 referenced this issue in jack603047588/Paddle Nov 9, 2022
slotpool  optimize ,  fix archive dio read and write,  add gperftools signal, add tensor resize by dim
xiaoguoguo626807 pushed a commit to xiaoguoguo626807/Paddle that referenced this issue Jan 5, 2023
add first-order autodiff composite logic for static graph
qizhaoaoe pushed a commit to qizhaoaoe/Paddle that referenced this issue Mar 3, 2023
zyfncg pushed a commit to zyfncg/Paddle that referenced this issue Sep 20, 2023
Upgrade the subgraph matching algorithm to PatternGraphMatchV2
zmxdream pushed a commit to zmxdream/Paddle that referenced this issue Jan 9, 2024
add the support of op partial_concat/grad
feifei-111 added a commit to feifei-111/Paddle that referenced this issue Mar 7, 2024
wwbitejotunn pushed a commit to wwbitejotunn/Paddle that referenced this issue Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants