Python dependency installation error

[ganghyun-y]

Warning messages:
1: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\stringi\libs\icudt69l.dat to 
C:\Users\ykh24\AppData\Local\R\win-library\4.2\stringi\libs\icudt69l.dat: Invalid argument

2: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\stringi\libs\x64\stringi.dll to C:\Users\ykh24\AppData\Local\R\win-library\4.2\stringi\libs\x64\stringi.dll: Permission denied

3: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\cli\libs\x64\cli.dll to C:\Users\ykh24\AppData\Local\R\win-library\4.2\cli\libs\x64\cli.dll: Permission denied

4: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\fs\libs\x64\fs.dll to C:\Users\ykh24\AppData\Local\R\win-library\4.2\fs\libs\x64\fs.dll: Permission denied

5: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\cachem\libs\x64\cachem.dll to C:\Users\ykh24\AppData\Local\R\win-library\4.2\cachem\libs\x64\cachem.dll: Permission denied

6: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\fastmap\libs\x64\fastmap.dll to C:\Users\ykh24\AppData\Local\R\win-library\4.2\fastmap\libs\x64\fastmap.dll: Permission denied

7: In file.copy(savedcopy, lib, recursive = TRUE) :
  problem copying C:\Users\ykh24\AppData\Local\R\win-library\4.2\00LOCK\curl\libs\x64\curl.dll to C:\Users\ykh24\AppData\Local\R\win-library\4.2\curl\libs\x64\curl.dll: Permission denied

Hi,
This warning message (above) appears after installing DO.utils.
There was nothing wrong with loading the DO.utils library.
It seemed to be going well, but there seemed to be a problem with installing the "adeft" packages, so I manually proceeded with setup.py and solved it.

The RStudio also outputs TRUE.

reticulate::configure_environment("DO.utils")
[1] TRUE

It seemed like the package problem was being solved, but... It doesn't work :(

extract_subtree(x, top_node, reload = FALSE)
List of 20
 $ python              : chr "C:/Users/ykh24/miniconda3/envs/myenv/python.exe"
 $ libpython           : chr "C:/Users/ykh24/miniconda3/envs/myenv/python39.dll"
 $ pythonhome          : chr "C:/Users/ykh24/miniconda3/envs/myenv"
 $ pythonpath          : chr "C:\\Users\\ykh24\\AppData\\Local\\R\\win-library\\4.2\\reticulate\\config;C:\\Users\\ykh24\\MINICO~1\\envs\\mye"| __truncated__
 $ prefix              : chr "C:\\Users\\ykh24\\MINICO~1\\envs\\myenv"
 $ exec_prefix         : chr "C:\\Users\\ykh24\\MINICO~1\\envs\\myenv"
 $ base_exec_prefix    : chr "C:\\Users\\ykh24\\MINICO~1\\envs\\myenv"
 $ virtualenv          : chr ""
 $ virtualenv_activate : chr ""
 $ version_string      : chr "3.9.16 (main, Jan 11 2023, 16:16:36) [MSC v.1916 64 bit (AMD64)]"
 $ version             : chr "3.9"
 $ architecture        : chr "64bit"
 $ anaconda            : logi FALSE
 $ conda               : logi FALSE
 $ numpy               :List of 2
  ..$ path   : chr "C:/Users/ykh24/miniconda3/envs/myenv/Lib/site-packages/numpy"
  ..$ version:Class 'numeric_version'  hidden list of 1
  .. ..$ : int [1:3] 1 23 5
 $ required_module     : chr "pyobo"
 $ required_module_path: chr "C:\\Users\\ykh24\\MINICO~1\\envs\\myenv\\lib\\site-packages\\pyobo\\__init__.p"
 $ available           : logi TRUE
 $ python_versions     : chr "C:/Users/ykh24/miniconda3/envs/myenv/python.exe"
 $ forced              : chr "use_python function"
- attr(*, "class")= chr "py_config"
Error: Python module pyDOID was not found.

Detected Python configuration:

[allenbaron]

Thanks for sharing this and for troubleshooting installation with me.

The first set of warning messages are an R package installation issue (see r-lib/remotes#368). It's best to install R packages in a fresh R session before any are loaded (when possible). I'm not sure if those are causing issues for you but it's a different issue than the python related error from DO.utils.

It seems that the pyDOID python package is still not available to DO.utils. Can you tell me a bit more about your Python setup? Did you have an installation of Python before installing DO.utils?

Can you share with me the results of running reticulate::py_discover_config()?

[ganghyun-y]

I installed the package in a virtual environment called myenv using miniconda.
The code below is a list of packages installed in my virtual environment (myenv).

# packages in environment at C:\Users\ykh24\miniconda3\envs\myenv:
#
# Name                    Version                   Build  Channel
adeft                     0.11.2                   pypi_0    pypi
aniso8601                 9.0.1                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
asttokens                 2.2.1                    pypi_0    pypi
attrs                     22.2.0                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
beautifulsoup4            4.11.2                   pypi_0    pypi
bioregistry               0.6.83                   pypi_0    pypi
bioversions               0.5.83                   pypi_0    pypi
blas                      1.0                         mkl
boto3                     1.26.79                  pypi_0    pypi
botocore                  1.29.79                  pypi_0    pypi
ca-certificates           2023.01.10           haa95532_0
cachier                   2.0.0                    pypi_0    pypi
certifi                   2022.12.7        py39haa95532_0
charset-normalizer        3.0.1                    pypi_0    pypi
chembl-downloader         0.4.2                    pypi_0    pypi
class-resolver            0.4.0                    pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
click-default-group       1.2.2                    pypi_0    pypi
cmake                     3.25.2                   pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
curies                    0.4.2                    pypi_0    pypi
dataclasses-json          0.5.7                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
dominate                  2.7.0                    pypi_0    pypi
drugbank-downloader       0.1.1                    pypi_0    pypi
enum34                    1.1.10                   pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fairsharing-client        0.1.0                    pypi_0    pypi
flask                     2.1.3                    pypi_0    pypi
flask-bootstrap           3.3.7.1                  pypi_0    pypi
flask-restx               1.0.6                    pypi_0    pypi
flask-wtf                 1.1.1                    pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
gilda                     0.10.3                   pypi_0    pypi
gitdb                     4.0.10                   pypi_0    pypi
gitpython                 3.1.31                   pypi_0    pypi
humanize                  4.6.0                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
ijson                     3.2.0.post0              pypi_0    pypi
importlib-metadata        6.0.0                    pypi_0    pypi
indra                     1.22.0                   pypi_0    pypi
intel-openmp              2021.4.0          haa95532_3556
ipython                   8.10.0                   pypi_0    pypi
isodate                   0.6.1                    pypi_0    pypi
itsdangerous              2.1.2                    pypi_0    pypi
jedi                      0.18.2                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
jsonschema                4.17.3                   pypi_0    pypi
lxml                      4.9.2                    pypi_0    pypi
markupsafe                2.0.1                    pypi_0    pypi
marshmallow               3.19.0                   pypi_0    pypi
marshmallow-enum          1.5.1                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mkl                       2021.4.0           haa95532_640
mkl-service               2.4.0            py39h2bbff1b_0
mkl_fft                   1.3.1            py39h277e83a_0
mkl_random                1.2.2            py39hf11a4ad_0
more-click                0.1.2                    pypi_0    pypi
more-itertools            9.0.0                    pypi_0    pypi
mpmath                    1.2.1                    pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
ndex2                     2.0.1                    pypi_0    pypi
networkx                  3.0                      pypi_0    pypi
nltk                      3.8.1                    pypi_0    pypi
numpy                     1.23.5           py39h3b20f71_0
numpy-base                1.23.5           py39h4da318b_0
objectpath                0.6.1                    pypi_0    pypi
obonet                    0.3.1                    pypi_0    pypi
openssl                   1.1.1t               h2bbff1b_0
packaging                 23.0                     pypi_0    pypi
pandas                    1.2.5                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pathtools                 0.1.2                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pip                       23.0.1                   pypi_0    pypi
portalocker               2.7.0                    pypi_0    pypi
prompt-toolkit            3.0.37                   pypi_0    pypi
protmapper                0.0.27                   pypi_0    pypi
psycopg2-binary           2.9.5                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pybiopax                  0.1.3                    pypi_0    pypi
pydantic                  1.10.5                   pypi_0    pypi
pydoid                    0.1.6                    pypi_0    pypi
pygments                  2.14.0                   pypi_0    pypi
pyobo                     0.8.12                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
pysb                      1.14.0                   pypi_0    pypi
pysolr                    3.9.0                    pypi_0    pypi
pystow                    0.5.0                    pypi_0    pypi
python                    3.9.16               h6244533_0
python-dateutil           2.8.2                    pypi_0    pypi
pytrie                    0.4.0                    pypi_0    pypi
pytz                      2022.7.1                 pypi_0    pypi
pywin32                   305                      pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
rdflib                    6.2.0                    pypi_0    pypi
regex                     2022.10.31               pypi_0    pypi
requests                  2.28.2                   pypi_0    pypi
requests-ftp              0.3.1                    pypi_0    pypi
requests-toolbelt         0.10.1                   pypi_0    pypi
s3transfer                0.6.0                    pypi_0    pypi
scikit-learn              1.2.1                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
setuptools                67.4.0                   pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1
smmap                     5.0.0                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
soupsieve                 2.4                      pypi_0    pypi
sqlite                    3.40.1               h2bbff1b_0
stack-data                0.6.2                    pypi_0    pypi
sympy                     1.11.1                   pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.9.0                    pypi_0    pypi
typing-extensions         4.5.0                    pypi_0    pypi
typing-inspect            0.8.0                    pypi_0    pypi
tzdata                    2022g                h04d1e81_0
umls-downloader           0.1.1                    pypi_0    pypi
unidecode                 1.3.6                    pypi_0    pypi
urllib3                   1.26.14                  pypi_0    pypi
vc                        14.2                 h21ff451_1
visitor                   0.1.3                    pypi_0    pypi
vs2015_runtime            14.27.29016          h5e58377_2
watchdog                  2.3.0                    pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
werkzeug                  2.1.2                    pypi_0    pypi
wheel                     0.38.4           py39haa95532_0
wincertstore              0.2              py39haa95532_2
wtforms                   3.0.1                    pypi_0    pypi
zenodo-client             0.3.0                    pypi_0    pypi
zipp                      3.15.0                   pypi_0    pypi

I installed Python before, and the image below shows python environments that can be linked to R.

> reticulate::py_discover_config()
python:         C:/Users/ykh24/miniconda3/envs/myenv/python.exe
libpython:      C:/Users/ykh24/miniconda3/envs/myenv/python39.dll
pythonhome:     C:/Users/ykh24/miniconda3/envs/myenv
version:        3.9.16 (main, Jan 11 2023, 16:16:36) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/ykh24/miniconda3/envs/myenv/Lib/site-packages/numpy
numpy_version:  1.23.5

[allenbaron]

Hmmm... that is strange. It seems like you should be good to go. I have noticed in the past that reticulate will sometimes activate the wrong environment and that may be what it's doing here. The only way I've found to get reticulate to use a consistent environment is to ensure the environment variable RETICULATE_PYTHON is set. This setting seems to override everything else.

Can you provide the output of reticulate::py_config()? Can you also share the result of reticulate::py_list_packages()? I know these seem redundant with what you've already sent but this will tell you which python environment is actually being activated (as opposed to those that are available) and whether it is "forced" by the RETICULATE_PYTHON environment variable and then list the of packages included in the loaded environment.

If you restart R to get a fresh session, then set the RETICULATE_PYTHON variable with Sys.setenv(RETICULATE_PYTHON = "C:/Users/ykh24/miniconda3/envs/myenv/python.exe") and then run reticulate::py_config() do you get a different result?

[allenbaron]

Looking at the reticulate issue tracker there are quite a few issues surrounding installation and some of these have been resolved in later releases than the 1.23 release used by DO.utils, so I'm going to bump the version of reticulate used by DO.utils to the latest release (v1.28) in the hope that it will improve things.

In general, I've been keeping an eye on tensorflow for how to ensure proper install. I noticed that they are still using an approach for installation that relies on a custom install function (install_tensorflow()). I was previously under the impression that they had moved to the automated python installation provided using the package DESCRIPTION file but that does not appear to be the case. I may need to add a custom install function to handle python dependencies as well.

Sorry you're having so much trouble with this.

[allenbaron]

This appears to be a problem a number of R packages that depend on python packages are having (rstudio/reticulate#997).

I need to check whether updating the reticulate version has fixed the problem but if it didn't I discovered a pretty simple workaround. Anyone who installs miniconda when prompted should have an r-reticulate conda environment automatically created (running reticulate::py_config() can confirm this; the python path will include 'envs/r-reticulate' in it). In this case, all that's needed is to install the needed python packages manually into the environment by running reticulate::py_install("pyDOID", pip = TRUE).

[ganghyun-y]

I tried to reticulate::py_install("pyDOID", pip = TRUE) , but it still can't find the pyDOID module.

Does it mean that DO.utils needs to be reinstalled in a virtual environment called r-retriculate?

[allenbaron]

DO.utils doesn't need to be in any specific environment but the python package(s) it sometimes uses do. I tested this recently on a Mac and it automatically created the r-reticulate virtual environment but didn't complete the install of pyDOID. It did work for me to simply tell it to install. In order to install more packages, you do have to have a virtual environment created and activated but it looks like you already do. Yours is just named myenv.

Did the install of pyDOID seem to work when you ran reticulate::py_install("pyDOID", pip = TRUE)? Or did it immediately produce error messages?

Since you had issues with the initial install and had all those 'problem copying...' errors it's possible that you have some corrupted files that are causing issues. It might be best to try re-installing from scratch. I'll try installing DO.utils on a PC later in the week and let you know what I get.

[ganghyun-y]

yes, I did.

I get a message that it's already installed, but it still doesn't work.
Next time, I will completely delete the package and proceed with the re-installation.

Thank you for your help. It's hard work, so I hope it's solved well.

> reticulate::py_config()
python:         C:/Users/ykh24/miniconda3/envs/r-reticulate/python.exe
libpython:      C:/Users/ykh24/miniconda3/envs/r-reticulate/python39.dll
pythonhome:     C:/Users/ykh24/miniconda3/envs/r-reticulate
version:        3.9.16 (main, Jan 11 2023, 16:16:36) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/ykh24/miniconda3/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.23.5
NOTE: Python version was forced by use_python function

> reticulate::py_install("pyDOID", pip = TRUE)

C:\Users\ykh24\Desktop\project\test_conda>CALL "C:\Users\ykh24\miniconda3\condabin\activate.bat" "C:\Users\ykh24\miniconda3\envs\r-reticulate" 

C:\Users\ykh24\Desktop\project\test_conda>conda.bat activate "C:\Users\ykh24\miniconda3\envs\r-reticulate" 

(r-reticulate) C:\Users\ykh24\Desktop\project\test_conda>"C:/Users/ykh24/miniconda3/envs/r-reticulate/python.exe" -m pip install --upgrade --no-user "pyDOID" 
Requirement already satisfied: pyDOID in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (0.1.6)
Requirement already satisfied: GitPython in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pyDOID) (3.1.31)
Requirement already satisfied: tqdm in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pyDOID) (4.64.1)
Requirement already satisfied: rdflib in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pyDOID) (6.2.0)
Requirement already satisfied: pandas in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pyDOID) (1.2.5)
Requirement already satisfied: gitdb<5,>=4.0.1 in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from GitPython->pyDOID) (4.0.10)
Requirement already satisfied: numpy>=1.16.5 in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pandas->pyDOID) (1.23.5)
Requirement already satisfied: python-dateutil>=2.7.3 in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pandas->pyDOID) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from pandas->pyDOID) (2022.7.1)
Requirement already satisfied: pyparsing in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from rdflib->pyDOID) (3.0.9)
Requirement already satisfied: isodate in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from rdflib->pyDOID) (0.6.1)
Requirement already satisfied: setuptools in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from rdflib->pyDOID) (67.4.0)
Requirement already satisfied: colorama in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from tqdm->pyDOID) (0.4.6)
Requirement already satisfied: smmap<6,>=3.0.1 in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from gitdb<5,>=4.0.1->GitPython->pyDOID) (5.0.0)
Requirement already satisfied: six>=1.5 in c:\users\ykh24\miniconda3\envs\r-reticulate\lib\site-packages (from python-dateutil>=2.7.3->pandas->pyDOID) (1.16.0)