Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

We need Ctranslate2 plz #1239

Open
Furtherking opened this issue Aug 28, 2024 · 8 comments · May be fixed by #1253
Open

We need Ctranslate2 plz #1239

Furtherking opened this issue Aug 28, 2024 · 8 comments · May be fixed by #1253
Labels
packages

Comments

@Furtherking
Copy link

We need Ctranslate2 plz
@mhsmith
Copy link
Member

mhsmith commented Sep 12, 2024

This package contains native components, so it would have to be built into an Android wheel file. If you'd like to try doing this yourself, follow the instructions here. And if you're successful, please make a pull request so we can add the package to the public repository.

If anyone else wants this package too, let us know by clicking the thumbs-up button above.

@aryan1165
Copy link

aryan1165 commented Sep 24, 2024 •
edited
Loading

@mhsmith I was trying to build it using mentioned instructions.

But when using ./build_wheel.py command i am getting following error :-

build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64
build-wheel: Using cached repository
build-wheel: tar -C /tmp/build-wheel-cgw4esx_ -xf ctranslate2-v4.1.0.tar.gz
build-wheel: mv /tmp/build-wheel-cgw4esx_ /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/src
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/include
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib
build-wheel: unzip -q -d /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy /home/aryan/chaquopy/maven/com/chaquo/python/target/3.11.6-0/target-3.11.6-0-x86_64.zip include/* jniLibs/*
build-wheel: mv /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/jniLibs/x86_64/* /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib
build-wheel: rm -r /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/jniLibs
build-wheel: ln -s libpython3.11.so /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/libpython3..so
build-wheel: ln -s libcrypto_chaquopy.so /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/libcrypto.so
build-wheel: ln -s libssl_chaquopy.so /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/libssl.so
build-wheel: ln -s libsqlite3_chaquopy.so /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/libsqlite3.so
build-wheel: /home/aryan/chaquopy/server/pypi/build/_bootstrap/3.11/bin/pip --version
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/env
build-wheel: python3.11 -m venv --without-pip /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/env
build-wheel: /home/aryan/chaquopy/server/pypi/build/_bootstrap/3.11/bin/pip --python /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/env/bin/python install wheel setuptools
Collecting wheel
  Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/1b/d1/9babe2ccaecff775992753d8686970b1e2755d21c8a63be73aba7a4e7d77/wheel-0.44.0-py3-none-any.whl.metadata
  Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
Collecting setuptools
  Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/ff/ae/f19306b5a221f6a436d8f2238d5b80925004093fa3edea59835b514d9057/setuptools-75.1.0-py3-none-any.whl.metadata
  Using cached setuptools-75.1.0-py3-none-any.whl.metadata (6.9 kB)
Using cached wheel-0.44.0-py3-none-any.whl (67 kB)
Using cached setuptools-75.1.0-py3-none-any.whl (1.2 MB)
Installing collected packages: wheel, setuptools
Successfully installed setuptools-75.1.0 wheel-0.44.0
build-wheel: abi=x86_64; api_level=21; prefix=/home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy; . /home/aryan/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/wrappers
No parent package detected, impossible to derive `name`
running egg_info
creating src/UNKNOWN.egg-info
writing src/UNKNOWN.egg-info/PKG-INFO
writing dependency_links to src/UNKNOWN.egg-info/dependency_links.txt
writing top-level names to src/UNKNOWN.egg-info/top_level.txt
writing manifest file 'src/UNKNOWN.egg-info/SOURCES.txt'
reading manifest file 'src/UNKNOWN.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'src/UNKNOWN.egg-info/SOURCES.txt'
build-wheel: abi=x86_64; api_level=21; prefix=/home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy; . /home/aryan/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/libpthread.a
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/requirements/chaquopy/lib/librt.a
build-wheel: cd /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/src
build-wheel: Building with PEP 517
No parent package detected, impossible to derive `name`
running bdist_wheel
running build
running build_py
installing to build/bdist.linux_x86_64/wheel
running install
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

running install_egg_info
running egg_info
writing src/UNKNOWN.egg-info/PKG-INFO
writing dependency_links to src/UNKNOWN.egg-info/dependency_links.txt
writing top-level names to src/UNKNOWN.egg-info/top_level.txt
reading manifest file 'src/UNKNOWN.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'src/UNKNOWN.egg-info/SOURCES.txt'
Copying src/UNKNOWN.egg-info to build/bdist.linux_x86_64/wheel/./UNKNOWN-0.0.0-py3.11.egg-info
running install_scripts
creating build/bdist.linux_x86_64/wheel/UNKNOWN-0.0.0.dist-info/WHEEL
creating '/home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/src/dist/.tmp-9ced2p_h/UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux_x86_64/wheel' to it
adding 'UNKNOWN-0.0.0.dist-info/LICENSE'
adding 'UNKNOWN-0.0.0.dist-info/METADATA'
adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
adding 'UNKNOWN-0.0.0.dist-info/RECORD'
removing build/bdist.linux_x86_64/wheel
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/fix_wheel
build-wheel: unzip -d /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/fix_wheel -q /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/src/dist/UNKNOWN-0.0.0-py3-none-any.whl
build-wheel: Error: /home/aryan/chaquopy/server/pypi/packages/ctranslate2/build/4.1.0/cp311-cp311-android_21_x86_64/fix_wheel/ctranslate2-4.1.0.dist-info does not exist

This is my meta.yaml file :-

  name: "ctranslate2"
  version: "4.1.0"  # The version of the package

source:
  git_url: https://github.com/OpenNMT/CTranslate2.git  # Use GitHub as the source
  git_rev: v4.1.0  # Specify the tag or commit

build:
  number: 0  # Build number

requirements:
  host:
    - python  # Python for host

Can you please guide me about what steps should i follow?

@mhsmith
Copy link
Member

mhsmith commented Sep 25, 2024

This package doesn't provide an sdist on PyPI, and the top level of its Git repository is not a valid Python source tree either. Instead, you'll have to make a local clone of the repository, and write your meta.yaml file as follows:

source:
  path: /path/to/CTranslate2/python

Based on this package's build instructions, you may also need to compile the underlying C++ library with a separate recipe, like the chaquopy-libzmq and pyzmq recipes that are mentioned in the README.

@aryan1165
Copy link

Hi @mhsmith , I tried to build c++ recipe first, naming it ctranslate2-lib

Here are its meta.yaml and build.sh files :-


# Set up the build directory
mkdir -p build
cd build

cmake .. -DCMAKE_INSTALL_PREFIX=$CHAQUOPY_LIB/ctranslate2-lib \
         -DCMAKE_LIBRARY_PATH=/home/aryan/llvm-project/openmp/runtime/src/libiomp5.so

# Compile the C++ library
make -j4
sudo make install
sudo ldconfig

package:
  name: "ctranslate2-lib"
  version: "4.1.0"

source:
  git_url: https://github.com/OpenNMT/CTranslate2.git
  git_rev: v4.1.0  # Use the correct version tag

build:
  number: 0

requirements:
  build:
    - cmake 3.30.2

Now it is generating .whl

./build-wheel.py --python 3.11 --abi arm64-v8a ctranslate2-lib
build-wheel: cd /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0
build-wheel: rm -rf /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a
build-wheel: Using cached repository
build-wheel: tar -C /tmp/build-wheel-771cjjy1 -xf ctranslate2-lib-v4.1.0.tar.gz
build-wheel: mv /tmp/build-wheel-771cjjy1 /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/requirements
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/include
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib
build-wheel: /home/aryan/chaquopy/server/pypi/build/_bootstrap/3.11/bin/pip --version
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/env
build-wheel: python3.11 -m venv --without-pip /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/env
build-wheel: /home/aryan/chaquopy/server/pypi/build/_bootstrap/3.11/bin/pip --python /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/env/bin/python install cmake==3.30.2
Collecting cmake==3.30.2
  Obtaining dependency information for cmake==3.30.2 from https://files.pythonhosted.org/packages/69/70/242937601f9ff9e6df4c0587b5a7702be4dbfd33420b409d80e2bccc276a/cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
Using cached cmake-3.30.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.9 MB)
Installing collected packages: cmake
Successfully installed cmake-3.30.2
build-wheel: abi=arm64-v8a; api_level=21; prefix=/home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy; . /home/aryan/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/wrappers
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib/libpthread.a
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib/librt.a
build-wheel: cd /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/chaquopy
build-wheel: /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build.sh
-- The C compiler identification is Clang 11.0.5
-- The CXX compiler identification is Clang 11.0.5
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Build spdlog: 1.10.0
-- Build type: Release
-- Compiling for multiple CPU ISA and enabling runtime dispatch
-- Found OpenMP_C: -fopenmp=libomp (found version "5.0")
-- Found OpenMP_CXX: -fopenmp=libomp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
CMake Error at CMakeLists.txt:276 (message):
  Intel OpenMP runtime libiomp5 not found


-- Configuring incomplete, errors occurred!
make: *** No targets specified and no makefile found.  Stop.
make: *** No rule to make target 'install'.  Stop.
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/ctranslate2_lib-4.1.0.dist-info
build-wheel: wheel pack /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix --dest-dir /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src --build-number 0
Repacking wheel as /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl...OK
build-wheel: mkdir -p /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel
build-wheel: unzip -d /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel -q /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl
build-wheel: cp -a /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src/LICENSE /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel/ctranslate2_lib-4.1.0.dist-info
build-wheel: Processing native binaries
build-wheel: wheel pack /home/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel --dest-dir /home/aryan/chaquopy/server/pypi/dist/ctranslate2-lib --build-number 0
Repacking wheel as /home/aryan/chaquopy/server/pypi/dist/ctranslate2-lib/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl...OK

but some warning about libiomp5.so was given so i built it using their git repo and passing the path also, but still its not able to find it.

Now when i try to build python library using -


# Set the path to the C++ library
export CTRANSLATE2_ROOT=$CHAQUOPY_LIB/ctranslate2-lib

# Build the Python wrapper
$PYTHON setup.py bdist_wheel

package:
  name: "ctranslate2-python"
  version: "4.1.0"

source:
  path: /home/aryan/CTranslate2/python

build:
  number: 0

requirements:
  build:
    - ctranslate2-lib 4.1.0 # Add dependency on the C++ library

It is not able to find ctranslate2-lib

aryan1165 pushed a commit to aryan1165/chaquopy that referenced this issue Sep 28, 2024
Added Ctranslate2-lib and Ctranslate2-python. Fixes chaquo#1239.
6c5d3a7
@aryan1165 aryan1165 linked a pull request Sep 28, 2024 that will close this issue
Open
@aryan1165
Copy link

Hi @mhsmith . Can you review my PR #1253.

@mhsmith
Copy link
Member

mhsmith commented Oct 4, 2024

Sorry for the delay, I'll review it as soon as I can.

@aryan1165
Copy link

aryan1165 commented Oct 12, 2024 •
edited by mhsmith
Loading

[Moved to #1253]

@chaquo chaquo deleted a comment from aryan1165 Oct 20, 2024
@chaquo chaquo deleted a comment from aryan1165 Oct 20, 2024
@chaquo chaquo deleted a comment from aryan1165 Oct 20, 2024
@chaquo chaquo deleted a comment from aryan1165 Oct 20, 2024
@chaquo chaquo deleted a comment from aryan1165 Oct 20, 2024
@chaquo chaquo deleted a comment from aryan1165 Oct 20, 2024
@mhsmith
Copy link
Member

mhsmith commented Oct 20, 2024

Please keep discussion about the PR in the PR itself, not here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
packages
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added Ctranslate2 aryan1165/chaquopy
3 participants
@mhsmith @Furtherking @aryan1165