Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requesting FreeBSD support for Python library #2105

Closed
lispmac opened this issue Jul 15, 2020 · 13 comments
Closed

Requesting FreeBSD support for Python library #2105

lispmac opened this issue Jul 15, 2020 · 13 comments
Assignees
Labels
Feature Request Missing Feature/Wrapper Lang: Python Python wrapper issue OS: FreeBSD FreeBSD OS
Milestone

Comments

@lispmac
Copy link

lispmac commented Jul 15, 2020

What language and solver does this apply to?

Python

Describe the problem you are trying to solve.

I would like to be able to easily install ortools on FreeBSD using pip.

Describe the solution you'd like

I would like to be able to write pip install ortools on FreeBSD and have ortools be installed.

Currently, running pip install ortools produces the following output on FreeBSD 12.1 STABLE:

Collecting ortools
  ERROR: Could not find a version that satisfies the requirement ortools (from versions: none)
ERROR: No matching distribution found for ortools

Describe alternatives you've considered

Approach 1:

I've tried installing the py3-ortools Python package via pip install py3-ortools.

However, running:

python3.7
from ortools.linear_solver import pywraplp

Produces error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/ortools/linear_solver/pywraplp.py", line 18, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: Shared object "libc.so.6" not found, required by "_pywraplp.so"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/ortools/linear_solver/pywraplp.py", line 21, in <module>
    _pywraplp = swig_import_helper()
  File "/usr/local/lib/python3.7/site-packages/ortools/linear_solver/pywraplp.py", line 20, in swig_import_helper
    return importlib.import_module('_pywraplp')
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywraplp'

Approach 2:

I've tried to build the library from source on FreeBSD (which required the installation of cmake and gcc, the latter to avoid build errors related to dependencies).

The procedure was (in a Vagrant FreeBSD image):

pkg install python37 py37-pip gcc cmake swig git
git clone https://github.com/google/or-tools
cd or-tools
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -DBUILD_DEPS=ON -DBUILD_PYTHON=ON .

The build succeeds.

Unfortunately, the following does not work for some reason (perhaps there are build steps that I am missing):

cd python/Release
pip install .
python3.7
import ortools

Which produces the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ortools'

Ideally one could just write pip install ortools and avoid the above entirely, just like on all the other major operating systems.

@Mizux Mizux added Feature Request Missing Feature/Wrapper Lang: Python Python wrapper issue OS: Linux GNU/Linux OS labels Jul 15, 2020
@Mizux Mizux added this to the Backlog milestone Jul 15, 2020
@Mizux
Copy link
Collaborator

Mizux commented Jul 15, 2020

Could you try to build out of the source ?

CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -DBUILD_DEPS=ON -DBUILD_PYTHON=ON -S. -Bbuild
cmake --build build
python3.7 -m pip install  --find-links=build/python/dist ortools

inspired by

COMMAND ${VENV_Python_EXECUTABLE} -m pip install --find-links=${CMAKE_CURRENT_BINARY_DIR}/python/dist ${PROJECT_NAME}

did you manage to run test ?
what is the name of the wheel package in the build dir ?

@lispmac
Copy link
Author

lispmac commented Jul 16, 2020

Running CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -DBUILD_DEPS=ON -DBUILD_PYTHON=ON -S. -Bbuild completes successfully.

At the second step, when I run cmake --build build I get an error:

[  0%] Generate C++ protocol buffer for ortools/constraint_solver/routing_parameters.proto
google/protobuf/duration.proto: File not found.
ortools/constraint_solver/routing_parameters.proto:24:1: Import "google/protobuf/duration.proto" was not found or had errors.
ortools/constraint_solver/routing_parameters.proto:393:3: "google.protobuf.Duration" is not defined.
ortools/constraint_solver/routing_parameters.proto:396:3: "google.protobuf.Duration" is not defined.
*** Error code 1

Stop.
make[2]: stopped in /home/vagrant/or-tools/build
*** Error code 1

Stop.
make[1]: stopped in /home/vagrant/or-tools/build
*** Error code 1

Stop.
make: stopped in /home/vagrant/or-tools/build

@Mizux
Copy link
Collaborator

Mizux commented Jul 16, 2020

can you try to find where "google/protobuf/duration.proto" is located ?

find build -type f -iname "duration.proto"

@Mizux Mizux self-assigned this Jul 16, 2020
@lispmac
Copy link
Author

lispmac commented Jul 16, 2020

vagrant@freebsd:~/or-tools % find build -type f -iname "duration.proto"
build/dependencies/install/include/google/protobuf/duration.proto
build/dependencies/Protobuf/source/src/google/protobuf/duration.proto

It's located in build/dependencies/install/include/google/protobuf/ and build/dependencies/Protobuf/source/src/google/protobuf/.

@lispmac
Copy link
Author

lispmac commented Jul 17, 2020

If I run:

cp -R ./build/dependencies/install/include/google ./
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake --build build

The above error is solved, but then I see another error:

Scanning dependencies of target ortools_util
[ 10%] Building CXX object ortools/util/CMakeFiles/ortools_util.dir/file_util.cc.o
[ 10%] Building CXX object ortools/util/CMakeFiles/ortools_util.dir/fp_utils.cc.o
In file included from /home/vagrant/or-tools/ortools/util/fp_utils.cc:14:
/home/vagrant/or-tools/ortools/util/fp_utils.h: In member function 'void operations_research::ScopedFloatingPointEnv::EnableExceptions(int)':
/home/vagrant/or-tools/ortools/util/fp_utils.h:87:11: error: 'struct fenv_t' has no member named '__control_word'
   87 |     fenv_.__control_word &= ~excepts;
      |           ^~~~~~~~~~~~~~
gmake[2]: *** [ortools/util/CMakeFiles/ortools_util.dir/build.make:122: ortools/util/CMakeFiles/ortools_util.dir/fp_utils.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1922: ortools/util/CMakeFiles/ortools_util.dir/all] Error 2
gmake: *** [Makefile:161: all] Error 2

If I comment out the line fenv_.__control_word &= ~excepts in ortools/util/fp_utils.h the build proceeds (not sure what the effect of that line is).

Then I see another error later in the build:

In file included from /home/vagrant/or-tools/ortools/graph/ebert_graph.h:182,
                 from /home/vagrant/or-tools/ortools/graph/linear_assignment.h:212,
                 from /home/vagrant/or-tools/ortools/constraint_solver/routing.cc:51:
/home/vagrant/or-tools/ortools/util/zvector.h:20:10: fatal error: endian.h: No such file or directory
   20 | #include <endian.h>
      |          ^~~~~~~~~~
compilation terminated.
gmake[2]: *** [ortools/constraint_solver/CMakeFiles/ortools_constraint_solver.dir/build.make:343: ortools/constraint_solver/CMakeFiles/ortools_constraint_solver.dir/routing.cc.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1706: ortools/constraint_solver/CMakeFiles/ortools_constraint_solver.dir/all] Error 2
gmake: *** [Makefile:161: all] Error 2

If I change the line to #include <machine/endian.h> the build proceeds.

The build succeeds.

If I then run python3.7 -m pip install --find-links=build/python/dist ortools

I see:

Looking in links: build/python/dist
Collecting ortools
Collecting protobuf>=3.12.2 (from ortools)
  Using cached https://files.pythonhosted.org/packages/47/90/27eac528bb6871336e86ed8bcbb72a9b412a4020bf69bb7be71d40e5eab0/protobuf-3.12.2-py2.py3-none-any.whl
Requirement already satisfied: six>=1.10 in /root/.local/lib/python3.7/site-packages (from ortools) (1.15.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from protobuf>=3.12.2->ortools) (44.0.0)
Installing collected packages: protobuf, ortools
Successfully installed ortools-7.7.7810 protobuf-3.12.2

It appears to have installed successfully.

@Mizux
Copy link
Collaborator

Mizux commented Jul 17, 2020

wow thanks a lot !

  1. For proto path, I may have found the issue, need to add
## Get Protobuf include dir
get_target_property(protobuf_dirs protobuf::libprotobuf INTERFACE_INCLUDE_DIRECTORIES)
foreach(dir IN LISTS protobuf_dirs)
  if ("${dir}" MATCHES "BUILD_INTERFACE")
    message(STATUS "Adding proto path: ${dir}")
    list(APPEND PROTO_DIRS "--proto_path=${dir}")
  endif()
endforeach()

It is done in cmake/cpp.cmake but not in python/java/dotnet, on my way to fix it...

  1. For struct fenv_t and endian.h it seems BSD has it own version
    e.g. https://github.com/freebsd/freebsd/blob/11af20b5ebba29c6ad1fea903d94de31fe509f3f/lib/msun/x86/fenv.h#L75-L82
    @lispmac do you know if there is a preprocessor macro to check if we are on BSD so we can fix the code ?
    EDIT: It seems we could use __FreeBSD__
    ref: https://svn.boost.org/svn/boost/trunk/boost/config/platform/bsd.hpp

@lperron
Copy link
Collaborator

lperron commented Jul 17, 2020 via email

@Mizux Mizux closed this as completed in db874af Jul 17, 2020
@Mizux
Copy link
Collaborator

Mizux commented Jul 17, 2020

Here, my Vagrantfile to test build on "FreeBSD" with comment and empty line removed...

$ cat Vagrantfile | sed "/^ *#/d" | sed "/^ *$/d"
Vagrant.configure("2") do |config|
  config.vm.guest = :freebsd
  config.vm.box = "freebsd/FreeBSD-11.4-STABLE"
  config.ssh.shell = "sh"
  config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
  config.vm.provision "shell", inline: <<-SHELL
     set -x
     pkg update -f
     pkg install -y git cmake
     git clone -b master https://github.com/google/or-tools.git
     cd or-tools
     cmake -S. -Bbuild -DBUILD_DEPS=ON
     cmake --build build
     set -x
  SHELL
end

Few dev notes:

$ vagrant destroy
$ export VAGRANT_EXPERIMENTAL="dependency_provisioners"
$ vagrant up
...
    default: /home/vagrant/or-tools/build/_deps/cgl-src/Cgl/src/CglLandP/CglLandPUtils.hpp
    default: :
    default: 88
    default: :
    default: 25
    default: :
    default:  
    default: error
    default: : 
    default: reinterpret_cast from 'nullptr_t' to 'OsiRowCut *' is not allowed
    default:         cuts_.resize(i, reinterpret_cast<OsiRowCut *> (NULL));
    default:                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    default: 1
    default:  error
    default:  generated

@lispmac
Copy link
Author

lispmac commented Jul 17, 2020

The fenv fix is almost correct.

On amd64, __control is defined on a nested __x87 struct: https://github.com/freebsd/freebsd/blob/11af20b5ebba29c6ad1fea903d94de31fe509f3f/lib/msun/x86/fenv.h#L86

So, the fenv fix (on amd64) would be:

#if defined(__APPLE__)
    fenv_.__control &= ~excepts;
#elif defined(__FreeBSD__)
    fenv_.__x87.__control &= ~excepts;
#else  // Linux                                                                                                                                                                                
    fenv_.__control_word &= ~excepts;
...

Regarding the Cgl build error -- even though the compilation fails with clang, the compilation with gcc succeeds. (Although clang is the preferred compiler on FreeBSD and gcc doesn't ship with the base install.)

Below is the Vagrantfile with the working (after db874af and the above fenv fix are merged to master) compilation with gcc.

Vagrant.configure("2") do |config|
  config.vm.guest = :freebsd
  config.vm.box = "freebsd/FreeBSD-11.4-STABLE"
  config.ssh.shell = "sh"
  config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
  config.vm.provision "shell", inline: <<-SHELL
     set -x
     pkg update -f
     pkg install -y git cmake gcc python37 py37-pip swig
     git clone -b master https://github.com/google/or-tools.git
     cd or-tools
     CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ cmake -S. -Bbuild -DBUILD_DEPS=ON -DBUILD_PYTHON=ON
     cmake --build build
     python3.7 -m pip install  --find-links=build/python/dist ortools
     set -x
  SHELL
end

Note that the provisioning script above also builds and installs the python package.

clang can hopefully be swapped in when the Cgl build error is fixed, assuming there are no other build errors with clang.

@Mizux
Copy link
Collaborator

Mizux commented Jul 17, 2020

I'll try to add the patch ASAP in the same time I'll try to fix the clang issue which seems to follow coin-or/Clp#93
note: dependencies chain should be CoinUtils -> Osi -> Clp -> Cgl -> Cbc so only Cgl and Cbc to fix ! :D

@lispmac
Copy link
Author

lispmac commented Jul 17, 2020

I've tested the above Vagrantfile on the generic/freebsd12 box and the build is successful (after manually patching with the fenv fix).

Mizux added a commit that referenced this issue Jul 17, 2020
@Mizux
Copy link
Collaborator

Mizux commented Jul 20, 2020

FYI:

  • I've just fixed Cgl and Cbc clang compilation issue on my fork (since coin-or Team refuse to support CMake) and create issue on coin-or if they want my patches...

  • I still have an issue when trying to link a binary since a "bad guy" reference "backtrace()" (from execinfo.h ?) function without linking to the corresponding library.
    after doing few grep in our build/_deps, I think it comes from absl::stacktrace.
    Unfortunately for me, absl-cpp only compile if you disable tests -> no binary generated -> you can't see the issue.
    My VagrantFile on absl-cpp

Vagrant.configure("2") do |config|
  config.vm.guest = :freebsd
  config.vm.box = "generic/freebsd12"
  config.ssh.shell = "sh"
  config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
  config.vm.provision "env", type: "shell", inline:<<-SHELL
  set -x
  pkg update -f
  pkg install -y git cmake vim
  SHELL
  config.vm.provision "devel", type: "shell", inline:<<-SHELL
  set -x
  git clone --depth 1 --branch 20200225.2 https://github.com/abseil/abseil-cpp.git
  cd abseil-cpp
  SHELL
  config.vm.provision "configure", type: "shell", inline:<<-SHELL
  set -x
  cd abseil-cpp
  cmake -S. -Bbuild -DABSL_USE_GOOGLETEST_HEAD=ON -DABSL_RUN_TESTS=ON
  SHELL
  config.vm.provision "build", type: "shell", inline:<<-SHELL
  set -x
  cd abseil-cpp
  cmake --build build -v
  SHELL
end

On my way to create a hello world CMake project which only link against absl-cpp to verify this assertion, unfortunately abseil-cpp didn't work with FetchContent() so we need some absl patches.
BUT this weekend all patches have been merged to abseil-cpp master (abseil/abseil-cpp#735, abseil/abseil-cpp#736)

  • EDIT: glog also use backtrace()
[0]─[~/work/master/build/_deps]
[^v^]─mizux@nuc10i7 %grepc -Rin "backtrace(" *-src
absl-src/absl/debugging/internal/stacktrace_generic-inl.inc:44:  backtrace(unused_stack, 1);
absl-src/absl/debugging/internal/stacktrace_generic-inl.inc:62:  size = backtrace(stack, kStackLength);
glog-src/src/stacktrace_windows-inl.h:47:  return CaptureStackBackTrace(skip_count, max_depth, result, NULL);
glog-src/src/stacktrace_generic-inl.h:46:  size = backtrace(stack, kStackLength);
glog-src/src/stacktrace_x86_64-inl.h:50:// Workaround for the malloc() in _Unwind_Backtrace() issue.
glog-src/src/stacktrace_x86_64-inl.h:51:static _Unwind_Reason_Code nop_backtrace(struct _Unwind_Context *uc, void *opq) {
glog-src/src/stacktrace_x86_64-inl.h:64:     _Unwind_Backtrace(nop_backtrace, NULL);
glog-src/src/stacktrace_x86_64-inl.h:100:  _Unwind_Backtrace(GetOneFrame, &targ);

@Mizux
Copy link
Collaborator

Mizux commented Jul 21, 2020

Strange, this morning using box generic/freebsd12 I managed to build using the default clang.

My current investigation

I wasn't able to reproduce the backtrace() symbol reference not found, on the two following CMake projects

  • One CMake project, with only absl-cpp as dependency, creating a simple binary and only linking against absl::backtrace and calling stacktrace..
  • One CMake project, with only glog as dependency, creating a simple binary and only linking against glog::glog and calling stacktrace
    note: projects are attached if someone want to reproduce it...
    glog_backtrace.tar.gz
    absl_backtrace.tar.gz

note: both project were done using the generic/freebsd12 since i was not able to "mount a directory" (ed using sync_folder) with the other freebsd/FreeBSD-11.4-STABLE box img, so potential workaround to test on this box would be to push these samples on a git repository then in the Vagrantfile clone the repository, thus we won't need sync_folder to have the files in the VM...

So I suspect freebsd/FreeBSD-11.4-STABLE to be broken on some way and won't follow this rabbit hole (already take too much time on it)

Fix Hypothesis

An other potential fix, path to investigate, is both libraries (glog and absl-cpp), AFAIK, don't use the FindBacktrace() CMake builtin module which:

Checks if OS supports backtrace(3) via either libc or custom library.

ref: https://cmake.org/cmake/help/v3.18/module/FindBacktrace.html

So i suspect the FreeBSD-1.14-STABLE box to use an extra library than the libc to provide the backtrace symbol and thus since we don't link against it, the error occurs...

Abseil-cpp master

FYI, currently absl-cpp master HEAD is not compiling on FreeBSD see: abseil/abseil-cpp#744

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Request Missing Feature/Wrapper Lang: Python Python wrapper issue OS: FreeBSD FreeBSD OS
Projects
None yet
Development

No branches or pull requests

3 participants