Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Making python3 work with cmake and the new python wrapper #1923

Merged
merged 1 commit into from
Feb 24, 2015
Merged

Making python3 work with cmake and the new python wrapper #1923

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ caffe_option(CPU_ONLY "Build Caffe wihtout CUDA support" OFF) # TODO: rename to
caffe_option(USE_CUDNN "Build Caffe with cuDNN libary support" ON IF NOT CPU_ONLY)
caffe_option(BUILD_SHARED_LIBS "Build shared libraries" ON)
caffe_option(BUILD_python "Build Python wrapper" ON)
set(python_version "2" CACHE STRING "Specify which python version to use")
caffe_option(BUILD_matlab "Build Matlab wrapper" OFF IF UNIX OR APPLE)
caffe_option(BUILD_docs "Build documentation" ON IF UNIX OR APPLE)

Expand Down
39 changes: 33 additions & 6 deletions cmake/Dependencies.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,39 @@ endif()

# ---[ Python
if(BUILD_python)
# disable Python 3 search
find_package(PythonInterp 2.7)
find_package(PythonLibs 2.7)
find_package(NumPy 1.7.1)
find_package(Boost 1.46 COMPONENTS python)

if(NOT "${python_version}" VERSION_LESS "3.0.0")
# use python3
find_package(PythonInterp 3.0)
find_package(PythonLibs 3.0)
find_package(NumPy 1.7.1)
# Find the matching boost python implementation
set(version ${PYTHONLIBS_VERSION_STRING})

STRING( REPLACE "." "" boost_py_version ${version} )
find_package(Boost 1.46 COMPONENTS "python-py${boost_py_version}")
set(Boost_PYTHON_FOUND ${Boost_PYTHON-PY${boost_py_version}_FOUND})

while(NOT "${version}" STREQUAL "" AND NOT Boost_PYTHON_FOUND)
STRING( REGEX REPLACE "([0-9.]+).[0-9]+" "\\1" version ${version} )
STRING( REGEX MATCHALL "([0-9.]+).[0-9]+" has_more_version ${version} )
if("${has_more_version}" STREQUAL "")
break()
endif()

STRING( REPLACE "." "" boost_py_version ${version} )
find_package(Boost 1.46 COMPONENTS "python-py${boost_py_version}")
set(Boost_PYTHON_FOUND ${Boost_PYTHON-PY${boost_py_version}_FOUND})
endwhile()
if(NOT Boost_PYTHON_FOUND)
find_package(Boost 1.46 COMPONENTS python)
endif()
else()
# disable Python 3 search
find_package(PythonInterp 2.7)
find_package(PythonLibs 2.7)
find_package(NumPy 1.7.1)
find_package(Boost 1.46 COMPONENTS python)
endif()
if(PYTHONLIBS_FOUND AND NUMPY_FOUND AND Boost_PYTHON_FOUND)
set(HAVE_PYTHON TRUE)
endif()
Expand Down
4 changes: 2 additions & 2 deletions docs/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Caffe has several dependencies.

Pycaffe and Matcaffe interfaces have their own natural needs.

* For Python Caffe: `Python 2.7`, `numpy (>= 1.7)`, boost-provided `boost.python`
* For Python Caffe: `Python 2.7` or `Python 3.3+`, `numpy (>= 1.7)`, boost-provided `boost.python`
* For MATLAB Caffe: MATLAB with the `mex` compiler.

**cuDNN Caffe**: for fastest operation Caffe is accelerated by drop-in integration of [NVIDIA cuDNN](https://developer.nvidia.com/cudnn). To speed up your Caffe models, install cuDNN then uncomment the `USE_CUDNN := 1` flag in `Makefile.config` when installing Caffe. Acceleration is automatic. For now cuDNN v1 is integrated but see [PR #1731](https://github.com/BVLC/caffe/pull/1731) for v2.
Expand Down Expand Up @@ -69,7 +69,7 @@ but we suggest first installing the [Anaconda](https://store.continuum.io/cshop/

To import the `caffe` Python module after completing the installation, add the module directory to your `$PYTHONPATH` by `export PYTHONPATH=/path/to/caffe/python:$PYTHONPATH` or the like. You should not import the module in the `caffe/python/caffe` directory!

*Caffe's Python interface works with Python 2.7. Python 3 or earlier Pythons are your own adventure.*
*Caffe's Python interface works with Python 2.7. Python 3.3+ should work out of the box without protobuf support. For protobuf support please install protobuf 3.0 alpha (https://developers.google.com/protocol-buffers/). Earlier Pythons are your own adventure.*

#### MATLAB

Expand Down
4 changes: 3 additions & 1 deletion python/caffe/_caffe.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,9 @@ BOOST_PYTHON_MODULE(_caffe) {
bp::class_<vector<bool> >("BoolVec")
.def(bp::vector_indexing_suite<vector<bool> >());

import_array();
// boost python expects a void (missing) return value, while import_array
// returns NULL for python3. import_array1() forces a void return value.
import_array1();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain/reference this? Everything I've found defines import_array1 as a unary function (hence the 1), and I can't find any reason there'd be a difference between 2 and 3, so I'm confused as to why this works at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import_array1 is the same as input_array, but it takes the return value of the statement as an argument. If we leave this return statement empty it will return "void". import_array will not compile in python3 (the boost function is defined as void ..., while import_array calls return some_int_constant).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that a boost issue? Surely boost should define an init function with the right type for the Python version? It does seem that import_array does the right thing, which is to return void for 2 and NULL for 3. What actually goes wrong with import_array? It sounds like boost thinks it's being built for 2...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here is what goes wrong:
/usr/include/python3.4m/numpy/__multiarray_api.h:1708:35: error: return-statement with a value, in function returning 'void' [-fpermissive] #define NUMPY_IMPORT_ARRAY_RETVAL NULL
The boost python function BOOST_PYTHON_MODULE(_caffe) is always defined as void BOOST_PP_CAT(init_module_, name)() which translates to something like void init_module__caffe(). This is irrespective of the python version (which is good).
Numpy switches out the return value of import_array from void to NULL, when switching from python2 to python3, for whatever reason. import_array1 fixes this.

}

} // namespace caffe
9 changes: 8 additions & 1 deletion python/caffe/io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,14 @@
from scipy.ndimage import zoom
from skimage.transform import resize

from caffe.proto import caffe_pb2
try:
# Python3 will most likely not be able to load protobuf
from caffe.proto import caffe_pb2
except:
if sys.version_info >= (3,0):
print("Failed to include caffe_pb2, things might go wrong!")
else:
raise

## proto / datum / ndarray conversion

Expand Down
5 changes: 4 additions & 1 deletion python/caffe/pycaffe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@
"""

from collections import OrderedDict
from itertools import izip_longest
try:
from itertools import izip_longest
except:
from itertools import zip_longest as izip_longest
import numpy as np

from ._caffe import Net, SGDSolver
Expand Down
6 changes: 3 additions & 3 deletions python/classify.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ def main(argv):
channel_swap=channel_swap)

if args.gpu:
print 'GPU mode'
print('GPU mode')

# Load numpy array (.npy), directory glob (*.jpg), or image file.
args.input_file = os.path.expanduser(args.input_file)
Expand All @@ -115,12 +115,12 @@ def main(argv):
else:
inputs = [caffe.io.load_image(args.input_file)]

print "Classifying %d inputs." % len(inputs)
print("Classifying %d inputs." % len(inputs))

# Classify.
start = time.time()
predictions = classifier.predict(inputs, not args.center_only)
print "Done in %.2f s." % (time.time() - start)
print("Done in %.2f s." % (time.time() - start))

# Save
np.save(args.output_file, predictions)
Expand Down
2 changes: 1 addition & 1 deletion python/detect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ def main(argv):
context_pad=args.context_pad)

if args.gpu:
print 'GPU mode'
print('GPU mode')

# Load input.
t = time.time()
Expand Down
2 changes: 1 addition & 1 deletion python/draw_net.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ def main():
args = parse_args()
net = caffe_pb2.NetParameter()
text_format.Merge(open(args.input_net_proto_file).read(), net)
print 'Drawing net to %s' % args.output_image_file
print('Drawing net to %s' % args.output_image_file)
caffe.draw.draw_net_to_file(net, args.output_image_file, args.rankdir)


Expand Down