A single CMake file with all build information #490

david-ragazzi · 2013-12-11T19:44:47Z

A new and improved CMake file.

Improvements:

Replacement of all CMake files to a single file in order to ease understanding and maintenance (this avoids several files scattered in the repository).
Environment variables (PATH, PYTHONPATH, NTA_ROOTDIR, etc) are set in own CMake file.
Now it use the own Swig executable and dependencies located on External folder. Before this, I used 'find_packages' which could set an incompatible version for Nupic.
Now it use the system python, i.e. the python package called from shell natively (similar to what Autotool version does). Before this, I used 'find_packages' which could set wrong python interpreter or libraries.

Build result: SUCCESS (including Swig modules!)

Tests results:
HtmTest: SUCCESS in all tests!!
TestEverything: 99.9107 %, only 1 (out of 2238) test failed: get spec of the dynamic library (i.e. libcpp_region).

I have worked hard to find out what is causing this, but without success.. :-( When I ignore TRY statement, it states that is not possible get a symbol from swig wrapped library:
dlopen(/Volumes/Data/Users/davidragazzi/nta/eng/lib/python2.7/site-packages/nupic/bindings/_math.so, 2): Symbol not found: __ZN14PySparseTensor3setERK13PyTensorIndexP7_object
Referenced from: /Volumes/Data/Users/davidragazzi/nta/eng/lib/python2.7/site-packages/nupic/bindings/_math.so
Expected in: flat namespace
in /Volumes/Data/Users/davidragazzi/nta/eng/lib/python2.7/site-packages/nupic/bindings/_math.so

If some numenta expert could debug this, it would be great!

A new and improved CMake file. Improvements: - Replacement of all CMake files to a single file in order to ease understanding and maintenance (this avoids several files scattered in the repository). - Environment variables (PATH, PYTHONPATH, NTA_ROOTDIR, etc) are set in own CMake file. - Now it use the own Swig executable and dependencies located on External folder. Before this, I used 'find_packages' which could set an incompatible version for Nupic. - Now it use the system python, i.e. the python package called from shell natively (similar to what Autotool version does). Before this, I used 'find_packages' which could set wrong python interpreter or libraries. Build result: SUCCESS (including Swig modules!) Tests results: HtmTest: SUCCESS in all tests!! TestEverything: 99.9107 %, only 1 (out of 2238) test failed: get spec of the dynamic library (i.e. libcpp_region). I have worked hard to find out what is causing this, but without success.. :-( When I ignore TRY statement, it states that is not possible get a symbol from swig wrapped library: dlopen(/Volumes/Data/Users/davidragazzi/nta/eng/lib/python2.7/site-packages/nupic/bindings/_math.so, 2): Symbol not found: __ZN14PySparseTensor3setERK13PyTensorIndexP7_object Referenced from: /Volumes/Data/Users/davidragazzi/nta/eng/lib/python2.7/site-packages/nupic/bindings/_math.so Expected in: flat namespace in /Volumes/Data/Users/davidragazzi/nta/eng/lib/python2.7/site-packages/nupic/bindings/_math.so If some numenta expert could debug this, it would be great!

david-ragazzi · 2013-12-11T19:55:16Z

@breznak
Hi Mark,

Now I believe we have a (99,9%) working version to use. Please test it and let me know any questions. It would be interesting if you or other numenta expert could debug the issue mentioned. I've checked all compiler, linker flags, includes, but the error remains. Maybe you find out a missing piece that I'm not realising it.. :-(

Best, David

subutai · 2013-12-13T21:51:19Z

Hi @david-ragazzi - thanks for doing this! I am intrigued by CMake - it looks like a very clean way to do things. With regards to the SparseTensor errors - it's very hard to tell. I do know that file uses a lot of complex template stuff so that might be getting in the way. However I'm pretty sure we don't use SparseTensor anywhere anymore. Could someone do a scan to see where we might be using it? If it is not being used I am ok removing it entirely from NuPIC. Maybe @breznak or someone else could create a separate PR to remove that, and that should resolve this issue too.

breznak · 2013-12-13T23:13:17Z

Hi @david-ragazzi ,

the cmake solution looks really pretty clear! :)

I've spotted two (hope minor) problems:

1/ compiler identification:

(py27)[marek@beruska cmake_build]$ cmake ..
-- The C compiler identification is GNU 4.8.2
-- The CXX compiler identification is GNU 4.8.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Checking platform...
-- Linux64 on x86_64
-- Configured for platform linux64
...
-- Checking if clang is the compiler...
-- ON

I have both gcc and clang avaliable, the C/CXX checks have found gcc, then the clang check has found clang. I haven't specified $CXX, but I think it would be good to default to gcc on linux. When I run make, it's clang that gets used.

2/ libpython problem:

from the configure:

Python 2.7.6
-- Python details:
-- Version: 2.7
-- Include dir: /home/marek/devel/nupic/py27/include/python2.7
-- Library dir: /home/marek/devel/nupic/py27/lib/python2.7
-- Library: /home/marek/devel/nupic/py27/lib/libpython2.7.so
-- Site packages dir: /tmp//nta/eng/lib/python2.7/site-packages
-- Numpy details:
-- Version: 1.8.0
-- Include dir: /home/marek/devel/nupic/py27/lib/python2.7/site-packages/numpy/core/include
-- Swig details:
-- Version: 1.3.36

The file -- Library: /home/marek/devel/nupic/py27/lib/libpython2.7.so does not exist on my system. Could it be you're assuming a name from how it's done on OSX? (I'm running python from viertualenv, but that should not matter).

Which is causing the compilation error later:
make[2]: *** No rule to make target '../py27/lib/libpython2.7.so', needed by '/tmp/nta/eng/lib/python2.7/site-packages/nupic/bindings/_algorithms.so'. Stop.
CMakeFiles/Makefile2:69: recipe for target 'CMakeFiles/_algorithms.dir/all' failed
make[1]: *** [CMakeFiles/_algorithms.dir/all] Error 2
Makefile:75: recipe for target 'all' failed
make: *** [all] Error 2

Otherwise it seems to be running fine, and even the swig seems compiling, that's 👍

Thank you very much, Mark

david-ragazzi · 2013-12-14T14:08:59Z

Namaste @subutai !

My intuition says that LD_LIBRARY_PATH could not be well configured (i.e. pointing to lib/python2.*/site_packages/bindings folder), so TestEverything executable could be not able to load dynamically these swig libraries. The problem is that set environment variables on OSX is very tricky because there are several scopes where they can or not be visible (a single 'export' on shell script cannot be enough to a env variable be visible on any xcode solution).

As Windows has a simplified form of manage env variables, I'll test this behavior on it.

david-ragazzi · 2013-12-14T14:22:45Z

Hi @breznak !

1/ compiler identification:
I actually dont know what could be causing this, maybe this configuring procedure is done before CLang is set as ON. I'll check this.

2/ libpython problem:
I had realised this issue after I create this PR, but now it is fixed. Now I include python library dir in order to avoid inform the name of the library (i.e. "-L{PYTHON_LIBRARY_DIR}" linker flag). I'll send a new version to you.

david-ragazzi · 2013-12-15T23:32:06Z

OH yeah!!! I finally found the error! I simply forgot add PySparseTensor.cpp as extra source file to _math swig library. Now all tests are 100%!!! After much work, we finally can have a clean and simple build process.

@subutai and @breznak:
I will submit the new version to Mark in order to he check if the mentioned minor issues remain on his linux.

rhyolight · 2013-12-16T06:31:39Z

@david-ragazzi That's great news! Way to go!

david-ragazzi · 2013-12-16T11:16:00Z

Thanks @rhyolight !
Since Mark give positive feedback, you guys could comit this file. I suggest leave it as a parallel build process on quarantine (1 month or less) and then remove all autotools stuff (and related scripts) after that.

I have thought in some files that could be removed in a full clean-up:

All makefiles.am, makefiles.in;
Entire build_system folder;
Env.sh, Build.sh, Clean.sh, and Autogen
Everything Xcode project.
Trunk .net solution.

The source folder will be perfectly clean after this clean-up, and new members will understand the build process only reading this file.

Once that comunity members do not relate issues with CMake build, we will feel confident in to do this clean-up.

sjmackenzie · 2013-12-16T11:52:49Z

Kudos to you David!

david-ragazzi · 2013-12-16T12:02:08Z

Thanks @sjmackenzie !

iandanforth · 2013-12-16T12:05:52Z

Woooo!

--Please excuse brevity, sent from phone.--
On Dec 15, 2013 3:32 PM, "DavidRagazzi" notifications@github.com wrote:

OH yeah!!! I finally found the error! I simply forget add
PySparseTensor.cpp as extra source file to _math swig library. Now all
tests are 100%!!! After much work, we finally we can have a clean and
simple build process.

@subutai https://github.com/subutai and @breznakhttps://github.com/breznak
:
I will submit the new version to Mark in order to he check if the
mentioned minor issues on his linux.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/490#issuecomment-30625104
.

david-ragazzi · 2013-12-16T12:51:01Z

Thanks @iandanforth !

@rhyolight :
I have writen a brief 'cake receipt' for update the readme file after quarantine (feel free for fix my spelling errors.. hehe):

(...)

Currently supported platforms:

Linux (32/64bit)
Mac OSX
Raspberry Pi (ARMv6)
VM images

Dependencies:

Python (2.6-2.7) (with development headers)
GCC (4.6-4.8), or Clang
CMake (2.8 or later)
Make or any IDE supported by CMake (VisualStudio, XCode, KDevelop, or Eclipse)

The dependencies are included in platform-specific repositories for convenience:

nupic-linux64 for 64-bit Linux systems
nupic-darwin64 for 64-bit OS X systems

Complete set of python requirements are documented in requirements.txt, compatible with pip:

pip install -r external/common/requirements.txt

Build and test NuPIC:

Command line (using 'make'):

Generate build files:
mkdir /path/to/build (we recommend you create a 'build' subfolder in source folder)
cd /path/to/build
ccmake ../path/to/source

Build the generated files:
make

Run the C++ tests:
$HOME/nta/eng/bin/htmtest
$HOME/nta/eng/bin/testeverything

Graphical interface:

Generate an IDE solution:

Open CMake executable;
Specify the source folder;
Specify the build folder, ie where IDE solution will be created (we recommend you create a 'build' subfolder in source folder);
Click 'Generate';
Choose the IDE that interest you (remember that IDE choice is limited to your OS, ie Visual Studio is available only on CMake for Windows)

Build the solution:

Open 'Nupic.*proj' solution file from IDE folder;
Run 'ALL_BUILD' project from your IDE.

Run the C++ tests:

Run 'HtmTest' or 'TestEverything' projects from your IDE (check 'output' panel to see the results).

(...)

Please let me know if is ok...

Best, David

PS:
Although '$HOME/nta/eng' being the binary folder, I think that put all binaries on '/path/to/source/bin' is more conventional.

rhyolight · 2013-12-16T16:39:04Z

@david-ragazzi Thanks! Would you mind putting the README updates into a separate PR we can look at after this one is merged? I will probably want to help out with that.

david-ragazzi · 2013-12-16T16:43:03Z

@rhyolight Of course, I will create a PR for this.

Update:
PR #499 created.

subutai · 2013-12-16T16:51:57Z

Very nice @david-ragazzi !

david-ragazzi · 2013-12-16T17:31:17Z

Thanks @subutai !

breznak · 2013-12-16T20:49:27Z

Hi @david-ragazzi ,

I tested the file you sent me (which is slightly different to this one) and my above error is fixed 👍 thanks!

There's still some issue during compilation:

Linking CXX shared module /home/marek/nta/eng/lib/python2.7/site-packages/nupic/bindings/_algorithms.so
c++: error: CoreFoundation: No such file or directory
c++: error: unrecognized command line option ‘-framework’
c++: error: unrecognized command line option ‘-module’
c++: error: unrecognized command line option ‘-avoid-version’
CMakeFiles/_algorithms.dir/build.make:107: recipe for target '/home/marek/nta/eng/lib/python2.7/site-packages/nupic/bindings/_algorithms.so' failed
make[2]: *** [/home/marek/nta/eng/lib/python2.7/site-packages/nupic/bindings/_algorithms.so] Error 1
CMakeFiles/Makefile2:69: recipe for target 'CMakeFiles/_algorithms.dir/all' failed
make[1]: *** [CMakeFiles/_algorithms.dir/all] Error 2
Makefile:75: recipe for target 'all' failed

...looks like OS-specific flags to the linker (?)

PS: sorry to always bring problems :)

Cheers, breznak!

breznak · 2013-12-16T21:48:29Z

CMakeLists.txt

+# Second, for multi-config builds (DEBUG or RELEASE, for example).
+#
+set(CMAKE_SWIG_OUTDIR "${PYTHON_SITE_PACKAGES_DIR}/nupic/bindings")
+set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_INSTALL_PREFIX}/bin)


the bin/, lib/ and build/ directories should then be added to .gitignore file

Absolutely.

david-ragazzi · 2013-12-16T21:56:19Z

@breznak
It's ok.. haha one hour or other they would arise... :-)

I'll will check this issue.

breznak · 2013-12-17T09:54:56Z

@david-ragazzi the new file you sent me runs just FINE!! 👍 👍 Thank you a lot.

Notes:
-your editor puts ^M at the end of each line, we'll remove that just before the file is ready to merge
-as cmake is now running on linux, we should set .travis.yml to use it

david-ragazzi · 2013-12-17T12:04:58Z

@breznak

👍

Ok, so it's time to ask you more a favor! hehe

I changed default directories structure according to our @subutai conversation.

Basically the CMake script configure $NUPIC and $NTA with default locations to the same repository dir BUT in case of the user already have these variables set on his system, it simply re-use them. This increases flexibility.

A sample of the code:
...

.#
.# Sets default locations.
.#
.# Default directories structure is:
.#
.# ....~/../repository (root directory with repository downloaded from internet)
.#........./docs (subdirectory with support documentation)
.#........./source (subdirectory with all source code)
.#........./release (subdirectory with executables and libraries generated by build proccess)
.#........./build_system (subdirectory with build process [makefiles or IDE solution] generated by CMake)
.#.........Readme.md
.#.........Licence.txt
.# ..../tmp/ntabuild (directory with temporary files generated by build proccess)
.#
.# Note that:
.# /repository/source is referenced by $NUPIC environment variable.
.# /repository/release is referenced by $NTA environment variable.
.# This way, users have flexibility to change default locations.
.#
.# Repository root is the parent directory which this CMake file is located
.# This CMake file must be always on /repository/source subdirectory.
get_filename_component(REPOSITORY_DIR ${CMAKE_SOURCE_DIR} PATH)
.# 'release' is a subdirectory created on repository root,
.# but if it already is specified on $NTA environment variable, simply re-use it!
if("$ENV{NTA}" STREQUAL "")
...set(CMAKE_INSTALL_PREFIX "${REPOSITORY_DIR}/release")
else()
...set(CMAKE_INSTALL_PREFIX "$ENV{NTA}")
endif()
.# 'ntabuild' is a subdirectory created on system temporary directory.
set(TEMP_BUILD_DIR $ENV{TMP}/ntabuild)
.# CMAKE_CURRENT_BINARY_DIR is used by swig module for put wrappers on, so don't erase this line!
set(CMAKE_CURRENT_BINARY_DIR ${TEMP_BUILD_DIR})

...

.#
.# Set environment variables of the operating system
.#
set_environment_variable(PATH "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}" ON)
set_environment_variable(PYTHONPATH "${PYTHON_SITE_PACKAGES_DIR}" OFF)
set_environment_variable(NUPIC "${CMAKE_SOURCE_DIR}" OFF)
set_environment_variable(LD_LIBRARY_PATH "${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}" ON)
set_environment_variable(DYLD_LIBRARY_PATH "${CMAKE_LIBRARY_OUTPUT_DIRECTORY}" ON)
.# If $NTA (ie 'release' directory) already is set, don't change it!
if("$ENV{NTA}" STREQUAL "")
...set_environment_variable(NTA "${CMAKE_INSTALL_PREFIX}" OFF)
...set_environment_variable(NTA_ROOTDIR "${CMAKE_INSTALL_PREFIX}" OFF)
...set_environment_variable(NTA_DATA_PATH "${CMAKE_INSTALL_PREFIX}/share/prediction/data" OFF)
endif()

..

Note that 'release' will be created in the parent directory of source dir. In soon, we would have to move some directories (in official nupic repository) to a new directory called 'source'. But this only can be done, after autotools process is replaced by CMake process.

I still dont test this version (I'm not on my computer now), so if you can help me, I would be thankful. :-) I'll send it by email..

david-ragazzi · 2013-12-17T13:32:38Z

New version of CMake file (now working on Linux):
https://github.com/david-ragazzi/nupic/blob/master/CMakeLists.txt

Update:
This version is presenting problems, today I'll fix it.. Please ignore it.

breznak · 2013-12-17T13:42:49Z

Is this the version with structure changes as Subutai suggested? I'll test it, so I'm supposted to create new folders source/, release/ and move everything in current $NUPIC to source?

david-ragazzi · 2013-12-17T14:32:44Z

@breznak

Is this the version with structure changes as Subutai suggested?

Yes!

so I'm supposted to create new folders source/, release/ and move everything in current $NUPIC to source?

Yes, would be great! Please move everything (except README.md, LICENSE.txt, and other files that you consider pertinent) to /source and set $NUPIC env variable to reference it. Also if you have find the /docs folder mentioned by Subutai please move it to /repository root.

Very thanks!!

david-ragazzi · 2013-12-19T02:35:05Z

@breznak

This version is ok:
https://github.com/david-ragazzi/nupic/blob/master/CMakeLists.txt

david-ragazzi · 2013-12-21T00:47:23Z

Suggestions applied and created new version of this file in #514

breznak reviewed Dec 16, 2013
View reviewed changes

david-ragazzi closed this Dec 21, 2013

david-ragazzi mentioned this pull request Dec 21, 2013

A single CMake file with all build information #514

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A single CMake file with all build information #490

A single CMake file with all build information #490

david-ragazzi commented Dec 11, 2013

david-ragazzi commented Dec 11, 2013

subutai commented Dec 13, 2013

breznak commented Dec 13, 2013

david-ragazzi commented Dec 14, 2013

david-ragazzi commented Dec 14, 2013

david-ragazzi commented Dec 15, 2013

rhyolight commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

sjmackenzie commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

iandanforth commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

rhyolight commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

subutai commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

breznak commented Dec 16, 2013

breznak Dec 16, 2013

david-ragazzi Dec 16, 2013

david-ragazzi commented Dec 16, 2013

breznak commented Dec 17, 2013

david-ragazzi commented Dec 17, 2013

david-ragazzi commented Dec 17, 2013

breznak commented Dec 17, 2013

david-ragazzi commented Dec 17, 2013

david-ragazzi commented Dec 19, 2013

david-ragazzi commented Dec 21, 2013

A single CMake file with all build information #490

A single CMake file with all build information #490

Conversation

david-ragazzi commented Dec 11, 2013

david-ragazzi commented Dec 11, 2013

subutai commented Dec 13, 2013

breznak commented Dec 13, 2013

david-ragazzi commented Dec 14, 2013

david-ragazzi commented Dec 14, 2013

david-ragazzi commented Dec 15, 2013

rhyolight commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

sjmackenzie commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

iandanforth commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

rhyolight commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

subutai commented Dec 16, 2013

david-ragazzi commented Dec 16, 2013

breznak commented Dec 16, 2013

breznak Dec 16, 2013

Choose a reason for hiding this comment

david-ragazzi Dec 16, 2013

Choose a reason for hiding this comment

david-ragazzi commented Dec 16, 2013

breznak commented Dec 17, 2013

david-ragazzi commented Dec 17, 2013

david-ragazzi commented Dec 17, 2013

breznak commented Dec 17, 2013

david-ragazzi commented Dec 17, 2013

david-ragazzi commented Dec 19, 2013

david-ragazzi commented Dec 21, 2013