How to load native linear algebra library

This post records how to load native linear algebra library successfully when you use breeze.

##Breeze, netlib-java, BLAS Breeze is a linear algebra library wrote in Scala, which now is the default linear algebra library used in Spark. Breeze uses netlib-jave to cross JVM to load native linear algebra, such as BLAS. If breeze can load native linear algebra library successfully, the performance can be promoted greatly.

However, although Spark use breeze as its default linear algebra library library, due to the LICENSES limit, Spark 1.0.x doesn't include all netlib-java components in pre-built assembly jar.(This condition may change as the Spark 1.1 release, here is the Spark PR to solve the problem).

##Solutions to load native linear algebra library As far as I know, the best way is rebuild spark assembly in your own environment.

The native_system.*.jar such as netlib-native_system-linux-x86_64-1.1-natives.jar in the maven central repository provided by the netlib-java author may not work, because the .so file included in the jar file may not fit in your own environment (The version it links to libblas.so.3 is much newer than that library on RHEL or CentOS), you'd better rebuild it follow these steps. Then replace them in your .ivy2 repository.

After that, you can follow the steps in the post with make-distribution.sh. In the end you can get your own re-built jar file. You can open the assembly jar file to see whether directory com.github.fommil.jniloader and netlib-native_system-xxx.so files are included.

To test whether having load native linear algebra library successfully ,you can test locally, and it can return info about whether load successfully.