Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML-44] [PIP] Update to oneAPI 2021.2 and Rework examples for validation #47

Merged
merged 62 commits into from
Apr 15, 2021
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
e920b98
remove hibench examples
xwu99 Mar 30, 2021
bce123d
Fix tbb linking
xwu99 Mar 31, 2021
04097cb
Add data
xwu99 Mar 31, 2021
b7a3073
Add env.sh.template
xwu99 Mar 31, 2021
bdbc4c7
Revise examples
xwu99 Mar 31, 2021
723a05a
Add ALS scala and modify als-pyspark.py
xwu99 Mar 31, 2021
bf00ffd
nit
xwu99 Mar 31, 2021
05681cc
Add build-all & run-all
xwu99 Mar 31, 2021
8717d3a
remove test-cluster/workloads and use examples for validation
xwu99 Mar 31, 2021
1eb3eb3
add setup-python3 and setup-cluster, fix paths
xwu99 Mar 31, 2021
6310e28
fix java home
xwu99 Mar 31, 2021
e904f38
add config ssh
xwu99 Mar 31, 2021
b118779
add config ssh
xwu99 Mar 31, 2021
f1310db
add config ssh
xwu99 Mar 31, 2021
ef919c2
add config ssh
xwu99 Mar 31, 2021
2aed68f
fix config-ssh
xwu99 Mar 31, 2021
c8594fe
fix config-ssh
xwu99 Mar 31, 2021
7b6aafc
fix config-ssh
xwu99 Mar 31, 2021
e7d65e6
fix config-ssh
xwu99 Mar 31, 2021
1042a15
fix config-ssh
xwu99 Mar 31, 2021
356e16f
fix config-ssh
xwu99 Mar 31, 2021
bd7eab2
set strict modes no
xwu99 Apr 1, 2021
6174a26
clear out comments
xwu99 Apr 1, 2021
b4cd169
Update oneCCL and oneDAL to oneAPI 2021.2.0, don't build oneCCL from …
xwu99 Apr 12, 2021
5d4197e
nit
xwu99 Apr 12, 2021
04aa72c
Fix install oneapi and source setvars
xwu99 Apr 12, 2021
59cddfc
nit
xwu99 Apr 12, 2021
c34e78e
Add spark.driver.host
xwu99 Apr 12, 2021
f541ce8
Add ci-build
xwu99 Apr 12, 2021
8738cbf
nit
xwu99 Apr 12, 2021
a1e786a
Update
xwu99 Apr 13, 2021
29de3dc
Update
xwu99 Apr 13, 2021
a12cffd
Add --ccl-configuration=cpu_icc
xwu99 Apr 13, 2021
572e308
Update
xwu99 Apr 13, 2021
fd59a2d
Update
xwu99 Apr 13, 2021
c05f52f
revert to build oneCCL from source and package related so
xwu99 Apr 13, 2021
e35c0d2
nit
xwu99 Apr 13, 2021
365b034
nit
xwu99 Apr 13, 2021
8fd68a2
Add ci-test-cluster
xwu99 Apr 13, 2021
6bc312e
update
xwu99 Apr 13, 2021
e1ffd2a
update
xwu99 Apr 13, 2021
848b02b
update
xwu99 Apr 13, 2021
37f6462
update
xwu99 Apr 14, 2021
c6dadf4
update
xwu99 Apr 14, 2021
7838283
Add check: OneCCL doesn't support loopback IP
xwu99 Apr 14, 2021
ec64f9e
update
xwu99 Apr 14, 2021
6385534
update
xwu99 Apr 14, 2021
30ab18b
update
xwu99 Apr 14, 2021
736dd21
update
xwu99 Apr 14, 2021
5138cdf
update
xwu99 Apr 14, 2021
4e1b138
update
xwu99 Apr 14, 2021
34823fd
update
xwu99 Apr 14, 2021
e8fb1a4
update
xwu99 Apr 14, 2021
f83cdac
update
xwu99 Apr 14, 2021
4aec7e0
update
xwu99 Apr 14, 2021
49f9140
update
xwu99 Apr 14, 2021
2a84d94
update README
xwu99 Apr 14, 2021
416c547
update
xwu99 Apr 14, 2021
041e87c
update
xwu99 Apr 14, 2021
108227a
update
xwu99 Apr 14, 2021
6422f8f
update
xwu99 Apr 15, 2021
57b5f55
Update README and nit changes
xwu99 Apr 15, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update
xwu99 committed Apr 14, 2021

Verified

This commit was signed with the committer’s verified signature.
frapell Franco Pellegrini
commit 37f6462606604e03c467cd74bcf4395886ec5f77
1 change: 1 addition & 0 deletions dev/ci-test.sh
Original file line number Diff line number Diff line change
@@ -31,6 +31,7 @@ echo =============================

cd $GITHUB_WORKSPACE/mllib-dal

# Build test
$GITHUB_WORKSPACE/dev/ci-build.sh

# Enable signal chaining support for JNI
4 changes: 2 additions & 2 deletions dev/test-cluster/ci-test-cluster.sh
Original file line number Diff line number Diff line change
@@ -15,8 +15,8 @@ cp $GITHUB_WORKSPACE/dev/test-cluster/env.sh $GITHUB_WORKSPACE/conf

cd $GITHUB_WORKSPACE/examples

hadoop fs -mkdir data
hadoop fs -copyFromLocal -f data
hadoop fs -mkdir -p /user/$USER
hadoop fs -copyFromLocal data
hadoop fs -ls data

./build-all.sh
6 changes: 5 additions & 1 deletion dev/test-cluster/setup-cluster.sh
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ cd $WORK_DIR

echo JAVA_HOME is $JAVA_HOME

mkdir ~/opt
[ -d ~/opt ] || mkdir ~/opt
cd ~/opt
[ -f spark-3.0.0-bin-hadoop2.7.tgz ] || wget --no-verbose https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
[ -d spark-3.0.0-bin-hadoop2.7 ] || tar -xzf spark-3.0.0-bin-hadoop2.7.tgz
@@ -30,7 +30,11 @@ mkdir -p /tmp/run/hdfs/datanode

export HADOOP_HOME=~/opt/hadoop-2.7.7
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native

export SPARK_HOME=~/opt/spark-3.0.0-bin-hadoop2.7
export PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH
export PYSPARK_PYTHON=python3

export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH