Migrate KMeans daal to DPC++ (oap-project#209)

* 1. add JNI wrapper 2. modify compiler config 3. add unit test Signed-off-by: minmingzhu <minming.zhu@intel.com> * add ci test Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * java code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * cpp code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-test.sh * Update ci-local-test.sh * Update install-build-deps-ubuntu.sh * Update install-build-deps-ubuntu.sh * Update ci-local-test.sh * Update ci-local-test.sh * Update ci-local-test.sh * Update install-build-deps-ubuntu.sh * Update install-build-deps-ubuntu.sh * Update install-build-deps-ubuntu.sh * Update install-build-deps-ubuntu.sh * Update install-build-deps-ubuntu.sh * Update prepare-build-deps.sh * add convert array, vector to homogentable Signed-off-by: minmingzhu <minming.zhu@intel.com> * 1. rollback ci config 2. remove CI test becaue homogentable only support GPU Signed-off-by: minmingzhu <minming.zhu@intel.com> * 1. update prepare-build-deps.sh Signed-off-by: minmingzhu <minming.zhu@intel.com> * remove ci GPU test Signed-off-by: minmingzhu <minming.zhu@intel.com> * 1. remove GPU CI test Signed-off-by: minmingzhu <minming.zhu@intel.com> * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * fix comments Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Makefile Signed-off-by: minmingzhu <minming.zhu@intel.com> * fix comments Signed-off-by: minmingzhu <minming.zhu@intel.com> * 1. fix ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update header Signed-off-by: minmingzhu <minming.zhu@intel.com> * update header Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-cluster-test.sh * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci conf Signed-off-by: minmingzhu <minming.zhu@intel.com> * update oneapi to 2022.1.2 Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci conf Signed-off-by: minmingzhu <minming.zhu@intel.com> * fix comment Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update build-jni.sh * update Makefile Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update test.sh * Update test.sh * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update test.sh * Update test.sh * update test.sh Signed-off-by: minmingzhu <minming.zhu@intel.com> * update test.sh Signed-off-by: minmingzhu <minming.zhu@intel.com> * update test.sh Signed-off-by: minmingzhu <minming.zhu@intel.com> * fix link libze_loader.so.1 Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci-local-dpc-test.sh Signed-off-by: minmingzhu <minming.zhu@intel.com> * update oneapi to 2022.1.2 Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-dpc-test.sh * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Makefile Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-dpc-test.sh * Update prepare-build-deps-gpu.sh * Update prepare-build-deps-gpu.sh * Update prepare-build-deps.sh * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-dpc-test.sh * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update install-build-level-zero-deps-ubuntun.sh * Update install-build-level-zero-deps-ubuntun.sh * Update install-build-level-zero-deps-ubuntun.sh * Update ci-local-dpc-test.sh * Update GPU.cpp * Update GPU.cpp * Update ci-build-test.sh * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update HomogenTableImpl.cpp Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-dpc-test.sh * Update ci-local-dpc-test.sh * Update ci-local-dpc-test.sh * update HomogenTable API Signed-off-by: minmingzhu <minming.zhu@intel.com> * update install-build-level-zero-deps-ubuntu.sh Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-dpc-test.sh * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-local-dpc-test.sh * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * update ci Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update run.sh * Update build-jni.sh * 1. Merge branch 'make_homogen_table' into convert_homogentable 2. update ci 3. add a new parameters for HomogenTable class Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update pom.xml * Update test.sh * Update Makefile * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ConvertHomogenTableSuite.scala * Update install-level-zero-ubuntu.sh * Update install-level-zero-ubuntu.sh * Update build.sh * Update test.sh * Update test.sh * Update HomogenTableImpl.cpp * Update install-build-deps-ubuntu.sh * Update HomogenTableImpl.cpp * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update install-build-deps-ubuntu.sh * fix comments Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update list-compute-devices.cpp * Convert homogentable to array/vector/matrix (#4) * 1. convert homogenTable to array/vector/matrix 2. add unit test Signed-off-by: minmingzhu <minming.zhu@intel.com> * Merge branch 'make_homogen_table' into convert_homogentable Signed-off-by: minmingzhu <minming.zhu@intel.com> * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * fix comments Signed-off-by: minmingzhu <minming.zhu@intel.com> * Convert RDD[Vector]/RDD[LabeledPoint] to HomogenTable (#8) * 1. convert rdd to HomogenTable * 2. add unit test Signed-off-by: minmingzhu <minming.zhu@intel.com> * add empty homogentable Signed-off-by: minmingzhu <minming.zhu@intel.com> * Mirgate kmeans daal to DPC++ (#7) Signed-off-by: minmingzhu <minming.zhu@intel.com> * 1. Mirgate kmeans daal to DPC++ * 2. add unit test * 1. move local host to distributed host 2. add cluster test 3. code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update ci-cluster-test.sh * Update ci-cluster-test.sh * Update build.sh * Update build.sh * Update OneDAL.cpp * Update OneDAL.cpp * Update OneDAL.cpp * Update com_intel_oap_mllib_OneDAL__.h * update code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * Update NaiveBayesResult.java * 1. update code style 2. When using the Host/CPU device throws a fail Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * rollback code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * rollback code style Signed-off-by: minmingzhu <minming.zhu@intel.com> * rollback checkstyle.xml Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * rollback checkstyle.xml Signed-off-by: minmingzhu <minming.zhu@intel.com> * update Signed-off-by: minmingzhu <minming.zhu@intel.com> * update OneDAL.scala Signed-off-by: minmingzhu <minming.zhu@intel.com> * revert ALS.scala * revert checkstyle.xml * :Revert "revert checkstyle.xml" This reverts commit 4f74bac. * disable cluster test for gpu * disable cluster test for gpu * nit Co-authored-by: minmingz <minming.zhu@inte.com> Co-authored-by: Wu, Xiaochang <xiaochang.wu@intel.com>
minmingzhu · Apr 23, 2023 · d02ac0f · d02ac0f
1 parent 9e8c13a
commit d02ac0f
Show file tree

Hide file tree

Showing 9 changed files with 10,233 additions and 0 deletions.
diff --git a/dev/ci/ci-build-test.sh b/dev/ci/ci-build-test.sh
@@ -8,6 +8,9 @@ trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG
 # echo an error message before exiting
 trap 'echo "\"${last_command}\" command filed with exit code $?."' EXIT
 
+# install level-zero
+$GITHUB_WORKSPACE/dev/install-level-zero-ubuntu.sh
+
 # Install dependencies for building
 $GITHUB_WORKSPACE/dev/install-build-deps-ubuntu.sh
 

diff --git a/dev/install-level-zero-ubuntu.sh b/dev/install-level-zero-ubuntu.sh
@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+echo "Installing level-zero components ..."
+sudo apt-get install -y gpg-agent wget
+wget -qO - https://repositories.intel.com/graphics/intel-graphics.key |
+sudo apt-key add -
+ sudo apt-add-repository \
+'deb [arch=amd64] https://repositories.intel.com/graphics/ubuntu focal main'
+sudo apt-get install -y \
+intel-opencl-icd \
+intel-level-zero-gpu level-zero 
diff --git a/dev/prepare-build-deps-gpu.sh b/dev/prepare-build-deps-gpu.sh
@@ -0,0 +1,80 @@
+#!/usr/bin/env bash
+
+# exit when any command fails
+set -e
+
+# keep track of the last executed command
+trap 'last_command=$current_command; current_command=$BASH_COMMAND' DEBUG
+# echo an error message before exiting
+trap 'echo "\"${last_command}\" command filed with exit code $?."' EXIT
+
+if [ -z ${ONEAPI_ROOT} ]; then
+  echo Please source Intel oneAPI Toolkit environments!
+  exit 1
+fi
+
+if [[ -z $DAALROOT ]]; then
+  echo DAALROOT not defined!
+  exit 1
+fi
+
+if [[ -z $TBBROOT ]]; then
+  echo TBBROOT not defined!
+  exit 1
+fi
+
+if [[ -z $I_MPI_ROOT ]]; then
+  echo I_MPI_ROOT not defined!
+  exit 1
+fi
+
+if [[ -z $CCL_ROOT ]]; then
+  echo CCL_ROOT not defined!
+  exit 1
+fi
+
+# Use patchelf to change SONAME for libfabric
+if [[ -z $(which patchelf) ]]; then
+  echo Please install \"patchelf\"!
+  exit 1
+fi
+
+if [[ $(basename $(pwd)) != "mllib-dal" ]]; then
+  echo Please execute the script from \"mllib-dal\" directory!
+  exit 1
+fi
+
+TARGET_DIR=./src/main/resources/lib
+
+rm -f $TARGET_DIR/*.so*
+
+cp $CCL_ROOT/lib/cpu_gpu_dpcpp/libccl.so.1.0 $TARGET_DIR/libccl.so.1
+
+
+cp $I_MPI_ROOT/libfabric/lib/libfabric.so.1 $TARGET_DIR/libfabric.so.1
+cp $I_MPI_ROOT/libfabric/lib/prov/libsockets-fi.so $TARGET_DIR
+
+# Workaround dlopen (libfabric.so) in oneCCL
+cp $I_MPI_ROOT/libfabric/lib/libfabric.so.1 $TARGET_DIR/libfabric.so
+patchelf --set-soname libfabric.so $TARGET_DIR/libfabric.so
+
+cp $I_MPI_ROOT/lib/release_mt/libmpi.so.12.0.0 $TARGET_DIR/libmpi.so.12
+
+cp $DAALROOT/lib/intel64/libJavaAPI.so.1.1 $TARGET_DIR/libJavaAPI.so
+
+cp $TBBROOT/lib/intel64/gcc4.8/libtbb.so.12.5 $TARGET_DIR/libtbb.so.12
+cp $TBBROOT/lib/intel64/gcc4.8/libtbbmalloc.so.2.5 $TARGET_DIR/libtbbmalloc.so.2
+
+# SYCL libs
+cp $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libintlc.so.5 $TARGET_DIR
+cp $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libsvml.so $TARGET_DIR
+
+# Workaround lib loading for JNI as libirng.so doesn't have soname
+cp $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libirng.so $TARGET_DIR
+patchelf --set-soname libirng.so $TARGET_DIR/libirng.so
+
+cp $CMPLR_ROOT/linux/compiler/lib/intel64_lin/libimf.so $TARGET_DIR
+cp $CMPLR_ROOT/linux/lib/libOpenCL.so.1 $TARGET_DIR
+cp $CMPLR_ROOT/linux/lib/libsycl.so.5 $TARGET_DIR
+
+echo oneAPI Toolkit version: $(basename $CCL_ROOT) > $TARGET_DIR/VERSION
diff --git a/mllib-dal/src/main/java/com/intel/oneapi/dal/table/HomogenTableImpl.java b/mllib-dal/src/main/java/com/intel/oneapi/dal/table/HomogenTableImpl.java
@@ -15,6 +15,7 @@ public class HomogenTableImpl implements HomogenTableIface {
     private long cObject;
     private TableMetadata metadata;
     private Common.ComputeDevice device;
+
     protected HomogenTableImpl(Common.ComputeDevice computeDevice) {
         super();
         this.device = computeDevice;

diff --git a/mllib-dal/src/main/native/KMeansOneAPIImpl.cpp b/mllib-dal/src/main/native/KMeansOneAPIImpl.cpp
@@ -0,0 +1,122 @@
+/*******************************************************************************
+ * Copyright 2020 Intel Corporation
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *******************************************************************************/
+
+#include <chrono>
+#include <iomanip>
+#include <iostream>
+#include <mutex>
+
+#ifdef CPU_GPU_PROFILE
+#include "GPU.h"
+#ifndef ONEDAL_DATA_PARALLEL
+#define ONEDAL_DATA_PARALLEL
+#endif
+
+#include "Communicator.hpp"
+#include "com_intel_oap_mllib_clustering_KMeansDALImpl.h"
+#include "oneapi/dal/algo/kmeans.hpp"
+#include "oneapi/dal/table/homogen.hpp"
+#include "service.h"
+
+using namespace std;
+using namespace oneapi::dal;
+const int ccl_root = 0;
+
+typedef std::shared_ptr<homogen_table> homogenPtr;
+
+std::mutex kmtx;
+std::vector<homogenPtr> cVector;
+
+static void saveShareHomogenPtrVector(const homogenPtr &ptr) {
+    kmtx.lock();
+    cVector.push_back(ptr);
+    kmtx.unlock();
+}
+
+static jlong doKMeansOneAPICompute(JNIEnv *env, jint rankId, jlong pNumTabData,
+                                   jlong pNumTabCenters, jint cluster_num,
+                                   jdouble tolerance, jint iteration_num,
+                                   jint executor_num, const ccl::string &ipPort,
+                                   jint cComputeDevice, jobject resultObj) {
+    std::cout << "oneDAL (native): OneAPI compute start , rankid %ld " << rankId
+              << std::endl;
+    const bool isRoot = (rankId == ccl_root);
+    compute_device device = getComputeDevice(cComputeDevice);
+    homogen_table htable =
+        *reinterpret_cast<const homogen_table *>(pNumTabData);
+    homogen_table centroids =
+        *reinterpret_cast<const homogen_table *>(pNumTabCenters);
+    const auto kmeans_desc = kmeans::descriptor<>()
+                                 .set_cluster_count(cluster_num)
+                                 .set_max_iteration_count(iteration_num)
+                                 .set_accuracy_threshold(tolerance);
+    kmeans::train_input local_input{htable, centroids};
+    auto queue = getQueue(device);
+    auto comm = preview::spmd::make_communicator<preview::spmd::backend::ccl>(
+        queue, executor_num, rankId, ipPort);
+    kmeans::train_result result_train =
+        preview::train(comm, kmeans_desc, local_input);
+    if (isRoot) {
+        std::cout << "iteration_num: " << iteration_num << std::endl;
+        std::cout << "Iteration count: " << result_train.get_iteration_count()
+                  << std::endl;
+        std::cout << "Objective function value: "
+                  << result_train.get_objective_function_value() << std::endl;
+        // Get the class of the input object
+        jclass clazz = env->GetObjectClass(resultObj);
+        // Get Field references
+        jfieldID totalCostField = env->GetFieldID(clazz, "totalCost", "D");
+        jfieldID iterationNumField =
+            env->GetFieldID(clazz, "iterationNum", "I");
+        // Set iteration num for result
+        env->SetIntField(resultObj, iterationNumField,
+                         result_train.get_iteration_count());
+        // Set cost for result
+        env->SetDoubleField(resultObj, totalCostField,
+                            result_train.get_objective_function_value());
+
+        homogenPtr centroidsPtr = std::make_shared<homogen_table>(
+            result_train.get_model().get_centroids());
+        saveShareHomogenPtrVector(centroidsPtr);
+        return (jlong)centroidsPtr.get();
+    } else {
+        return (jlong)0;
+    }
+}
+
+/*
+ * Class:     com_intel_oap_mllib_clustering_KMeansDALImpl
+ * Method:    cKMeansOneapiComputeWithInitCenters
+ * Signature: (JJIDIIILcom/intel/oap/mllib/clustering/KMeansResult;)J
+ */
+JNIEXPORT jlong JNICALL
+Java_com_intel_oap_mllib_clustering_KMeansDALImpl_cKMeansOneapiComputeWithInitCenters(
+    JNIEnv *env, jobject obj, jlong pNumTabData, jlong pNumTabCenters,
+    jint cluster_num, jdouble tolerance, jint iteration_num, jint executor_num,
+    jint cComputeDevice, jint rankId, jstring ip_port, jobject resultObj) {
+    std::cout << "oneDAL (native): use GPU DPC++ kernels with " << std::endl;
+    const char *ipport = env->GetStringUTFChars(ip_port, 0);
+    std::string ipPort = std::string(ipport);
+    jlong ret = 0L;
+    printf("oneDAL (native):  KMeansOneapiComputeWithInitCenters %d \n",
+           cComputeDevice);
+    ret = doKMeansOneAPICompute(
+        env, rankId, pNumTabData, pNumTabCenters, cluster_num, tolerance,
+        iteration_num, executor_num, ipPort, cComputeDevice, resultObj);
+    env->ReleaseStringUTFChars(ip_port, ipport);
+    return ret;
+}
+#endif
diff --git a/mllib-dal/src/main/native/Makefile b/mllib-dal/src/main/native/Makefile
@@ -49,6 +49,8 @@ endif
 INCS := -I $(JAVA_HOME)/include \
         -I $(JAVA_HOME)/include/linux \
         -I $(DAALROOT)/include \
+        -I /opt/rh/devtoolset-7/root/usr/include/c++/7 \
+        -I /opt/rh/devtoolset-7/root/usr/include/c++/7/x86_64-redhat-linux \
         -I ./javah \
         -I ./
 
@@ -118,6 +120,7 @@ OBJS += \
 
 DEFINES=-D$(PLATFORM_PROFILE)
 
+
 ifeq ($(PLATFORM_PROFILE),CPU_GPU_PROFILE)
   CPP_SRCS += ./GPU.cpp
   OBJS += ./GPU.o

diff --git a/mllib-dal/src/main/scala/com/intel/oap/mllib/clustering/KMeansDALImpl.scala b/mllib-dal/src/main/scala/com/intel/oap/mllib/clustering/KMeansDALImpl.scala
@@ -17,7 +17,11 @@
 package com.intel.oap.mllib.clustering
 
 import com.intel.oap.mllib.Utils.getOneCCLIPPort
+<<<<<<< HEAD
 import com.intel.oap.mllib.{OneCCL, OneDAL, Utils}
+=======
+import com.intel.oap.mllib.{OneCCL, OneDAL}
+>>>>>>> Migrate KMeans  daal to DPC++ (#209)
 import com.intel.oneapi.dal.table.Common
 import org.apache.spark.TaskContext
 import org.apache.spark.internal.Logging

diff --git a/mllib-dal/src/test/scala/data/kmeans_dense_train_centroids.csv b/mllib-dal/src/test/scala/data/kmeans_dense_train_centroids.csv
@@ -0,0 +1,10 @@
+-31.504,-73.089,-38.436,-94.426,-74.766,-67.435,4.703,6.929,6.232,14.258,-32.189,46.390,36.269,49.638,-30.474,90.697,-53.270,-55.981,-76.285,107.620
+28.634,55.676,-8.076,-59.468,-58.356,-59.880,89.226,3.375,88.767,91.344,31.533,45.158,89.224,-77.296,7.158,-27.333,88.878,-29.220,-77.884,-61.412
+-7.891,-48.543,-39.840,-113.667,-69.113,-75.884,3.101,108.083,-13.944,11.159,-20.248,63.385,43.436,64.571,-45.034,43.182,-83.274,-44.630,-103.643,75.750
+-47.760,39.908,-72.153,55.267,-21.681,98.050,52.326,22.514,10.478,-20.894,76.173,-15.170,44.295,64.886,-43.488,24.998,-38.863,-14.418,4.036,-94.388
+-0.012,25.333,15.920,102.457,88.853,17.935,41.462,-68.552,64.139,124.873,8.922,103.287,-53.089,65.990,-36.423,15.801,10.300,38.470,-14.507,4.895
+-55.098,-38.414,19.163,31.976,-59.315,-97.125,-43.348,72.491,-19.681,34.969,-29.560,-22.753,-25.411,-90.476,-81.657,17.736,68.008,-76.612,-38.012,-7.251
+-4.233,100.762,-80.907,-6.668,-22.292,-53.800,25.021,52.130,-62.667,22.358,-22.819,98.560,70.029,-36.080,-12.062,-55.113,19.833,84.475,54.979,-81.076
+-4.680,-46.987,-87.867,101.028,-112.063,-57.149,78.864,64.595,-16.539,-30.937,34.636,-49.476,24.540,26.411,-79.789,93.406,-48.819,42.862,-61.290,-83.510
+30.809,37.382,-84.056,-9.649,-12.509,-83.705,47.871,58.416,-71.769,13.849,-29.367,104.843,84.317,-10.185,-54.414,-61.872,29.629,77.375,49.537,-101.110
+-1.990,-40.990,46.589,-3.508,9.232,50.130,-32.612,19.911,12.406,13.907,52.201,14.932,-27.741,-1.314,104.201,-59.687,-82.769,12.405,2.996,68.074