Wave Direction
By Howard Wang
"The method I came up with to calculate and determine the direction of waves in the general surf region can be described simply as a physical analysis of the overall movement of the Smartfin, currently using buoy calibrated data for controlled behaviour. Basically, the movement of the Smartfin emulates the movement of the waves." - Howard Wang
# Howard Wang 05/14/19
# CSE 145: Embedded systems and design
# Main Database: https://surf.smartfin.org/
# Analyzing data from Buoy Calibration experiment to get wave direction.
# First, parse the data from the .CSV file containing ocean and wave motion data.
# This data comes from a controlled experiment (CE3), so we are assuming that
# all of the vertical accelerations are contained in IMUA2.
# MATPLOTLIB
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.animation as animation
#from mpl_toolkits.basemap import Basemap
# DATAFRAMES
import pandas as pd
import numpy as np
# SCIPY
from scipy import stats
from scipy import constants
from scipy import signal #added
from scipy.interpolate import CubicSpline
from scipy.interpolate import interp1d
from scipy.integrate import simps
from scipy.integrate import cumtrapz
import pylab as pylab
# SYSTEM and CONVERSION TOOLS
import math
import abc
import sys
import csv
import io
import os
import datetime
import pytz
import re
# MODELING AND GRAPHS
import peakutils
import statsmodels.api as sm
# URL REQUESTS
import requests
# VECTORS AND GRAPHICS
import mpld3
import folium
# import cmocean
import skinematics as skin
from skinematics import quat, vector, misc, rotmat, imus, view
import pygame
# PLOTTING TOOLS
from plotly import tools
import plotly.offline
import plotly.graph_objs as go
%matplotlib notebook
%matplotlib inline
print("Done!")Done!
ride_ids = ['14888']
# 14743 - Motion Control July 10th
# 14750 - Magnetometer Control July 11th
# 14814 - Pool Displacement Control July 17th
# 14815 - Compass Orientation (Lying on Charger Side) July 19th
# 14816 - Orientation w Higher Sampling (Lying on Charger Side) July 20th
# 14827 - Pool Displacement Control w Higher Sampling (Jul 23)
# 14888 - First Buoy Calibration Experiment (July 30)
print("Done!")Done!
#%% Fin ID scraper
# Input fin ID, get all ride IDs
# base URL to which we'll append given fin IDs
fin_url_base = 'http://surf.smartfin.org/fin/'
# Look for the following text in the HTML contents in fcn below
str_id_ride = 'rideId = \'' # backslash allows us to look for single quote
str_id_date = 'var date = \'' # backslash allows us to look for single quote
#%% Ride ID scraper
# Input ride ID, get ocean and motion CSVs
# Base URL to which we'll append given ride IDs
ride_url_base = 'https://surf.smartfin.org/ride/'
# Look for the following text in the HTML contents in fcn below
str_id_csv = 'img id="temperatureChart" class="chart" src="'
def get_csv_from_ride_id(rid):
# Build URL for each individual ride
ride_url = ride_url_base+str(rid)
print(ride_url)
# Get contents of ride_url
html_contents = requests.get(ride_url).text
# Find CSV identifier
loc_csv_id = html_contents.find(str_id_csv)
# Different based on whether user logged in with FB or Google
offset_googleOAuth = [46, 114]
offset_facebkOAuth = [46, 112]
if html_contents[loc_csv_id+59] == 'f': # Facebook login
off0 = offset_facebkOAuth[0]
off1 = offset_facebkOAuth[1]
else: # Google login
off0 = offset_googleOAuth[0]
off1 = offset_googleOAuth[1]
csv_id_longstr = html_contents[loc_csv_id+off0:loc_csv_id+off1]
# Stitch together full URL for CSV
if ("media" in csv_id_longstr) & ("Calibration" not in html_contents): # other junk URLs can exist and break everything
ocean_csv_url = 'https://surf.smartfin.org/'+csv_id_longstr+'Ocean.CSV'
motion_csv_url = 'https://surf.smartfin.org/'+csv_id_longstr+'Motion.CSV'
print(ocean_csv_url)
# Go to ocean_csv_url and grab contents (theoretically, a CSV)
oceanCSV = requests.get(ocean_csv_url).content
ocean_df_small = pd.read_csv(io.StringIO(oceanCSV.decode('utf-8')))
# Grab CSV from motion url
motionCSV = requests.get(motion_csv_url).content
motion_df_small = pd.read_csv(io.StringIO(motionCSV.decode('utf-8')))
return ocean_df_small, motion_df_small
else:
ocean_df_small_resample = pd.DataFrame() # empty DF just so something is returned
motion_df_small_resample = pd.DataFrame()
return ocean_df_small_resample, motion_df_small_resample
print("Done!")Done!
appended_ocean_list = [] # list of DataFrames from original CSVs
appended_motion_list = []
appended_multiIndex = [] # fin_id & ride_id used to identify each DataFrame
## Nested loops (for each fin ID, find all ride IDs, then build a DataFrame from all ride CSVs)
## (Here, ride IDS are either ocean or motion dataframes)
count_good_fins = 0
# Loop over ride_ids and find CSVs
for rid in ride_ids:
try:
new_ocean_df, new_motion_df = get_csv_from_ride_id(rid) # get given ride's CSV from its ride ID using function above
#print(len(new_ocean_df))
#print(len(new_motion_df))
if not new_ocean_df.empty: # Calibration rides, for example
# Append only if DF isn't empty. There may be a better way to control empty DFs which are created above
appended_multiIndex.append(str(rid)) # build list to be multiIndex of future DataFrame
appended_ocean_list.append(new_ocean_df)
appended_motion_list.append(new_motion_df)
print("Ride data has been uploaded.")
#print("Ride: ", rid, "data has been uploaded.")
count_good_fins += 1
except:
print("Ride threw an exception!")
#print("Ride ", rid, "threw an exception!")
#%% Build the "Master" DataFrame
# appended_ocean_df.summary()
df_keys = tuple(appended_multiIndex) # keys gotta be a tuple, a list which data in it cannot be changed
ocean_df = pd.concat(appended_ocean_list, keys = df_keys, names=['ride_id'])
motion_df = pd.concat(appended_motion_list, keys = df_keys, names = ['ride_id'])
# Print motion and Ocean dataframes
print(motion_df)
print(ocean_df)https://surf.smartfin.org/ride/14888
https://surf.smartfin.org/media/201807/google_117589279598321562176_000666D321BE_180730171718_Ocean.CSV
Ride data has been uploaded.
UTC Time IMU A1 IMU A2 \
ride_id
14888 0 2018-07-30T17:17:24.3320+00:00 2067196596 -34.0 -520.0
1 2018-07-30T17:17:24.3650+00:00 2067196629 -30.0 -517.0
2 2018-07-30T17:17:24.3990+00:00 2067196662 -28.0 -517.0
3 2018-07-30T17:17:24.4320+00:00 2067196695 -28.0 -518.0
4 2018-07-30T17:17:24.4650+00:00 2067196728 -30.0 -517.0
5 2018-07-30T17:17:24.4980+00:00 2067196761 -31.0 -518.0
6 2018-07-30T17:17:24.5310+00:00 2067196794 -33.0 -520.0
7 2018-07-30T17:17:24.5650+00:00 2067196828 -30.0 -519.0
8 2018-07-30T17:17:24.5970+00:00 2067196860 -30.0 -517.0
9 2018-07-30T17:17:24.6310+00:00 2067196894 -29.0 -517.0
10 2018-07-30T17:17:24.6630+00:00 2067196926 -30.0 -518.0
11 2018-07-30T17:17:24.6960+00:00 2067196959 -32.0 -520.0
12 2018-07-30T17:17:24.7300+00:00 2067196992 -32.0 -518.0
13 2018-07-30T17:17:24.7630+00:00 2067197025 -30.0 -517.0
14 2018-07-30T17:17:24.7960+00:00 2067197058 -28.0 -516.0
15 2018-07-30T17:17:24.8290+00:00 2067197091 -27.0 -516.0
16 2018-07-30T17:17:24.8620+00:00 2067197124 -31.0 -518.0
17 2018-07-30T17:17:24.8950+00:00 2067197157 -34.0 -518.0
18 2018-07-30T17:17:24.9290+00:00 2067197191 -33.0 -519.0
19 2018-07-30T17:17:24.9610+00:00 2067197223 -30.0 -517.0
20 2018-07-30T17:17:24.9950+00:00 2067197257 -27.0 -516.0
21 2018-07-30T17:17:25.0280+00:00 2067197290 -28.0 -516.0
22 2018-07-30T17:17:25.0620+00:00 2067197323 7.0 -516.0
23 2018-07-30T17:17:25.0950+00:00 2067197356 -50.0 -518.0
24 2018-07-30T17:17:25.1280+00:00 2067197389 -38.0 -517.0
25 2018-07-30T17:17:25.1610+00:00 2067197422 -39.0 -516.0
26 2018-07-30T17:17:25.1940+00:00 2067197455 -29.0 -517.0
27 2018-07-30T17:17:25.2280+00:00 2067197489 -29.0 -517.0
28 2018-07-30T17:17:25.2600+00:00 2067197521 -29.0 -517.0
29 2018-07-30T17:17:25.2930+00:00 2067197554 -30.0 -517.0
... ... ... ... ...
7559 2018-07-30T17:30:50.4950+00:00 2068000375 -58.0 -553.0
7560 2018-07-30T17:30:50.6940+00:00 2068000574 -65.0 -549.0
7561 2018-07-30T17:30:50.8930+00:00 2068000772 -73.0 -539.0
7562 2018-07-30T17:30:51.0920+00:00 2068000971 -73.0 -530.0
7563 2018-07-30T17:30:51.2910+00:00 2068001169 -76.0 -522.0
7564 2018-07-30T17:30:51.4910+00:00 2068001368 -78.0 -510.0
7565 2018-07-30T17:30:51.6900+00:00 2068001567 -73.0 -497.0
7566 2018-07-30T17:30:51.8890+00:00 2068001765 -73.0 -494.0
7567 2018-07-30T17:30:52.0980+00:00 2068001974 -64.0 -480.0
7568 2018-07-30T17:30:52.2970+00:00 2068002172 -57.0 -480.0
7569 2018-07-30T17:30:52.4960+00:00 2068002371 -50.0 -477.0
7570 2018-07-30T17:30:52.6960+00:00 2068002570 -40.0 -472.0
7571 2018-07-30T17:30:52.8950+00:00 2068002768 -32.0 -469.0
7572 2018-07-30T17:30:53.0930+00:00 2068002966 -20.0 -469.0
7573 2018-07-30T17:30:53.3030+00:00 2068003175 -14.0 -472.0
7574 2018-07-30T17:30:53.5020+00:00 2068003374 -4.0 -475.0
7575 2018-07-30T17:30:53.7020+00:00 2068003573 3.0 -480.0
7576 2018-07-30T17:30:53.9010+00:00 2068003771 8.0 -491.0
7577 2018-07-30T17:30:54.1000+00:00 2068003970 12.0 -498.0
7578 2018-07-30T17:30:54.3000+00:00 2068004169 14.0 -509.0
7579 2018-07-30T17:30:54.4980+00:00 2068004367 17.0 -514.0
7580 2018-07-30T17:30:54.6980+00:00 2068004566 16.0 -523.0
7581 2018-07-30T17:30:54.9080+00:00 2068004775 14.0 -530.0
7582 2018-07-30T17:30:54.9970+00:00 2068004864 NaN NaN
7583 2018-07-30T17:31:00.4840+00:00 2068010335 NaN NaN
7584 2018-07-30T17:31:05.5020+00:00 2068015338 NaN NaN
7585 2018-07-30T17:31:10.9830+00:00 2068020803 NaN NaN
7586 2018-07-30T17:31:16.0010+00:00 2068025806 NaN NaN
7587 2018-07-30T17:31:21.4850+00:00 2068031274 NaN NaN
7588 2018-07-30T17:31:26.5000+00:00 2068036274 NaN NaN
IMU A3 IMU G1 IMU G2 IMU G3 IMU M1 IMU M2 IMU M3 \
ride_id
14888 0 -48.0 82.0 -19.0 12.0 208.0 394.0 438.0
1 -57.0 32.0 -14.0 9.0 220.0 400.0 438.0
2 -59.0 -7.0 -11.0 6.0 218.0 398.0 430.0
3 -62.0 -27.0 -9.0 5.0 210.0 400.0 428.0
4 -62.0 64.0 -13.0 9.0 212.0 394.0 426.0
5 -54.0 157.0 -24.0 17.0 215.0 399.0 433.0
6 -41.0 29.0 -14.0 9.0 220.0 400.0 432.0
7 -46.0 -101.0 -3.0 2.0 216.0 406.0 432.0
8 -55.0 -121.0 -1.0 1.0 213.0 401.0 429.0
9 -63.0 -26.0 -8.0 6.0 218.0 396.0 428.0
10 -57.0 -7.0 -11.0 5.0 209.0 407.0 427.0
11 -54.0 -3.0 -11.0 6.0 212.0 398.0 440.0
12 -57.0 -22.0 -8.0 7.0 220.0 400.0 432.0
13 -60.0 -27.0 -9.0 6.0 208.0 400.0 434.0
14 -58.0 11.0 -12.0 9.0 216.0 396.0 430.0
15 -58.0 -4.0 -10.0 6.0 213.0 403.0 437.0
16 -55.0 -32.0 -8.0 4.0 211.0 405.0 441.0
17 -62.0 -71.0 -7.0 3.0 209.0 403.0 437.0
18 -62.0 -32.0 -9.0 5.0 203.0 395.0 433.0
19 -66.0 -68.0 14.0 5.0 213.0 395.0 441.0
20 -66.0 -97.0 25.0 3.0 218.0 392.0 436.0
21 -68.0 -60.0 10.0 6.0 216.0 392.0 440.0
22 -71.0 -24.0 4.0 5.0 217.0 397.0 441.0
23 -73.0 -47.0 13.0 5.0 217.0 399.0 445.0
24 -72.0 -43.0 6.0 8.0 210.0 396.0 442.0
25 -74.0 -29.0 -1.0 7.0 219.0 389.0 441.0
26 -73.0 -7.0 -11.0 6.0 214.0 396.0 448.0
27 -75.0 -53.0 9.0 6.0 210.0 402.0 440.0
28 -77.0 -40.0 6.0 7.0 208.0 392.0 442.0
29 -76.0 -11.0 -9.0 5.0 224.0 386.0 440.0
... ... ... ... ... ... ... ...
7559 -65.0 -9.0 -10.0 5.0 199.0 405.0 439.0
7560 -66.0 -7.0 -10.0 5.0 199.0 409.0 459.0
7561 -65.0 -8.0 -10.0 4.0 201.0 395.0 469.0
7562 -63.0 -9.0 -9.0 3.0 205.0 387.0 467.0
7563 -61.0 -8.0 -9.0 4.0 197.0 381.0 465.0
7564 -63.0 -8.0 -9.0 4.0 209.0 377.0 457.0
7565 -65.0 -9.0 -10.0 5.0 201.0 383.0 445.0
7566 -62.0 -8.0 -9.0 5.0 211.0 377.0 441.0
7567 -60.0 -7.0 -9.0 5.0 208.0 384.0 438.0
7568 -57.0 -6.0 -9.0 6.0 204.0 374.0 434.0
7569 -62.0 -7.0 -10.0 5.0 207.0 385.0 435.0
7570 -56.0 -6.0 -10.0 6.0 210.0 396.0 432.0
7571 -56.0 -6.0 -11.0 4.0 203.0 397.0 429.0
7572 -56.0 -7.0 -11.0 5.0 196.0 396.0 438.0
7573 -56.0 -6.0 -11.0 8.0 205.0 389.0 433.0
7574 -58.0 -7.0 -12.0 5.0 211.0 397.0 429.0
7575 -57.0 -6.0 -12.0 4.0 201.0 395.0 429.0
7576 -58.0 -6.0 -12.0 4.0 210.0 392.0 432.0
7577 -57.0 -6.0 -12.0 5.0 216.0 402.0 426.0
7578 -55.0 -6.0 -13.0 7.0 208.0 394.0 424.0
7579 -59.0 -7.0 -12.0 7.0 207.0 389.0 433.0
7580 -56.0 -8.0 -11.0 6.0 213.0 395.0 429.0
7581 -60.0 -8.0 -13.0 5.0 212.0 400.0 424.0
7582 NaN NaN NaN NaN NaN NaN NaN
7583 NaN NaN NaN NaN NaN NaN NaN
7584 NaN NaN NaN NaN NaN NaN NaN
7585 NaN NaN NaN NaN NaN NaN NaN
7586 NaN NaN NaN NaN NaN NaN NaN
7587 NaN NaN NaN NaN NaN NaN NaN
7588 NaN NaN NaN NaN NaN NaN NaN
Latitude Longitude
ride_id
14888 0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 NaN NaN
11 NaN NaN
12 NaN NaN
13 NaN NaN
14 NaN NaN
15 NaN NaN
16 NaN NaN
17 NaN NaN
18 NaN NaN
19 NaN NaN
20 NaN NaN
21 NaN NaN
22 NaN NaN
23 NaN NaN
24 NaN NaN
25 NaN NaN
26 NaN NaN
27 NaN NaN
28 NaN NaN
29 NaN NaN
... ... ...
7559 NaN NaN
7560 NaN NaN
7561 NaN NaN
7562 NaN NaN
7563 NaN NaN
7564 NaN NaN
7565 NaN NaN
7566 NaN NaN
7567 NaN NaN
7568 NaN NaN
7569 NaN NaN
7570 NaN NaN
7571 NaN NaN
7572 NaN NaN
7573 NaN NaN
7574 NaN NaN
7575 NaN NaN
7576 NaN NaN
7577 NaN NaN
7578 NaN NaN
7579 NaN NaN
7580 NaN NaN
7581 NaN NaN
7582 3286943.0 -11725208.0
7583 3286942.0 -11725208.0
7584 3286941.0 -11725207.0
7585 3286941.0 -11725206.0
7586 3286941.0 -11725206.0
7587 3286942.0 -11725206.0
7588 3286943.0 -11725207.0
[7589 rows x 13 columns]
UTC Time Temperature 1 \
ride_id
14888 0 2018-07-30T17:17:23.8620+00:00 2067196127 463
1 2018-07-30T17:17:29.9020+00:00 2067202149 464
2 2018-07-30T17:17:35.9400+00:00 2067208169 464
3 2018-07-30T17:17:41.9790+00:00 2067214190 464
4 2018-07-30T17:17:48.0180+00:00 2067220212 464
5 2018-07-30T17:17:54.0550+00:00 2067226231 464
6 2018-07-30T17:18:00.0930+00:00 2067232251 464
7 2018-07-30T17:18:06.1310+00:00 2067238271 464
8 2018-07-30T17:18:12.1690+00:00 2067244291 464
9 2018-07-30T17:18:18.2070+00:00 2067250311 464
10 2018-07-30T17:18:24.2440+00:00 2067256331 465
11 2018-07-30T17:18:30.2830+00:00 2067262352 465
12 2018-07-30T17:18:36.3230+00:00 2067268374 465
13 2018-07-30T17:18:42.3610+00:00 2067274394 465
14 2018-07-30T17:18:48.3980+00:00 2067280413 465
15 2018-07-30T17:18:54.4350+00:00 2067286432 466
16 2018-07-30T17:19:00.4730+00:00 2067292452 466
17 2018-07-30T17:19:06.5100+00:00 2067298472 466
18 2018-07-30T17:19:12.5480+00:00 2067304492 465
19 2018-07-30T17:19:18.5890+00:00 2067310515 465
20 2018-07-30T17:19:24.6280+00:00 2067316536 466
21 2018-07-30T17:19:30.6670+00:00 2067322557 466
22 2018-07-30T17:19:36.7050+00:00 2067328577 466
23 2018-07-30T17:19:42.7430+00:00 2067334597 467
24 2018-07-30T17:19:48.7800+00:00 2067340617 467
25 2018-07-30T17:19:54.8210+00:00 2067346640 467
26 2018-07-30T17:20:00.8600+00:00 2067352661 468
27 2018-07-30T17:20:06.8980+00:00 2067358681 468
28 2018-07-30T17:20:12.9350+00:00 2067364700 468
29 2018-07-30T17:20:18.9740+00:00 2067370721 468
... ... ... ...
110 2018-07-30T17:28:28.0700+00:00 2067858371 465
111 2018-07-30T17:28:34.1090+00:00 2067864393 465
112 2018-07-30T17:28:40.1460+00:00 2067870412 465
113 2018-07-30T17:28:46.1850+00:00 2067876433 464
114 2018-07-30T17:28:52.2230+00:00 2067882453 464
115 2018-07-30T17:28:58.2610+00:00 2067888473 464
116 2018-07-30T17:29:04.3000+00:00 2067894494 464
117 2018-07-30T17:29:10.3380+00:00 2067900515 465
118 2018-07-30T17:29:16.3760+00:00 2067906535 464
119 2018-07-30T17:29:22.4140+00:00 2067912555 465
120 2018-07-30T17:29:28.4510+00:00 2067918574 464
121 2018-07-30T17:29:34.4900+00:00 2067924595 464
122 2018-07-30T17:29:40.5290+00:00 2067930616 465
123 2018-07-30T17:29:46.5680+00:00 2067936637 464
124 2018-07-30T17:29:52.6040+00:00 2067942656 464
125 2018-07-30T17:29:58.6410+00:00 2067948675 464
126 2018-07-30T17:30:04.6800+00:00 2067954696 464
127 2018-07-30T17:30:10.7180+00:00 2067960716 464
128 2018-07-30T17:30:16.7560+00:00 2067966736 464
129 2018-07-30T17:30:22.7940+00:00 2067972756 464
130 2018-07-30T17:30:28.8350+00:00 2067978779 464
131 2018-07-30T17:30:34.8720+00:00 2067984799 464
132 2018-07-30T17:30:40.9120+00:00 2067990821 465
133 2018-07-30T17:30:46.9500+00:00 2067996841 465
134 2018-07-30T17:30:52.9890+00:00 2068002862 465
135 2018-07-30T17:30:59.0290+00:00 2068008884 465
136 2018-07-30T17:31:05.0680+00:00 2068014905 465
137 2018-07-30T17:31:11.1050+00:00 2068020925 465
138 2018-07-30T17:31:17.1430+00:00 2068026945 465
139 2018-07-30T17:31:23.1810+00:00 2068032965 465
Calibrated Temperature 1 Temperature 1 Stable Temperature 2 \
ride_id
14888 0 28.938 False 7189
1 29.000 False 7307
2 29.000 False 7284
3 29.000 False 7276
4 29.000 False 7268
5 29.000 False 7265
6 29.000 False 7256
7 29.000 False 7256
8 29.000 False 7253
9 29.000 False 7251
10 29.062 False 7249
11 29.062 False 7287
12 29.062 False 7339
13 29.062 False 7359
14 29.062 False 7358
15 29.125 False 7347
16 29.125 False 7330
17 29.125 False 7303
18 29.062 False 7295
19 29.062 False 7286
20 29.125 False 7291
21 29.125 False 7288
22 29.125 False 7355
23 29.188 False 7414
24 29.188 False 7414
25 29.188 False 7416
26 29.250 False 7416
27 29.250 False 7437
28 29.250 False 7501
29 29.250 False 7489
... ... ... ...
110 29.062 False 7095
111 29.062 False 7087
112 29.062 False 7082
113 29.000 False 7080
114 29.000 False 7079
115 29.000 False 7066
116 29.000 False 7069
117 29.062 False 7075
118 29.000 False 7073
119 29.062 False 7065
120 29.000 False 7065
121 29.000 False 7067
122 29.062 False 7062
123 29.000 False 7054
124 29.000 False 7059
125 29.000 False 7062
126 29.000 False 7065
127 29.000 False 7073
128 29.000 False 7075
129 29.000 False 7078
130 29.000 False 7082
131 29.000 False 7094
132 29.062 False 7093
133 29.062 False 7099
134 29.062 False 7103
135 29.062 False 7108
136 29.062 False 7102
137 29.062 False 7100
138 29.062 False 7096
139 29.062 False 7099
Calibrated Temperature 2 Temperature 2 Stable salinity \
ride_id
14888 0 28.271 False NaN
1 28.733 False NaN
2 28.643 False NaN
3 28.612 False NaN
4 28.580 False NaN
5 28.569 False NaN
6 28.533 False NaN
7 28.533 False NaN
8 28.522 False NaN
9 28.514 False NaN
10 28.506 False NaN
11 28.655 False NaN
12 28.859 False NaN
13 28.937 False NaN
14 28.933 False NaN
15 28.890 False NaN
16 28.824 False NaN
17 28.718 False NaN
18 28.686 False NaN
19 28.651 False NaN
20 28.671 False NaN
21 28.659 False NaN
22 28.922 False NaN
23 29.153 False NaN
24 29.153 False NaN
25 29.161 False NaN
26 29.161 False NaN
27 29.243 False NaN
28 29.494 False NaN
29 29.447 False NaN
... ... ... ...
110 27.902 False NaN
111 27.871 False NaN
112 27.851 False NaN
113 27.843 False NaN
114 27.839 False NaN
115 27.788 False NaN
116 27.800 False NaN
117 27.824 False NaN
118 27.816 False NaN
119 27.784 False NaN
120 27.784 False NaN
121 27.792 False NaN
122 27.773 False NaN
123 27.741 False NaN
124 27.761 False NaN
125 27.773 False NaN
126 27.784 False NaN
127 27.816 False NaN
128 27.824 False NaN
129 27.835 False NaN
130 27.851 False NaN
131 27.898 False NaN
132 27.894 False NaN
133 27.918 False NaN
134 27.933 False NaN
135 27.953 False NaN
136 27.930 False NaN
137 27.922 False NaN
138 27.906 False NaN
139 27.918 False NaN
Calibrated Salinity Salinity Stable pH Calibrated pH \
ride_id
14888 0 NaN NaN NaN NaN
1 NaN NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN NaN NaN NaN
4 NaN NaN NaN NaN
5 NaN NaN NaN NaN
6 NaN NaN NaN NaN
7 NaN NaN NaN NaN
8 NaN NaN NaN NaN
9 NaN NaN NaN NaN
10 NaN NaN NaN NaN
11 NaN NaN NaN NaN
12 NaN NaN NaN NaN
13 NaN NaN NaN NaN
14 NaN NaN NaN NaN
15 NaN NaN NaN NaN
16 NaN NaN NaN NaN
17 NaN NaN NaN NaN
18 NaN NaN NaN NaN
19 NaN NaN NaN NaN
20 NaN NaN NaN NaN
21 NaN NaN NaN NaN
22 NaN NaN NaN NaN
23 NaN NaN NaN NaN
24 NaN NaN NaN NaN
25 NaN NaN NaN NaN
26 NaN NaN NaN NaN
27 NaN NaN NaN NaN
28 NaN NaN NaN NaN
29 NaN NaN NaN NaN
... ... ... .. ...
110 NaN NaN NaN NaN
111 NaN NaN NaN NaN
112 NaN NaN NaN NaN
113 NaN NaN NaN NaN
114 NaN NaN NaN NaN
115 NaN NaN NaN NaN
116 NaN NaN NaN NaN
117 NaN NaN NaN NaN
118 NaN NaN NaN NaN
119 NaN NaN NaN NaN
120 NaN NaN NaN NaN
121 NaN NaN NaN NaN
122 NaN NaN NaN NaN
123 NaN NaN NaN NaN
124 NaN NaN NaN NaN
125 NaN NaN NaN NaN
126 NaN NaN NaN NaN
127 NaN NaN NaN NaN
128 NaN NaN NaN NaN
129 NaN NaN NaN NaN
130 NaN NaN NaN NaN
131 NaN NaN NaN NaN
132 NaN NaN NaN NaN
133 NaN NaN NaN NaN
134 NaN NaN NaN NaN
135 NaN NaN NaN NaN
136 NaN NaN NaN NaN
137 NaN NaN NaN NaN
138 NaN NaN NaN NaN
139 NaN NaN NaN NaN
pH Stable
ride_id
14888 0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
6 NaN
7 NaN
8 NaN
9 NaN
10 NaN
11 NaN
12 NaN
13 NaN
14 NaN
15 NaN
16 NaN
17 NaN
18 NaN
19 NaN
20 NaN
21 NaN
22 NaN
23 NaN
24 NaN
25 NaN
26 NaN
27 NaN
28 NaN
29 NaN
... ...
110 NaN
111 NaN
112 NaN
113 NaN
114 NaN
115 NaN
116 NaN
117 NaN
118 NaN
119 NaN
120 NaN
121 NaN
122 NaN
123 NaN
124 NaN
125 NaN
126 NaN
127 NaN
128 NaN
129 NaN
130 NaN
131 NaN
132 NaN
133 NaN
134 NaN
135 NaN
136 NaN
137 NaN
138 NaN
139 NaN
[140 rows x 14 columns]
#print(motion_df)
saved_copy_motion_df = motion_df.copy(deep=True) #make a copy of the dataframe with raw data
print(saved_copy_motion_df) UTC Time IMU A1 IMU A2 \
ride_id
14888 0 2018-07-30T17:17:24.3320+00:00 2067196596 -34.0 -520.0
1 2018-07-30T17:17:24.3650+00:00 2067196629 -30.0 -517.0
2 2018-07-30T17:17:24.3990+00:00 2067196662 -28.0 -517.0
3 2018-07-30T17:17:24.4320+00:00 2067196695 -28.0 -518.0
4 2018-07-30T17:17:24.4650+00:00 2067196728 -30.0 -517.0
5 2018-07-30T17:17:24.4980+00:00 2067196761 -31.0 -518.0
6 2018-07-30T17:17:24.5310+00:00 2067196794 -33.0 -520.0
7 2018-07-30T17:17:24.5650+00:00 2067196828 -30.0 -519.0
8 2018-07-30T17:17:24.5970+00:00 2067196860 -30.0 -517.0
9 2018-07-30T17:17:24.6310+00:00 2067196894 -29.0 -517.0
10 2018-07-30T17:17:24.6630+00:00 2067196926 -30.0 -518.0
11 2018-07-30T17:17:24.6960+00:00 2067196959 -32.0 -520.0
12 2018-07-30T17:17:24.7300+00:00 2067196992 -32.0 -518.0
13 2018-07-30T17:17:24.7630+00:00 2067197025 -30.0 -517.0
14 2018-07-30T17:17:24.7960+00:00 2067197058 -28.0 -516.0
15 2018-07-30T17:17:24.8290+00:00 2067197091 -27.0 -516.0
16 2018-07-30T17:17:24.8620+00:00 2067197124 -31.0 -518.0
17 2018-07-30T17:17:24.8950+00:00 2067197157 -34.0 -518.0
18 2018-07-30T17:17:24.9290+00:00 2067197191 -33.0 -519.0
19 2018-07-30T17:17:24.9610+00:00 2067197223 -30.0 -517.0
20 2018-07-30T17:17:24.9950+00:00 2067197257 -27.0 -516.0
21 2018-07-30T17:17:25.0280+00:00 2067197290 -28.0 -516.0
22 2018-07-30T17:17:25.0620+00:00 2067197323 7.0 -516.0
23 2018-07-30T17:17:25.0950+00:00 2067197356 -50.0 -518.0
24 2018-07-30T17:17:25.1280+00:00 2067197389 -38.0 -517.0
25 2018-07-30T17:17:25.1610+00:00 2067197422 -39.0 -516.0
26 2018-07-30T17:17:25.1940+00:00 2067197455 -29.0 -517.0
27 2018-07-30T17:17:25.2280+00:00 2067197489 -29.0 -517.0
28 2018-07-30T17:17:25.2600+00:00 2067197521 -29.0 -517.0
29 2018-07-30T17:17:25.2930+00:00 2067197554 -30.0 -517.0
... ... ... ... ...
7559 2018-07-30T17:30:50.4950+00:00 2068000375 -58.0 -553.0
7560 2018-07-30T17:30:50.6940+00:00 2068000574 -65.0 -549.0
7561 2018-07-30T17:30:50.8930+00:00 2068000772 -73.0 -539.0
7562 2018-07-30T17:30:51.0920+00:00 2068000971 -73.0 -530.0
7563 2018-07-30T17:30:51.2910+00:00 2068001169 -76.0 -522.0
7564 2018-07-30T17:30:51.4910+00:00 2068001368 -78.0 -510.0
7565 2018-07-30T17:30:51.6900+00:00 2068001567 -73.0 -497.0
7566 2018-07-30T17:30:51.8890+00:00 2068001765 -73.0 -494.0
7567 2018-07-30T17:30:52.0980+00:00 2068001974 -64.0 -480.0
7568 2018-07-30T17:30:52.2970+00:00 2068002172 -57.0 -480.0
7569 2018-07-30T17:30:52.4960+00:00 2068002371 -50.0 -477.0
7570 2018-07-30T17:30:52.6960+00:00 2068002570 -40.0 -472.0
7571 2018-07-30T17:30:52.8950+00:00 2068002768 -32.0 -469.0
7572 2018-07-30T17:30:53.0930+00:00 2068002966 -20.0 -469.0
7573 2018-07-30T17:30:53.3030+00:00 2068003175 -14.0 -472.0
7574 2018-07-30T17:30:53.5020+00:00 2068003374 -4.0 -475.0
7575 2018-07-30T17:30:53.7020+00:00 2068003573 3.0 -480.0
7576 2018-07-30T17:30:53.9010+00:00 2068003771 8.0 -491.0
7577 2018-07-30T17:30:54.1000+00:00 2068003970 12.0 -498.0
7578 2018-07-30T17:30:54.3000+00:00 2068004169 14.0 -509.0
7579 2018-07-30T17:30:54.4980+00:00 2068004367 17.0 -514.0
7580 2018-07-30T17:30:54.6980+00:00 2068004566 16.0 -523.0
7581 2018-07-30T17:30:54.9080+00:00 2068004775 14.0 -530.0
7582 2018-07-30T17:30:54.9970+00:00 2068004864 NaN NaN
7583 2018-07-30T17:31:00.4840+00:00 2068010335 NaN NaN
7584 2018-07-30T17:31:05.5020+00:00 2068015338 NaN NaN
7585 2018-07-30T17:31:10.9830+00:00 2068020803 NaN NaN
7586 2018-07-30T17:31:16.0010+00:00 2068025806 NaN NaN
7587 2018-07-30T17:31:21.4850+00:00 2068031274 NaN NaN
7588 2018-07-30T17:31:26.5000+00:00 2068036274 NaN NaN
IMU A3 IMU G1 IMU G2 IMU G3 IMU M1 IMU M2 IMU M3 \
ride_id
14888 0 -48.0 82.0 -19.0 12.0 208.0 394.0 438.0
1 -57.0 32.0 -14.0 9.0 220.0 400.0 438.0
2 -59.0 -7.0 -11.0 6.0 218.0 398.0 430.0
3 -62.0 -27.0 -9.0 5.0 210.0 400.0 428.0
4 -62.0 64.0 -13.0 9.0 212.0 394.0 426.0
5 -54.0 157.0 -24.0 17.0 215.0 399.0 433.0
6 -41.0 29.0 -14.0 9.0 220.0 400.0 432.0
7 -46.0 -101.0 -3.0 2.0 216.0 406.0 432.0
8 -55.0 -121.0 -1.0 1.0 213.0 401.0 429.0
9 -63.0 -26.0 -8.0 6.0 218.0 396.0 428.0
10 -57.0 -7.0 -11.0 5.0 209.0 407.0 427.0
11 -54.0 -3.0 -11.0 6.0 212.0 398.0 440.0
12 -57.0 -22.0 -8.0 7.0 220.0 400.0 432.0
13 -60.0 -27.0 -9.0 6.0 208.0 400.0 434.0
14 -58.0 11.0 -12.0 9.0 216.0 396.0 430.0
15 -58.0 -4.0 -10.0 6.0 213.0 403.0 437.0
16 -55.0 -32.0 -8.0 4.0 211.0 405.0 441.0
17 -62.0 -71.0 -7.0 3.0 209.0 403.0 437.0
18 -62.0 -32.0 -9.0 5.0 203.0 395.0 433.0
19 -66.0 -68.0 14.0 5.0 213.0 395.0 441.0
20 -66.0 -97.0 25.0 3.0 218.0 392.0 436.0
21 -68.0 -60.0 10.0 6.0 216.0 392.0 440.0
22 -71.0 -24.0 4.0 5.0 217.0 397.0 441.0
23 -73.0 -47.0 13.0 5.0 217.0 399.0 445.0
24 -72.0 -43.0 6.0 8.0 210.0 396.0 442.0
25 -74.0 -29.0 -1.0 7.0 219.0 389.0 441.0
26 -73.0 -7.0 -11.0 6.0 214.0 396.0 448.0
27 -75.0 -53.0 9.0 6.0 210.0 402.0 440.0
28 -77.0 -40.0 6.0 7.0 208.0 392.0 442.0
29 -76.0 -11.0 -9.0 5.0 224.0 386.0 440.0
... ... ... ... ... ... ... ...
7559 -65.0 -9.0 -10.0 5.0 199.0 405.0 439.0
7560 -66.0 -7.0 -10.0 5.0 199.0 409.0 459.0
7561 -65.0 -8.0 -10.0 4.0 201.0 395.0 469.0
7562 -63.0 -9.0 -9.0 3.0 205.0 387.0 467.0
7563 -61.0 -8.0 -9.0 4.0 197.0 381.0 465.0
7564 -63.0 -8.0 -9.0 4.0 209.0 377.0 457.0
7565 -65.0 -9.0 -10.0 5.0 201.0 383.0 445.0
7566 -62.0 -8.0 -9.0 5.0 211.0 377.0 441.0
7567 -60.0 -7.0 -9.0 5.0 208.0 384.0 438.0
7568 -57.0 -6.0 -9.0 6.0 204.0 374.0 434.0
7569 -62.0 -7.0 -10.0 5.0 207.0 385.0 435.0
7570 -56.0 -6.0 -10.0 6.0 210.0 396.0 432.0
7571 -56.0 -6.0 -11.0 4.0 203.0 397.0 429.0
7572 -56.0 -7.0 -11.0 5.0 196.0 396.0 438.0
7573 -56.0 -6.0 -11.0 8.0 205.0 389.0 433.0
7574 -58.0 -7.0 -12.0 5.0 211.0 397.0 429.0
7575 -57.0 -6.0 -12.0 4.0 201.0 395.0 429.0
7576 -58.0 -6.0 -12.0 4.0 210.0 392.0 432.0
7577 -57.0 -6.0 -12.0 5.0 216.0 402.0 426.0
7578 -55.0 -6.0 -13.0 7.0 208.0 394.0 424.0
7579 -59.0 -7.0 -12.0 7.0 207.0 389.0 433.0
7580 -56.0 -8.0 -11.0 6.0 213.0 395.0 429.0
7581 -60.0 -8.0 -13.0 5.0 212.0 400.0 424.0
7582 NaN NaN NaN NaN NaN NaN NaN
7583 NaN NaN NaN NaN NaN NaN NaN
7584 NaN NaN NaN NaN NaN NaN NaN
7585 NaN NaN NaN NaN NaN NaN NaN
7586 NaN NaN NaN NaN NaN NaN NaN
7587 NaN NaN NaN NaN NaN NaN NaN
7588 NaN NaN NaN NaN NaN NaN NaN
Latitude Longitude
ride_id
14888 0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 NaN NaN
11 NaN NaN
12 NaN NaN
13 NaN NaN
14 NaN NaN
15 NaN NaN
16 NaN NaN
17 NaN NaN
18 NaN NaN
19 NaN NaN
20 NaN NaN
21 NaN NaN
22 NaN NaN
23 NaN NaN
24 NaN NaN
25 NaN NaN
26 NaN NaN
27 NaN NaN
28 NaN NaN
29 NaN NaN
... ... ...
7559 NaN NaN
7560 NaN NaN
7561 NaN NaN
7562 NaN NaN
7563 NaN NaN
7564 NaN NaN
7565 NaN NaN
7566 NaN NaN
7567 NaN NaN
7568 NaN NaN
7569 NaN NaN
7570 NaN NaN
7571 NaN NaN
7572 NaN NaN
7573 NaN NaN
7574 NaN NaN
7575 NaN NaN
7576 NaN NaN
7577 NaN NaN
7578 NaN NaN
7579 NaN NaN
7580 NaN NaN
7581 NaN NaN
7582 3286943.0 -11725208.0
7583 3286942.0 -11725208.0
7584 3286941.0 -11725207.0
7585 3286941.0 -11725206.0
7586 3286941.0 -11725206.0
7587 3286942.0 -11725206.0
7588 3286943.0 -11725207.0
[7589 rows x 13 columns]
#Drop the "nan" values from the columns that we care about.
dropped_motion_df = motion_df.dropna(subset=['Time', 'IMU A1', 'IMU A2', 'IMU A3', 'IMU G1', 'IMU G2', 'IMU G3', 'IMU M1',
'IMU M2', 'IMU M3'])
print(dropped_motion_df) UTC Time IMU A1 IMU A2 \
ride_id
14888 0 2018-07-30T17:17:24.3320+00:00 2067196596 -34.0 -520.0
1 2018-07-30T17:17:24.3650+00:00 2067196629 -30.0 -517.0
2 2018-07-30T17:17:24.3990+00:00 2067196662 -28.0 -517.0
3 2018-07-30T17:17:24.4320+00:00 2067196695 -28.0 -518.0
4 2018-07-30T17:17:24.4650+00:00 2067196728 -30.0 -517.0
5 2018-07-30T17:17:24.4980+00:00 2067196761 -31.0 -518.0
6 2018-07-30T17:17:24.5310+00:00 2067196794 -33.0 -520.0
7 2018-07-30T17:17:24.5650+00:00 2067196828 -30.0 -519.0
8 2018-07-30T17:17:24.5970+00:00 2067196860 -30.0 -517.0
9 2018-07-30T17:17:24.6310+00:00 2067196894 -29.0 -517.0
10 2018-07-30T17:17:24.6630+00:00 2067196926 -30.0 -518.0
11 2018-07-30T17:17:24.6960+00:00 2067196959 -32.0 -520.0
12 2018-07-30T17:17:24.7300+00:00 2067196992 -32.0 -518.0
13 2018-07-30T17:17:24.7630+00:00 2067197025 -30.0 -517.0
14 2018-07-30T17:17:24.7960+00:00 2067197058 -28.0 -516.0
15 2018-07-30T17:17:24.8290+00:00 2067197091 -27.0 -516.0
16 2018-07-30T17:17:24.8620+00:00 2067197124 -31.0 -518.0
17 2018-07-30T17:17:24.8950+00:00 2067197157 -34.0 -518.0
18 2018-07-30T17:17:24.9290+00:00 2067197191 -33.0 -519.0
19 2018-07-30T17:17:24.9610+00:00 2067197223 -30.0 -517.0
20 2018-07-30T17:17:24.9950+00:00 2067197257 -27.0 -516.0
21 2018-07-30T17:17:25.0280+00:00 2067197290 -28.0 -516.0
22 2018-07-30T17:17:25.0620+00:00 2067197323 7.0 -516.0
23 2018-07-30T17:17:25.0950+00:00 2067197356 -50.0 -518.0
24 2018-07-30T17:17:25.1280+00:00 2067197389 -38.0 -517.0
25 2018-07-30T17:17:25.1610+00:00 2067197422 -39.0 -516.0
26 2018-07-30T17:17:25.1940+00:00 2067197455 -29.0 -517.0
27 2018-07-30T17:17:25.2280+00:00 2067197489 -29.0 -517.0
28 2018-07-30T17:17:25.2600+00:00 2067197521 -29.0 -517.0
29 2018-07-30T17:17:25.2930+00:00 2067197554 -30.0 -517.0
... ... ... ... ...
7551 2018-07-30T17:30:49.0880+00:00 2067998973 3.0 -548.0
7552 2018-07-30T17:30:49.2880+00:00 2067999172 -2.0 -560.0
7553 2018-07-30T17:30:49.4880+00:00 2067999371 -9.0 -567.0
7554 2018-07-30T17:30:49.6860+00:00 2067999569 -24.0 -564.0
7555 2018-07-30T17:30:49.8860+00:00 2067999768 -31.0 -561.0
7557 2018-07-30T17:30:50.0850+00:00 2067999967 -37.0 -563.0
7558 2018-07-30T17:30:50.2840+00:00 2068000165 -50.0 -561.0
7559 2018-07-30T17:30:50.4950+00:00 2068000375 -58.0 -553.0
7560 2018-07-30T17:30:50.6940+00:00 2068000574 -65.0 -549.0
7561 2018-07-30T17:30:50.8930+00:00 2068000772 -73.0 -539.0
7562 2018-07-30T17:30:51.0920+00:00 2068000971 -73.0 -530.0
7563 2018-07-30T17:30:51.2910+00:00 2068001169 -76.0 -522.0
7564 2018-07-30T17:30:51.4910+00:00 2068001368 -78.0 -510.0
7565 2018-07-30T17:30:51.6900+00:00 2068001567 -73.0 -497.0
7566 2018-07-30T17:30:51.8890+00:00 2068001765 -73.0 -494.0
7567 2018-07-30T17:30:52.0980+00:00 2068001974 -64.0 -480.0
7568 2018-07-30T17:30:52.2970+00:00 2068002172 -57.0 -480.0
7569 2018-07-30T17:30:52.4960+00:00 2068002371 -50.0 -477.0
7570 2018-07-30T17:30:52.6960+00:00 2068002570 -40.0 -472.0
7571 2018-07-30T17:30:52.8950+00:00 2068002768 -32.0 -469.0
7572 2018-07-30T17:30:53.0930+00:00 2068002966 -20.0 -469.0
7573 2018-07-30T17:30:53.3030+00:00 2068003175 -14.0 -472.0
7574 2018-07-30T17:30:53.5020+00:00 2068003374 -4.0 -475.0
7575 2018-07-30T17:30:53.7020+00:00 2068003573 3.0 -480.0
7576 2018-07-30T17:30:53.9010+00:00 2068003771 8.0 -491.0
7577 2018-07-30T17:30:54.1000+00:00 2068003970 12.0 -498.0
7578 2018-07-30T17:30:54.3000+00:00 2068004169 14.0 -509.0
7579 2018-07-30T17:30:54.4980+00:00 2068004367 17.0 -514.0
7580 2018-07-30T17:30:54.6980+00:00 2068004566 16.0 -523.0
7581 2018-07-30T17:30:54.9080+00:00 2068004775 14.0 -530.0
IMU A3 IMU G1 IMU G2 IMU G3 IMU M1 IMU M2 IMU M3 \
ride_id
14888 0 -48.0 82.0 -19.0 12.0 208.0 394.0 438.0
1 -57.0 32.0 -14.0 9.0 220.0 400.0 438.0
2 -59.0 -7.0 -11.0 6.0 218.0 398.0 430.0
3 -62.0 -27.0 -9.0 5.0 210.0 400.0 428.0
4 -62.0 64.0 -13.0 9.0 212.0 394.0 426.0
5 -54.0 157.0 -24.0 17.0 215.0 399.0 433.0
6 -41.0 29.0 -14.0 9.0 220.0 400.0 432.0
7 -46.0 -101.0 -3.0 2.0 216.0 406.0 432.0
8 -55.0 -121.0 -1.0 1.0 213.0 401.0 429.0
9 -63.0 -26.0 -8.0 6.0 218.0 396.0 428.0
10 -57.0 -7.0 -11.0 5.0 209.0 407.0 427.0
11 -54.0 -3.0 -11.0 6.0 212.0 398.0 440.0
12 -57.0 -22.0 -8.0 7.0 220.0 400.0 432.0
13 -60.0 -27.0 -9.0 6.0 208.0 400.0 434.0
14 -58.0 11.0 -12.0 9.0 216.0 396.0 430.0
15 -58.0 -4.0 -10.0 6.0 213.0 403.0 437.0
16 -55.0 -32.0 -8.0 4.0 211.0 405.0 441.0
17 -62.0 -71.0 -7.0 3.0 209.0 403.0 437.0
18 -62.0 -32.0 -9.0 5.0 203.0 395.0 433.0
19 -66.0 -68.0 14.0 5.0 213.0 395.0 441.0
20 -66.0 -97.0 25.0 3.0 218.0 392.0 436.0
21 -68.0 -60.0 10.0 6.0 216.0 392.0 440.0
22 -71.0 -24.0 4.0 5.0 217.0 397.0 441.0
23 -73.0 -47.0 13.0 5.0 217.0 399.0 445.0
24 -72.0 -43.0 6.0 8.0 210.0 396.0 442.0
25 -74.0 -29.0 -1.0 7.0 219.0 389.0 441.0
26 -73.0 -7.0 -11.0 6.0 214.0 396.0 448.0
27 -75.0 -53.0 9.0 6.0 210.0 402.0 440.0
28 -77.0 -40.0 6.0 7.0 208.0 392.0 442.0
29 -76.0 -11.0 -9.0 5.0 224.0 386.0 440.0
... ... ... ... ... ... ... ...
7551 -60.0 -8.0 -11.0 7.0 208.0 396.0 428.0
7552 -65.0 -11.0 -11.0 6.0 204.0 396.0 434.0
7553 -61.0 -9.0 -11.0 4.0 207.0 401.0 423.0
7554 -63.0 -8.0 -10.0 4.0 215.0 395.0 429.0
7555 -66.0 -9.0 -10.0 4.0 205.0 403.0 425.0
7557 -66.0 -9.0 -10.0 6.0 200.0 402.0 440.0
7558 -66.0 -9.0 -10.0 6.0 202.0 398.0 446.0
7559 -65.0 -9.0 -10.0 5.0 199.0 405.0 439.0
7560 -66.0 -7.0 -10.0 5.0 199.0 409.0 459.0
7561 -65.0 -8.0 -10.0 4.0 201.0 395.0 469.0
7562 -63.0 -9.0 -9.0 3.0 205.0 387.0 467.0
7563 -61.0 -8.0 -9.0 4.0 197.0 381.0 465.0
7564 -63.0 -8.0 -9.0 4.0 209.0 377.0 457.0
7565 -65.0 -9.0 -10.0 5.0 201.0 383.0 445.0
7566 -62.0 -8.0 -9.0 5.0 211.0 377.0 441.0
7567 -60.0 -7.0 -9.0 5.0 208.0 384.0 438.0
7568 -57.0 -6.0 -9.0 6.0 204.0 374.0 434.0
7569 -62.0 -7.0 -10.0 5.0 207.0 385.0 435.0
7570 -56.0 -6.0 -10.0 6.0 210.0 396.0 432.0
7571 -56.0 -6.0 -11.0 4.0 203.0 397.0 429.0
7572 -56.0 -7.0 -11.0 5.0 196.0 396.0 438.0
7573 -56.0 -6.0 -11.0 8.0 205.0 389.0 433.0
7574 -58.0 -7.0 -12.0 5.0 211.0 397.0 429.0
7575 -57.0 -6.0 -12.0 4.0 201.0 395.0 429.0
7576 -58.0 -6.0 -12.0 4.0 210.0 392.0 432.0
7577 -57.0 -6.0 -12.0 5.0 216.0 402.0 426.0
7578 -55.0 -6.0 -13.0 7.0 208.0 394.0 424.0
7579 -59.0 -7.0 -12.0 7.0 207.0 389.0 433.0
7580 -56.0 -8.0 -11.0 6.0 213.0 395.0 429.0
7581 -60.0 -8.0 -13.0 5.0 212.0 400.0 424.0
Latitude Longitude
ride_id
14888 0 NaN NaN
1 NaN NaN
2 NaN NaN
3 NaN NaN
4 NaN NaN
5 NaN NaN
6 NaN NaN
7 NaN NaN
8 NaN NaN
9 NaN NaN
10 NaN NaN
11 NaN NaN
12 NaN NaN
13 NaN NaN
14 NaN NaN
15 NaN NaN
16 NaN NaN
17 NaN NaN
18 NaN NaN
19 NaN NaN
20 NaN NaN
21 NaN NaN
22 NaN NaN
23 NaN NaN
24 NaN NaN
25 NaN NaN
26 NaN NaN
27 NaN NaN
28 NaN NaN
29 NaN NaN
... ... ...
7551 NaN NaN
7552 NaN NaN
7553 NaN NaN
7554 NaN NaN
7555 NaN NaN
7557 NaN NaN
7558 NaN NaN
7559 NaN NaN
7560 NaN NaN
7561 NaN NaN
7562 NaN NaN
7563 NaN NaN
7564 NaN NaN
7565 NaN NaN
7566 NaN NaN
7567 NaN NaN
7568 NaN NaN
7569 NaN NaN
7570 NaN NaN
7571 NaN NaN
7572 NaN NaN
7573 NaN NaN
7574 NaN NaN
7575 NaN NaN
7576 NaN NaN
7577 NaN NaN
7578 NaN NaN
7579 NaN NaN
7580 NaN NaN
7581 NaN NaN
[7463 rows x 13 columns]
# To store time elapsed between each measurement, and time offset from 0s
time_e_list = []
time_o_list = []
#Remove all nan instances in time:
time_array_nans = np.array(dropped_motion_df.loc[:,"Time"], dtype=float)
time_array = []
imuA1_array_nans = np.array(dropped_motion_df.loc[:,"IMU A1"], dtype=float)
imu_array_A1 = []
imuA2_array_nans = np.array(dropped_motion_df.loc[:,"IMU A2"], dtype=float)
imu_array_A2 = []
imuA3_array_nans = np.array(dropped_motion_df.loc[:,"IMU A3"], dtype=float)
imu_array_A3 = []
imuG1_array_nans = np.array(dropped_motion_df.loc[:,"IMU G1"], dtype=float)
imu_array_G1 = []
imuG2_array_nans = np.array(dropped_motion_df.loc[:,"IMU G2"], dtype=float)
imu_array_G2 = []
imuG3_array_nans = np.array(dropped_motion_df.loc[:,"IMU G3"], dtype=float)
imu_array_G3 = []
imuM1_array_nans = np.array(dropped_motion_df.loc[:,"IMU M1"], dtype=float)
imu_array_M1 = []
imuM2_array_nans = np.array(dropped_motion_df.loc[:,"IMU M2"], dtype=float)
imu_array_M2 = []
imuM3_array_nans = np.array(dropped_motion_df.loc[:,"IMU M3"], dtype=float)
imu_array_M3 = []
#Get all the times and imus where time, imu a1, imu a2, and imu a3 are NOT nan values:
for t,x,y,z,a,b,c,d,e,f in zip(time_array_nans, imuA1_array_nans, imuA2_array_nans, imuA3_array_nans, imuG1_array_nans,
imuG2_array_nans, imuG3_array_nans, imuM1_array_nans, imuM2_array_nans, imuM3_array_nans):
if (np.isnan(t)==0 and np.isnan(x)==0 and np.isnan(y)==0 and np.isnan(z)==0):
time_array.append(t)
imu_array_A1.append(x)
imu_array_A2.append(y)
imu_array_A3.append(z)
imu_array_G1.append(a)
imu_array_G2.append(b)
imu_array_G3.append(c)
imu_array_M1.append(d)
imu_array_M2.append(e)
imu_array_M3.append(f)
#for x in time_array:
# print(x)
start_time = time_array[0]
time_len = len(time_array)
i = 0
while (i < time_len - 1):
prev = time_array[i]
after = time_array[i+1]
offset = after - prev
#if (np.isnan(offset)==0):
time_o_list.append(offset)
elapsed = time_array[i] - start_time
#if (np.isnan(elapsed)==0):
time_e_list.append(elapsed)
i = i + 1
##Check to make sure there are no "nan" values:
i = 0
while (i < len(time_o_list)):
if (np.isnan(time_o_list[i])):
print("Error! Value at index: ", i, " is nan")
i = i + 1
#Drop the last value from each of the imu lists to make it match the time list.
del(imu_array_A1[-1])
del(imu_array_A2[-1])
del(imu_array_A3[-1])
del(imu_array_G1[-1])
del(imu_array_G2[-1])
del(imu_array_G3[-1])
del(imu_array_M1[-1])
del(imu_array_M2[-1])
del(imu_array_M3[-1])
print(len(time_e_list))
print(len(time_o_list))
print(len(imu_array_A1))
print(len(imu_array_A2))
print(len(imu_array_A3))
print(len(imu_array_G1))
print(len(imu_array_G2))
print(len(imu_array_G3))
print(len(imu_array_M1))
print(len(imu_array_M2))
print(len(imu_array_M3))
CheckLength = len(time_e_list) + len(time_o_list) + len(imu_array_A1) + len(imu_array_A2) + len(imu_array_A3) + len(imu_array_G1) + len(imu_array_G2) + len(imu_array_G3) + len(imu_array_M1)+ len(imu_array_M2) + len(imu_array_M3)
if CheckLength//11 == len(time_e_list):
print("All columns are matching!")7462
7462
7462
7462
7462
7462
7462
7462
7462
7462
7462
All columns are matching!
#Raw acceleration constant 512 = 1g (accelerometer's measured force due to gravity)
g_const = 512
#Approximate measurement for gravity:
gravity = 9.80665
# Correct the IMU Acceleration columns into units of meters
def convert_acc_units(acc_array):
ret_array = []
for a in acc_array:
#Acceleration is now in m/s^2, need to subtract gravity from vertical axis. (??)
new_a = a / g_const * gravity + gravity
ret_array.append(new_a)
return ret_array
imu1_array = convert_acc_units(imu_array_A1) #new units in m/s^2
imu2_array = convert_acc_units(imu_array_A2) #new units in m/s^2
imu3_array = convert_acc_units(imu_array_A3) #new units in m/s^2
# To check:
#for x,y in zip(imu2_array, imu_array_A2):
# print(x,y)
def convert_time_units(time_array):
ret_array = []
for t in time_array:
new_t = t * (10**(-3)) #converting units in milliseconds to seconds
ret_array.append(new_t)
return ret_array
time_o_array = convert_time_units(time_o_list) #new units in seconds
time_e_array = convert_time_units(time_e_list) #new units in seconds
# To check:
# for t in time_e_array:
# print(t)
print("Done!")Done!
# Square x and z acceleration values
accel_xz_array = [];
for i in range(0, len(imu1_array)):
currMagnitude = math.sqrt((math.pow(imu1_array[i], 2) + math.pow(imu3_array[i], 2)))
accel_xz_array.append(currMagnitude)
print("Graph of XZ Acceleration vs. Time")
plt.plot(time_e_array, accel_xz_array)
plt.xlabel("Time (s)")
plt.ylabel("Acceleration-XZ (m/s^2)")
plt.show()Graph of XZ Acceleration vs. Time
# Offset variables help in recentering the magnetic data in order to define direction and use trig functions
M1_offset_var = 219.786
M2_offset_var = 180
M3_offset_var = 280
def calibrate_magn_data(magn_array, offset_value):
ret_array = []
for m in magn_array:
new_m = m - offset_value
ret_array.append(new_m)
return ret_array
imuM1_array = calibrate_magn_data(imu_array_M1, M1_offset_var)
imuM2_array = calibrate_magn_data(imu_array_M2, M2_offset_var)
imuM3_array = calibrate_magn_data(imu_array_M3, M3_offset_var)
# Check
# print(imuM1_array)
print("Done.")Done.
# Create N x 3 arrays for functions that need them later on, such as Scikit Kinematics
magn_height = len(imuM1_array)
acc_height = len(imu1_array)
acc_array = np.zeros(shape=(acc_height,3))
magn_array = np.zeros(shape=(magn_height,3))
print("For Accelerometer: ")
for x in range(len(acc_array)):
acc_array[x,0] = imu1_array[x]
acc_array[x,1] = imu2_array[x]
acc_array[x,2] = imu3_array[x]
print(acc_array)
print("\nFor Magnetometer: ")
for x in range(len(magn_array)):
magn_array[x,0] = imuM1_array[x]
magn_array[x,1] = imuM2_array[x]
magn_array[x,2] = imuM3_array[x]
print(magn_array)
print("Done.")For Accelerometer:
[[ 9.15542715 -0.15322891 8.88727656]
[ 9.2320416 -0.09576807 8.71489404]
[ 9.27034883 -0.09576807 8.67658682]
...
[10.07480059 0.05746084 8.75320127]
[10.13226143 -0.03830723 8.67658682]
[10.11310781 -0.21068975 8.73404766]]
For Magnetometer:
[[-1.1786e+01 2.1400e+02 1.5800e+02]
[ 2.1400e-01 2.2000e+02 1.5800e+02]
[-1.7860e+00 2.1800e+02 1.5000e+02]
...
[-1.1786e+01 2.1400e+02 1.4400e+02]
[-1.2786e+01 2.0900e+02 1.5300e+02]
[-6.7860e+00 2.1500e+02 1.4900e+02]]
Done.
Orientation from here onwards will be from the board/surfers reference frame (yaw left = turning left)
x = -IMU1, y = -IMU3, z = -IMU2
# The new array for board reference frame will have the IMUs in columns according to X,Y,Z directions
board_acc = acc_array.copy() # Reassign to the correct axes as stated above
temp_x_acc = board_acc[:,0] * (-1)
temp_y_acc = board_acc[:,1] * (-1)
temp_z_acc = board_acc[:,2] * (-1)
board_acc[:,0] = temp_x_acc # X acceleration
board_acc[:,1] = temp_y_acc # Y acceleration
board_acc[:,2] = temp_z_acc # Z acceleration
print(board_acc)
print("\n")
board_magn = magn_array.copy()
temp_x_magn = board_magn[:,0] * (-1)
temp_y_magn = board_magn[:,1] * (-1)
temp_z_magn = board_magn[:,2] * (-1)
board_magn[:,0] = temp_x_magn
board_magn[:,1] = temp_y_magn
board_magn[:,2] = temp_z_magn
print(board_magn)
print("Done.")[[ -9.15542715 0.15322891 -8.88727656]
[ -9.2320416 0.09576807 -8.71489404]
[ -9.27034883 0.09576807 -8.67658682]
...
[-10.07480059 -0.05746084 -8.75320127]
[-10.13226143 0.03830723 -8.67658682]
[-10.11310781 0.21068975 -8.73404766]]
[[ 1.1786e+01 -2.1400e+02 -1.5800e+02]
[-2.1400e-01 -2.2000e+02 -1.5800e+02]
[ 1.7860e+00 -2.1800e+02 -1.5000e+02]
...
[ 1.1786e+01 -2.1400e+02 -1.4400e+02]
[ 1.2786e+01 -2.0900e+02 -1.5300e+02]
[ 6.7860e+00 -2.1500e+02 -1.4900e+02]]
Done.
# Square x, y and z acceleration values
accel_xyz_array = [];
for i in range(0, len(board_acc)):
currMagnitude = math.sqrt(math.pow(board_acc[i][0], 2) + math.pow(board_acc[i][1], 2) + math.pow(board_acc[i][2], 2))
accel_xyz_array.append(currMagnitude)
# Check length
print(len(accel_xyz_array))
print("Graph of XYZ Acceleration vs. Time")
plt.plot(time_e_array, accel_xyz_array)
plt.xlabel("Time (s)")
plt.ylabel("Acceleration-XYZ (m/s^2)")
plt.show()7462
Graph of XYZ Acceleration vs. Time
# Azimuth and Altitude LEGEND:
# Altitude is the angle between the ground and the vector
# Azimuth is the angle going clockwise from 0 deg North:
# N - 0/360deg, E - 90deg, S - 180deg, W - 270deg
# This will get complicated (ie make cases or lots of if statements) when rotations about the heading become more prevalent
def azimuth(x,y,z):
real_y = y * (-1) # This is to account for y
return (180/math.pi * math.atan2(real_y,x)) % 360
def altitude(x,y,z):
h = math.hypot(y, x)
return 180/math.pi * math.atan2(z,h)
def printAltAzi(alt, azi):
print ("Alt:", alt, "\n", "Azi:",azi,"\n")
# These values are uncorrected values: still need to add or subtract 'declination'
# (for AziMuth) and 'inclination' (for Altitude) correction values for geographical location
heading_altitude = board_magn[:,0].copy()
heading_azimuth = board_magn[:,0].copy()
i = 0 #iterator
#for i in range(len(M1_no_out)):
while i < len(heading_altitude):
heading_altitude[i] = altitude(board_magn[i,0], board_magn[i,1], board_magn[i,2])
heading_azimuth[i] = azimuth(board_magn[i,0], board_magn[i,1], board_magn[i,2])
#printAltAzi(heading_altitude[i],heading_azimuth[i])
i += 1
heading_azi_plot = plt.figure(figsize=(10,5))
azi_plot = heading_azi_plot.add_subplot(111)
azi_plot.plot(time_e_array, heading_azimuth)
azi_plot.set_title("Azimuth vs. Time")
azi_plot.set_xlabel("Time Elapsed[sec]")
azi_plot.set_ylabel("Azimuth [deg]")
azi_plot.set_ylim([-50,400])
heading_alt_plot = plt.figure(figsize=(10,5))
alt_plot = heading_alt_plot.add_subplot(111)
alt_plot.plot(time_e_array, heading_altitude)
alt_plot.set_title("Altitude vs. Time")
alt_plot.set_xlabel("Time Elapsed [sec]")
alt_plot.set_ylabel("Altitude [deg]")
plt.show()
#for t in range(len(time_e_array)):
#printAltAzi(heading_altitude[t], heading_azimuth[t])
# Fixing random state for reproducibility
np.random.seed(19680801)
# Compute areas and colors
r = [i for i in range(0, len(board_magn))]
theta = heading_azimuth/360 * 2 * np.pi
area = 1
colors = theta
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111, projection='polar')
c = ax.scatter(theta, r, c=colors, s=area, cmap='hsv', alpha=1)
# Compute areas and colors
alt_r = [i for i in range(0, len(board_magn))]
alt_theta = heading_altitude/360 * 2 * np.pi
alt_area = 1
alt_colors = theta
fig2 = plt.figure(figsize=(10,10))
ax2 = fig2.add_subplot(111, projection='polar')
c2 = ax2.scatter(alt_theta, alt_r, c=alt_colors, s=alt_area, cmap='hsv', alpha=1)
# Plot first 10 directional changes
magnfig = plt.figure(figsize=(20,20))
magnaxi = magnfig.add_subplot(111, projection='3d')
magnaxi.scatter(board_magn[0,0], board_magn[0,2], board_magn[0,1], c='black', s=300, marker = "<")
magnaxi.scatter(board_magn[1:,0], board_magn[1:,2], board_magn[1:,1], c='black', s=10, marker = "o")
magnaxi.plot(board_magn[:,0], board_magn[:,2], board_magn[:,1], color='red')
magnaxi.set_xlabel("X Axis ->", fontsize=30)
magnaxi.set_ylabel("Y Axis ->", fontsize=30)
magnaxi.set_zlabel("Z Axis ->", fontsize=30)
plt.show()
As displayed in the diagram, the method for determining peak wave direction is to pool all the data from each of these Smartfins so that we can conduct a cumulative spectral analysis. This means instead of generating a frequency graph for a single ride, we will generate multiple graphs that represents the data for all the Smartfins, and then analyze the frequency transitions between close-by Smartfins.
As the graph suggests, the only difference between this frequency graph and the one used to determine wave height is the parameter wave density, which is essentially the average acceleration per unit of frequency. It represents the energy density per wave unit. Overall, we can compare each frequency graph side-by-side to see how the wave behaves when it starts from one Smartfin and reaches the next. Using this data, we can generalize the directional propagation and behaviour of the waves in the region covered by all the Smartfins. The projected result of my method can be visualized below:
-
Currently, the most popular methods to determine wave direction involve using Buoys, which don't give sufficient data in terms of determining wave direction in a small region. Since buoys are spaced out hundred of meters apart and way offshore, it is almost impossible to determine proximal wave direction without the help of Smartfins.
-
One way to generate a directional spectrum is to measure the same parameter - such as pressure - at a series of nearby locations using pressor sensors. I considered this, but realized that we didn't have the resources to conduct such an experiment.
-
The other way to produce a directional spectrum is by measuring different parameters at the same point. This is the approach used in directional buoys, which measure pitch and roll in addition to vertical heave. However, even this could easily be achieved using an array of Smartfins. It would essentially accomplish the same task as the directional buoys, and it would be much easier to gather wave data for smaller regions.
-
Using spectral analysis and frequency graphs, this method reuses the one for determining wave height, and is much simpler to execute.
-
The only disadvantage I can forsee is the level of noise and inaccuracy presented by Smartfins and the potential for human error, for example when a surfer decides to paddle upstream, or push their surfboard deeper underwater into a wave.