Skip to content

Commit efc597d

Browse files
ctraliectralie
committedJul 16, 2016
Finished inline 4 video experiment
1 parent 5c863f7 commit efc597d

5 files changed

+151
-6
lines changed
 

‎4-Video.ipynb

+151-5
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
},
1414
{
1515
"cell_type": "code",
16-
"execution_count": 1,
16+
"execution_count": null,
1717
"metadata": {
1818
"collapsed": false
1919
},
@@ -218,7 +218,7 @@
218218
},
219219
{
220220
"cell_type": "code",
221-
"execution_count": 4,
221+
"execution_count": null,
222222
"metadata": {
223223
"collapsed": false
224224
},
@@ -240,7 +240,7 @@
240240
},
241241
{
242242
"cell_type": "code",
243-
"execution_count": 5,
243+
"execution_count": null,
244244
"metadata": {
245245
"collapsed": false
246246
},
@@ -259,8 +259,154 @@
259259
"source": [
260260
"As you can see, the maximum persistence peaks at around 40 frames, which is the period of each hand wave. This is what the theory we developed for 1D time series would have predicted as the roundest window.<BR>\n",
261261
"\n",
262-
"<h2>KTH Dataset Rankings</h2><BR>\n",
263-
"For the final experiment, students will split up into groups and run code on different subsets of the KTH dataset which ranks the video clips in decreasing order of periodicity. As an example, <a href = \"VideoResults/index.html\">click here</a> to see the rankings of all activities for the first 4 subjects. Groups will run the code in <a href = \"KTHTests.py\">KTHTests.py</a> after modifying it to go through the appropriate subset of the database by changing lines 53 through 55. A new web page will be generated <a href = \"VideoResults/index.html\">here</a> to show the resulting rankings. Note that a fixed window length of 20 frames is maintained throughout all of the experiments. Feel free to tweak the window size (\"win\" on line 72), the dimension of the embedding (\"dim\" on line 73), or any other parameters you think would make the results more meaningful."
262+
"<h2>KTH 4 Video Ranking</h2>\n",
263+
"<BR>\n",
264+
"Now, students will rank 4 videos from the database from most periodic to least periodic visually, and then we will run TDA on the videos and compare the ranking based on maximum persistence to the rank aggregated class estimate. Click below to load the four videos.\n"
265+
]
266+
},
267+
{
268+
"cell_type": "code",
269+
"execution_count": null,
270+
"metadata": {
271+
"collapsed": false
272+
},
273+
"outputs": [],
274+
"source": [
275+
"#First video\n",
276+
"videos = ['KTH/handwaving/person01_handwaving_d1_uncomp.ogg']\n",
277+
"video = io.open(videos[-1], 'r+b').read()\n",
278+
"encoded = base64.b64encode(video)\n",
279+
"HTML(data='''<h1>Video 1</h1><video alt=\"test\" controls>\n",
280+
" <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
281+
" </video>'''.format(encoded.decode('ascii')))"
282+
]
283+
},
284+
{
285+
"cell_type": "code",
286+
"execution_count": null,
287+
"metadata": {
288+
"collapsed": false
289+
},
290+
"outputs": [],
291+
"source": [
292+
"videos.append('KTH/boxing/person01_boxing_d1_uncomp.ogg')\n",
293+
"video = io.open(videos[-1], 'r+b').read()\n",
294+
"encoded = base64.b64encode(video)\n",
295+
"HTML(data='''<h1>Video 2</h2><video alt=\"test\" controls>\n",
296+
" <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
297+
" </video>'''.format(encoded.decode('ascii')))"
298+
]
299+
},
300+
{
301+
"cell_type": "code",
302+
"execution_count": null,
303+
"metadata": {
304+
"collapsed": false
305+
},
306+
"outputs": [],
307+
"source": [
308+
"videos.append('KTH/walking/person01_walking_d1_uncomp.ogg')\n",
309+
"video = io.open(videos[-1], 'r+b').read()\n",
310+
"encoded = base64.b64encode(video)\n",
311+
"HTML(data='''<h1>Video 3</h1><video alt=\"test\" controls>\n",
312+
" <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
313+
" </video>'''.format(encoded.decode('ascii')))"
314+
]
315+
},
316+
{
317+
"cell_type": "code",
318+
"execution_count": null,
319+
"metadata": {
320+
"collapsed": false
321+
},
322+
"outputs": [],
323+
"source": [
324+
"videos.append('KTH/handwaving/person02_handwaving_d2_uncomp.ogg')\n",
325+
"video = io.open(videos[-1], 'r+b').read()\n",
326+
"encoded = base64.b64encode(video)\n",
327+
"HTML(data='''<h1>Video 4</h1><video alt=\"test\" controls>\n",
328+
" <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
329+
" </video>'''.format(encoded.decode('ascii')))"
330+
]
331+
},
332+
{
333+
"cell_type": "markdown",
334+
"metadata": {},
335+
"source": [
336+
"Now that you have loaded in and watched the videos, go to the following Google doc and input your ranking, from most periodic to least:<BR><BR>\n",
337+
"\n",
338+
"<a href = \"https://docs.google.com/spreadsheets/d/1L59t7oO6jiFHlrxSMnMGwnrKNdqYebTO_30I7nhcFgg/edit?usp=sharing\">https://docs.google.com/spreadsheets/d/1L59t7oO6jiFHlrxSMnMGwnrKNdqYebTO_30I7nhcFgg/edit?usp=sharing</a><BR><BR>\n",
339+
"\n",
340+
"Now, run the code below to compute a ranking based on delay embeddings and persistent homology. The code goes through the video in blocks equal to 160 frames, hopping forward 80 frames to the next block, and it uses a fixed window size of 20 frames for all videos. You can tweak these parameters if you'd like. At the end, it records the maximum maximum persistence over all blocks of 160 frames."
341+
]
342+
},
343+
{
344+
"cell_type": "code",
345+
"execution_count": 25,
346+
"metadata": {
347+
"collapsed": false
348+
},
349+
"outputs": [],
350+
"source": [
351+
"from KTHTests import *\n",
352+
"\n",
353+
"#Setup video blocks\n",
354+
"BlockLen = 160\n",
355+
"BlockHop = 80\n",
356+
"win = 20\n",
357+
"dim = 20\n",
358+
"\n",
359+
"scores = np.zeros(len(videos))\n",
360+
"#Loop through each video and record the maximum persistence score\n",
361+
"for i in range(len(videos)):\n",
362+
" (XOrig, FrameDims) = loadVideo(videos[i])\n",
363+
" X = getPCAVideo(XOrig)\n",
364+
" [X, validIdx] = getTimeDerivative(X, 10)\n",
365+
"\n",
366+
" idxs = []\n",
367+
" N = X.shape[0]\n",
368+
" NBlocks = int(np.ceil(1 + (N - BlockLen)/BlockHop))\n",
369+
" print(\"NBlocks = \", NBlocks)\n",
370+
" for k in range(NBlocks):\n",
371+
" thisidxs = np.arange(k*BlockHop, k*BlockHop+BlockLen, dtype=np.int64)\n",
372+
" thisidxs = thisidxs[thisidxs < N]\n",
373+
" idxs.append(thisidxs)\n",
374+
"\n",
375+
" res = np.zeros(NBlocks)\n",
376+
"\n",
377+
" #Get sliding window video in blocks\n",
378+
" maxXS = []\n",
379+
" maxPD = []\n",
380+
" for j in range(len(idxs)):\n",
381+
" idx = idxs[j]\n",
382+
" Tau = win/float(dim-1)\n",
383+
" dT = (len(idx)-dim*Tau)/float(len(idx))\n",
384+
" XS = getSlidingWindowVideo(X[idx, :], dim, Tau, dT)\n",
385+
"\n",
386+
" #Mean-center and normalize sliding window\n",
387+
" XS = XS - np.mean(XS, 1)[:, None]\n",
388+
" XS = XS/np.sqrt(np.sum(XS**2, 1))[:, None]\n",
389+
"\n",
390+
" PDs = doRipsFiltration(XS, 1)\n",
391+
" if len(PDs) < 2:\n",
392+
" continue\n",
393+
" if PDs[1].size > 0:\n",
394+
" res[j] = np.max(PDs[1][:, 1] - PDs[1][:, 0])\n",
395+
" if res[j] > scores[i]:\n",
396+
" scores[i] = res[j]\n",
397+
" maxXS = np.array(XS)\n",
398+
" maxPD = np.array(PDs[1])\n",
399+
"\n",
400+
"print(\"\\n\\n\\n\\n-----------------------\\n RESULTS\\n-----------------------\\n\")\n",
401+
"print(np.argsort(-scores)+1)"
402+
]
403+
},
404+
{
405+
"cell_type": "markdown",
406+
"metadata": {},
407+
"source": [
408+
"<h2>KTH Dataset Batch Rankings (Optional)</h2><BR>\n",
409+
"For an optional final experiment, students will split up into groups and run code on different subsets of the KTH dataset which ranks the video clips in decreasing order of periodicity. As an example, <a href = \"VideoResults/index.html\">click here</a> to see the rankings of all activities for the first 4 subjects. Groups will run the code in <a href = \"KTHTests.py\">KTHTests.py</a> after modifying it to go through the appropriate subset of the database by changing lines 53 through 55. A new web page will be generated <a href = \"VideoResults/index.html\">here</a> to show the resulting rankings. Note that a fixed window length of 20 frames is maintained throughout all of the experiments. Feel free to tweak the window size (\"win\" on line 72), the dimension of the embedding (\"dim\" on line 73), or any other parameters you think would make the results more meaningful."
264410
]
265411
},
266412
{

‎KTH/boxing/person01_boxing_d1_uncomp.ogg

731 KB
Binary file not shown.

‎KTH/handwaving/person02_handwaving_d2_uncomp.ogg

1.02 MB
Binary file not shown.

‎KTH/walking/person01_walking_d1_uncomp.ogg

1.01 MB
Binary file not shown.

‎KTHTests.py

-1
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,6 @@ def getTimeDerivative(I, Win):
8080
thisidxs = thisidxs[thisidxs < N]
8181
idxs.append(thisidxs)
8282

83-
wins = np.arange(2, 50)
8483
res = np.zeros(NBlocks)
8584

8685
#Get sliding window video in blocks

0 commit comments

Comments
 (0)
Please sign in to comment.