1.working_with_rdd_1 (1).ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# python 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Spark & Python: Working with RDDs (I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- DataSets :  KDD Cup 1999 ( http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html )\n",
    "\n",
    "This is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, which was held in conjunction with KDD-99 The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between ``bad'' connections, called intrusions or attacks, and ``good'' normal connections. This database contains a standard set of data to be audited, which includes a wide variety of intrusions simulated in a military network environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RDD Creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) Getting the data files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import urllib\n",
    "\n",
    "f = urllib.request.urlretrieve(\"http://kdd.ics.uci.edu/databases/kddcup99/kddcup.data_10_percent.gz\", \n",
    "                       \"kddcup.data_10_percent.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (2) Creating a RDD from a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_file = \"./kddcup.data_10_percent.gz\"\n",
    "raw_data = sc.textFile(data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- SparkContext.textFile은 압축된 파일을 직접 다룰 수 있음."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "494021"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data count\n",
    "\n",
    "raw_data.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0,tcp,http,SF,181,5450,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,9,9,1.00,0.00,0.11,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,239,486,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,19,19,1.00,0.00,0.05,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,235,1337,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,29,29,1.00,0.00,0.03,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,219,1337,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,39,39,1.00,0.00,0.03,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,217,2032,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,49,49,1.00,0.00,0.02,0.00,0.00,0.00,0.00,0.00,normal.']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 처음 다섯 줄\n",
    "\n",
    "raw_data.take(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (3) filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "normal_raw_data = raw_data.filter(lambda x: 'normal.' in x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal count :  97278\n",
      "걸린 시간 : 1.009 seconds\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "\n",
    "start = time()\n",
    "\n",
    "normal_count = normal_raw_data.count()\n",
    "\n",
    "diff = time() - start\n",
    "\n",
    "print('normal count : ', normal_count),\n",
    "print('걸린 시간 :', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- filter의 결과만 남음"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (4) map\n",
    "- 모든 요소에 특정 함수 적용 가능"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 : 0.138 seconds\n",
      "[['0', 'tcp', 'http', 'SF', '181', '5450', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '8', '8', '0.00', '0.00', '0.00', '0.00', '1.00', '0.00', '0.00', '9', '9', '1.00', '0.00', '0.11', '0.00', '0.00', '0.00', '0.00', '0.00', 'normal.'], ['0', 'tcp', 'http', 'SF', '239', '486', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '8', '8', '0.00', '0.00', '0.00', '0.00', '1.00', '0.00', '0.00', '19', '19', '1.00', '0.00', '0.05', '0.00', '0.00', '0.00', '0.00', '0.00', 'normal.'], ['0', 'tcp', 'http', 'SF', '235', '1337', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '8', '8', '0.00', '0.00', '0.00', '0.00', '1.00', '0.00', '0.00', '29', '29', '1.00', '0.00', '0.03', '0.00', '0.00', '0.00', '0.00', '0.00', 'normal.'], ['0', 'tcp', 'http', 'SF', '219', '1337', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '6', '6', '0.00', '0.00', '0.00', '0.00', '1.00', '0.00', '0.00', '39', '39', '1.00', '0.00', '0.03', '0.00', '0.00', '0.00', '0.00', '0.00', 'normal.'], ['0', 'tcp', 'http', 'SF', '217', '2032', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '6', '6', '0.00', '0.00', '0.00', '0.00', '1.00', '0.00', '0.00', '49', '49', '1.00', '0.00', '0.02', '0.00', '0.00', '0.00', '0.00', '0.00', 'normal.']]\n"
     ]
    }
   ],
   "source": [
    "csv_data = raw_data.map(lambda x: x.split(\",\"))\n",
    "start = time()\n",
    "head_rows = csv_data.take(5)\n",
    "diff = time() - start\n",
    "print('걸린 시간 :', round(diff,3), 'seconds')\n",
    "print(head_rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 : 2.212 seconds\n"
     ]
    }
   ],
   "source": [
    "# 100000\n",
    "start = time()\n",
    "head_rows = csv_data.take(100000)\n",
    "diff = time() - start\n",
    "print('걸린 시간 :', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** - Using map with predefined functions **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('normal.', ['0', 'tcp', 'http', 'SF', '181', '5450', '0', '0', '0', '0', '0', '1', '0', '0', '0', '0', '0', '0', '0', '0', '0', '0', '8', '8', '0.00', '0.00', '0.00', '0.00', '1.00', '0.00', '0.00', '9', '9', '1.00', '0.00', '0.11', '0.00', '0.00', '0.00', '0.00', '0.00', 'normal.'])\n"
     ]
    }
   ],
   "source": [
    "def parse_interaction(line):\n",
    "    elems = line.split(\",\")\n",
    "    # normal 위치\n",
    "    tag = elems[41]\n",
    "    return (tag, elems)\n",
    "\n",
    "key_csv_data = raw_data.map(parse_interaction)\n",
    "head_rows = key_csv_data.take(5)\n",
    "print(head_rows[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- map을 function과 함께 이용하면 action을 할 때 바로 적용 가능"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (5) collect\n",
    "- RDD에 있는 모든 요소를 메모리로 올림"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 : 4.562 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "\n",
    "all_raw_data = raw_data.collect()\n",
    "\n",
    "diff = time() - start\n",
    "\n",
    "print('걸린 시간 :', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(all_raw_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0,tcp,http,SF,181,5450,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,9,9,1.00,0.00,0.11,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,239,486,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,19,19,1.00,0.00,0.05,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,235,1337,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,29,29,1.00,0.00,0.03,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,219,1337,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,39,39,1.00,0.00,0.03,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,217,2032,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,49,49,1.00,0.00,0.02,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,217,2032,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,6,6,0.00,0.00,0.00,0.00,1.00,0.00,0.00,59,59,1.00,0.00,0.02,0.00,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,212,1940,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1,2,0.00,0.00,0.00,0.00,1.00,0.00,1.00,1,69,1.00,0.00,1.00,0.04,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,159,4087,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,5,5,0.00,0.00,0.00,0.00,1.00,0.00,0.00,11,79,1.00,0.00,0.09,0.04,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,210,151,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,8,89,1.00,0.00,0.12,0.04,0.00,0.00,0.00,0.00,normal.',\n",
       " '0,tcp,http,SF,212,786,0,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,8,8,0.00,0.00,0.00,0.00,1.00,0.00,0.00,8,99,1.00,0.00,0.12,0.05,0.00,0.00,0.00,0.00,normal.']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_raw_data[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pyspark.rdd.RDD"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(raw_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 모든 spark 워커 노드들은 RDD의 조각들을 가지고 있는데, collect를 실행 시 모든 RDD 조각들을 호출(?)하고 하나로 합친다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 :  4.706 seconds\n",
      "normal 개수 :  97278\n"
     ]
    }
   ],
   "source": [
    "# get data from file\n",
    "data_file = \"./kddcup.data_10_percent.gz\"\n",
    "raw_data = sc.textFile(data_file)\n",
    "\n",
    "# parse into key-value pairs\n",
    "key_csv_data = raw_data.map(parse_interaction)\n",
    "\n",
    "# filter normal key interactions\n",
    "normal_key_interactions = key_csv_data.filter(lambda x: x[0] == \"normal.\")\n",
    "\n",
    "# collect all\n",
    "\n",
    "start = time()\n",
    "all_normal = normal_key_interactions.collect()\n",
    "diff = time() - start\n",
    "normal_count = len(all_normal)\n",
    "\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')\n",
    "print ('normal 개수 : ', normal_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 :  2.566 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "normal_key_interactions.count()\n",
    "diff = time() - start\n",
    "\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 :  0.0 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "normal_count = len(all_normal)\n",
    "diff = time() - start\n",
    "\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- count를 사용해서 개수를 세는 것보다 collet후 개수를 세는 것이 훨씬 빠름\n",
    "- 반복 연산이 필요할 때는 count를 반복해서 시행하는 것보다 collect후 len()을 사용하는 것이 더 나음."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling RDDs\n",
    "- sample\n",
    "- takeSample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) sample\n",
    "- transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "parameters : withReplacement, fraction, seed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sample size : 49351\n",
      "total size :  494021\n"
     ]
    }
   ],
   "source": [
    "raw_data_sample = raw_data.sample(False, 0.1, 122)\n",
    "sample_size = raw_data_sample.count()\n",
    "total_size = raw_data.count()\n",
    "\n",
    "print ('sample size :', sample_size),\n",
    "print ('total size : ', total_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 추가 transformation의 일부로 사용될 때 유용함.\n",
    "- aggregation과 key-value 페어 연산에는 더 강력함.\n",
    "- MLlib를 이용할 때 특히 유용함."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal이 포함된 row 비율 :  0.195\n",
      "걸린 시간 :  1.281 seconds\n"
     ]
    }
   ],
   "source": [
    "# sample data에서 normal이 포함된 row의 개수를 세어보자.\n",
    "\n",
    "# transformations to be applied\n",
    "raw_data_sample_items = raw_data_sample.map(lambda x: x.split(\",\"))\n",
    "sample_normal_tags = raw_data_sample_items.filter(lambda x: \"normal.\" in x)\n",
    "\n",
    "# actions + time\n",
    "start = time()\n",
    "sample_normal_tags_count = sample_normal_tags.count()\n",
    "diff = time() - start\n",
    "\n",
    "sample_normal_ratio = sample_normal_tags_count / float(sample_size)\n",
    "\n",
    "print ('normal이 포함된 row 비율 : ', round(sample_normal_ratio,3))\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal이 포함된 row 비율 :  0.195\n",
      "걸린 시간 :  2.825 seconds\n"
     ]
    }
   ],
   "source": [
    "# raw data에서 normal이 포함된 row의 개수를 세어보자.\n",
    "\n",
    "# transformations to be applied\n",
    "raw_data_items = raw_data.map(lambda x: x.split(\",\"))\n",
    "normal_tags = raw_data_items.filter(lambda x: \"normal.\" in x)\n",
    "\n",
    "# actions + time\n",
    "start = time()\n",
    "normal_tags_count = normal_tags.count()\n",
    "diff = time() - start\n",
    "\n",
    "normal_ratio = normal_tags_count / float(total_size)\n",
    "\n",
    "print ('normal이 포함된 row 비율 : ', round(sample_normal_ratio,3))\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- raw data에서 수행시 시간이 더 걸림. 당연한 결과..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (2) takeSample\n",
    "- sample 과 비슷하지만 샘플사이즈를 비율이 아닌 크기로 정의할 수 있음.\n",
    "- action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "normal이 포함된 row 비율 :  0.197\n",
      "걸린 시간 :  3.733 seconds\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "raw_data_sample = raw_data.takeSample(False, 400000, 1234)\n",
    "normal_data_sample = [x.split(\",\") for x in raw_data_sample if \"normal.\" in x]\n",
    "diff = time() - start\n",
    "\n",
    "normal_sample_size = len(normal_data_sample)\n",
    "\n",
    "normal_ratio = normal_sample_size / 400000.0\n",
    "\n",
    "print ('normal이 포함된 row 비율 : ', round(normal_ratio,3))\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- sample transformation보다 시간이 더 걸린 이유는 샘플링 프로세스를 분산시켜서 수행했기 때문.\n",
    "- filtering과 splitting을 싱글 노드에서 수행함. ?????\n",
    "\n",
    "However, it took longer, even with a slightly smaller sample. The reason is that Spark just distributed the execution of the sampling process. The filtering and splitting of the results were done locally in a single node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set operations on RDDs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- RDD에서 union과 intersection을 수행할 수 있음.\n",
    "- RDD는 실제하는 데이터 셋이 아니므로 union을 수행시 중복 데이터를 제거하지 않음.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) Getting attack interactions using substract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# filter noraml\n",
    "\n",
    "normal_raw_data = raw_data.filter(lambda x: \"normal.\" in x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# filter \"attact\" by using subtract\n",
    "\n",
    "attack_raw_data = raw_data.subtract(normal_raw_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 :  1.22 seconds\n"
     ]
    }
   ],
   "source": [
    "# count all\n",
    "\n",
    "start = time()\n",
    "raw_data_count = raw_data.count()\n",
    "diff = time() - start\n",
    "\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 :  0.935 seconds\n"
     ]
    }
   ],
   "source": [
    "# count normal\n",
    "\n",
    "start = time()\n",
    "normal_raw_data_count = normal_raw_data.count()\n",
    "diff = time() - start\n",
    "\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "걸린 시간 :  7.432 seconds\n"
     ]
    }
   ],
   "source": [
    "# count attacks\n",
    "\n",
    "start = time()\n",
    "attack_raw_data_count = attack_raw_data.count()\n",
    "diff = time() - start\n",
    "\n",
    "print ('걸린 시간 : ', round(diff,3), 'seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "noraml data :  97278\n",
      "attact data :  396743\n"
     ]
    }
   ],
   "source": [
    "print('noraml data : ', normal_raw_data_count)\n",
    "print('attact data : ', attack_raw_data_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) Protocol and service combinations using cartesian\n",
    "- Cartesian product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['icmp', 'udp', 'tcp']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "csv_data = raw_data.map(lambda x: x.split(\",\"))\n",
    "protocols = csv_data.map(lambda x: x[1]).distinct()\n",
    "protocols.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['finger',\n",
       " 'http',\n",
       " 'netbios_dgm',\n",
       " 'name',\n",
       " 'hostnames',\n",
       " 'vmnet',\n",
       " 'systat',\n",
       " 'shell',\n",
       " 'netbios_ssn',\n",
       " 'urh_i',\n",
       " 'pop_3',\n",
       " 'ctf',\n",
       " 'domain',\n",
       " 'mtp',\n",
       " 'remote_job',\n",
       " 'exec',\n",
       " 'supdup',\n",
       " 'http_443',\n",
       " 'sunrpc',\n",
       " 'urp_i',\n",
       " 'pop_2',\n",
       " 'csnet_ns',\n",
       " 'smtp',\n",
       " 'whois',\n",
       " 'ldap',\n",
       " 'daytime',\n",
       " 'imap4',\n",
       " 'nntp',\n",
       " 'klogin',\n",
       " 'rje',\n",
       " 'IRC',\n",
       " 'link',\n",
       " 'eco_i',\n",
       " 'tftp_u',\n",
       " 'iso_tsap',\n",
       " 'uucp_path',\n",
       " 'auth',\n",
       " 'ecr_i',\n",
       " 'other',\n",
       " 'domain_u',\n",
       " 'courier',\n",
       " 'discard',\n",
       " 'red_i',\n",
       " 'tim_i',\n",
       " 'time',\n",
       " 'login',\n",
       " 'ftp',\n",
       " 'telnet',\n",
       " 'ntp_u',\n",
       " 'sql_net',\n",
       " 'echo',\n",
       " 'private',\n",
       " 'gopher',\n",
       " 'efs',\n",
       " 'netbios_ns',\n",
       " 'ftp_data',\n",
       " 'nnsp',\n",
       " 'ssh',\n",
       " 'netstat',\n",
       " 'uucp',\n",
       " 'Z39_50',\n",
       " 'kshell',\n",
       " 'X11',\n",
       " 'bgp',\n",
       " 'pm_dump',\n",
       " 'printer']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "services = csv_data.map(lambda x: x[2]).distinct()\n",
    "services.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "protocol, services 조합 개수 :  198\n"
     ]
    }
   ],
   "source": [
    "product = protocols.cartesian(services).collect()\n",
    "\n",
    "print ('protocol, services 조합 개수 : ', len(product))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}