Lab2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## OS Lab2\n",
    "\n",
    "Producer-consumer problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Producer-consumer problem\n",
    "- 一或多個 producer thread 製造工作\n",
    "- 一或多個 consumer thread 完成工作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 例如:\n",
    "- producer 掃描檔案目錄，將巢狀目錄名稱給 consumer\n",
    "- consumer 計算目錄中所含檔案的數量"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$producer(s)\\rightarrow (buffer)\\rightarrow consumer(s)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- buffer具有容量限制\n",
    "- buffer已滿時producer不能加入資料\n",
    "- buffer空時consumer不取出資料"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lock in Python\n",
    "- 回憶前一天範例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 x 3 =1 0\n",
      " x 3 = 3\n",
      "23 x 3 = 6\n",
      " x 3 = 9\n"
     ]
    }
   ],
   "source": [
    "import threading\n",
    "\n",
    "def thread_func(num1, num2):\n",
    "    print(num1,'x',num2,'=',num1 * num2)\n",
    "\n",
    "def main():\n",
    "    # How many thread you want to use\n",
    "    thread_num = 4\n",
    "    threads = []\n",
    "\n",
    "    # Assign job to threads\n",
    "    for i in range(thread_num):\n",
    "        # Pass argument to function with tuple\n",
    "        thread = threading.Thread(target = thread_func, args = (i, 3))\n",
    "        threads.append(thread)\n",
    "\n",
    "    # run all threads\n",
    "    for thread in threads:\n",
    "        thread.start()\n",
    "\n",
    "    # Wait for threads finish\n",
    "    for thread in threads:\n",
    "        thread.join()\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 未使用lock造成輸出混亂"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 x 3 = 0\n",
      "1 x 3 = 3\n",
      "2 x 3 = 6\n",
      "3 x 3 = 9\n"
     ]
    }
   ],
   "source": [
    "import threading\n",
    "\n",
    "lock = threading.Lock()\n",
    "\n",
    "def thread_func(num1, num2):\n",
    "    lock.acquire()\n",
    "    print(num1,'x',num2,'=',num1 * num2)\n",
    "    lock.release()\n",
    "    \n",
    "def main():\n",
    "    # How many thread you want to use\n",
    "    thread_num = 4\n",
    "    threads = []\n",
    "\n",
    "    # Assign job to threads\n",
    "    for i in range(thread_num):\n",
    "        # Pass argument to function with tuple\n",
    "        thread = threading.Thread(target = thread_func, args = (i, 3))\n",
    "        threads.append(thread)\n",
    "\n",
    "    # run all threads\n",
    "    for thread in threads:\n",
    "        thread.start()\n",
    "\n",
    "    # Wait for threads finish\n",
    "    for thread in threads:\n",
    "        thread.join()\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## File operation in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- `os.listdir(path)`: 以list形式列出path中所有檔案\n",
    "- `os.path.join(path1, path2)`: 將path1和path2連結後傳回新路徑\n",
    "- `os.path.isdir(path) 和 os.path.isfile(path)`: 檢查`path`是檔案或是目錄\n",
    "- [Official document](https://docs.python.org/3.6/library/os.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thread-safe queue in python\n",
    "- 特別注意`put()`及`get()`函數\n",
    "- [Official document](https://docs.python.org/3/library/queue.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lab2\n",
    "- 建立maxsize=5的queue(`queue.Queue(5)`)作為buffer\n",
    "- 定義`producer(top_dir, queue_buffer)`和`consumer(queue_buffer)`\n",
    "    - `producer`找出`top_dir`中的所有sub-directory，將路徑放入`queue_buffer`\n",
    "    - `consumer`從`queue_buffer`中取出路徑並計算其中的檔案數量(不含sub-directory)，並將數量加總至一個global變數`file_count`\n",
    "- Main thread在工作完成後印出總檔案數量\n",
    "- 使用`Lock`保護必要資料\n",
    "- 請使用助教提供的`testdata`資料夾，正確答案為114個檔案"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 files found.\n"
     ]
    }
   ],
   "source": [
    "import threading\n",
    "import queue\n",
    "import os\n",
    "\n",
    "buffer_size = 5\n",
    "\n",
    "lock = threading.Lock()\n",
    "queue = queue.Queue(buffer_size)\n",
    "file_count = 0\n",
    "\n",
    "def producer(top_dir, queue_buffer):\n",
    "    # Search sub-dir in top_dir and put them in queue\n",
    "    pass\n",
    "\n",
    "def consumer(queue_buffer):\n",
    "    # search file in directory\n",
    "    pass\n",
    "\n",
    "\n",
    "def main():\n",
    "    producer_thread = threading.Thread(target = producer, args = ('./testdata', queue))\n",
    "\n",
    "    consumer_count = 20\n",
    "    consumers = []\n",
    "    for i in range(consumer_count):\n",
    "        consumers.append(threading.Thread(target = consumer, args = (queue,)))\n",
    "\n",
    "    producer_thread.start()\n",
    "    for c in consumers:\n",
    "        c.start()\n",
    "\n",
    "    producer_thread.join()\n",
    "    for c in consumers:\n",
    "        c.join()\n",
    "\n",
    "    print(file_count, 'files found.')\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}