hw_12.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPfcehD71HqUI8kWWPeUaU+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Kostiantyn-Liapkalo/nltk_spacy/blob/Main/hw_12.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "CnxKlIfYG0fP"
      },
      "outputs": [],
      "source": [
        "import spacy\n",
        "import nltk\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize, sent_tokenize\n",
        "from heapq import nlargest\n",
        "import string\n",
        "from spacy.lang.en.stop_words import STOP_WORDS"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**NLTK**"
      ],
      "metadata": {
        "id": "H2m6lQSTHFKl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text1 = 'The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering. Because it has achieved significance within the past fifty years, Criteria Consideration G applies. Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA. Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station. Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions. She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents. Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II and Block I SSME. Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory. She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission. In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle. She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew. Under Criterion C, Discovery is significant as a feat of engineering. According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials. The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield. Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System. As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.'"
      ],
      "metadata": {
        "id": "huilc7p3HGPz"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('stopwords')\n",
        "nltk.download('punkt')\n",
        "nl_stop_words = set(stopwords.words('english'))\n",
        "nl_tokens = word_tokenize(text1)\n",
        "nl_sentences = sent_tokenize(text1)\n",
        "punctuation = string.punctuation + '\\n'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w00Ry2_8HdPb",
        "outputId": "fd1db8e7-9cdc-40ee-f1c2-c157b83264d4"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('NLTK tokens =', len(nl_tokens))\n",
        "print('NLTK sentences =', len(nl_sentences))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6F6WzgVZHoEy",
        "outputId": "e759387c-43a5-4192-e2b4-a6c49f847f14"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "NLTK tokens = 519\n",
            "NLTK sentences = 16\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Порахуємо частоту використання кожного слова у тексті та нормалізуємо дані значення**"
      ],
      "metadata": {
        "id": "oNmGWbJEIKt5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nl_word_frequencies = {}\n",
        "\n",
        "for word in nl_tokens:\n",
        "  if word.lower() not in nl_stop_words:\n",
        "    if word.lower() not in punctuation:\n",
        "      if word not in nl_word_frequencies.keys():\n",
        "        nl_word_frequencies[word] = 1\n",
        "      else:\n",
        "        nl_word_frequencies[word] += 1\n",
        "\n",
        "nl_max_word = max(nl_word_frequencies.values())\n",
        "\n",
        "for w in nl_word_frequencies.keys():\n",
        "  nl_word_frequencies[w] = nl_word_frequencies[w] / nl_max_word"
      ],
      "metadata": {
        "id": "uAqpR-VHIMya"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(nl_word_frequencies)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qPKs6P1iIU4K",
        "outputId": "db643331-8372-4874-8cec-6be7f67ef1f4"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'Orbiter': 0.1, 'Discovery': 0.7, 'OV-103': 0.1, 'considered': 0.1, 'eligible': 0.1, 'listing': 0.1, 'National': 0.1, 'Register': 0.1, 'Historic': 0.1, 'Places': 0.1, 'NRHP': 0.1, 'context': 0.1, 'U.S.': 0.2, 'Space': 1.0, 'Shuttle': 0.6, 'Program': 0.2, '1969-2011': 0.1, 'Criterion': 0.4, 'areas': 0.1, 'Exploration': 0.1, 'Transportation': 0.1, 'C': 0.2, 'area': 0.1, 'Engineering': 0.1, 'achieved': 0.1, 'significance': 0.1, 'within': 0.1, 'past': 0.1, 'fifty': 0.1, 'years': 0.1, 'Criteria': 0.1, 'Consideration': 0.1, 'G': 0.1, 'applies': 0.1, 'significant': 0.2, 'oldest': 0.1, 'three': 0.1, 'extant': 0.1, 'orbiter': 0.6, 'vehicles': 0.2, 'constructed': 0.1, 'SSP': 0.2, 'longest': 0.1, 'running': 0.1, 'American': 0.1, 'space': 0.3, 'program': 0.1, 'date': 0.1, 'third': 0.1, 'five': 0.2, 'orbiters': 0.2, 'built': 0.1, 'NASA': 0.1, 'Unlike': 0.1, 'Mercury': 0.1, 'Gemini': 0.1, 'Apollo': 0.1, 'programs': 0.1, '’': 0.1, 'emphasis': 0.1, 'cost': 0.1, 'effectiveness': 0.1, 'reusability': 0.1, 'eventually': 0.1, 'construction': 0.2, 'station': 0.2, 'Including': 0.1, 'maiden': 0.1, 'voyage': 0.1, 'launched': 0.1, 'August': 0.1, '30': 0.1, '1984': 0.1, 'flew': 0.3, 'thirty-nine': 0.1, 'times': 0.2, 'four': 0.1, 'also': 0.3, 'first': 1.0, 'fly': 0.3, 'twenty': 0.1, 'missions': 0.4, 'honor': 0.1, 'chosen': 0.1, 'Return': 0.1, 'Flight': 0.1, 'vehicle': 0.1, 'Challenger': 0.2, 'Columbia': 0.1, 'accidents': 0.1, 'shuttle': 0.2, 'redesigned': 0.1, 'SRBs': 0.1, 'result': 0.1, 'accident': 0.1, 'Phase': 0.1, 'II': 0.1, 'Block': 0.1, 'SSME': 0.1, 'carried': 0.1, 'Hubble': 0.1, 'Telescope': 0.1, 'orbit': 0.1, 'performed': 0.1, 'two': 0.1, 'servicing': 0.1, 'observatory': 0.1, 'last': 0.1, 'dedicated': 0.1, 'Department': 0.1, 'Defense': 0.1, 'DoD': 0.1, 'well': 0.1, 'unclassified': 0.1, 'defense-related': 0.1, 'mission': 0.1, 'addition': 0.1, 'vital': 0.1, 'International': 0.1, 'Station': 0.1, 'ISS': 0.2, 'thirteen': 0.1, 'thirty-seven': 0.1, 'total': 0.1, 'flown': 0.2, 'dock': 0.1, 'perform': 0.1, 'exchange': 0.1, 'resident': 0.1, 'crew': 0.1, 'feat': 0.1, 'engineering': 0.2, 'According': 0.1, 'Wayne': 0.1, 'Hale': 0.2, 'flight': 0.1, 'director': 0.1, 'Johnson': 0.1, 'Center': 0.1, 'represents': 0.1, '“': 0.2, 'huge': 0.1, 'technological': 0.1, 'leap': 0.1, 'expendable': 0.1, 'rockets': 0.1, 'capsules': 0.1, 'reusable': 0.3, 'winged': 0.2, 'hypersonic': 0.2, 'cargo-carrying': 0.2, 'spacecraft.': 0.1, '”': 0.2, 'Although': 0.1, 'base': 0.2, 'structure': 0.1, 'followed': 0.1, 'conventional': 0.1, 'aircraft': 0.2, 'design': 0.1, 'used': 0.1, 'advanced': 0.1, 'materials': 0.2, 'minimized': 0.1, 'weight': 0.1, 'purposes': 0.1, 'featured': 0.2, 'low': 0.1, 'thermal': 0.1, 'expansion': 0.1, 'ratios': 0.1, 'provided': 0.1, 'stable': 0.1, 'Thermal': 0.1, 'Protection': 0.1, 'System': 0.2, 'TPS': 0.2, 'previous': 0.1, 'spaceflight': 0.1, 'single-use': 0.1, 'ablative': 0.1, 'heat': 0.1, 'shield': 0.1, 'notable': 0.1, 'achievements': 0.1, 'included': 0.1, 'orbital': 0.1, 'propulsion': 0.1, 'system': 0.1, 'two-fault-tolerant': 0.1, 'Integrated': 0.1, 'Avionics': 0.1, 'stated': 0.1, 'remains': 0.1, 'largest': 0.1, 'fastest': 0.1, 'history': 0.1, 'regularly': 0.1, 'twenty-five': 0.1, 'speed': 0.1, 'sound': 0.1}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Проведемо оцінку речень, просумувавши нормалізовані значення частоти слів всього тексту у кожному реченні**"
      ],
      "metadata": {
        "id": "62rEmXJSIZQ6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nl_sentence_scores = {}\n",
        "\n",
        "for sent in nl_sentences:\n",
        "    for word in word_tokenize(sent.lower()):\n",
        "        if word.lower() in nl_word_frequencies.keys():\n",
        "            if sent not in nl_sentence_scores:\n",
        "                nl_sentence_scores[sent] = nl_word_frequencies[word.lower()]\n",
        "            else:\n",
        "                nl_sentence_scores[sent] += nl_word_frequencies[word.lower()]\n",
        "\n",
        "display(nl_sentence_scores)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 382
        },
        "id": "PYBuOdIwIbO6",
        "outputId": "0735d701-c937-4470-becb-5cab128727f9"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "{'The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.': 2.4000000000000004,\n",
              " 'Because it has achieved significance within the past fifty years, Criteria Consideration G applies.': 0.7,\n",
              " 'Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.': 3.3000000000000007,\n",
              " 'Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station.': 1.4,\n",
              " 'Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions.': 4.4,\n",
              " 'She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents.': 0.5,\n",
              " 'Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II and Block I SSME.': 3.3000000000000003,\n",
              " 'Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory.': 1.8000000000000003,\n",
              " 'She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission.': 3.3000000000000007,\n",
              " 'In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.': 2.8000000000000007,\n",
              " 'She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew.': 3.1000000000000005,\n",
              " 'Under Criterion C, Discovery is significant as a feat of engineering.': 0.5,\n",
              " 'According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.': 6.799999999999996,\n",
              " 'The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield.': 3.7000000000000006,\n",
              " 'Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System.': 3.9000000000000004,\n",
              " 'As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.': 2.800000000000001}"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Виведемо 4 найважливіші речення, що мають слова, які найчастіше зустрічаються у документі**"
      ],
      "metadata": {
        "id": "Kr3FySmAIjo6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "summary = nlargest(4, nl_sentence_scores, key=nl_sentence_scores.get)\n",
        "print(\"Summary:\")\n",
        "for sum in summary:\n",
        "  print(sum)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wKjYq8vVIo_x",
        "outputId": "4b722fc0-5cef-4ef4-db62-8939a6c23525"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Summary:\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.\n",
            "Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions.\n",
            "Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System.\n",
            "The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**SPACY**"
      ],
      "metadata": {
        "id": "UJDuzCO8IxQj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text = 'The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering. Because it has achieved significance within the past fifty years, Criteria Consideration G applies. Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA. Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station. Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions. She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents. Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II and Block I SSME. Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory. She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission. In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle. She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew. Under Criterion C, Discovery is significant as a feat of engineering. According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.” Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials. The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield. Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System. As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.'"
      ],
      "metadata": {
        "id": "fziWqKN9IzXJ"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sp = spacy.load('en_core_web_sm')\n",
        "doc = sp(text)\n",
        "sp_tokens = word_tokenize(text)\n",
        "stop_words = list(STOP_WORDS)"
      ],
      "metadata": {
        "id": "u6ZGu_TZI_WK"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(stop_words)\n",
        "print(punctuation)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YX0gpTkUJIMS",
        "outputId": "ead4c337-aa23-434d-8102-d80a4c7d909b"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['they', 'everywhere', 'but', 'yourself', 'see', 'because', 'made', 'throughout', 'whoever', 'onto', 'more', 'if', 'perhaps', \"'ll\", 'you', 'became', 'at', 'without', 'hereby', 'are', 'him', 'put', 'whence', 'those', 'something', 'beforehand', 'ten', 'whereupon', 'whereafter', 'nine', 'up', 'could', 'hundred', 'ourselves', 'around', 'among', 'go', 'forty', '’m', 'someone', 'will', \"'ve\", 'nothing', 'thereafter', 'between', 'been', 'who', 'often', 'still', 'same', 'even', 'bottom', 'most', 'did', 'get', 'fifteen', 'seems', '’re', 'mine', 'alone', 'every', 'give', 'myself', 'there', 'themselves', 'eight', 'meanwhile', 'amount', 'becoming', 'none', 'had', 'anything', 'again', '‘m', 'us', 'move', 'anyhow', 'two', 'seemed', 'whom', 'whither', 'mostly', 'does', 'on', 'along', 'not', \"'re\", 'very', 'over', 'otherwise', 'each', 'also', '’s', 'latter', 'serious', 'indeed', 'done', 'nevertheless', 'am', 'being', 'this', 'afterwards', 'himself', 'seem', 'least', 'therein', '‘ve', '’d', 'through', 'whose', 'next', 'five', 'was', 'which', 'any', 'less', 'formerly', 'about', 'although', 'may', 'below', 'his', 'four', 'full', 'out', \"'d\", 'your', 'no', 'under', 'for', 'yourselves', 'in', 'and', 'or', 'quite', 'fifty', 're', 'much', 'would', 'ever', 'former', 'n’t', 'via', 'part', 'after', 'enough', 'into', 'ours', 'hence', 'per', 'wherein', 'should', 'one', 'we', 'herein', 'latterly', 'become', 'might', 'hereupon', 'what', 'becomes', 'where', 'as', 'whether', 'itself', 'whenever', 'rather', 'have', 'both', 'he', 'some', 'call', 'beside', 'doing', 'me', 'anyone', 'nowhere', 'well', 'besides', 'toward', 'everything', 'once', 'though', 'either', 'wherever', 'too', 'twelve', 'it', '’ll', 'moreover', 'that', 'various', 'while', 'else', 'hereafter', 'nor', 'front', 'first', 'then', 'neither', 'several', 'due', 'eleven', '‘s', 'others', 'the', 'almost', 'do', 'another', 'sometimes', 'must', 'their', 'when', '’ve', 'ca', 'since', 'really', 'thru', 'yet', 'by', 'used', 'cannot', 'say', 'side', 'further', 'seeming', 'whereas', 'back', 'therefore', 'were', 'elsewhere', 'an', 'thereby', 'down', 'whole', 'be', 'whereby', 'above', 'last', 'empty', 'twenty', 'my', 'before', 'is', 'towards', 'only', 'n‘t', 'keep', 'please', 'somehow', 'why', '‘d', 'has', 'many', \"'m\", 'behind', 'her', \"'s\", 'take', 'than', 'anyway', 'however', 'until', 'always', 'them', 'top', 'few', 'from', 'with', 'to', 'its', 'can', 'except', 'namely', 'upon', 'yours', 'against', 'all', 'noone', 'never', 'unless', 'across', 'make', 'our', 'nobody', 'sometime', 'of', 'third', \"n't\", 'these', 'anywhere', 'using', 'everyone', 'other', 'a', 'such', 'thus', 'herself', 'off', 'within', 'thereupon', 'hers', 'so', 'together', 'she', 'three', '‘re', 'regarding', 'own', 'just', 'thence', 'beyond', 'amongst', 'name', 'how', 'here', 'sixty', 'somewhere', 'already', 'i', 'show', '‘ll', 'six', 'during', 'now', 'whatever']\n",
            "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Порахуємо частоту використання кожного слова у тексті та нормалізуємо дані значення**"
      ],
      "metadata": {
        "id": "yCHqkCDLJOC_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "word_frequencies = {}\n",
        "for word in doc:\n",
        "  if word.text.lower() not in stop_words:\n",
        "    if word.text.lower() not in punctuation:\n",
        "      if word.text not in word_frequencies.keys():\n",
        "        word_frequencies[word.text] = 1\n",
        "      else:\n",
        "        word_frequencies[word.text] += 1\n",
        "\n",
        "max_word = max(word_frequencies.values())\n",
        "\n",
        "for w in word_frequencies.keys():\n",
        "  word_frequencies[w] = word_frequencies[w] / max_word"
      ],
      "metadata": {
        "id": "8F67Ka9kJPj7"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(word_frequencies)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jr5i0i21JUij",
        "outputId": "88cf03bc-38bd-4e37-958b-85da45656cc9"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'Orbiter': 0.1, 'Discovery': 0.7, 'OV-103': 0.1, 'considered': 0.1, 'eligible': 0.1, 'listing': 0.1, 'National': 0.1, 'Register': 0.1, 'Historic': 0.1, 'Places': 0.1, 'NRHP': 0.1, 'context': 0.1, 'U.S.': 0.2, 'Space': 1.0, 'Shuttle': 0.6, 'Program': 0.2, '1969': 0.1, '2011': 0.1, 'Criterion': 0.4, 'areas': 0.1, 'Exploration': 0.1, 'Transportation': 0.1, 'C': 0.2, 'area': 0.1, 'Engineering': 0.1, 'achieved': 0.1, 'significance': 0.1, 'past': 0.1, 'years': 0.1, 'Criteria': 0.1, 'Consideration': 0.1, 'G': 0.1, 'applies': 0.1, 'significant': 0.2, 'oldest': 0.1, 'extant': 0.1, 'orbiter': 0.6, 'vehicles': 0.2, 'constructed': 0.1, 'SSP': 0.2, 'longest': 0.1, 'running': 0.1, 'American': 0.1, 'space': 0.3, 'program': 0.1, 'date': 0.1, 'orbiters': 0.2, 'built': 0.1, 'NASA': 0.1, 'Unlike': 0.1, 'Mercury': 0.1, 'Gemini': 0.1, 'Apollo': 0.1, 'programs': 0.1, 'emphasis': 0.1, 'cost': 0.1, 'effectiveness': 0.1, 'reusability': 0.1, 'eventually': 0.1, 'construction': 0.2, 'station': 0.2, 'Including': 0.1, 'maiden': 0.1, 'voyage': 0.1, 'launched': 0.1, 'August': 0.1, '30': 0.1, '1984': 0.1, 'flew': 0.3, 'thirty': 0.2, 'times': 0.2, 'fly': 0.3, 'missions': 0.4, 'honor': 0.1, 'chosen': 0.1, 'Return': 0.1, 'Flight': 0.1, 'vehicle': 0.1, 'Challenger': 0.2, 'Columbia': 0.1, 'accidents': 0.1, 'shuttle': 0.2, 'redesigned': 0.1, 'SRBs': 0.1, 'result': 0.1, 'accident': 0.1, 'Phase': 0.1, 'II': 0.1, 'Block': 0.1, 'SSME': 0.1, 'carried': 0.1, 'Hubble': 0.1, 'Telescope': 0.1, 'orbit': 0.1, 'performed': 0.1, 'servicing': 0.1, 'observatory': 0.1, 'dedicated': 0.1, 'Department': 0.1, 'Defense': 0.1, 'DoD': 0.1, 'unclassified': 0.1, 'defense': 0.1, 'related': 0.1, 'mission': 0.1, 'addition': 0.1, 'vital': 0.1, 'International': 0.1, 'Station': 0.1, 'ISS': 0.2, 'thirteen': 0.1, 'seven': 0.1, 'total': 0.1, 'flown': 0.2, 'dock': 0.1, 'perform': 0.1, 'exchange': 0.1, 'resident': 0.1, 'crew': 0.1, 'feat': 0.1, 'engineering': 0.2, 'According': 0.1, 'Wayne': 0.1, 'Hale': 0.2, 'flight': 0.1, 'director': 0.1, 'Johnson': 0.1, 'Center': 0.1, 'represents': 0.1, '“': 0.2, 'huge': 0.1, 'technological': 0.1, 'leap': 0.1, 'expendable': 0.1, 'rockets': 0.1, 'capsules': 0.1, 'reusable': 0.3, 'winged': 0.2, 'hypersonic': 0.2, 'cargo': 0.2, 'carrying': 0.2, 'spacecraft': 0.1, '”': 0.2, 'base': 0.2, 'structure': 0.1, 'followed': 0.1, 'conventional': 0.1, 'aircraft': 0.2, 'design': 0.1, 'advanced': 0.1, 'materials': 0.2, 'minimized': 0.1, 'weight': 0.1, 'purposes': 0.1, 'featured': 0.2, 'low': 0.1, 'thermal': 0.1, 'expansion': 0.1, 'ratios': 0.1, 'provided': 0.1, 'stable': 0.1, 'Thermal': 0.1, 'Protection': 0.1, 'System': 0.2, 'TPS': 0.2, 'previous': 0.1, 'spaceflight': 0.1, 'single': 0.1, 'use': 0.1, 'ablative': 0.1, 'heat': 0.1, 'shield': 0.1, 'notable': 0.1, 'achievements': 0.1, 'included': 0.1, 'orbital': 0.1, 'propulsion': 0.1, 'system': 0.1, 'fault': 0.1, 'tolerant': 0.1, 'Integrated': 0.1, 'Avionics': 0.1, 'stated': 0.1, 'remains': 0.1, 'largest': 0.1, 'fastest': 0.1, 'history': 0.1, 'having': 0.1, 'regularly': 0.1, 'speed': 0.1, 'sound': 0.1}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Проведемо оцінку речень, просумувавши нормалізовані значення частоти слів всього тексту у кожному реченні**"
      ],
      "metadata": {
        "id": "BAA86kloJbLU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence_tokens = list(doc.sents)\n",
        "\n",
        "sentence_scores = {}\n",
        "\n",
        "for sent in sentence_tokens:\n",
        "    for word in sent:\n",
        "        if word.text.lower() in word_frequencies.keys():\n",
        "            if sent not in sentence_scores:\n",
        "                sentence_scores[sent] = word_frequencies[word.text.lower()]\n",
        "            else:\n",
        "                sentence_scores[sent] += word_frequencies[word.text.lower()]"
      ],
      "metadata": {
        "id": "sj9YSrplJcap"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "display(sentence_scores)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 416
        },
        "id": "GqZZC9CPJm5a",
        "outputId": "26b86786-fc7a-40d1-dd1d-1f1bc9380d14"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "{The Orbiter Discovery, OV-103, is considered eligible for listing in the National Register of Historic Places (NRHP) in the context of the U.S. Space Shuttle Program (1969-2011) under Criterion A in the areas of Space Exploration and Transportation and under Criterion C in the area of Engineering.: 2.5000000000000004,\n",
              " Because it has achieved significance within the past fifty years, Criteria Consideration G applies.: 0.5,\n",
              " Under Criterion A, Discovery is significant as the oldest of the three extant orbiter vehicles constructed for the Space Shuttle Program (SSP), the longest running American space program to date; she was the third of five orbiters built by NASA.: 2.9000000000000004,\n",
              " Unlike the Mercury, Gemini, and Apollo programs, the SSP’s emphasis was on cost effectiveness and reusability, and eventually the construction of a space station.: 1.3,\n",
              " Including her maiden voyage (launched August 30, 1984), Discovery flew to space thirty-nine times, more than any of the other four orbiters; she was also the first orbiter to fly twenty missions.: 2.9999999999999996,\n",
              " She had the honor of being chosen as the Return to Flight vehicle after both the Challenger and Columbia accidents.: 0.5,\n",
              " Discovery was the first shuttle to fly with the redesigned SRBs, a result of the Challenger accident, and the first shuttle to fly with the Phase II and Block I SSME.: 1.3,\n",
              " Discovery also carried the Hubble Space Telescope to orbit and performed two of the five servicing missions to the observatory.: 1.2000000000000002,\n",
              " She flew the first and last dedicated Department of Defense (DoD) missions, as well as the first unclassified defense-related mission.: 1.3000000000000003,\n",
              " In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.: 3.0000000000000004,\n",
              " She was the first orbiter to dock to the ISS, and the first to perform an exchange of a resident crew.: 1.0999999999999999,\n",
              " Under Criterion C, Discovery is significant as a feat of engineering.: 0.5,\n",
              " According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.”: 3.9000000000000012,\n",
              " Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.: 3.200000000000001,\n",
              " The Space Shuttle orbiter also featured the first reusable TPS; all previous spaceflight vehicles had a single-use, ablative heat shield.: 2.500000000000001,\n",
              " Other notable engineering achievements of the orbiter included the first reusable orbital propulsion system, and the first two-fault-tolerant Integrated Avionics System.: 2.0000000000000004,\n",
              " As Hale stated, the Space Shuttle remains “the largest, fastest, winged hypersonic aircraft in history,” having regularly flown at twenty-five times the speed of sound.: 2.800000000000001}"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Виведемо 3 найважливіші речення, що мають слова, які найчастіше зустрічаються у документі**"
      ],
      "metadata": {
        "id": "u7TMnXg0JuTq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "summary = nlargest(3, sentence_scores, key=sentence_scores.get)\n",
        "print(\"Summary:\")\n",
        "for sum in summary:\n",
        "  print(sum)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-NtPmwFuJvWa",
        "outputId": "74af6db3-050f-41a6-f021-c8a4e7981ad1"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Summary:\n",
            "According to Wayne Hale, a flight director from Johnson Space Center, the Space Shuttle orbiter represents a “huge technological leap from expendable rockets and capsules to a reusable, winged, hypersonic, cargo-carrying spacecraft.”\n",
            "Although her base structure followed a conventional aircraft design, she used advanced materials that both minimized her weight for cargo-carrying purposes and featured low thermal expansion ratios, which provided a stable base for her Thermal Protection System (TPS) materials.\n",
            "In addition, Discovery was vital to the construction of the International Space Station (ISS); she flew thirteen of the thirty-seven total missions flown to the station by a U.S. Space Shuttle.\n"
          ]
        }
      ]
    }
  ]
}