learn_pytorch_52_pytorch自然语言处理实战前的准备之gensim简介.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1 为什么要使用gensim? \n",
    "因为gensim中的word2vec可以实现中文文本向量化！！！  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 gensim开源自然语言处理库  \n",
    "官网：https://radimrehurek.com/gensim/    \n",
    "gensim的核心概念：   \n",
    "(1) Document:  字符串    \n",
    "(2) Corpus:  无监督学习语料库是Document对象的集合,作用1用作模型训练的输入，作用2用作主题提取器。  \n",
    "(3) Vector： 将语料库的每个文档表示为特征向量。    \n",
    "(4) Model ：使用模型对向量化的语料库进行转换。   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 gensim中的重要用法  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  3.1 Dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "from gensim import corpora\n",
    "from pprint import pprint\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = [\n",
    "   \"CNTK formerly known as Computational Network Toolkit\",\n",
    "   \"is a free easy-to-use open-source commercial-grade toolkit\",\n",
    "   \"that enable us to train deep learning algorithms to learn like the human brain.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_tokens = [[text for text in doc.split()] for doc in doc]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict_LoS = corpora.Dictionary(text_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(27 unique tokens: ['CNTK', 'Computational', 'Network', 'Toolkit', 'as']...)\n"
     ]
    }
   ],
   "source": [
    "print(dict_LoS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'CNTK': 0, 'Computational': 1, 'Network': 2, 'Toolkit': 3, 'as': 4, 'formerly': 5, 'known': 6, 'a': 7, 'commercial-grade': 8, 'easy-to-use': 9, 'free': 10, 'is': 11, 'open-source': 12, 'toolkit': 13, 'algorithms': 14, 'brain.': 15, 'deep': 16, 'enable': 17, 'human': 18, 'learn': 19, 'learning': 20, 'like': 21, 'that': 22, 'the': 23, 'to': 24, 'train': 25, 'us': 26}\n"
     ]
    }
   ],
   "source": [
    "print(dict_LoS.token2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2  BoW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "import pprint\n",
    "from gensim import corpora\n",
    "from gensim.utils import simple_preprocess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_list = [\n",
    "   \"Hello, how are you?\", \"How do you do?\", \n",
    "   \"Hey what are you doing? yes you What are you doing?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_tokenized = [simple_preprocess(doc) for doc in doc_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dictionary = corpora.Dictionary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "BoW_corpus = [dictionary.doc2bow(doc, allow_update=True) for doc in doc_tokenized]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[(0, 1), (1, 1), (2, 1), (3, 1)], [(2, 1), (3, 1), (4, 2)], [(0, 2), (3, 3), (5, 2), (6, 1), (7, 2), (8, 1)]]\n"
     ]
    }
   ],
   "source": [
    "print(BoW_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[('are', 1), ('hello', 1), ('how', 1), ('you', 1)], [('how', 1), ('you', 1), ('do', 2)], [('are', 2), ('you', 3), ('doing', 2), ('hey', 1), ('what', 2), ('yes', 1)]]\n"
     ]
    }
   ],
   "source": [
    "id_words = [[(dictionary[id], count) for id, count in line] for line in BoW_corpus]\n",
    "print(id_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}