chapter4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第四章：文本和字节"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 介绍：\n",
    "- 字符码位和字节表述\n",
    "- bytes、bytearray和memoryview\n",
    "- 编解码器\n",
    "- 避免和处理编解码错误\n",
    "- 处理文本文件\n",
    "- 默认编码的陷阱和标准ＩＯ问题\n",
    "- 规范化unicode文本，进行安全比较\n",
    "- locale和pyUCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## python3中的str对象获得的元素是unicode字符"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 码位和字节序列之间转换即编解码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "b'caf\\xc3\\xa9'\n",
      "5\n",
      "b'caf\\xc3\\xa9'\n"
     ]
    }
   ],
   "source": [
    "s='café'\n",
    "print(len(s))\n",
    "b=s.encode('utf8')\n",
    "print(b)\n",
    "print(len(b))\n",
    "b.decode('utf8')\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## bytes或bytearray对象是介于０－２５５之间的整数，切片类型与原类型相同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'caf\\xc3\\xa9'\n",
      "99\n",
      "b'c'\n",
      "bytearray(b'caf\\xc3\\xa9')\n",
      "bytearray(b'\\xa9')\n"
     ]
    }
   ],
   "source": [
    "cafe=bytes('café',encoding='utf-8')\n",
    "print(cafe)\n",
    "print(cafe[0])\n",
    "print(cafe[:1])\n",
    "cafe_arr=bytearray(cafe)\n",
    "print(cafe_arr)\n",
    "print(cafe_arr[-1:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## bytes,fromhex可用于解析十六进制数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'1K\\xce\\xa9'\n"
     ]
    }
   ],
   "source": [
    "print(bytes.fromhex('31 4B CE A9'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建bytes、bytearray:\n",
    "- 使用str和encoding关键字作为参数\n",
    "- 一个可迭代对象，提供０～２５５之间的数值\n",
    "- 一个实现了缓冲协议的对象(bytes,bytearray,memoryview,array.array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'\\xfe\\xff\\xff\\xff\\x00\\x00\\x01\\x00\\x02\\x00'\n"
     ]
    }
   ],
   "source": [
    "import array\n",
    "numbers=array.array('h',[-2,-1,0,1,2])\n",
    "octets=bytes(numbers)\n",
    "print(octets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## struct模块把打包的字节序列转换成不同类型字段组成的元组，能处理bytes,bytearray,memoryview对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(b'just', b'a', b'test', 12336)\n"
     ]
    }
   ],
   "source": [
    "import struct\n",
    "\n",
    "fmt='<4s1s4sH'\n",
    "header=bytes('justatest00',encoding='utf-8')\n",
    "print(struct.unpack(fmt,header))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 尝试不同的编解码器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "latin_1\tb'EL Nino'\n",
      "utf-8\tb'EL Nino'\n",
      "utf-16\tb'\\xff\\xfeE\\x00L\\x00 \\x00N\\x00i\\x00n\\x00o\\x00'\n"
     ]
    }
   ],
   "source": [
    "for codec in ['latin_1','utf-8','utf-16']:\n",
    "    print(codec,'EL Nino'.encode(codec),sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理UnicodeEncodeError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'S&#227;o Paulo'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "city='São Paulo'\n",
    "# city.encode('cp437')\n",
    "city.encode('cp437',errors='ignore')\n",
    "city.encode('cp437',errors='replace')\n",
    "city.encode('cp437',errors='xmlcharrefreplace')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理UnicodeDecodeError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Monrt�al'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "octets=b'Monrt\\xe9al'\n",
    "octets.decode('cp1252')\n",
    "octets.decode('iso8859_7')\n",
    "octets.decode('koi8_r')\n",
    "octets.decode('utf-8',errors='replace')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理SyntaxError\n",
    "- 打开py文件出错时在文件顶部加入encoding注释\n",
    "- \\#encoding:utf-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 找出字节序列的源码：使用chardet!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 处理文本文件\n",
    "- 尽早将输入的字节序列解码成字符串\n",
    "- 在业务逻辑中只处理字符串对象\n",
    "- 尽量晚把字符串编码成字节序列\n",
    "- 需要在在不同平台运行代码，则打开文件时应始终明确encoding参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用unicodedata.normalize函数进行Unicode字符串规范化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 5\n",
      "4 4\n",
      "5 5\n"
     ]
    }
   ],
   "source": [
    "from unicodedata import normalize\n",
    "s1='café'\n",
    "s2='cafe\\u0301'\n",
    "print(len(s1),len(s2))\n",
    "print(len(normalize('NFC',s1)),len(normalize('NFC',s2)))\n",
    "print(len(normalize('NFD',s1)),len(normalize('NFD',s2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 大小写折叠"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bjblbjlhjaiubsbdjbb\n"
     ]
    }
   ],
   "source": [
    "caps='bjBLBJLHJaiUBsbdjbB'\n",
    "micro=caps.casefold()\n",
    "print(micro)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 规范化文本匹配"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "def nfc_equal(str1,str2):\n",
    "    return normalize('NFC',str1)==normalize('NFC',str2)\n",
    "def fold_equal(str1,str2):\n",
    "    return normalize('NFC',str1).casefold()==normalize('NFC',str2).casefold()\n",
    "\n",
    "s1='cafe'\n",
    "s2='Cafe'\n",
    "\n",
    "print(nfc_equal(s1,s2))\n",
    "print(fold_equal(s1,s2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unicode文本排序\n",
    "使用pyuca.Collator.sort_key方法进行排序，保证对不同语言按照当地规则进行排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['张三', '李四', '王三', '王五']\n"
     ]
    }
   ],
   "source": [
    "import pyuca\n",
    "coll=pyuca.Collator()\n",
    "sort_list_cn=['张三','李四','王五','王三']\n",
    "sorted_list_cn=sorted(sort_list_cn,key=coll.sort_key)\n",
    "print(sorted_list_cn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 中文排序貌似还是locale.strxfrm作为排序键更靠谱"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['张三', '李四', '王五', '王三']\n"
     ]
    }
   ],
   "source": [
    "import locale\n",
    "locale.setlocale(locale.LC_COLLATE,'zh_CN.UTF-8')\n",
    "sort_list_cn=['张三','李四','王五','王三']\n",
    "sorted_list_cn=sorted(sort_list_cn,key=locale.strxfrm)\n",
    "print(sort_list_cn)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}