{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Beautiful Soup 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 环境准备"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在开始之前，你需要先安装 Beautiful Soup 4。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以通过一下代码检测你是否安装成功："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4.11.1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import bs4\n",
    "bs4.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup 4 对 HTML/XML 文件的解析是依赖于解析器的，支持 Python 标准库中的 HTML 解析器，还支持第三方的解析器 lxml 和 html5lib。下表列出了不同解析器的优缺点，你可以根据需求选择合适的解析器，并使用 pip 或其他方式安装。\n",
    "\n",
    "| 解析器               | 使用方法                                                     | 优点                                                         | 缺点                                 |\n",
    "| -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------ |\n",
    "| Python’s html.parser | `BeautifulSoup(markup, \"html.parser\")`                       | 1.Python 标准库内置 2.速度不错 3.容错性不错                  | 不如 lxml 快，不如 html5lib 容错性好 |\n",
    "| lxml’s HTML parser   | `BeautifulSoup(markup, \"lxml\")`                              | 1.极快的速度 2.容错性不错                                    | 需要安装额外的C语言库                |\n",
    "| lxml’s XML parser    | `BeautifulSoup(markup, \"lxml-xml\")`/`BeautifulSoup(markup, \"xml\")` | 1.极快的速度 2.目前唯一支持的 XML 解析器                     | 需要安装额外的C语言库                |\n",
    "| html5lib             | `BeautifulSoup(markup, \"html5lib\")`                          | 1.极佳的容错性（以浏览器的方式解析文档）2.生成 HTML5 格式文档 | 速度慢                               |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你图方便，可以选择 Python 内置的解析器，其速度和容错性也足以满足大多数需求；如果你追求速度，那么 lxml 一定是不二之选；如果待解析的 HTML 文件非常复杂，需要较高容错性，那么选择 html5lib ;如果你要解析的是 XML，就只能选择 lxml 了。\n",
    "\n",
    "另外需要注意的是，不同解析器对同一文档的解析结果仍可能不同，若文档存在不符合标准的语法，那么它被不同解析器解析后可能会生成不同结构的树型文档。因此，在团队合作中最好将使用的解析器进行说明并统一。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基本使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我们传入字符串或文件句柄到`BeautifulSoup`构造方法中，我们便可以轻松得到`soup`对象。在这个过程中，字符串/文档被转换为 Unicode，并且 HTML 实例都被转换成 Unicode 编码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html><body><p>data</p></body></html>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(\"<html>data</html>\")\n",
    "soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 对象的种类\n",
    "\n",
    "Beautiful Soup 将复杂 HTML 文档转换成一个复杂的树形结构，每个节点都是 Python 对象，所有对象可以归纳为4种: `Tag` ， `NavigableString` ， `BeautifulSoup` ， `Comment` 。我们下面来逐一介绍。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tag\n",
    "\n",
    "`Tag` 对象和原生 HTML 或 XML 文档中的 tag 相同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.element.Tag"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup('<b class=\"boldest\">Extremely bold</b>')\n",
    "tag = soup.b\n",
    "type(tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tag 中最重要的属性有 name 和 attributes。每个 tag 都有它自己的名字，可以用 `.name` 方法获取。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'b'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果改变了 tag 的 name，那将影响所有通过当前 Beautiful Soup 对象生成的文档:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html><body><blockquote class=\"boldest\">Extremely bold</blockquote></body></html>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag.name = \"blockquote\"\n",
    "soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个 tag 可能有多个属性。tag `<b class=\"boldest\">` 有一个 `class` 的属性，值为 `boldest` 。tag 的属性的操作方法与字典类似："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['boldest']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag['class']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "也可以用 `.attrs` 方法获取所有属性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'class': ['boldest']}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag.attrs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tag 的属性可以被添加，删除或修改。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<blockquote class=\"verybold\" id=\"1\">Extremely bold</blockquote>\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "tag['class'] = 'verybold'\n",
    "tag['id'] = 1\n",
    "print(tag)\n",
    "\n",
    "del tag['class']\n",
    "del tag['id']\n",
    "print(tag.get('class'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NavigableString\n",
    "\n",
    "字符串常被包含在 tag 内。Beautiful Soup 用 `NavigableString` 类来包装 tag 中的字符串："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.element.NavigableString'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Extremely bold'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(type(tag.string))\n",
    "tag.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过 `str()` 方法可以直接将 `NavigableString` 对象转换成Unicode字符串："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unicode_string = str(tag.string)\n",
    "type(unicode_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "字符串是不能直接被编辑的，但是可以通过 `replace_with()` 方法来替换字符串："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<blockquote>No longer bold</blockquote>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag.string.replace_with(\"No longer bold\")\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` 对象将已解析的文档作为一个整体表示。大多数时候可以把它看作 `Tag` 对象。你可以通过一些操作将两个已解析的文档进行合并："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
       "<document><content/><footer>Here's the footer</footer></document>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = BeautifulSoup(\"<document><content/>INSERT FOOTER HERE</document\", \"xml\")\n",
    "footer = BeautifulSoup(\"<footer>Here's the footer</footer>\", \"xml\")\n",
    "doc.find(text=\"INSERT FOOTER HERE\").replace_with(footer)\n",
    "doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` 对象并不是真正的 HTML 或 XML 的 tag，但可以使用它的 `.name` 方法, `BeautifulSoup` 对象包含了一个值为 `[document]` 的特殊属性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[document]'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comments and other special strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然上述三种对象可以几乎覆盖 HTML 和 XML 中的所有内容，但仍然会存在一些其他的特殊对象，比如注释。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.element.Comment"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = \"<b><!--Hey, buddy. Want to buy a used parser?--></b>\"\n",
    "soup = BeautifulSoup(markup, 'html.parser')\n",
    "comment = soup.b.string\n",
    "type(comment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该 `Comment` 对象只是一种特殊类型 `NavigableString`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hey, buddy. Want to buy a used parser?'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Navigating the tree\n",
    "\n",
    "以“三姐妹”的文档为例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\"\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(html_doc, 'lxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going down"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Navigating using tag names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个 Tag 可能包含多个字符串或其它的 Tag，这些都是这个 Tag 的子节点。Beautiful Soup 提供了许多操作和遍历子节点的方法。\n",
    "\n",
    "操作文档树最简单的方法就是告诉它你想获取的 tag 的 name。如果想获取 `<head>` 标签，只要用 `soup.head` ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<head><title>The Dormouse's story</title></head>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你想获取某个 tag 的子节点，可以重复调用该方法，比如 `soup.head.title` ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>The Dormouse's story</title>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.head.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.contents` and `.children`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tag 的 `.contents` 属性可以将 tag 的子节点以列表的方式输出："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<title>The Dormouse's story</title>]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"The Dormouse's story\"]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "head_tag = soup.head\n",
    "print(head_tag.contents)\n",
    "title_tag = head_tag.contents[0]\n",
    "title_tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` 对象也有子节点，在这个例子中，`<html>`标签就是它的一个子节点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'html'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(soup.contents))\n",
    "soup.contents[0].name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，string 对象没有 `.contents` 属性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以通过 `.children` generator 来遍历所有子节点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Dormouse's story\n"
     ]
    }
   ],
   "source": [
    "for child in title_tag.children:\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.descendants`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.descendants` 属性可以对 tag 的所有子孙节点进行递归遍历："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>The Dormouse's story</title>\n",
      "The Dormouse's story\n"
     ]
    }
   ],
   "source": [
    "for child in head_tag.descendants:\n",
    "    print(child)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.string`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果一个 tag 仅有一个子节点,那么这个 tag 也可以使用 `.string` 方法得到子节点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The Dormouse's story\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "title_tag.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果包含了多个子节点，那么就会返回 `None` ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    }
   ],
   "source": [
    "print(soup.html.string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.strings` and `stripped_strings`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果 tag 包含了多个子节点，可以使用 `.strings` 或 `stripped_strings` 来获取所有子节点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"The Dormouse's story\"\n",
      "'\\n'\n",
      "'\\n'\n",
      "\"The Dormouse's story\"\n",
      "'\\n'\n",
      "'Once upon a time there were three little sisters; and their names were\\n'\n",
      "'Elsie'\n",
      "',\\n'\n",
      "'Lacie'\n",
      "' and\\n'\n",
      "'Tillie'\n",
      "';\\nand they lived at the bottom of a well.'\n",
      "'\\n'\n",
      "'...'\n",
      "'\\n'\n"
     ]
    }
   ],
   "source": [
    "for string in soup.strings:\n",
    "    print(repr(string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`stripped_strings` 的作用是清除空格和空行："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"The Dormouse's story\"\n",
      "\"The Dormouse's story\"\n",
      "'Once upon a time there were three little sisters; and their names were'\n",
      "'Elsie'\n",
      "','\n",
      "'Lacie'\n",
      "'and'\n",
      "'Tillie'\n",
      "';\\nand they lived at the bottom of a well.'\n",
      "'...'\n"
     ]
    }
   ],
   "source": [
    "for string in soup.stripped_strings:\n",
    "    print(repr(string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.parent`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.parent` 属性用来获取 tag 的父节点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<head><title>The Dormouse's story</title></head>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "title_tag = soup.title\n",
    "title_tag.parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文档的根节点是 `BeautifulSoup` 对象，它的父节点是 `None` ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.BeautifulSoup'>\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "html_tag = soup.html\n",
    "print(type(html_tag.parent))\n",
    "print(soup.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.parents`\n",
    "\n",
    "`.parents` 属性可以对 tag 的所有父节点进行递归遍历："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p\n",
      "body\n",
      "html\n",
      "[document]\n"
     ]
    }
   ],
   "source": [
    "link = soup.a\n",
    "for parent in link.parents:\n",
    "    if parent is None:\n",
    "        print(parent)\n",
    "    else:\n",
    "        print(parent.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going sideways"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用一个简单的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <body>\n",
      "  <a>\n",
      "   <b>\n",
      "    text1\n",
      "   </b>\n",
      "   <c>\n",
      "    text2\n",
      "   </c>\n",
      "  </a>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "sibling_soup = BeautifulSoup(\"<a><b>text1</b><c>text2</c></b></a>\", 'lxml')\n",
    "print(sibling_soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为 `<b>` 标签和 `<c>` 标签是同一层：他们是同一个元素的子节点，所以 `<b>` 和 `<c>` 可以被称为兄弟节点。一段文档以标准格式输出时，兄弟节点有相同的缩进级别。在代码中也可以使用这种关系。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.next_sibling` and `.previous_sibling`\n",
    "\n",
    "`.next_sibling` 和 `.previous_sibling` 属性可以来查询兄弟节点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<c>text2</c>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(sibling_soup.b.previous_sibling)\n",
    "sibling_soup.b.next_sibling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "None\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<b>text1</b>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(sibling_soup.c.next_sibling)\n",
    "sibling_soup.c.previous_sibling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "值得注意的是，实际文档中的 tag 的 `.next_sibling` 和 `.previous_sibling` 属性通常可能是空白。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.next_siblings` and `.previous_siblings`\n",
    "\n",
    "`.next_siblings` 和 `.previous_siblings` 属性可以对 tag 的所有兄弟节点进行递归遍历："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "',\\n'\n",
      "<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>\n",
      "' and\\n'\n",
      "<a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>\n",
      "';\\nand they lived at the bottom of a well.'\n"
     ]
    }
   ],
   "source": [
    "for sibling in soup.a.next_siblings:\n",
    "    print(repr(sibling))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "' and\\n'\n",
      "<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>\n",
      "',\\n'\n",
      "<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>\n",
      "'Once upon a time there were three little sisters; and their names were\\n'\n"
     ]
    }
   ],
   "source": [
    "for sibling in soup.find(id=\"link3\").previous_siblings:\n",
    "    print(repr(sibling))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going back and forth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.next_element` and `.previous_element`\n",
    "\n",
    "`.next_element` 属性指向解析过程中下一个被解析的对象，结果可能与 `.next_sibling` 相同，但通常不同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ";\n",
      "and they lived at the bottom of a well.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Tillie'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_a_tag = soup.find(\"a\", id=\"link3\")\n",
    "print(last_a_tag.next_sibling)\n",
    "last_a_tag.next_element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.previous_element` 属性则指向解析过程中上一个被解析的对象。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " and\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(last_a_tag.previous_element)\n",
    "last_a_tag.previous_element.next_element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `.next_elements` and `.previous_elements`\n",
    "\n",
    "`.next_elements` 和 `.previous_elements` 迭代器就可以向前或向后访问文档的解析内容，就好像文档正在被解析一样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Tillie'\n",
      "';\\nand they lived at the bottom of a well.'\n",
      "'\\n'\n",
      "<p class=\"story\">...</p>\n",
      "'...'\n",
      "'\\n'\n"
     ]
    }
   ],
   "source": [
    "for element in last_a_tag.next_elements:\n",
    "    print(repr(element))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching the tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与遍历文档树相比，我们更常用的是搜索文档树。其中最重要的两个方法就是 `.find()` 和 `.find_all()` ，我们将在接下来主要介绍它们。我们仍然用之前的例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_doc = \"\"\"\n",
    "<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\"\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(html_doc, 'lxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kinds of filters\n",
    "\n",
    "在介绍 `find_all()` 方法之前，我们首先来了解一些常用的 filter。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A string\n",
    "\n",
    "传入一个字符串参数，将会查找与字符串完整匹配的内容，默认为 UTF-8 编码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<b>The Dormouse's story</b>]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A regular expression\n",
    "\n",
    "传入正则表达式，将会调用 `re.search()` 方法，并返回匹配的结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "body\n",
      "b\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "for tag in soup.find_all(re.compile(\"^b\")):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A list\n",
    "\n",
    "传入列表，将会返回与列表中任一元素匹配的内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<b>The Dormouse's story</b>,\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all([\"a\", \"b\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `True`\n",
    "\n",
    "`True` 可以匹配任何值，但不会返回字符串节点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "html\n",
      "head\n",
      "title\n",
      "body\n",
      "p\n",
      "b\n",
      "p\n",
      "a\n",
      "a\n",
      "a\n",
      "p\n"
     ]
    }
   ],
   "source": [
    "for tag in soup.find_all(True):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A function\n",
    "\n",
    "你还可以自定义一个函数，接收一个元素参数，并返回一个布尔值。下面是一个例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"title\"><b>The Dormouse's story</b></p>,\n",
       " <p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a> and\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>;\n",
       " and they lived at the bottom of a well.</p>,\n",
       " <p class=\"story\">...</p>]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def has_class_but_no_id(tag):\n",
    "    return tag.has_attr('class') and not tag.has_attr('id')\n",
    "soup.find_all(has_class_but_no_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以发挥想象力，创造更多满足自己需要的 filter。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find_all()`\n",
    "\n",
    "```python\n",
    "find_all(name, attrs, recursive, string, limit, **kwargs)\n",
    "```\n",
    "\n",
    "下面我们来解释一下各个参数的意义："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `name` 参数\n",
    "\n",
    "查找所有名为 `name` 的 tag。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<title>The Dormouse's story</title>]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all(\"title\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### keyword 参数\n",
    "\n",
    "如果一个指定名字的参数不是搜索内置的参数名，搜索时会把该参数当作指定名字 tag 的属性来搜索，如果包含一个名字为  id 的参数，Beautiful Soup 会搜索每个 tag 的 id 属性."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all(id=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用多个指定名字的参数可以同时过滤 tag 的多个属性："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all(href=re.compile(\"elsie\"), id='link1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有些 tag 属性在搜索不能使用，比如 HTML5 中的 data-* 属性，但是可以通过 `find_all()` 方法的 `attrs` 参数定义一个字典参数来搜索包含特殊属性的 tag："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<div data-foo=\"value\">foo!</div>]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_soup = BeautifulSoup('<div data-foo=\"value\">foo!</div>')\n",
    "data_soup.find_all(attrs={\"data-foo\": \"value\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `string` 参数\n",
    "\n",
    "通过 `string` 参数搜索文档中的字符串内容。同样接受不同的 filter。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Elsie']\n",
      "['Elsie', 'Lacie', 'Tillie']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"The Dormouse's story\", \"The Dormouse's story\"]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(soup.find_all(string=\"Elsie\"))\n",
    "print(soup.find_all(string=[\"Tillie\", \"Elsie\", \"Lacie\"]))\n",
    "soup.find_all(string=re.compile(\"Dormouse\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `limit` 参数\n",
    "\n",
    "`limit` 参数指定搜索的最大数量。对于大文档，如果我们不需要所有的结果，那么这个参数是非常有用的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all(\"a\", limit=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `recursive` 参数\n",
    "\n",
    "`find_all()` 方法默认搜索所有子孙节点，如果只想搜索当前节点的直接子节点，可以设置 `recursive`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<title>The Dormouse's story</title>]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(soup.html.find_all(\"title\"))\n",
    "soup.html.find_all(\"title\", recursive=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling a tag is like calling `find_all()`\n",
    "\n",
    "由于 `find_all()` 几乎是 Beautiful Soup 中最常用的搜索方法，所以定义了简写方法。下面左右的代码是等价的："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all(\"a\") == soup(\"a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `find()`\n",
    "\n",
    "```python\n",
    "find(name, attrs, recursive, string, **kwargs)\n",
    "```\n",
    "\n",
    "当我们只想要一个搜索结果时，使用 `find()` 方法或许是更好的选择。`find()` 方法在找不到匹配对象时，返回 `None`。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了 `find()` 和 `find_all()` 方法，Beautiful Soup 还提供了十个用于搜索的 API，它们是 `find_parents()`, `find_parent()`, `find_next_siblings()`, `find_next_sibling()`, `find_previous_siblings()`, `find_previous_sibling()`, `find_all_next()`, `find_next()`, `find_all_previous()`, `find_previous()`.它们的用法和参数都 `find_all()` 类似，通过函数名也可以大致知道它们的作用，相信你可以很快掌握它们的用法。\n",
    "\n",
    "`find()` 和 `find_all()` 方法虽然很常用，用来搜索一些结构简单的文档非常方便，但是如果你想搜索一个结构非常复杂的文档，那么 `find()` 和 `find_all()` 方法是不够的。\n",
    "\n",
    "比如，在一些场景下，你想搜索的 tag 可能除了包含的内容，和其他一些 tag 完全一样，包括 tag 的名字和其他的属性，但是它们的内容不都是你想要的。这时如果你用 `find_all()` 方法，得到的结果并不尽人意，同时筛选起来也非常困难。对于这种情况，我的建议是打开浏览器开发者模式，然后查看 HTML 源码，厘清你想要的 tag 和其他内容的父子、兄弟等层级关系，最后用 `find()` 和 `find_all()` 结合上面提到的其他十个 API 来搜索。总之，灵活应用这些搜索的 API 能使你对文档的解析更快也更轻松。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSS 选择器\n",
    "\n",
    "Beautiful Soup 支持大部分的 CSS 选择器，详见 http://www.w3.org/TR/CSS2/selector.html， 在 Tag 或 BeautifulSoup 对象的 `.select()` 方法中传入字符串参数， 即可使用 CSS 选择器的语法找到 tag。对于熟悉CSS选择器语法的人来说这是个非常方便的方法。具体用法可以参考官方文档，在此不再赘述。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying the tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup 的强项是文档树的搜索，但同时也可以方便的修改文档树"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 修改 Tag 的名称和属性\n",
    "\n",
    "在 Tag 的部分已经讲解过，请参考前面对应内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 修改 `.string`\n",
    "\n",
    "给 tag 的 `.string` 属性赋值，就相当于用当前的内容替代了原来的内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"http://example.com/\">New link text.</a>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "tag = soup.a\n",
    "tag.string = \"New link text.\"\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.append()`\n",
    "\n",
    "向 tag 中添加内容，就像 Python 列表的 `append()` 方法一样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html><body><a>FooBar</a></body></html>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Foo', 'Bar']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(\"<a>Foo</a>\")\n",
    "soup.a.append(\"Bar\")\n",
    "\n",
    "print(soup)\n",
    "soup.a.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `extend()`\n",
    "\n",
    "按顺序向 tag 中添加列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a>Soup's on</a>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Soup', \"'s\", ' ', 'on']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(\"<a>Soup</a>\", 'html.parser')\n",
    "soup.a.extend([\"'s\", \" \", \"on\"])\n",
    "\n",
    "print(soup)\n",
    "soup.a.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `NavigableString()` 和 `.new_tag()`\n",
    "\n",
    "`NavigableString()` 可以添加一段文本内容到文档中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Hello', ' there']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bs4 import NavigableString\n",
    "soup = BeautifulSoup(\"<b></b>\")\n",
    "tag = soup.b\n",
    "tag.append(\"Hello\")\n",
    "new_string = NavigableString(\" there\")\n",
    "tag.append(new_string)\n",
    "tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果想要创建一段注释，或 `NavigableString` 的任何子类，只要调用 NavigableString 的构造方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Hello', ' there', 'Nice to see you.']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bs4 import Comment\n",
    "new_comment = soup.new_string(\"Nice to see you.\", Comment)\n",
    "tag.append(new_comment)\n",
    "tag\n",
    "tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建一个 tag 最好的方法是调用工厂方法 `BeautifulSoup.new_tag()`："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<b><a href=\"http://www.example.com\"></a></b>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<b><a href=\"http://www.example.com\">Link text.</a></b>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(\"<b></b>\")\n",
    "original_tag = soup.b\n",
    "\n",
    "new_tag = soup.new_tag(\"a\", href=\"http://www.example.com\")\n",
    "original_tag.append(new_tag)\n",
    "print(original_tag)\n",
    "\n",
    "new_tag.string = \"Link text.\"\n",
    "original_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `insert()`\n",
    "\n",
    "`insert()` 方法与 `append()` 方法类似，区别是不会把新元素添加到父节点 `.contents` 属性的最后，而是把元素插入到指定的位置："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a href=\"http://example.com/\">I linked to but did not endorse <i>example.com</i></a>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['I linked to ', 'but did not endorse ', <i>example.com</i>]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "tag = soup.a\n",
    "\n",
    "tag.insert(1, \"but did not endorse \")\n",
    "print(tag)\n",
    "tag.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `insert_before` 和 `insert_after`\n",
    "\n",
    "在当前 tag 或字符串之前或之后插入内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<b><i>Don't</i>stop</b>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(\"<b>stop</b>\")\n",
    "tag = soup.new_tag(\"i\")\n",
    "tag.string = \"Don't\"\n",
    "soup.b.string.insert_before(tag)\n",
    "soup.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<b><i>Don't</i> ever stop</b>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<i>Don't</i>, ' ever ', 'stop']"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.b.i.insert_after(soup.new_string(\" ever \"))\n",
    "print(soup.b)\n",
    "soup.b.contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `clear()`\n",
    "\n",
    "清空 tag 中的内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"http://example.com/\"></a>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "tag = soup.a\n",
    "\n",
    "tag.clear()\n",
    "tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `extract()`\n",
    "\n",
    "将当前 tag 或字符串从文档树中移除，并返回移除的内容："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a href=\"http://example.com/\">I linked to </a>\n",
      "<i>example.com</i>\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup, 'html.parser')\n",
    "a_tag = soup.a\n",
    "i_tag = soup.i.extract()\n",
    "\n",
    "print(a_tag)\n",
    "print(i_tag)\n",
    "print(i_tag.parent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `decompose()`\n",
    "\n",
    "将当前节点从文档树中移除并完全销毁："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"http://example.com/\">I linked to </a>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "a_tag = soup.a\n",
    "i_tag = soup.i\n",
    "soup.i.decompose()\n",
    "\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以用 `.decomposed` 属性检查节点是否已经被销毁。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(i_tag.decomposed)\n",
    "a_tag.decomposed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `replace_with()`\n",
    "\n",
    "移除文档树中的某段内容，并用新 tag 或文本节点替代它："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a href=\"http://example.com/\">I linked to <b>example.com</b></a>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<a href=\"http://example.com/\">I linked to <b>example</b>.<i>net</i></a>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup, 'html.parser')\n",
    "a_tag = soup.a\n",
    "\n",
    "new_tag = soup.new_tag(\"b\")\n",
    "new_tag.string = \"example.com\"\n",
    "a_tag.i.replace_with(new_tag)\n",
    "\n",
    "print(a_tag)\n",
    "\n",
    "bold_tag = soup.new_tag(\"b\")\n",
    "bold_tag.string = \"example\"\n",
    "i_tag = soup.new_tag(\"i\")\n",
    "i_tag.string = \"net\"\n",
    "a_tag.b.replace_with(bold_tag, \".\", i_tag)\n",
    "\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `wrap()`\n",
    "\n",
    "对指定的 tag 元素进行包装，并返回包装后的结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div><p><b>I wish I was bold.</b></p></div>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(\"<p>I wish I was bold.</p>\")\n",
    "soup.p.string.wrap(soup.new_tag(\"b\"))\n",
    "\n",
    "soup.p.wrap(soup.new_tag(\"div\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `unwrap()`\n",
    "\n",
    "将包装的 tag 元素从包装的 tag 中移除，常用来进行标记的解包："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a href=\"http://example.com/\">I linked to example.com</a>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "a_tag = soup.a\n",
    "\n",
    "a_tag.i.unwrap()\n",
    "a_tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 输出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pretty-printing\n",
    "\n",
    "`prettify()` 方法将 Beautiful Soup 的文档树格式化后以 Unicode 编码输出，每个 XML/HTML 标签都独占一行，十分优雅。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <body>\n",
      "  <a href=\"http://example.com/\">\n",
      "   I linked to\n",
      "   <i>\n",
      "    example.com\n",
      "   </i>\n",
      "  </a>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">I linked to <i>example.com</i></a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` 对象和它的 tag 节点都可以调用 `prettify()` 方法:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a href=\"http://example.com/\">\n",
      " I linked to\n",
      " <i>\n",
      "  example.com\n",
      " </i>\n",
      "</a>\n"
     ]
    }
   ],
   "source": [
    "print(soup.a.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，由于这种输出方法会添加换行符和空白字符，所以不要用它去 reform 文档。它的意义在于让你较为清晰的看清楚文档的结构。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output formatters\n",
    "\n",
    "Beautiful Soup 输出是会将 HTML 中的特殊字符转换成 Unicode，比如 &lquot："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<html><body><p>“Hello World!” he said.</p></body></html>'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(\"&ldquo;Hello World!&rdquo; he said.\")\n",
    "str(soup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你也可以将 Unicode 编码转换为 UTF-8 编码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'<html><body><p>\\xe2\\x80\\x9cHello World!\\xe2\\x80\\x9d he said.</p></body></html>'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.encode(\"utf8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在输出过程中，你可以通过设置 `formatter` 属性来指定输出的格式。由于这部分在实际工程中应用并不多，如果你想进一步学习，请参考官方文档或阅读源代码。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `get_text()`\n",
    "\n",
    "如果只想得到 tag 中包含的文本内容，那么可以调用 `get_text()` 方法，这个方法获取到 tag 中包含的所有文版内容包括子孙 tag 中的内容，并将结果作为 Unicode 字符串返回："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nI linked to example.com\\n'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markup = '<a href=\"http://example.com/\">\\nI linked to <i>example.com</i>\\n</a>'\n",
    "soup = BeautifulSoup(markup)\n",
    "\n",
    "soup.get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以通过参数指定 tag 的文本内容的分隔符："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nI linked to |example.com|\\n'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.get_text(\"|\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "还可以去除获得文本内容的前后空白字符："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I linked to|example.com'"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.get_text(\"|\", strip=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "或者使用 `.stripped_strings` generator，获得文本列表后手动处理列表："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I linked to', 'example.com']"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[text for text in soup.stripped_strings]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 编码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beautiful Soup 会自动检测文档的编码，并将文档转换为 Unicode。如果你安装了 Python 库 `charset-normalizer`， `chardet` 或 `cchardet`，猜测编码的准确率会大幅提高。你可以通过传入 `from_encoding` 参数来指定编码方式。\n",
    "\n",
    "同时需要注意，无论输入文档的编码方式是什么，Beautiful Soup 的输出编码均为 UTF-8。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 解析部分文档\n",
    "\n",
    "`SoupStrainer` 类可以定义文档的某段内容，这样搜索文档时就不必先解析整篇文档，只会解析在 `SoupStrainer` 中定义过的文档。创建一个 `SoupStrainer` 对象并作为 parse_only 参数给 BeautifulSoup 的构造方法即可。\n",
    "\n",
    "下面我们给出三个例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import SoupStrainer\n",
    "\n",
    "only_a_tags = SoupStrainer(\"a\")\n",
    "\n",
    "only_tags_with_id_link2 = SoupStrainer(id=\"link2\")\n",
    "\n",
    "def is_short_string(string):\n",
    "    return string is not None and len(string) < 10\n",
    "only_short_strings = SoupStrainer(string=is_short_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">\n",
      " Elsie\n",
      "</a>\n",
      "<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">\n",
      " Lacie\n",
      "</a>\n",
      "<a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">\n",
      " Tillie\n",
      "</a>\n",
      "<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">\n",
      " Lacie\n",
      "</a>\n",
      "Elsie\n",
      ",\n",
      "Lacie\n",
      "and\n",
      "Tillie\n",
      "...\n",
      "\n"
     ]
    }
   ],
   "source": [
    "html_doc = \"\"\"<html><head><title>The Dormouse's story</title></head>\n",
    "<body>\n",
    "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
    "\n",
    "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
    "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
    "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
    "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
    "and they lived at the bottom of a well.</p>\n",
    "\n",
    "<p class=\"story\">...</p>\n",
    "\"\"\"\n",
    "\n",
    "print(BeautifulSoup(html_doc, \"html.parser\", parse_only=only_a_tags).prettify())\n",
    "print(BeautifulSoup(html_doc, \"html.parser\", parse_only=only_tags_with_id_link2).prettify())\n",
    "print(BeautifulSoup(html_doc, \"html.parser\", parse_only=only_short_strings).prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Debug\n",
    "\n",
    "如果想知道 Beautiful Soup 到底怎样处理一份文档，可以将文档传入 diagnose() 方法，Beautiful Soup 会输出一份报告，说明不同的解析器会怎样处理这段文档，并标出当前的解析过程会使用哪种解析器。通过如下方式开始使用：\n",
    "\n",
    "```python\n",
    "from bs4.diagnose import diagnose\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 参考及更多学习资源\n",
    "\n",
    "以上就是关于 Beautiful Soup 4 的全部内容了，希望能在解析文档的过程中帮助你。下面列出了一些参考资料和更多的学习资源：\n",
    "\n",
    "- [Beautiful Soup Documentation](https://beautiful-soup-4.readthedocs.io/en/latest/)\n",
    "- [HTML 语言基础](https://docs.net9.org/languages/html/)\n",
    "- [Python爬虫利器之Beautiful Soup的用法](https://cuiqingcai.com/1319.html)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "3d6b570fd79ac60e08d7cb15c4920cf068ddf265ddaa1ba9d441106fe4b8e39c"
  },
  "kernelspec": {
   "display_name": "Python 3.8.13 ('summer')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}