\n",
- "
\n", - "
\n",
- " \n",
- " \n",
- "\"\"\"\n",
- "\n",
- "headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n",
- "\n",
- "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
- "html_header_splits = html_splitter.split_text(html_string)\n",
- "html_header_splits"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "e29b4aade2a0070c",
- "metadata": {
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "source": [
- "### 2) How to constrain chunk sizes:\n",
- "\n",
- "`HTMLSectionSplitter` can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "id": "6ada8ea093ea0475",
- "metadata": {
- "ExecuteTime": {
- "end_time": "2023-10-02T18:57:51.016141300Z",
- "start_time": "2023-10-02T18:57:50.647495400Z"
- },
- "collapsed": false,
- "jupyter": {
- "outputs_hidden": false
- }
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "[Document(page_content='Foo \\n Some intro text about Foo.', metadata={'Header 1': 'Foo'}),\n",
- " Document(page_content='Bar main section \\n Some intro text about Bar.', metadata={'Header 2': 'Bar main section'}),\n",
- " Document(page_content='Bar subsection 1 \\n Some text about the first subtopic of Bar.', metadata={'Header 3': 'Bar subsection 1'}),\n",
- " Document(page_content='Bar subsection 2 \\n Some text about the second subtopic of Bar.', metadata={'Header 3': 'Bar subsection 2'}),\n",
- " Document(page_content='Baz \\n Some text about Baz \\n \\n \\n Some concluding text about Foo', metadata={'Header 2': 'Baz'})]"
- ]
- },
- "execution_count": 3,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
- "\n",
- "html_string = \"\"\"\n",
- " \n",
- " \n",
- " \n",
- " Foo
\n", - "Some intro text about Foo.
\n", - "\n",
- "
\n",
- " Bar main section
\n", - "Some intro text about Bar.
\n", - "Bar subsection 1
\n", - "Some text about the first subtopic of Bar.
\n", - "Bar subsection 2
\n", - "Some text about the second subtopic of Bar.
\n", - "\n",
- "
\n",
- " Baz
\n", - "Some text about Baz
\n", - "\n", - "
Some concluding text about Foo
\n", - "\n",
- "
\n", - "
\n",
- " \n",
- " \n",
- "\"\"\"\n",
- "\n",
- "headers_to_split_on = [\n",
- " (\"h1\", \"Header 1\"),\n",
- " (\"h2\", \"Header 2\"),\n",
- " (\"h3\", \"Header 3\"),\n",
- " (\"h4\", \"Header 4\"),\n",
- "]\n",
- "\n",
- "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
- "\n",
- "html_header_splits = html_splitter.split_text(html_string)\n",
- "\n",
- "chunk_size = 500\n",
- "chunk_overlap = 30\n",
- "text_splitter = RecursiveCharacterTextSplitter(\n",
- " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
- ")\n",
- "\n",
- "# Split\n",
- "splits = text_splitter.split_documents(html_header_splits)\n",
- "splits"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/docs/docs/how_to/index.mdx b/docs/docs/how_to/index.mdx
index 51f906c434b43..d7d1a8f2fce1a 100644
--- a/docs/docs/how_to/index.mdx
+++ b/docs/docs/how_to/index.mdx
@@ -143,8 +143,7 @@ What LangChain calls [LLMs](/docs/concepts/text_llms) are older forms of languag
[Text Splitters](/docs/concepts/text_splitters) take a document and split into chunks that can be used for retrieval.
- [How to: recursively split text](/docs/how_to/recursive_text_splitter)
-- [How to: split by HTML headers](/docs/how_to/HTML_header_metadata_splitter)
-- [How to: split by HTML sections](/docs/how_to/HTML_section_aware_splitter)
+- [How to: split HTML](/docs/how_to/split_html)
- [How to: split by character](/docs/how_to/character_text_splitter)
- [How to: split code](/docs/how_to/code_splitter)
- [How to: split Markdown by headers](/docs/how_to/markdown_header_metadata_splitter)
diff --git a/docs/docs/how_to/split_html.ipynb b/docs/docs/how_to/split_html.ipynb
new file mode 100644
index 0000000000000..c93dc66471e73
--- /dev/null
+++ b/docs/docs/how_to/split_html.ipynb
@@ -0,0 +1,963 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "7fb27b941602401d91542211134fc71a",
+ "metadata": {},
+ "source": [
+ "# How to split HTML"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "acae54e37e7d407bbb7b55eff062a284",
+ "metadata": {},
+ "source": [
+ "Splitting HTML documents into manageable chunks is essential for various text processing tasks such as natural language processing, search indexing, and more. In this guide, we will explore three different text splitters provided by LangChain that you can use to split HTML content effectively:\n",
+ "\n",
+ "- [**HTMLHeaderTextSplitter**](#using-htmlheadertextsplitter)\n",
+ "- [**HTMLSectionSplitter**](#using-htmlsectionsplitter)\n",
+ "- [**HTMLSemanticPreservingSplitter**](#using-htmlsemanticpreservingsplitter)\n",
+ "\n",
+ "Each of these splitters has unique features and use cases. This guide will help you understand the differences between them, why you might choose one over the others, and how to use them effectively."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e48a7480-5ec3-47e6-9e6a-f36c74d16f4f",
+ "metadata": {},
+ "source": [
+ "```\n",
+ "%pip install -qU langchain-text-splitters\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9a63283cbaf04dbcab1f6479b197f3a8",
+ "metadata": {},
+ "source": [
+ "## Overview of the Splitters"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8dd0d8092fe74a7c96281538738b07e2",
+ "metadata": {},
+ "source": [
+ "### [HTMLHeaderTextSplitter](#using-htmlheadertextsplitter)\n",
+ "\n",
+ ":::info\n",
+ "Useful when you want to preserve the hierarchical structure of a document based on its headings.\n",
+ ":::\n",
+ "\n",
+ "**Description**: Splits HTML text based on header tags (e.g., `Foo
\n", - "Some intro text about Foo.
\n", - "\n",
- "
\n",
- " Bar main section
\n", - "Some intro text about Bar.
\n", - "Bar subsection 1
\n", - "Some text about the first subtopic of Bar.
\n", - "Bar subsection 2
\n", - "Some text about the second subtopic of Bar.
\n", - "\n",
- "
\n",
- " Baz
\n", - "Some text about Baz
\n", - "\n", - "
Some concluding text about Foo
\n", - "`, ``, ``, etc.), and adds metadata for each header relevant to any given chunk.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Splits text at the HTML element level.\n",
+ "- Preserves context-rich information encoded in document structures.\n",
+ "- Can return chunks element by element or combine elements with the same metadata.\n",
+ "\n",
+ "___\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "72eea5119410473aa328ad9291626812",
+ "metadata": {},
+ "source": [
+ "### [HTMLSectionSplitter](#using-htmlsectionsplitter)\n",
+ "\n",
+ ":::info \n",
+ "Useful when you want to split HTML documents into larger sections, such as ``, ``, or custom-defined sections. \n",
+ ":::\n",
+ "\n",
+ "**Description**: Similar to HTMLHeaderTextSplitter but focuses on splitting HTML into sections based on specified tags.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Uses XSLT transformations to detect and split sections.\n",
+ "- Internally uses `RecursiveCharacterTextSplitter` for large sections.\n",
+ "- Considers font sizes to determine sections.\n",
+ "___"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8edb47106e1a46a883d545849b8ab81b",
+ "metadata": {},
+ "source": [
+ "### [HTMLSemanticPreservingSplitter](#using-htmlsemanticpreservingsplitter)\n",
+ "\n",
+ ":::info \n",
+ "Ideal when you need to ensure that structured elements are not split across chunks, preserving contextual relevancy. \n",
+ ":::\n",
+ "\n",
+ "**Description**: Splits HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Preserves tables, lists, and other specified HTML elements.\n",
+ "- Allows custom handlers for specific HTML tags.\n",
+ "- Ensures that the semantic meaning of the document is maintained.\n",
+ "- Built in normalization & stopword removal\n",
+ "\n",
+ "___"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "10185d26023b46108eb7d9f57d49d2b3",
+ "metadata": {},
+ "source": [
+ "### Choosing the Right Splitter\n",
+ "\n",
+ "- **Use `HTMLHeaderTextSplitter` when**: You need to split an HTML document based on its header hierarchy and maintain metadata about the headers.\n",
+ "- **Use `HTMLSectionSplitter` when**: You need to split the document into larger, more general sections, possibly based on custom tags or font sizes.\n",
+ "- **Use `HTMLSemanticPreservingSplitter` when**: You need to split the document into chunks while preserving semantic elements like tables and lists, ensuring that they are not split and that their context is maintained."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "19d42e40-015c-4bfa-b9ce-783e8377af2b",
+ "metadata": {},
+ "source": [
+ "| Feature | HTMLHeaderTextSplitter | HTMLSectionSplitter | HTMLSemanticPreservingSplitter |\n",
+ "|--------------------------------------------|------------------------|---------------------|-------------------------------|\n",
+ "| Splits based on headers | Yes | Yes | Yes |\n",
+ "| Preserves semantic elements (tables, lists) | No | No | Yes |\n",
+ "| Adds metadata for headers | Yes | Yes | Yes |\n",
+ "| Custom handlers for HTML tags | No | No | Yes |\n",
+ "| Preserves media (images, videos) | No | No | Yes |\n",
+ "| Considers font sizes | No | Yes | No |\n",
+ "| Uses XSLT transformations | No | Yes | No |"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8763a12b2bbd4a93a75aff182afb95dc",
+ "metadata": {},
+ "source": [
+ "## Example HTML Document\n",
+ "\n",
+ "Let's use the following HTML document as an example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "c9ca2682",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "html_string = \"\"\"\n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Fancy Example HTML Page \n",
+ " \n",
+ " \n",
+ " Main Title
\n",
+ " This is an introductory paragraph with some basic content.
\n",
+ " \n",
+ " Section 1: Introduction
\n",
+ " This section introduces the topic. Below is a list:
\n",
+ " \n",
+ " - First item
\n",
+ " - Second item
\n",
+ " - Third item with bold text and a link
\n",
+ "
\n",
+ " \n",
+ " Subsection 1.1: Details
\n",
+ " This subsection provides additional details. Here's a table:
\n",
+ " \n",
+ " \n",
+ " \n",
+ " Header 1 \n",
+ " Header 2 \n",
+ " Header 3 \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Row 1, Cell 1 \n",
+ " Row 1, Cell 2 \n",
+ " Row 1, Cell 3 \n",
+ " \n",
+ " \n",
+ " Row 2, Cell 1 \n",
+ " Row 2, Cell 2 \n",
+ " Row 2, Cell 3 \n",
+ " \n",
+ " \n",
+ "
\n",
+ " \n",
+ " Section 2: Media Content
\n",
+ " This section contains an image and a video:
\n",
+ " 
\n",
+ " \n",
+ "\n",
+ " Section 3: Code Example
\n",
+ " This section contains a code block:
\n",
+ " \n",
+ " <div>\n",
+ " <p>This is a paragraph inside a div.</p>\n",
+ " </div>\n",
+ "
\n",
+ "\n",
+ " Conclusion
\n",
+ " This is the conclusion of the document.
\n",
+ " \n",
+ " \n",
+ "\"\"\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "86c25b86",
+ "metadata": {},
+ "source": [
+ "## Using HTMLHeaderTextSplitter\n",
+ "\n",
+ "[HTMLHeaderTextSplitter](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html) is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the HTML element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n",
+ "\n",
+ "It is analogous to the [MarkdownHeaderTextSplitter](/docs/how_to/markdown_header_metadata_splitter) for markdown files.\n",
+ "\n",
+ "To specify what headers to split on, specify `headers_to_split_on` when instantiating `HTMLHeaderTextSplitter` as shown below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "23361e55",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a list: \\nFirst item Second item Third item with bold text and a link'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction', 'Header 3': 'Subsection 1.1: Details'}, page_content=\"This subsection provides additional details. Here's a table:\"),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block:'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLHeaderTextSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "html_header_splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f7e9cfef-5387-4ffe-b1a8-4b9214b9debd",
+ "metadata": {},
+ "source": [
+ "To return each element together with their associated headers, specify `return_each_element=True` when instantiating `HTMLHeaderTextSplitter`:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "adb0da17-d1d5-4913-aacd-dc5d70db66cf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "html_splitter = HTMLHeaderTextSplitter(\n",
+ " headers_to_split_on,\n",
+ " return_each_element=True,\n",
+ ")\n",
+ "html_header_splits_elements = html_splitter.split_text(html_string)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fa781f86-d04f-4c09-a4a1-26aac88d4fc3",
+ "metadata": {},
+ "source": [
+ "Comparing with the above, where elements are aggregated by their headers:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "d9d7b61f-f927-49fc-a592-1b0f049d1a2f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n",
+ "page_content='This section introduces the topic. Below is a list: \n",
+ "First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "for element in html_header_splits[:2]:\n",
+ " print(element)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4b5869e0-3a9b-4fa2-82ec-dbde33464a52",
+ "metadata": {},
+ "source": [
+ "Now each element is returned as a distinct `Document`:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "e2677a27-875a-4455-8e3f-6a6c7706be20",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n",
+ "page_content='This section introduces the topic. Below is a list:' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n",
+ "page_content='First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "for element in html_header_splits_elements[:3]:\n",
+ " print(element)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "24200c53-6a52-436e-ba4d-8d0514cfa87c",
+ "metadata": {},
+ "source": [
+ "### How to split from a URL or HTML file:\n",
+ "\n",
+ "To read directly from a URL, pass the URL string into the `split_text_from_url` method.\n",
+ "\n",
+ "Similarly, a local HTML file can be passed to the `split_text_from_file` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "581eeddf-7e88-48a7-999d-da56304e3522",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "url = \"https://plato.stanford.edu/entries/goedel/\"\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ " (\"h4\", \"Header 4\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "\n",
+ "# for local file use html_splitter.split_text_from_file()\n",
+ "html_header_splits = html_splitter.split_text_from_url(url)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88ba9878-068e-434f-bdac-17972b2aa9ad",
+ "metadata": {},
+ "source": [
+ "### How to constrain chunk sizes:\n",
+ "\n",
+ "`HTMLHeaderTextSplitter`, which splits based on HTML headers, can be composed with another splitter which constrains splits based on character lengths, such as `RecursiveCharacterTextSplitter`.\n",
+ "\n",
+ "This can be done using the `.split_documents` method of the second splitter:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "3efc1fb8-f264-4ae6-883d-694a4d252f86",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "chunk_size = 500\n",
+ "chunk_overlap = 30\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
+ ")\n",
+ "\n",
+ "# Split\n",
+ "splits = text_splitter.split_documents(html_header_splits)\n",
+ "splits[80:85]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "27363092-c6b2-4290-9e12-f6aece503bfb",
+ "metadata": {},
+ "source": [
+ "### Limitations\n",
+ "\n",
+ "There can be quite a bit of structural variation from one HTML document to another, and while `HTMLHeaderTextSplitter` will attempt to attach all \"relevant\" headers to any given chunk, it can sometimes miss certain headers. For example, the algorithm assumes an informational hierarchy in which headers are always at nodes \"above\" associated text, i.e. prior siblings, ancestors, and combinations thereof. In the following news article (as of the writing of this document), the document is structured such that the text of the top-level headline, while tagged \"h1\", is in a *distinct* subtree from the text elements that we'd expect it to be *\"above\"*—so we can observe that the \"h1\" element and its associated text do not show up in the chunk metadata (but, where applicable, we do see \"h2\" and its associated text): \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "332043f6-ec39-4fcd-aa57-4dec5252684b",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "No two El Niño winters are the same, but many have temperature and precipitation trends in common. \n",
+ "Average conditions during an El Niño winter across the continental US. \n",
+ "One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA. \n",
+ "Because the jet stream is essentially a river of air that storms flow through, they c\n"
+ ]
+ }
+ ],
+ "source": [
+ "url = \"https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html\"\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text_from_url(url)\n",
+ "print(html_header_splits[1].page_content[:500])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa8ba422",
+ "metadata": {},
+ "source": [
+ "## Using HTMLSectionSplitter\n",
+ "\n",
+ "Similar in concept to the [HTMLHeaderTextSplitter](#using-htmlheadertextsplitter), the `HTMLSectionSplitter` is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk. It lets you split HTML by sections.\n",
+ "\n",
+ "It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures.\n",
+ "\n",
+ "Use `xslt_path` to provide an absolute path to transform the HTML so that it can detect sections based on provided tags. The default is to use the `converting_to_header.xslt` file in the `data_connection/document_transformers` directory. This is for converting the html to a format/layout that is easier to detect sections. For example, `span` based on their font size can be converted to header tags to be detected as a section.\n",
+ "\n",
+ "### How to split HTML strings:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "65376c86",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title \\n This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\"Section 1: Introduction \\n This section introduces the topic. Below is a list: \\n \\n First item \\n Second item \\n Third item with bold text and a link \\n \\n \\n Subsection 1.1: Details \\n This subsection provides additional details. Here's a table: \\n \\n \\n \\n Header 1 \\n Header 2 \\n Header 3 \\n \\n \\n \\n \\n Row 1, Cell 1 \\n Row 1, Cell 2 \\n Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1 \\n Row 2, Cell 2 \\n Row 2, Cell 3\"),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content \\n This section contains an image and a video: \\n \\n \\n Your browser does not support the video tag.'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example \\n This section contains a code block: \\n \\n \\n This is a paragraph inside a div.
\\n '),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion \\n This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLSectionSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "html_header_splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5aa8627f-0af3-48c2-b9ed-bfbc46f8030d",
+ "metadata": {},
+ "source": [
+ "### How to constrain chunk sizes:\n",
+ "\n",
+ "`HTMLSectionSplitter` can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "5a9759b2-f6ff-413e-b42d-9d8967f3f3e6",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title'),\n",
+ " Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some'),\n",
+ " Document(metadata={'Header 1': 'Main Title'}, page_content='some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Section 1: Introduction'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='is a list:'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='First item \\n Second item'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Third item with bold text and a link'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Subsection 1.1: Details'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='This subsection provides additional details.'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content=\"Here's a table:\"),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Header 1 \\n Header 2 \\n Header 3'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 1 \\n Row 1, Cell 2'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 2, Cell 2 \\n Row 2, Cell 3'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Your browser does not support the video'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='tag.'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: \\n \\n '),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This is a paragraph inside a div.
'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content=''),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
+ "\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "\n",
+ "chunk_size = 50\n",
+ "chunk_overlap = 5\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
+ ")\n",
+ "\n",
+ "# Split\n",
+ "splits = text_splitter.split_documents(html_header_splits)\n",
+ "splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6fb9f81a",
+ "metadata": {},
+ "source": [
+ "## Using HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "The `HTMLSemanticPreservingSplitter` is designed to split HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components. This ensures that such elements are not split across chunks, causing loss of contextual relevancy such as table headers, list headers etc.\n",
+ "\n",
+ "This splitter is designed at its heart, to create contextually relevant chunks. General Recursive splitting with `HTMLHeaderTextSplitter` can cause tables, lists and other structered elements to be split in the middle, losing signifcant context and creating bad chunks.\n",
+ "\n",
+ "The `HTMLSemanticPreservingSplitter` is essential for splitting HTML content that includes structured elements like tables and lists, especially when it's critical to preserve these elements intact. Additionally, its ability to define custom handlers for specific HTML tags makes it a versatile tool for processing complex HTML documents.\n",
+ "\n",
+ "**IMPORTANT**: `max_chunk_size` is not a definite maximum size of a chunk, the calculation of max size, occurs when the preserved content is not apart of the chunk, to ensure it is not split. When we add the preserved data back in to the chunk, there is a chance the chunk size will exceed the `max_chunk_size`. This is crucial to ensure we maintain the structure of the original document\n",
+ "\n",
+ ":::info \n",
+ "\n",
+ "Notes:\n",
+ "\n",
+ "1. We have defined a custom handler to re-format the contents of code blocks\n",
+ "2. We defined a deny list for specific html elements, to decompose them and their contents pre-processing\n",
+ "3. We have intentionally set a small chunk size to demonstrate the non-splitting of elements\n",
+ "\n",
+ ":::"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "6cd119a8-c3d1-48a8-b569-469cd1607169",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3\"),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:  '),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: This is a paragraph inside a div.
'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# BeautifulSoup is required to use the custom handlers\n",
+ "from bs4 import Tag\n",
+ "from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "\n",
+ "def code_handler(element: Tag) -> str:\n",
+ " data_lang = element.get(\"data-lang\")\n",
+ " code_format = f\"{element.get_text()}\"\n",
+ "\n",
+ " return code_format\n",
+ "\n",
+ "\n",
+ "splitter = HTMLSemanticPreservingSplitter(\n",
+ " headers_to_split_on=headers_to_split_on,\n",
+ " separators=[\"\\n\\n\", \"\\n\", \". \", \"! \", \"? \"],\n",
+ " max_chunk_size=50,\n",
+ " preserve_images=True,\n",
+ " preserve_videos=True,\n",
+ " elements_to_preserve=[\"table\", \"ul\", \"ol\", \"code\"],\n",
+ " denylist_tags=[\"script\", \"style\", \"head\"],\n",
+ " custom_handlers={\"code\": code_handler},\n",
+ ")\n",
+ "\n",
+ "documents = splitter.split_text(html_string)\n",
+ "documents"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8c5413e2-3c50-435a-bda7-11e574a3fbab",
+ "metadata": {},
+ "source": [
+ "### Preserving Tables and Lists\n",
+ "In this example, we will demonstrate how the `HTMLSemanticPreservingSplitter` can preserve a table and a large list within an HTML document. The chunk size will be set to 50 characters to illustrate how the splitter ensures that these elements are not split, even when they exceed the maximum defined chunk size."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "8d97a2a6-9c73-4396-a922-f7a4eebe47f8",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[Document(metadata={'Header 1': 'Section 1'}, page_content='This section contains an important table and list'), Document(metadata={'Header 1': 'Section 1'}, page_content='that should not be split across chunks.'), Document(metadata={'Header 1': 'Section 1'}, page_content='Item Quantity Price Apples 10 $1.00 Oranges 5 $0.50 Bananas 50 $1.50'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='Additional text in subsection 1.1 that is'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='separated from the table and list. Here is a'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content=\"detailed list: Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.\")]\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "html_string = \"\"\"\n",
+ "\n",
+ "\n",
+ " \n",
+ " \n",
+ " Section 1
\n",
+ " This section contains an important table and list that should not be split across chunks.
\n",
+ " \n",
+ " \n",
+ " Item \n",
+ " Quantity \n",
+ " Price \n",
+ " \n",
+ " \n",
+ " Apples \n",
+ " 10 \n",
+ " $1.00 \n",
+ " \n",
+ " \n",
+ " Oranges \n",
+ " 5 \n",
+ " $0.50 \n",
+ " \n",
+ " \n",
+ " Bananas \n",
+ " 50 \n",
+ " $1.50 \n",
+ " \n",
+ "
\n",
+ " Subsection 1.1
\n",
+ " Additional text in subsection 1.1 that is separated from the table and list.
\n",
+ " Here is a detailed list:
\n",
+ " \n",
+ " - Item 1: Description of item 1, which is quite detailed and important.
\n",
+ " - Item 2: Description of item 2, which also contains significant information.
\n",
+ " - Item 3: Description of item 3, another item that we don't want to split across chunks.
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n",
+ "\n",
+ "splitter = HTMLSemanticPreservingSplitter(\n",
+ " headers_to_split_on=headers_to_split_on,\n",
+ " max_chunk_size=50,\n",
+ " elements_to_preserve=[\"table\", \"ul\"],\n",
+ ")\n",
+ "\n",
+ "documents = splitter.split_text(html_string)\n",
+ "print(documents)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e44bab7f-5a81-4ec4-a7d8-0413c8e15b33",
+ "metadata": {},
+ "source": [
+ "#### Explanation\n",
+ "In this example, the `HTMLSemanticPreservingSplitter` ensures that the entire table and the unordered list (``) are preserved within their respective chunks. Even though the chunk size is set to 50 characters, the splitter recognizes that these elements should not be split and keeps them intact.\n",
+ "\n",
+ "This is particularly important when dealing with data tables or lists, where splitting the content could lead to loss of context or confusion. The resulting `Document` objects retain the full structure of these elements, ensuring that the contextual relevance of the information is maintained."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0c0a83c9-1177-4f64-9f94-688b5ec4fafd",
+ "metadata": {},
+ "source": [
+ "### Using a Custom Handler\n",
+ "The `HTMLSemanticPreservingSplitter` allows you to define custom handlers for specific HTML elements. Some platforms, have custom HTML tags that are not natively parsed by `BeautifulSoup`, when this occurs, you can utilize custom handlers to add the formatting logic easily. \n",
+ "\n",
+ "This can be particularly useful for elements that require special processing, such as `\n",
+ " Some text after the iframe.
\n",
+ " \n",
+ " - Item 1: Description of item 1, which is quite detailed and important.
\n",
+ " - Item 2: Description of item 2, which also contains significant information.
\n",
+ " - Item 3: Description of item 3, another item that we don't want to split across chunks.
\n",
+ "
\n",
+ "
\n",
+ "
`, etc.), and adds metadata for each header relevant to any given chunk.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Splits text at the HTML element level.\n",
+ "- Preserves context-rich information encoded in document structures.\n",
+ "- Can return chunks element by element or combine elements with the same metadata.\n",
+ "\n",
+ "___\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "72eea5119410473aa328ad9291626812",
+ "metadata": {},
+ "source": [
+ "### [HTMLSectionSplitter](#using-htmlsectionsplitter)\n",
+ "\n",
+ ":::info \n",
+ "Useful when you want to split HTML documents into larger sections, such as ``, ``, or custom-defined sections. \n",
+ ":::\n",
+ "\n",
+ "**Description**: Similar to HTMLHeaderTextSplitter but focuses on splitting HTML into sections based on specified tags.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Uses XSLT transformations to detect and split sections.\n",
+ "- Internally uses `RecursiveCharacterTextSplitter` for large sections.\n",
+ "- Considers font sizes to determine sections.\n",
+ "___"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8edb47106e1a46a883d545849b8ab81b",
+ "metadata": {},
+ "source": [
+ "### [HTMLSemanticPreservingSplitter](#using-htmlsemanticpreservingsplitter)\n",
+ "\n",
+ ":::info \n",
+ "Ideal when you need to ensure that structured elements are not split across chunks, preserving contextual relevancy. \n",
+ ":::\n",
+ "\n",
+ "**Description**: Splits HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Preserves tables, lists, and other specified HTML elements.\n",
+ "- Allows custom handlers for specific HTML tags.\n",
+ "- Ensures that the semantic meaning of the document is maintained.\n",
+ "- Built in normalization & stopword removal\n",
+ "\n",
+ "___"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "10185d26023b46108eb7d9f57d49d2b3",
+ "metadata": {},
+ "source": [
+ "### Choosing the Right Splitter\n",
+ "\n",
+ "- **Use `HTMLHeaderTextSplitter` when**: You need to split an HTML document based on its header hierarchy and maintain metadata about the headers.\n",
+ "- **Use `HTMLSectionSplitter` when**: You need to split the document into larger, more general sections, possibly based on custom tags or font sizes.\n",
+ "- **Use `HTMLSemanticPreservingSplitter` when**: You need to split the document into chunks while preserving semantic elements like tables and lists, ensuring that they are not split and that their context is maintained."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "19d42e40-015c-4bfa-b9ce-783e8377af2b",
+ "metadata": {},
+ "source": [
+ "| Feature | HTMLHeaderTextSplitter | HTMLSectionSplitter | HTMLSemanticPreservingSplitter |\n",
+ "|--------------------------------------------|------------------------|---------------------|-------------------------------|\n",
+ "| Splits based on headers | Yes | Yes | Yes |\n",
+ "| Preserves semantic elements (tables, lists) | No | No | Yes |\n",
+ "| Adds metadata for headers | Yes | Yes | Yes |\n",
+ "| Custom handlers for HTML tags | No | No | Yes |\n",
+ "| Preserves media (images, videos) | No | No | Yes |\n",
+ "| Considers font sizes | No | Yes | No |\n",
+ "| Uses XSLT transformations | No | Yes | No |"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8763a12b2bbd4a93a75aff182afb95dc",
+ "metadata": {},
+ "source": [
+ "## Example HTML Document\n",
+ "\n",
+ "Let's use the following HTML document as an example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "c9ca2682",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "html_string = \"\"\"\n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Fancy Example HTML Page \n",
+ " \n",
+ " \n",
+ " Main Title
\n",
+ " This is an introductory paragraph with some basic content.
\n",
+ " \n",
+ " Section 1: Introduction
\n",
+ " This section introduces the topic. Below is a list:
\n",
+ " \n",
+ " - First item
\n",
+ " - Second item
\n",
+ " - Third item with bold text and a link
\n",
+ "
\n",
+ " \n",
+ " Subsection 1.1: Details
\n",
+ " This subsection provides additional details. Here's a table:
\n",
+ " \n",
+ " \n",
+ " \n",
+ " Header 1 \n",
+ " Header 2 \n",
+ " Header 3 \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Row 1, Cell 1 \n",
+ " Row 1, Cell 2 \n",
+ " Row 1, Cell 3 \n",
+ " \n",
+ " \n",
+ " Row 2, Cell 1 \n",
+ " Row 2, Cell 2 \n",
+ " Row 2, Cell 3 \n",
+ " \n",
+ " \n",
+ "
\n",
+ " \n",
+ " Section 2: Media Content
\n",
+ " This section contains an image and a video:
\n",
+ " \n",
+ " \n",
+ "\n",
+ " Section 3: Code Example
\n",
+ " This section contains a code block:
\n",
+ " \n",
+ " <div>\n",
+ " <p>This is a paragraph inside a div.</p>\n",
+ " </div>\n",
+ "
\n",
+ "\n",
+ " Conclusion
\n",
+ " This is the conclusion of the document.
\n",
+ " \n",
+ " \n",
+ "\"\"\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "86c25b86",
+ "metadata": {},
+ "source": [
+ "## Using HTMLHeaderTextSplitter\n",
+ "\n",
+ "[HTMLHeaderTextSplitter](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html) is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the HTML element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n",
+ "\n",
+ "It is analogous to the [MarkdownHeaderTextSplitter](/docs/how_to/markdown_header_metadata_splitter) for markdown files.\n",
+ "\n",
+ "To specify what headers to split on, specify `headers_to_split_on` when instantiating `HTMLHeaderTextSplitter` as shown below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "23361e55",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a list: \\nFirst item Second item Third item with bold text and a link'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction', 'Header 3': 'Subsection 1.1: Details'}, page_content=\"This subsection provides additional details. Here's a table:\"),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block:'),\n",
+ " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLHeaderTextSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "html_header_splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f7e9cfef-5387-4ffe-b1a8-4b9214b9debd",
+ "metadata": {},
+ "source": [
+ "To return each element together with their associated headers, specify `return_each_element=True` when instantiating `HTMLHeaderTextSplitter`:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "adb0da17-d1d5-4913-aacd-dc5d70db66cf",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "html_splitter = HTMLHeaderTextSplitter(\n",
+ " headers_to_split_on,\n",
+ " return_each_element=True,\n",
+ ")\n",
+ "html_header_splits_elements = html_splitter.split_text(html_string)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fa781f86-d04f-4c09-a4a1-26aac88d4fc3",
+ "metadata": {},
+ "source": [
+ "Comparing with the above, where elements are aggregated by their headers:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "d9d7b61f-f927-49fc-a592-1b0f049d1a2f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n",
+ "page_content='This section introduces the topic. Below is a list: \n",
+ "First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "for element in html_header_splits[:2]:\n",
+ " print(element)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4b5869e0-3a9b-4fa2-82ec-dbde33464a52",
+ "metadata": {},
+ "source": [
+ "Now each element is returned as a distinct `Document`:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "e2677a27-875a-4455-8e3f-6a6c7706be20",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n",
+ "page_content='This section introduces the topic. Below is a list:' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n",
+ "page_content='First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n"
+ ]
+ }
+ ],
+ "source": [
+ "for element in html_header_splits_elements[:3]:\n",
+ " print(element)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "24200c53-6a52-436e-ba4d-8d0514cfa87c",
+ "metadata": {},
+ "source": [
+ "### How to split from a URL or HTML file:\n",
+ "\n",
+ "To read directly from a URL, pass the URL string into the `split_text_from_url` method.\n",
+ "\n",
+ "Similarly, a local HTML file can be passed to the `split_text_from_file` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "581eeddf-7e88-48a7-999d-da56304e3522",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "url = \"https://plato.stanford.edu/entries/goedel/\"\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ " (\"h4\", \"Header 4\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "\n",
+ "# for local file use html_splitter.split_text_from_file()\n",
+ "html_header_splits = html_splitter.split_text_from_url(url)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88ba9878-068e-434f-bdac-17972b2aa9ad",
+ "metadata": {},
+ "source": [
+ "### How to constrain chunk sizes:\n",
+ "\n",
+ "`HTMLHeaderTextSplitter`, which splits based on HTML headers, can be composed with another splitter which constrains splits based on character lengths, such as `RecursiveCharacterTextSplitter`.\n",
+ "\n",
+ "This can be done using the `.split_documents` method of the second splitter:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "3efc1fb8-f264-4ae6-883d-694a4d252f86",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "chunk_size = 500\n",
+ "chunk_overlap = 30\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
+ ")\n",
+ "\n",
+ "# Split\n",
+ "splits = text_splitter.split_documents(html_header_splits)\n",
+ "splits[80:85]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "27363092-c6b2-4290-9e12-f6aece503bfb",
+ "metadata": {},
+ "source": [
+ "### Limitations\n",
+ "\n",
+ "There can be quite a bit of structural variation from one HTML document to another, and while `HTMLHeaderTextSplitter` will attempt to attach all \"relevant\" headers to any given chunk, it can sometimes miss certain headers. For example, the algorithm assumes an informational hierarchy in which headers are always at nodes \"above\" associated text, i.e. prior siblings, ancestors, and combinations thereof. In the following news article (as of the writing of this document), the document is structured such that the text of the top-level headline, while tagged \"h1\", is in a *distinct* subtree from the text elements that we'd expect it to be *\"above\"*—so we can observe that the \"h1\" element and its associated text do not show up in the chunk metadata (but, where applicable, we do see \"h2\" and its associated text): \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "332043f6-ec39-4fcd-aa57-4dec5252684b",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "No two El Niño winters are the same, but many have temperature and precipitation trends in common. \n",
+ "Average conditions during an El Niño winter across the continental US. \n",
+ "One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA. \n",
+ "Because the jet stream is essentially a river of air that storms flow through, they c\n"
+ ]
+ }
+ ],
+ "source": [
+ "url = \"https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html\"\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text_from_url(url)\n",
+ "print(html_header_splits[1].page_content[:500])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa8ba422",
+ "metadata": {},
+ "source": [
+ "## Using HTMLSectionSplitter\n",
+ "\n",
+ "Similar in concept to the [HTMLHeaderTextSplitter](#using-htmlheadertextsplitter), the `HTMLSectionSplitter` is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk. It lets you split HTML by sections.\n",
+ "\n",
+ "It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures.\n",
+ "\n",
+ "Use `xslt_path` to provide an absolute path to transform the HTML so that it can detect sections based on provided tags. The default is to use the `converting_to_header.xslt` file in the `data_connection/document_transformers` directory. This is for converting the html to a format/layout that is easier to detect sections. For example, `span` based on their font size can be converted to header tags to be detected as a section.\n",
+ "\n",
+ "### How to split HTML strings:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "65376c86",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title \\n This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\"Section 1: Introduction \\n This section introduces the topic. Below is a list: \\n \\n First item \\n Second item \\n Third item with bold text and a link \\n \\n \\n Subsection 1.1: Details \\n This subsection provides additional details. Here's a table: \\n \\n \\n \\n Header 1 \\n Header 2 \\n Header 3 \\n \\n \\n \\n \\n Row 1, Cell 1 \\n Row 1, Cell 2 \\n Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1 \\n Row 2, Cell 2 \\n Row 2, Cell 3\"),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content \\n This section contains an image and a video: \\n \\n \\n Your browser does not support the video tag.'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example \\n This section contains a code block: \\n \\n \\n This is a paragraph inside a div.
\\n '),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion \\n This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLSectionSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "html_header_splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5aa8627f-0af3-48c2-b9ed-bfbc46f8030d",
+ "metadata": {},
+ "source": [
+ "### How to constrain chunk sizes:\n",
+ "\n",
+ "`HTMLSectionSplitter` can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "5a9759b2-f6ff-413e-b42d-9d8967f3f3e6",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title'),\n",
+ " Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some'),\n",
+ " Document(metadata={'Header 1': 'Main Title'}, page_content='some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Section 1: Introduction'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='is a list:'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='First item \\n Second item'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Third item with bold text and a link'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Subsection 1.1: Details'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='This subsection provides additional details.'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content=\"Here's a table:\"),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Header 1 \\n Header 2 \\n Header 3'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 1 \\n Row 1, Cell 2'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 2, Cell 2 \\n Row 2, Cell 3'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Your browser does not support the video'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='tag.'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: \\n \\n '),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This is a paragraph inside a div.
'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content=''),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
+ "\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "\n",
+ "chunk_size = 50\n",
+ "chunk_overlap = 5\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
+ ")\n",
+ "\n",
+ "# Split\n",
+ "splits = text_splitter.split_documents(html_header_splits)\n",
+ "splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6fb9f81a",
+ "metadata": {},
+ "source": [
+ "## Using HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "The `HTMLSemanticPreservingSplitter` is designed to split HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components. This ensures that such elements are not split across chunks, causing loss of contextual relevancy such as table headers, list headers etc.\n",
+ "\n",
+ "This splitter is designed at its heart, to create contextually relevant chunks. General Recursive splitting with `HTMLHeaderTextSplitter` can cause tables, lists and other structered elements to be split in the middle, losing signifcant context and creating bad chunks.\n",
+ "\n",
+ "The `HTMLSemanticPreservingSplitter` is essential for splitting HTML content that includes structured elements like tables and lists, especially when it's critical to preserve these elements intact. Additionally, its ability to define custom handlers for specific HTML tags makes it a versatile tool for processing complex HTML documents.\n",
+ "\n",
+ "**IMPORTANT**: `max_chunk_size` is not a definite maximum size of a chunk, the calculation of max size, occurs when the preserved content is not apart of the chunk, to ensure it is not split. When we add the preserved data back in to the chunk, there is a chance the chunk size will exceed the `max_chunk_size`. This is crucial to ensure we maintain the structure of the original document\n",
+ "\n",
+ ":::info \n",
+ "\n",
+ "Notes:\n",
+ "\n",
+ "1. We have defined a custom handler to re-format the contents of code blocks\n",
+ "2. We defined a deny list for specific html elements, to decompose them and their contents pre-processing\n",
+ "3. We have intentionally set a small chunk size to demonstrate the non-splitting of elements\n",
+ "\n",
+ ":::"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "6cd119a8-c3d1-48a8-b569-469cd1607169",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3\"),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:  '),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: This is a paragraph inside a div.
'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# BeautifulSoup is required to use the custom handlers\n",
+ "from bs4 import Tag\n",
+ "from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "\n",
+ "def code_handler(element: Tag) -> str:\n",
+ " data_lang = element.get(\"data-lang\")\n",
+ " code_format = f\"{element.get_text()}\"\n",
+ "\n",
+ " return code_format\n",
+ "\n",
+ "\n",
+ "splitter = HTMLSemanticPreservingSplitter(\n",
+ " headers_to_split_on=headers_to_split_on,\n",
+ " separators=[\"\\n\\n\", \"\\n\", \". \", \"! \", \"? \"],\n",
+ " max_chunk_size=50,\n",
+ " preserve_images=True,\n",
+ " preserve_videos=True,\n",
+ " elements_to_preserve=[\"table\", \"ul\", \"ol\", \"code\"],\n",
+ " denylist_tags=[\"script\", \"style\", \"head\"],\n",
+ " custom_handlers={\"code\": code_handler},\n",
+ ")\n",
+ "\n",
+ "documents = splitter.split_text(html_string)\n",
+ "documents"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8c5413e2-3c50-435a-bda7-11e574a3fbab",
+ "metadata": {},
+ "source": [
+ "### Preserving Tables and Lists\n",
+ "In this example, we will demonstrate how the `HTMLSemanticPreservingSplitter` can preserve a table and a large list within an HTML document. The chunk size will be set to 50 characters to illustrate how the splitter ensures that these elements are not split, even when they exceed the maximum defined chunk size."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "8d97a2a6-9c73-4396-a922-f7a4eebe47f8",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[Document(metadata={'Header 1': 'Section 1'}, page_content='This section contains an important table and list'), Document(metadata={'Header 1': 'Section 1'}, page_content='that should not be split across chunks.'), Document(metadata={'Header 1': 'Section 1'}, page_content='Item Quantity Price Apples 10 $1.00 Oranges 5 $0.50 Bananas 50 $1.50'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='Additional text in subsection 1.1 that is'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='separated from the table and list. Here is a'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content=\"detailed list: Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.\")]\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "html_string = \"\"\"\n",
+ "\n",
+ "\n",
+ " \n",
+ " \n",
+ " Section 1
\n",
+ " This section contains an important table and list that should not be split across chunks.
\n",
+ " \n",
+ " \n",
+ " Item \n",
+ " Quantity \n",
+ " Price \n",
+ " \n",
+ " \n",
+ " Apples \n",
+ " 10 \n",
+ " $1.00 \n",
+ " \n",
+ " \n",
+ " Oranges \n",
+ " 5 \n",
+ " $0.50 \n",
+ " \n",
+ " \n",
+ " Bananas \n",
+ " 50 \n",
+ " $1.50 \n",
+ " \n",
+ "
\n",
+ " Subsection 1.1
\n",
+ " Additional text in subsection 1.1 that is separated from the table and list.
\n",
+ " Here is a detailed list:
\n",
+ " \n",
+ " - Item 1: Description of item 1, which is quite detailed and important.
\n",
+ " - Item 2: Description of item 2, which also contains significant information.
\n",
+ " - Item 3: Description of item 3, another item that we don't want to split across chunks.
\n",
+ "
\n",
+ " \n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n",
+ "\n",
+ "splitter = HTMLSemanticPreservingSplitter(\n",
+ " headers_to_split_on=headers_to_split_on,\n",
+ " max_chunk_size=50,\n",
+ " elements_to_preserve=[\"table\", \"ul\"],\n",
+ ")\n",
+ "\n",
+ "documents = splitter.split_text(html_string)\n",
+ "print(documents)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e44bab7f-5a81-4ec4-a7d8-0413c8e15b33",
+ "metadata": {},
+ "source": [
+ "#### Explanation\n",
+ "In this example, the `HTMLSemanticPreservingSplitter` ensures that the entire table and the unordered list (``) are preserved within their respective chunks. Even though the chunk size is set to 50 characters, the splitter recognizes that these elements should not be split and keeps them intact.\n",
+ "\n",
+ "This is particularly important when dealing with data tables or lists, where splitting the content could lead to loss of context or confusion. The resulting `Document` objects retain the full structure of these elements, ensuring that the contextual relevance of the information is maintained."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0c0a83c9-1177-4f64-9f94-688b5ec4fafd",
+ "metadata": {},
+ "source": [
+ "### Using a Custom Handler\n",
+ "The `HTMLSemanticPreservingSplitter` allows you to define custom handlers for specific HTML elements. Some platforms, have custom HTML tags that are not natively parsed by `BeautifulSoup`, when this occurs, you can utilize custom handlers to add the formatting logic easily. \n",
+ "\n",
+ "This can be particularly useful for elements that require special processing, such as `\n",
+ " Some text after the iframe.
\n",
+ " \n",
+ " - Item 1: Description of item 1, which is quite detailed and important.
\n",
+ " - Item 2: Description of item 2, which also contains significant information.
\n",
+ " - Item 3: Description of item 3, another item that we don't want to split across chunks.
\n",
+ "
\n",
+ "
\n",
+ "
`, or custom-defined sections. \n",
+ ":::\n",
+ "\n",
+ "**Description**: Similar to HTMLHeaderTextSplitter but focuses on splitting HTML into sections based on specified tags.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Uses XSLT transformations to detect and split sections.\n",
+ "- Internally uses `RecursiveCharacterTextSplitter` for large sections.\n",
+ "- Considers font sizes to determine sections.\n",
+ "___"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8edb47106e1a46a883d545849b8ab81b",
+ "metadata": {},
+ "source": [
+ "### [HTMLSemanticPreservingSplitter](#using-htmlsemanticpreservingsplitter)\n",
+ "\n",
+ ":::info \n",
+ "Ideal when you need to ensure that structured elements are not split across chunks, preserving contextual relevancy. \n",
+ ":::\n",
+ "\n",
+ "**Description**: Splits HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components.\n",
+ "\n",
+ "**Capabilities**:\n",
+ "- Preserves tables, lists, and other specified HTML elements.\n",
+ "- Allows custom handlers for specific HTML tags.\n",
+ "- Ensures that the semantic meaning of the document is maintained.\n",
+ "- Built in normalization & stopword removal\n",
+ "\n",
+ "___"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "10185d26023b46108eb7d9f57d49d2b3",
+ "metadata": {},
+ "source": [
+ "### Choosing the Right Splitter\n",
+ "\n",
+ "- **Use `HTMLHeaderTextSplitter` when**: You need to split an HTML document based on its header hierarchy and maintain metadata about the headers.\n",
+ "- **Use `HTMLSectionSplitter` when**: You need to split the document into larger, more general sections, possibly based on custom tags or font sizes.\n",
+ "- **Use `HTMLSemanticPreservingSplitter` when**: You need to split the document into chunks while preserving semantic elements like tables and lists, ensuring that they are not split and that their context is maintained."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "19d42e40-015c-4bfa-b9ce-783e8377af2b",
+ "metadata": {},
+ "source": [
+ "| Feature | HTMLHeaderTextSplitter | HTMLSectionSplitter | HTMLSemanticPreservingSplitter |\n",
+ "|--------------------------------------------|------------------------|---------------------|-------------------------------|\n",
+ "| Splits based on headers | Yes | Yes | Yes |\n",
+ "| Preserves semantic elements (tables, lists) | No | No | Yes |\n",
+ "| Adds metadata for headers | Yes | Yes | Yes |\n",
+ "| Custom handlers for HTML tags | No | No | Yes |\n",
+ "| Preserves media (images, videos) | No | No | Yes |\n",
+ "| Considers font sizes | No | Yes | No |\n",
+ "| Uses XSLT transformations | No | Yes | No |"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8763a12b2bbd4a93a75aff182afb95dc",
+ "metadata": {},
+ "source": [
+ "## Example HTML Document\n",
+ "\n",
+ "Let's use the following HTML document as an example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "c9ca2682",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "html_string = \"\"\"\n",
+ "\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Fancy Example HTML Page \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ "
\n",
+ " \n",
+ " \n",
+ " \n",
+ "\n",
+ " )\n",
+ "html_header_splits = html_splitter.split_text_from_url(url)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "88ba9878-068e-434f-bdac-17972b2aa9ad",
+ "metadata": {},
+ "source": [
+ "### How to constrain chunk sizes:\n",
+ "\n",
+ "`HTMLHeaderTextSplitter`, which splits based on HTML headers, can be composed with another splitter which constrains splits based on character lengths, such as `RecursiveCharacterTextSplitter`.\n",
+ "\n",
+ "This can be done using the `.split_documents` method of the second splitter:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "3efc1fb8-f264-4ae6-883d-694a4d252f86",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),\n",
+ " Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "chunk_size = 500\n",
+ "chunk_overlap = 30\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
+ ")\n",
+ "\n",
+ "# Split\n",
+ "splits = text_splitter.split_documents(html_header_splits)\n",
+ "splits[80:85]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "27363092-c6b2-4290-9e12-f6aece503bfb",
+ "metadata": {},
+ "source": [
+ "### Limitations\n",
+ "\n",
+ "There can be quite a bit of structural variation from one HTML document to another, and while `HTMLHeaderTextSplitter` will attempt to attach all \"relevant\" headers to any given chunk, it can sometimes miss certain headers. For example, the algorithm assumes an informational hierarchy in which headers are always at nodes \"above\" associated text, i.e. prior siblings, ancestors, and combinations thereof. In the following news article (as of the writing of this document), the document is structured such that the text of the top-level headline, while tagged \"h1\", is in a *distinct* subtree from the text elements that we'd expect it to be *\"above\"*—so we can observe that the \"h1\" element and its associated text do not show up in the chunk metadata (but, where applicable, we do see \"h2\" and its associated text): \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "332043f6-ec39-4fcd-aa57-4dec5252684b",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "No two El Niño winters are the same, but many have temperature and precipitation trends in common. \n",
+ "Average conditions during an El Niño winter across the continental US. \n",
+ "One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA. \n",
+ "Because the jet stream is essentially a river of air that storms flow through, they c\n"
+ ]
+ }
+ ],
+ "source": [
+ "url = \"https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html\"\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text_from_url(url)\n",
+ "print(html_header_splits[1].page_content[:500])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aa8ba422",
+ "metadata": {},
+ "source": [
+ "## Using HTMLSectionSplitter\n",
+ "\n",
+ "Similar in concept to the [HTMLHeaderTextSplitter](#using-htmlheadertextsplitter), the `HTMLSectionSplitter` is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk. It lets you split HTML by sections.\n",
+ "\n",
+ "It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures.\n",
+ "\n",
+ "Use `xslt_path` to provide an absolute path to transform the HTML so that it can detect sections based on provided tags. The default is to use the `converting_to_header.xslt` file in the `data_connection/document_transformers` directory. This is for converting the html to a format/layout that is easier to detect sections. For example, `span` based on their font size can be converted to header tags to be detected as a section.\n",
+ "\n",
+ "### How to split HTML strings:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "65376c86",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title \\n This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\"Section 1: Introduction \\n This section introduces the topic. Below is a list: \\n \\n First item \\n Second item \\n Third item with bold text and a link \\n \\n \\n Subsection 1.1: Details \\n This subsection provides additional details. Here's a table: \\n \\n \\n \\n Header 1 \\n Header 2 \\n Header 3 \\n \\n \\n \\n \\n Row 1, Cell 1 \\n Row 1, Cell 2 \\n Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1 \\n Row 2, Cell 2 \\n Row 2, Cell 3\"),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content \\n This section contains an image and a video: \\n \\n \\n Your browser does not support the video tag.'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example \\n This section contains a code block: \\n \\n '),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# BeautifulSoup is required to use the custom handlers\n",
+ "from bs4 import Tag\n",
+ "from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "\n",
+ "def code_handler(element: Tag) -> str:\n",
+ " data_lang = element.get(\"data-lang\")\n",
+ " code_format = f\"{element.get_text()}\"\n",
+ "\n",
+ " return code_format\n",
+ "\n",
+ "\n",
+ "splitter = HTMLSemanticPreservingSplitter(\n",
+ " headers_to_split_on=headers_to_split_on,\n",
+ " separators=[\"\\n\\n\", \"\\n\", \". \", \"! \", \"? \"],\n",
+ " max_chunk_size=50,\n",
+ " preserve_images=True,\n",
+ " preserve_videos=True,\n",
+ " elements_to_preserve=[\"table\", \"ul\", \"ol\", \"code\"],\n",
+ " denylist_tags=[\"script\", \"style\", \"head\"],\n",
+ " custom_handlers={\"code\": code_handler},\n",
+ ")\n",
+ "\n",
+ "documents = splitter.split_text(html_string)\n",
+ "documents"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8c5413e2-3c50-435a-bda7-11e574a3fbab",
+ "metadata": {},
+ "source": [
+ "### Preserving Tables and Lists\n",
+ "In this example, we will demonstrate how the `HTMLSemanticPreservingSplitter` can preserve a table and a large list within an HTML document. The chunk size will be set to 50 characters to illustrate how the splitter ensures that these elements are not split, even when they exceed the maximum defined chunk size."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "8d97a2a6-9c73-4396-a922-f7a4eebe47f8",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[Document(metadata={'Header 1': 'Section 1'}, page_content='This section contains an important table and list'), Document(metadata={'Header 1': 'Section 1'}, page_content='that should not be split across chunks.'), Document(metadata={'Header 1': 'Section 1'}, page_content='Item Quantity Price Apples 10 $1.00 Oranges 5 $0.50 Bananas 50 $1.50'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='Additional text in subsection 1.1 that is'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content='separated from the table and list. Here is a'), Document(metadata={'Header 2': 'Subsection 1.1'}, page_content=\"detailed list: Item 1: Description of item 1, which is quite detailed and important. Item 2: Description of item 2, which also contains significant information. Item 3: Description of item 3, another item that we don't want to split across chunks.\")]\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "html_string = \"\"\"\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ " Main Title
\n", + "This is an introductory paragraph with some basic content.
\n", + " \n", + "Section 1: Introduction
\n", + "This section introduces the topic. Below is a list:
\n", + "- \n",
+ "
- First item \n", + "
- Second item \n", + "
- Third item with bold text and a link \n", + "
Subsection 1.1: Details
\n", + "This subsection provides additional details. Here's a table:
\n", + "| Header 1 | \n", + "Header 2 | \n", + "Header 3 | \n", + "
|---|---|---|
| Row 1, Cell 1 | \n", + "Row 1, Cell 2 | \n", + "Row 1, Cell 3 | \n", + "
| Row 2, Cell 1 | \n", + "Row 2, Cell 2 | \n", + "Row 2, Cell 3 | \n", + "
Section 2: Media Content
\n", + "This section contains an image and a video:
\n", + "Section 3: Code Example
\n", + "This section contains a code block:
\n", + "\n",
+ " <div>\n",
+ " <p>This is a paragraph inside a div.</p>\n",
+ " </div>\n",
+ " Conclusion
\n", + "This is the conclusion of the document.
\n", + " \n", + " \n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "id": "86c25b86", + "metadata": {}, + "source": [ + "## Using HTMLHeaderTextSplitter\n", + "\n", + "[HTMLHeaderTextSplitter](https://python.langchain.com/api_reference/text_splitters/html/langchain_text_splitters.html.HTMLHeaderTextSplitter.html) is a \"structure-aware\" [text splitter](/docs/concepts/text_splitters/) that splits text at the HTML element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n", + "\n", + "It is analogous to the [MarkdownHeaderTextSplitter](/docs/how_to/markdown_header_metadata_splitter) for markdown files.\n", + "\n", + "To specify what headers to split on, specify `headers_to_split_on` when instantiating `HTMLHeaderTextSplitter` as shown below." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "23361e55", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n", + " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a list: \\nFirst item Second item Third item with bold text and a link'),\n", + " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction', 'Header 3': 'Subsection 1.1: Details'}, page_content=\"This subsection provides additional details. Here's a table:\"),\n", + " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n", + " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block:'),\n", + " Document(metadata={'Header 1': 'Main Title', 'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from langchain_text_splitters import HTMLHeaderTextSplitter\n", + "\n", + "headers_to_split_on = [\n", + " (\"h1\", \"Header 1\"),\n", + " (\"h2\", \"Header 2\"),\n", + " (\"h3\", \"Header 3\"),\n", + "]\n", + "\n", + "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n", + "html_header_splits = html_splitter.split_text(html_string)\n", + "html_header_splits" + ] + }, + { + "cell_type": "markdown", + "id": "f7e9cfef-5387-4ffe-b1a8-4b9214b9debd", + "metadata": {}, + "source": [ + "To return each element together with their associated headers, specify `return_each_element=True` when instantiating `HTMLHeaderTextSplitter`:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "adb0da17-d1d5-4913-aacd-dc5d70db66cf", + "metadata": {}, + "outputs": [], + "source": [ + "html_splitter = HTMLHeaderTextSplitter(\n", + " headers_to_split_on,\n", + " return_each_element=True,\n", + ")\n", + "html_header_splits_elements = html_splitter.split_text(html_string)" + ] + }, + { + "cell_type": "markdown", + "id": "fa781f86-d04f-4c09-a4a1-26aac88d4fc3", + "metadata": {}, + "source": [ + "Comparing with the above, where elements are aggregated by their headers:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d9d7b61f-f927-49fc-a592-1b0f049d1a2f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n", + "page_content='This section introduces the topic. Below is a list: \n", + "First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n" + ] + } + ], + "source": [ + "for element in html_header_splits[:2]:\n", + " print(element)" + ] + }, + { + "cell_type": "markdown", + "id": "4b5869e0-3a9b-4fa2-82ec-dbde33464a52", + "metadata": {}, + "source": [ + "Now each element is returned as a distinct `Document`:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "e2677a27-875a-4455-8e3f-6a6c7706be20", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "page_content='This is an introductory paragraph with some basic content.' metadata={'Header 1': 'Main Title'}\n", + "page_content='This section introduces the topic. Below is a list:' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n", + "page_content='First item Second item Third item with bold text and a link' metadata={'Header 1': 'Main Title', 'Header 2': 'Section 1: Introduction'}\n" + ] + } + ], + "source": [ + "for element in html_header_splits_elements[:3]:\n", + " print(element)" + ] + }, + { + "cell_type": "markdown", + "id": "24200c53-6a52-436e-ba4d-8d0514cfa87c", + "metadata": {}, + "source": [ + "### How to split from a URL or HTML file:\n", + "\n", + "To read directly from a URL, pass the URL string into the `split_text_from_url` method.\n", + "\n", + "Similarly, a local HTML file can be passed to the `split_text_from_file` method." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "581eeddf-7e88-48a7-999d-da56304e3522", + "metadata": {}, + "outputs": [], + "source": [ + "url = \"https://plato.stanford.edu/entries/goedel/\"\n", + "\n", + "headers_to_split_on = [\n", + " (\"h1\", \"Header 1\"),\n", + " (\"h2\", \"Header 2\"),\n", + " (\"h3\", \"Header 3\"),\n", + " (\"h4\", \"Header 4\"),\n", + "]\n", + "\n", + "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on)\n", + "\n", + "# for local file use html_splitter.split_text_from_file(\\n
'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion \\n This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import HTMLSectionSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "html_header_splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5aa8627f-0af3-48c2-b9ed-bfbc46f8030d",
+ "metadata": {},
+ "source": [
+ "### How to constrain chunk sizes:\n",
+ "\n",
+ "`HTMLSectionSplitter` can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "5a9759b2-f6ff-413e-b42d-9d8967f3f3e6",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='Main Title'),\n",
+ " Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some'),\n",
+ " Document(metadata={'Header 1': 'Main Title'}, page_content='some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Section 1: Introduction'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic. Below is a'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='is a list:'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='First item \\n Second item'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='Third item with bold text and a link'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Subsection 1.1: Details'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='This subsection provides additional details.'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content=\"Here's a table:\"),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Header 1 \\n Header 2 \\n Header 3'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 1 \\n Row 1, Cell 2'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 1, Cell 3 \\n \\n \\n Row 2, Cell 1'),\n",
+ " Document(metadata={'Header 3': 'Subsection 1.1: Details'}, page_content='Row 2, Cell 2 \\n Row 2, Cell 3'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Section 2: Media Content'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='Your browser does not support the video'),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='tag.'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='Section 3: Code Example'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: \\n \\n This is a paragraph inside a div.
\\n'),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='
'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='Conclusion'),\n",
+ " Document(metadata={'Header 2': 'Conclusion'}, page_content='This is the conclusion of the document.')]"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "headers_to_split_on = [\n",
+ " (\"h1\", \"Header 1\"),\n",
+ " (\"h2\", \"Header 2\"),\n",
+ " (\"h3\", \"Header 3\"),\n",
+ "]\n",
+ "\n",
+ "html_splitter = HTMLSectionSplitter(headers_to_split_on)\n",
+ "\n",
+ "html_header_splits = html_splitter.split_text(html_string)\n",
+ "\n",
+ "chunk_size = 50\n",
+ "chunk_overlap = 5\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=chunk_size, chunk_overlap=chunk_overlap\n",
+ ")\n",
+ "\n",
+ "# Split\n",
+ "splits = text_splitter.split_documents(html_header_splits)\n",
+ "splits"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6fb9f81a",
+ "metadata": {},
+ "source": [
+ "## Using HTMLSemanticPreservingSplitter\n",
+ "\n",
+ "The `HTMLSemanticPreservingSplitter` is designed to split HTML content into manageable chunks while preserving the semantic structure of important elements like tables, lists, and other HTML components. This ensures that such elements are not split across chunks, causing loss of contextual relevancy such as table headers, list headers etc.\n",
+ "\n",
+ "This splitter is designed at its heart, to create contextually relevant chunks. General Recursive splitting with `HTMLHeaderTextSplitter` can cause tables, lists and other structered elements to be split in the middle, losing signifcant context and creating bad chunks.\n",
+ "\n",
+ "The `HTMLSemanticPreservingSplitter` is essential for splitting HTML content that includes structured elements like tables and lists, especially when it's critical to preserve these elements intact. Additionally, its ability to define custom handlers for specific HTML tags makes it a versatile tool for processing complex HTML documents.\n",
+ "\n",
+ "**IMPORTANT**: `max_chunk_size` is not a definite maximum size of a chunk, the calculation of max size, occurs when the preserved content is not apart of the chunk, to ensure it is not split. When we add the preserved data back in to the chunk, there is a chance the chunk size will exceed the `max_chunk_size`. This is crucial to ensure we maintain the structure of the original document\n",
+ "\n",
+ ":::info \n",
+ "\n",
+ "Notes:\n",
+ "\n",
+ "1. We have defined a custom handler to re-format the contents of code blocks\n",
+ "2. We defined a deny list for specific html elements, to decompose them and their contents pre-processing\n",
+ "3. We have intentionally set a small chunk size to demonstrate the non-splitting of elements\n",
+ "\n",
+ ":::"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "6cd119a8-c3d1-48a8-b569-469cd1607169",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[Document(metadata={'Header 1': 'Main Title'}, page_content='This is an introductory paragraph with some basic content.'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='This section introduces the topic'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content='. Below is a list: First item Second item Third item with bold text and a link Subsection 1.1: Details This subsection provides additional details'),\n",
+ " Document(metadata={'Header 2': 'Section 1: Introduction'}, page_content=\". Here's a table: Header 1 Header 2 Header 3 Row 1, Cell 1 Row 1, Cell 2 Row 1, Cell 3 Row 2, Cell 1 Row 2, Cell 2 Row 2, Cell 3\"),\n",
+ " Document(metadata={'Header 2': 'Section 2: Media Content'}, page_content='This section contains an image and a video:  '),\n",
+ " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This section contains a code block: This is a paragraph inside a div.
'),\n", + " Document(metadata={'Header 2': 'Section 3: Code Example'}, page_content='This is a paragraph inside a div.
\n",
+ " \n",
+ "
\n",
+ "
\n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n",
+ "\n",
+ "splitter = HTMLSemanticPreservingSplitter(\n",
+ " headers_to_split_on=headers_to_split_on,\n",
+ " max_chunk_size=50,\n",
+ " elements_to_preserve=[\"table\", \"ul\"],\n",
+ ")\n",
+ "\n",
+ "documents = splitter.split_text(html_string)\n",
+ "print(documents)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e44bab7f-5a81-4ec4-a7d8-0413c8e15b33",
+ "metadata": {},
+ "source": [
+ "#### Explanation\n",
+ "In this example, the `HTMLSemanticPreservingSplitter` ensures that the entire table and the unordered list (`Section 1
\n", + "This section contains an important table and list that should not be split across chunks.
\n", + "| Item | \n", + "Quantity | \n", + "Price | \n", + "
|---|---|---|
| Apples | \n", + "10 | \n", + "$1.00 | \n", + "
| Oranges | \n", + "5 | \n", + "$0.50 | \n", + "
| Bananas | \n", + "50 | \n", + "$1.50 | \n", + "
Subsection 1.1
\n", + "Additional text in subsection 1.1 that is separated from the table and list.
\n", + "Here is a detailed list:
\n", + "- \n",
+ "
- Item 1: Description of item 1, which is quite detailed and important. \n", + "
- Item 2: Description of item 2, which also contains significant information. \n", + "
- Item 3: Description of item 3, another item that we don't want to split across chunks. \n", + "
- `) are preserved within their respective chunks. Even though the chunk size is set to 50 characters, the splitter recognizes that these elements should not be split and keeps them intact.\n",
+ "\n",
+ "This is particularly important when dealing with data tables or lists, where splitting the content could lead to loss of context or confusion. The resulting `Document` objects retain the full structure of these elements, ensuring that the contextual relevance of the information is maintained."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0c0a83c9-1177-4f64-9f94-688b5ec4fafd",
+ "metadata": {},
+ "source": [
+ "### Using a Custom Handler\n",
+ "The `HTMLSemanticPreservingSplitter` allows you to define custom handlers for specific HTML elements. Some platforms, have custom HTML tags that are not natively parsed by `BeautifulSoup`, when this occurs, you can utilize custom handlers to add the formatting logic easily. \n",
+ "\n",
+ "This can be particularly useful for elements that require special processing, such as `\n",
+ "
- Item 1: Description of item 1, which is quite detailed and important. \n", + "
- Item 2: Description of item 2, which also contains significant information. \n", + "
- Item 3: Description of item 3, another item that we don't want to split across chunks. \n", + "
Some text after the iframe.
\n", + "- \n",
+ "