Skip to content

Commit

Permalink
text_splitters: Add HTMLSemanticPreservingSplitter (#25911)
Browse files Browse the repository at this point in the history
**Description:** 

With current HTML splitters, they rely on secondary use of the
`RecursiveCharacterSplitter` to further chunk the document into
manageable chunks. The issue with this is it fails to maintain important
structures such as tables, lists, etc within HTML.

This Implementation of a HTML splitter, allows the user to define a
maximum chunk size, HTML elements to preserve in full, options to
preserve `<a>` href links in the output and custom handlers.

The core splitting begins with headers, similar to `HTMLHeaderSplitter`.
If these sections exceed the length of the `max_chunk_size` further
recursive splitting is triggered. During this splitting, elements listed
to preserve, will be excluded from the splitting process. This can cause
chunks to be slightly larger then the max size, depending on preserved
length. However, all contextual relevance of the preserved item remains
intact.

**Custom Handlers**: Sometimes, companies such as Atlassian have custom
HTML elements, that are not parsed by default with `BeautifulSoup`.
Custom handlers allows a user to provide a function to be ran whenever a
specific html tag is encountered. This allows the user to preserve and
gather information within custom html tags that `bs4` will potentially
miss during extraction.

**Dependencies:** User will need to install `bs4` in their project to
utilise this class

I have also added in `how_to` and unit tests, which require `bs4` to
run, otherwise they will be skipped.

Flowchart of process:


![HTMLSemanticPreservingSplitter](https://github.com/user-attachments/assets/20873c36-22ed-4c80-884b-d3c6f433f5a7)

---------

Co-authored-by: Bagatur <[email protected]>
Co-authored-by: Chester Curme <[email protected]>
  • Loading branch information
3 people authored Dec 19, 2024
1 parent 24bfa06 commit f696950
Show file tree
Hide file tree
Showing 13 changed files with 1,835 additions and 574 deletions.

This file was deleted.

This file was deleted.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/docs/concepts/text_splitters.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Examples of structure-based splitting:
* See the how-to guide for [Markdown splitting](/docs/how_to/markdown_header_metadata_splitter/).
* See the how-to guide for [Recursive JSON splitting](/docs/how_to/recursive_json_splitter/).
* See the how-to guide for [Code splitting](/docs/how_to/code_splitter/).
* See the how-to guide for [HTML splitting](/docs/how_to/HTML_header_metadata_splitter/).
* See the how-to guide for [HTML splitting](/docs/how_to/split_html/).

:::

Expand Down
359 changes: 0 additions & 359 deletions docs/docs/how_to/HTML_header_metadata_splitter.ipynb

This file was deleted.

Loading

0 comments on commit f696950

Please sign in to comment.