First integration of the refactoring of PDFloader/parser

langchain-ai · Dec 10, 2024 · c348a50 · c348a50
1 parent 24292c4
commit c348a50
Show file tree

Hide file tree

Showing 13 changed files with 10,609 additions and 936 deletions.
diff --git a/docs/docs/how_to/document_loader_custom.ipynb b/docs/docs/how_to/document_loader_custom.ipynb
diff --git a/docs/docs/how_to/document_loader_pdf.ipynb b/docs/docs/how_to/document_loader_pdf.ipynb
diff --git a/docs/docs/integrations/document_loaders/pdfminer.ipynb b/docs/docs/integrations/document_loaders/pdfminer.ipynb
diff --git a/docs/docs/integrations/document_loaders/pdfminer_fasthtml.ipynb b/docs/docs/integrations/document_loaders/pdfminer_fasthtml.ipynb
diff --git a/docs/docs/integrations/document_loaders/pdfplumber.ipynb b/docs/docs/integrations/document_loaders/pdfplumber.ipynb
diff --git a/docs/docs/integrations/document_loaders/pymupdf.ipynb b/docs/docs/integrations/document_loaders/pymupdf.ipynb
diff --git a/docs/docs/integrations/document_loaders/pypdfdirectory.ipynb b/docs/docs/integrations/document_loaders/pypdfdirectory.ipynb
@@ -4,7 +4,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# PyPDFDirectoryLoader\n",
+    "# PyPDFDirectoryLoader (Deprecated)\n",
+    "\n",
+    "Note: This loader is deprecated. Please use [GenericLoader](https://python.langchain.com/docs/how_to/document_loader_custom/#overview) instead.\n",
+    "\n",
+    "<br>\n",
     "\n",
     "This loader loads all PDF files from a specific directory.\n",
     "\n",
@@ -36,13 +40,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
    "metadata": {},
-   "outputs": [],
    "source": [
     "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",
     "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""
-   ]
+   ],
+   "outputs": [],
+   "execution_count": null
   },
   {
    "cell_type": "markdown",
@@ -55,12 +59,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
    "metadata": {},
-   "outputs": [],
    "source": [
-    "%pip install -qU langchain_community"
-   ]
+    "%%capture\n",
+    "%pip install -qU langchain_community pypdf pillow"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "outputs": [],
+   "execution_count": null,
+   "source": "%pip install -qU ../../../../dist/patch_langchain_pdf_loader-0.0.0-py3-none-any.whl"
   },
   {
    "cell_type": "markdown",
@@ -73,17 +85,31 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-11-26T16:13:22.559920Z",
+     "start_time": "2024-11-26T16:13:21.497502Z"
+    }
+   },
    "source": [
     "from langchain_community.document_loaders import PyPDFDirectoryLoader\n",
     "\n",
     "directory_path = (\n",
     "    \"../../docs/integrations/document_loaders/example_data/layout-parser-paper.pdf\"\n",
     ")\n",
     "loader = PyPDFDirectoryLoader(\"example_data/\")"
-   ]
+   ],
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_53439/4208923121.py:6: LangChainDeprecationWarning: The class `PyPDFDirectoryLoader` was deprecated in LangChain 0.3.X and will be removed in 1.0. Use :class:`~langchain_community.document_loaders.generic.GenericLoader` instead.\n",
+      "  loader = PyPDFDirectoryLoader(\"example_data/\")\n"
+     ]
+    }
+   ],
+   "execution_count": 1
   },
   {
    "cell_type": "markdown",
@@ -94,41 +120,51 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-11-26T16:13:23.170371Z",
+     "start_time": "2024-11-26T16:13:22.627303Z"
+    }
+   },
+   "source": [
+    "docs = loader.load()\n",
+    "docs[0]"
+   ],
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "Document(metadata={'source': 'example_data/layout-parser-paper.pdf', 'page': 0}, page_content='LayoutParser : A Uniﬁed Toolkit for Deep\\nLearning Based Document Image Analysis\\nZejiang Shen1( \\x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\\nLee4, Jacob Carlson3, and Weining Li5\\n1Allen Institute for AI\\[email protected]\\n2Brown University\\nruochen [email protected]\\n3Harvard University\\n{melissadell,jacob carlson }@fas.harvard.edu\\n4University of Washington\\[email protected]\\n5University of Waterloo\\[email protected]\\nAbstract. Recent advances in document image analysis (DIA) have been\\nprimarily driven by the application of neural networks. Ideally, research\\noutcomes could be easily deployed in production and extended for further\\ninvestigation. However, various factors like loosely organized codebases\\nand sophisticated model conﬁgurations complicate the easy reuse of im-\\nportant innovations by a wide audience. Though there have been on-going\\neﬀorts to improve reusability and simplify deep learning (DL) model\\ndevelopment in disciplines like natural language processing and computer\\nvision, none of them are optimized for challenges in the domain of DIA.\\nThis represents a major gap in the existing toolkit, as DIA is central to\\nacademic research across a wide range of disciplines in the social sciences\\nand humanities. This paper introduces LayoutParser , an open-source\\nlibrary for streamlining the usage of DL in DIA research and applica-\\ntions. The core LayoutParser library comes with a set of simple and\\nintuitive interfaces for applying and customizing DL models for layout de-\\ntection, character recognition, and many other document processing tasks.\\nTo promote extensibility, LayoutParser also incorporates a community\\nplatform for sharing both pre-trained models and full document digiti-\\nzation pipelines. We demonstrate that LayoutParser is helpful for both\\nlightweight and large-scale digitization pipelines in real-word use cases.\\nThe library is publicly available at https://layout-parser.github.io .\\nKeywords: Document Image Analysis ·Deep Learning ·Layout Analysis\\n·Character Recognition ·Open Source library ·Toolkit.\\n1 Introduction\\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\\ndocument image analysis (DIA) tasks including document image classiﬁcation [ 11,arXiv:2103.15348v2  [cs.CV]  21 Jun 2021')"
+       "Document(metadata={'author': '', 'creationdate': '2021-06-22T01:27:10+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2021-06-22T01:27:10+00:00', 'ptex.fullbanner': 'This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2', 'producer': 'pdfTeX-1.40.21', 'subject': '', 'title': '', 'trapped': '/False', 'source': 'example_data/layout-parser-paper.pdf', 'total_pages': 16, 'page': 0}, page_content='LayoutParser: A Uniﬁed Toolkit for Deep\\nLearning Based Document Image Analysis\\nZejiang Shen1 (\\x00 ), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\\nLee4, Jacob Carlson3, and Weining Li5\\n1 Allen Institute for AI\\[email protected]\\n2 Brown University\\nruochen [email protected]\\n3 Harvard University\\n{melissadell,jacob carlson}@fas.harvard.edu\\n4 University of Washington\\[email protected]\\n5 University of Waterloo\\[email protected]\\nAbstract. Recent advances in document image analysis (DIA) have been\\nprimarily driven by the application of neural networks. Ideally, research\\noutcomes could be easily deployed in production and extended for further\\ninvestigation. However, various factors like loosely organized codebases\\nand sophisticated model conﬁgurations complicate the easy reuse of im-\\nportant innovations by a wide audience. Though there have been on-going\\neﬀorts to improve reusability and simplify deep learning (DL) model\\ndevelopment in disciplines like natural language processing and computer\\nvision, none of them are optimized for challenges in the domain of DIA.\\nThis represents a major gap in the existing toolkit, as DIA is central to\\nacademic research across a wide range of disciplines in the social sciences\\nand humanities. This paper introduces LayoutParser, an open-source\\nlibrary for streamlining the usage of DL in DIA research and applica-\\ntions. The core LayoutParser library comes with a set of simple and\\nintuitive interfaces for applying and customizing DL models for layout de-\\ntection, character recognition, and many other document processing tasks.\\nTo promote extensibility, LayoutParser also incorporates a community\\nplatform for sharing both pre-trained models and full document digiti-\\nzation pipelines. We demonstrate that LayoutParser is helpful for both\\nlightweight and large-scale digitization pipelines in real-word use cases.\\nThe library is publicly available at https://layout-parser.github.io.\\nKeywords: Document Image Analysis · Deep Learning · Layout Analysis\\n· Character Recognition · Open Source library · Toolkit.\\n1 Introduction\\nDeep Learning(DL)-based approaches are the state-of-the-art for a wide range of\\ndocument image analysis (DIA) tasks including document image classiﬁcation [11,\\narXiv:2103.15348v2  [cs.CV]  21 Jun 2021')"
       ]
      },
      "execution_count": 2,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
-   "source": [
-    "docs = loader.load()\n",
-    "docs[0]"
-   ]
+   "execution_count": 2
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-11-26T16:13:23.324756Z",
+     "start_time": "2024-11-26T16:13:23.321488Z"
+    }
+   },
+   "source": [
+    "print(docs[0].metadata)"
+   ],
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "{'source': 'example_data/layout-parser-paper.pdf', 'page': 0}\n"
+      "{'author': '', 'creationdate': '2021-06-22T01:27:10+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2021-06-22T01:27:10+00:00', 'ptex.fullbanner': 'This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2', 'producer': 'pdfTeX-1.40.21', 'subject': '', 'title': '', 'trapped': '/False', 'source': 'example_data/layout-parser-paper.pdf', 'total_pages': 16, 'page': 0}\n"
      ]
     }
    ],
-   "source": [
-    "print(docs[0].metadata)"
-   ]
+   "execution_count": 3
   },
   {
    "cell_type": "markdown",
@@ -139,9 +175,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [],
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-11-26T16:13:23.919646Z",
+     "start_time": "2024-11-26T16:13:23.375102Z"
+    }
+   },
    "source": [
     "page = []\n",
     "for doc in loader.lazy_load():\n",
@@ -151,7 +190,9 @@
     "        # index.upsert(page)\n",
     "\n",
     "        page = []"
-   ]
+   ],
+   "outputs": [],
+   "execution_count": 4
   },
   {
    "cell_type": "markdown",