Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Community]: Image Extraction Fixed for PDFPlumberParser #28491

Merged
merged 13 commits into from
Dec 18, 2024
+24 −4
Merged

[Community]: Image Extraction Fixed for PDFPlumberParser #28491

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a3a72c8
[fix]: Image Extraction Fixed
keenborder786 Dec 3, 2024
b5ec3fb
[chore]: New Package Added
keenborder786 Dec 3, 2024
8aa57a3
[tests]: Added Pillow in Required Dependencies
keenborder786 Dec 3, 2024
68599a1
[fix]: Corrected the if condition
keenborder786 Dec 10, 2024
f7fd399
[chore]: Removing the Pillow Dependency
keenborder786 Dec 15, 2024
3b54263
[chore]: Removing Pillow as a Global dependency
keenborder786 Dec 15, 2024
2ce8cef
Merge branch 'master' into fix/_extract_images_from_page
keenborder786 Dec 15, 2024
a7ff56f
[chore]: Reverted to default poetry config
keenborder786 Dec 15, 2024
b9b3e86
[chore]: Reverted to default poetry config
keenborder786 Dec 15, 2024
4bfc4b3
[chore]: removing the added dependencies
keenborder786 Dec 15, 2024
f193726
[chore]: removing the added dependencies
keenborder786 Dec 15, 2024
144a6f8
retain old behavior for images that are not 1 bit
ccurme Dec 18, 2024
2860a6f
Merge branch 'master' into fix/_extract_images_from_page
ccurme Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions libs/community/langchain_community/document_loaders/parsers/pdf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

import numpy as np
from langchain_core.documents import Document
from PIL import Image

from langchain_community.document_loaders.base import BaseBlobParser
from langchain_community.document_loaders.blob_loaders import Blob
Expand Down Expand Up @@ -453,8 +454,12 @@ def _extract_images_from_page(self, page: pdfplumber.page.Page) -> str:
for img in page.images:
if img["stream"]["Filter"].name in _PDF_FILTER_WITHOUT_LOSS:
images.append(
np.frombuffer(img["stream"].get_data(), dtype=np.uint8).reshape(
img["stream"]["Height"], img["stream"]["Width"], -1
np.array(
Image.frombytes(
"1",
(img["stream"]["Width"], img["stream"]["Height"]),
img["stream"].get_data(),
).convert("L")
)
)
elif img["stream"]["Filter"].name in _PDF_FILTER_WITH_LOSS:
Expand Down
105 changes: 104 additions & 1 deletion libs/community/poetry.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions libs/community/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ dataclasses-json = ">= 0.5.7, < 0.7"
pydantic-settings = "^2.4.0"
langsmith = "^0.1.125"
httpx-sse = "^0.4.0"
pillow = "^11.0.0"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't be adding any new required dependencies — is there any other way to fix

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baskaryan The way I see it no.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baskaryan let me know what I can do then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this fix, I need pillow otherwise it won't work.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keenborder786, optional dependencies can be OK, but not required ones

Also please get in the habit of including tests with PRs. We want to avoid a situation where a bug fix, is introducing new bug fixes (e.g., #28480 (comment))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eyurtsev okay.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eyurtsev @baskaryan I have removed pillow as a required dependency. Please check now.

[[tool.poetry.dependencies.numpy]]
version = ">=1.22.4,<2"
python = "<3.12"
Expand Down
5 changes: 3 additions & 2 deletions libs/community/tests/unit_tests/test_dependencies.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,15 @@ def test_required_dependencies(poetry_conf: Mapping[str, Any]) -> None:
"aiohttp",
"dataclasses-json",
"httpx-sse",
"langchain",
"langchain-core",
"langsmith",
"numpy",
"pillow",
"pydantic-settings",
"python",
"requests",
"pydantic-settings",
"tenacity",
"langchain",
]
)

Expand Down
Loading