Rag opensearch usecase with Beam's MLTransform #32018

itsayushpandey · 2024-07-30T06:06:28Z

Added Insertion and Enrichment pipeline for RAG (Retrival Augmented Generation) usecase using OpenSearch vector database #GSOC-259

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

itsayushpandey · 2024-07-30T06:07:58Z

R: @damccorm @jrmccluskey

github-actions · 2024-07-30T06:09:15Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py

jrmccluskey · 2024-07-31T17:35:03Z

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py

+
+
+
+


Run yapf manually to clean up the whitespace (see https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-Runningyapfformattermanually)

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py

jrmccluskey · 2024-07-31T17:44:08Z

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py

+        if self.batch_counter == 0:
+            return
+
+        with _InsertEmbeddingInOpenSearchSink(self.host, self.port, self.username, self.password, self.embedded_columns) as sink:


Is there a reason you're instantiating a new instance of the internal class here every time? You wind up creating a new class + client every time

I reason I did that because I don't want to bulk push on a stale connection. Given it's a batch push I thought it might be better instead of try catching an exception and reinstantiating a connection after error.
What do you think ?

examples/notebooks/beam-ml/rag_usecase/opensearch_enrichment.py

jrmccluskey · 2024-08-12T14:39:08Z

I'd recommend rebasing the PR now that the initial RAG PR is merged

jrmccluskey

Please rebase this PR to remove the duplicate code (anything from #31657 that has already been reviewed and merged)

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py

examples/notebooks/beam-ml/rag_usecase/opensearch_enrichment.py

examples/notebooks/beam-ml/rag_usecase/opensearch_rag_pipeline.ipynb

itsayushpandey · 2024-08-21T03:01:34Z

@jrmccluskey - thanks for the review suggestions! I have made changes as per your feedback, please let me know if these look good.

jrmccluskey

Code looks good, probably time to focus on cleaning up the notebook

Same note on rebasing the branch, see https://stackoverflow.com/questions/7929369/how-to-rebase-local-branch-onto-remote-master/18442755#18442755

itsayushpandey · 2024-08-23T01:30:01Z

Code looks good, probably time to focus on cleaning up the notebook

Same note on rebasing the branch, see https://stackoverflow.com/questions/7929369/how-to-rebase-local-branch-onto-remote-master/18442755#18442755

@jrmccluskey - the branch seems upto date with master This branch has no conflicts with the base branch. Am I misunderstanding something?

damccorm · 2024-09-05T15:46:28Z

@itsayushpandey I think this is the problem:

Your branch doesn't contain the most recent set of changes. To fix, you can:

git remote add upstream https://github.com/apache/beam.git
git fetch upstream
git rebase upstream/main

itsayushpandey · 2024-09-14T00:05:36Z

Hi @damccorm - the trailing commits of branch should have been updated. Can you please check if this looks good to go now?

examples/notebooks/beam-ml/rag_usecase/credentials.env

examples/notebooks/beam-ml/rag_usecase/opensearch_rag_pipeline.ipynb

damccorm

Sorry for the slow review - has been a very busy stretch. This looks good to me - just had one minor comment, once that is addressed I'll merge

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py

damccorm

Thanks!

* Adding insertion and enrichment pipeline * Enhanced Data Schema * Added Apache Licensed to the notebook * Adding Chunking Strategy * removed unused imports * Modified insertion logic in redis for incorporating chunking strategy * refacted redis code * code review changes * Added chunking code in notebook * Added code review changes * Code review changes: using chunking strategy as enum * Added Code Review Changes * Code review changes * Added code review changes * Added Code Review Changes * Code review changes * Ingestion and Enrichment pipeline for OpenSearch Vector DB * Added logic for reading password from .env file * Added opensearch vector notebook * Update credentials.env * Added code review changes * Added Description in opensearch notebook * Added description in opensearch notebook * Code review changes

itsayushpandey added 12 commits June 20, 2024 00:20

Adding insertion and enrichment pipeline

1980ff8

Enhanced Data Schema

f511706

Added Apache Licensed to the notebook

d96c7b3

Adding Chunking Strategy

89a651a

removed unused imports

acceb6f

Modified insertion logic in redis for incorporating chunking strategy

d766f48

refacted redis code

e3d9ca7

code review changes

2158fab

Added chunking code in notebook

d4eaf1b

Added code review changes

12a4297

Code review changes: using chunking strategy as enum

0167f81

Added Code Review Changes

b37b5a8

github-actions bot added the examples label Jul 30, 2024

itsayushpandey added 2 commits July 31, 2024 11:37

Code review changes

471580c

Added code review changes

5175714

jrmccluskey requested changes Jul 31, 2024

View reviewed changes

itsayushpandey marked this pull request as draft July 31, 2024 17:49

itsayushpandey added 3 commits August 9, 2024 13:59

Added Code Review Changes

8466e47

Code review changes

6054399

Ingestion and Enrichment pipeline for OpenSearch Vector DB

3affaad

itsayushpandey force-pushed the rag_opensearch branch from 331df69 to 3affaad Compare August 10, 2024 04:19

Added logic for reading password from .env file

2aaf074

itsayushpandey and others added 2 commits August 17, 2024 12:41

Added opensearch vector notebook

26d1243

Update credentials.env

fb644e0

itsayushpandey marked this pull request as ready for review August 18, 2024 02:42

jrmccluskey requested changes Aug 19, 2024

View reviewed changes

Added code review changes

845acb6

itsayushpandey changed the title ~~Rag opensearch~~ Rag opensearch usecase with Beam's MLTransform Aug 21, 2024

itsayushpandey requested a review from jrmccluskey August 21, 2024 03:01

jrmccluskey reviewed Aug 22, 2024

View reviewed changes

itsayushpandey requested a review from jrmccluskey August 23, 2024 01:49

itsayushpandey added 2 commits September 11, 2024 20:04

Merge branch 'apache:master' into rag_opensearch

5db3fd0

Merge branch 'apache:master' into rag_opensearch

c1e8d6d

damccorm reviewed Sep 19, 2024

View reviewed changes

examples/notebooks/beam-ml/rag_usecase/credentials.env Outdated Show resolved Hide resolved

examples/notebooks/beam-ml/rag_usecase/opensearch_rag_pipeline.ipynb Show resolved Hide resolved

examples/notebooks/beam-ml/rag_usecase/opensearch_rag_pipeline.ipynb Outdated Show resolved Hide resolved

itsayushpandey added 2 commits September 20, 2024 22:07

Added Description in opensearch notebook

f972959

Added description in opensearch notebook

c4c27bd

itsayushpandey requested a review from damccorm September 21, 2024 04:10

damccorm reviewed Sep 30, 2024

View reviewed changes

examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py Outdated Show resolved Hide resolved

itsayushpandey and others added 2 commits October 2, 2024 20:45

Merge branch 'apache:master' into rag_opensearch

8ddf529

Code review changes

3968fc6

itsayushpandey requested a review from damccorm October 3, 2024 00:58

damccorm approved these changes Oct 3, 2024

View reviewed changes

damccorm merged commit 1d7b0bc into apache:master Oct 3, 2024
3 checks passed

itsayushpandey deleted the rag_opensearch branch October 3, 2024 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rag opensearch usecase with Beam's MLTransform #32018

Rag opensearch usecase with Beam's MLTransform #32018

itsayushpandey commented Jul 30, 2024 •

edited

Loading

itsayushpandey commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

jrmccluskey Jul 31, 2024

jrmccluskey Jul 31, 2024

itsayushpandey Aug 10, 2024

jrmccluskey commented Aug 12, 2024

jrmccluskey left a comment

itsayushpandey commented Aug 21, 2024

jrmccluskey left a comment

itsayushpandey commented Aug 23, 2024 •

edited

Loading

damccorm commented Sep 5, 2024

itsayushpandey commented Sep 14, 2024

damccorm left a comment

damccorm left a comment

Rag opensearch usecase with Beam's MLTransform #32018

Rag opensearch usecase with Beam's MLTransform #32018

Conversation

itsayushpandey commented Jul 30, 2024 • edited Loading

GitHub Actions Tests Status (on master branch)

itsayushpandey commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

jrmccluskey Jul 31, 2024

Choose a reason for hiding this comment

jrmccluskey Jul 31, 2024

Choose a reason for hiding this comment

itsayushpandey Aug 10, 2024

Choose a reason for hiding this comment

jrmccluskey commented Aug 12, 2024

jrmccluskey left a comment

Choose a reason for hiding this comment

itsayushpandey commented Aug 21, 2024

jrmccluskey left a comment

Choose a reason for hiding this comment

itsayushpandey commented Aug 23, 2024 • edited Loading

damccorm commented Sep 5, 2024

itsayushpandey commented Sep 14, 2024

damccorm left a comment

Choose a reason for hiding this comment

damccorm left a comment

Choose a reason for hiding this comment

itsayushpandey commented Jul 30, 2024 •

edited

Loading

itsayushpandey commented Aug 23, 2024 •

edited

Loading