Initial commit for RAG pipeline scripts #427
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hmumtazz
requested review from
dhrubo-os,
greaa-aws,
ylwu-amzn,
b4sjoo,
jngz-es and
rbhavna
as code owners
|
DCO is missing |
hmumtazz
force-pushed
the
rag_pipeline
branch
from
43e01d3
to
db78131
Compare
ylwu-amzn
reviewed
Nov 15, 2024
ylwu-amzn
reviewed
Nov 15, 2024
ylwu-amzn
reviewed
Nov 15, 2024
ylwu-amzn
reviewed
Nov 15, 2024
opensearch_py_ml/ml_commons/rag_pipeline/rag/opensearch_class.py
Outdated
Show resolved
Hide resolved
ylwu-amzn
reviewed
Nov 16, 2024
| }, | ||
| "settings": { | ||
| "index": { | ||
| "number_of_shards": 2, |
brianf-aws
reviewed
Nov 16, 2024
brianf-aws
suggested changes
Nov 17, 2024
There was a problem hiding this comment.
really cool stuff here! 📖 🤖 💬 Lets try and refactor some code so its reusable across classes also lets apply SRP so that a class isnt burdened by doing a lot at once.
Make sure to add documentation to files and methods.
Also Im seeing there exists a query python file within py-ml https://github.com/opensearch-project/opensearch-py-ml/blame/main/opensearch_py_ml/query.py if possible maybe we can aggregate to existing code?
Lastly lets come up with a great description of the feature. You put a lot of effort so lets make it visually appealing in the PR description (diagrams, how to use, gifs, concise summary; emojis) You can use this as influence opensearch-project/neural-search#933 talk to @dhrubo-os if maybe we want to open up a Issue and then link this PR not sure if thats too much.
Great work!
Comment on lines
+27
to
+51
| self.aws_region = config.get('region') | ||
| self.index_name = config.get('index_name') |
There was a problem hiding this comment.
Is there input validation involved? maybe we can catch this earlier so it doesn't have to be a headache later?
dhrubo-os
reviewed
Nov 19, 2024
dhrubo-os
reviewed
Nov 19, 2024
dhrubo-os
reviewed
Nov 19, 2024
dhrubo-os
reviewed
Nov 19, 2024
| print(f"Failed to initialize clients: {e}") | ||
| return False | ||
|
|
||
| def process_file(self, file_path: str) -> List[Dict[str, str]]: |
There was a problem hiding this comment.
check docstring for other methods in this repo and follow the same style.
dhrubo-os
reviewed
Nov 19, 2024
dhrubo-os
reviewed
Nov 19, 2024
opensearch_py_ml/ml_commons/rag_pipeline/rag/opensearch_class.py
Outdated
Show resolved
Hide resolved
dhrubo-os
reviewed
Nov 19, 2024
|
DCO is missing. |
hmumtazz
force-pushed
the
rag_pipeline
branch
2 times, most recently
from
ced6a59
to
3260f3a
Compare
Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
…ffering a suggested default value with the flexibility for users to enter a custom value if needed. Signed-off-by: hmumtazz <[email protected]>
…, addressed comments, fixed upload csv method Signed-off-by: hmumtazz <[email protected]>
This commit serves as a blanket sign-off for all previous commits in this pull request, in compliance with the Developer Certificate of Origin (DCO). Signed-off-by: hmumtazz <[email protected]>
hmumtazz
force-pushed
the
rag_pipeline
branch
from
a05e767
to
2635751
Compare
- Add functionality for users to upload entire folders by specifying a file path, enabling batch processing of multiple files - Show path to config file - Implement a visual confirmation system for setup completion, using red/green indicators similar to the document ingestion process - Combine the RAG setup and model registration into one unified process, prompting users if they want to register a model, offering options or if CX wants to use their already made custom model ID input, and providing clear confirmation when a model ID is saved, all while streamlining the overall setup for improved user experience - Update user interface starting from setup Signed-off-by: hmumtazz <[email protected]>
…pre-existing one, allowing users to run a single setup process - Update requirements.txt to support RAG dependencies Signed-off-by: hmumtazz <[email protected]>
dhrubo-os
reviewed
Nov 27, 2024
|
|
||
| if domain_endpoint: | ||
| # Construct the full domain URL | ||
| self.opensearch_domain_url = f'https://{domain_endpoint}' |
There was a problem hiding this comment.
better to get the total domain url I think. What if somebody wants to use http?
dhrubo-os
reviewed
Nov 27, 2024
opensearch_py_ml/ml_commons/rag_pipeline/rag/AIConnectorHelper.py
Outdated
Show resolved
Hide resolved
dhrubo-os
reviewed
Nov 27, 2024
opensearch_py_ml/ml_commons/rag_pipeline/rag/AIConnectorHelper.py
Outdated
Show resolved
Hide resolved
dhrubo-os
reviewed
Nov 27, 2024
dhrubo-os
reviewed
Nov 27, 2024
…s can now specify their model payload through JSON object Signed-off-by: hmumtazz <[email protected]>
- Remove requirements.txt and setup.py from Git repository - Add these files to .gitignore to prevent future tracking - Keep local copies of these files for development purposes Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
- Delete opensearch_py_ml/ml_commons/rag_pipeline/rag/.gitignore - Consolidate gitignore rules in the root .gitignore file Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
Signed-off-by: hmumtazz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
[Describe what this change achieves]
Issues Resolved
[List any issues this PR will resolve]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.