Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: few shot example optimzier #1739

Merged
merged 10 commits into from
Dec 9, 2024

Conversation

jjmachan
Copy link
Member

@jjmachan jjmachan commented Dec 7, 2024

optimize with few short examples

from ragas.metrics import AspectCritic
from ragas.llms import llm_factory

# define metric
llm = llm_factory("gpt-4o")
metric = AspectCritic(
    name="answer_correctness",
    definition="Given the user_input, reference and response. Is the response correct compared with the reference",
    llm=llm,
)

# optimize with annotation
from ragas.config import DemonstrationConfig
demonstration_config = DemonstrationConfig()
metric.train(
    "alignment_sample.json",
    demonstration_config=demonstration_config,
)

@jjmachan jjmachan marked this pull request as ready for review December 8, 2024 13:27
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Dec 8, 2024
@jjmachan jjmachan requested a review from shahules786 December 8, 2024 13:31
def add_example(self, input: BaseModel, output: BaseModel):
# get json string for input
input_json = input.model_dump_json()
self._embeddings_of_examples.append(self.embeddings.embed_query(input_json))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my experiments embedding user_input only and whole input (containing user_input, response,etc) gave almost identical results. But I think logically it makes more sense to only embed user_input. Although this has to be revisited again later. Let us leave it like this for now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but what if there is no user_input?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's something we have to figure out, ie what's the most accurate way to embed few shot examples. Let's leave it like this now.

src/ragas/config.py Outdated Show resolved Hide resolved
src/ragas/config.py Outdated Show resolved Hide resolved
@shahules786 shahules786 self-requested a review December 9, 2024 04:59
Copy link
Member

@shahules786 shahules786 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good. Logic and everything is perfect. Suggested few changes.

@jjmachan jjmachan requested a review from shahules786 December 9, 2024 14:51
@shahules786 shahules786 merged commit d432ed0 into explodinggradients:main Dec 9, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants