v0.2.8

What's Changed

• fix: Update README.md by @avanco in #1738
• feat: genetic algo based optimizer by @shahules786 in #1724
• Added native support for Gemini models generation completion signals in LangchainLLMWrapper class / is_finished method by @cnemri in #1727
• Added custom is_finished_parser logic to Google Vertex AI customizati… by @cnemri in #1728
• feat: added basics for upload by @jjmachan in #1741
• feat: few shot example optimzier by @jjmachan in #1739
• fix: correct url for upload() by @jjmachan in #1742
• fix: prompt naming related issues by @shahules786 in #1743
• feat: load from annotated testset by @jjmachan in #1747
• docs: add demo example for metric training by @shahules786 in #1746
• docs: demo related fixes by @shahules786 in #1750

New Contributors

• @avanco made their first contribution in #1738
• @cnemri made their first contribution in #1727

Full Changelog: v0.2.7...v0.2.8

v0.2.7

What's Changed

• feat: removed critics and rubrics from examples by @jjmachan in #1691
• feat: test-generation improvements by @shahules786 in #1698
• chore: updated tracking for single scores by @jjmachan in #1716
• feat: remove examples by @jjmachan in #1708
• Bug in documentation rag_testset_generation.md: API changes required arguments by @bmerkle in #1714
• feat: add output type to metrics by @shahules786 in #1722
• feat: dataloader for annotated json by @shahules786 in #1723
• feat: train your own evaluators by @shahules786 in #1701
• fix: attempting a simple fix for #1718 by @ayulockin in #1721
• fix: add missing query qualifier by @shahules786 in #1733
• feat: Test generation for non-english corpus by @shahules786 in #1734

New Contributors

• @bmerkle made their first contribution in #1714
• @ayulockin made their first contribution in #1721

Full Changelog: v0.2.6...v0.2.7

v0.2.6

What's Changed

• Avoid unnecessary decompose_claims call when mode='precision' in _factual_correctness.py by @dfhssilva in #1669
• fixes: instruction format by @shahules786 in #1672
• docs: updated readme by @jjmachan in #1674
• docs: update readme by @jjmachan in #1675
• fixes: add extraction limit to Extractors by @shahules786 in #1673
• fix: upload for testset by @jjmachan in #1676
• removed MetricWithLLM from SemanticSimilarity by @Luka958Pixion in #1678
• feat: improvements in default test generation by @shahules786 in #1661
• Update rag.md by @TimWue in #1686
• fix: missing embeddings argument in testset and some E2E tests by @jjmachan in #1690
• fixes: handle long context extraction by @shahules786 in #1680
• feat: make general purpose metrics more general by @jjmachan in #1666
• Fixes ToolCallAccuracy raises ZeroDivisionError when called without any arguments. by @sahusiddharth in #1685

New Contributors

• @dfhssilva made their first contribution in #1669
• @Luka958Pixion made their first contribution in #1678
• @TimWue made their first contribution in #1686

Full Changelog: v0.2.5...v0.2.6

v0.2.5

What's Changed

• feat: Unify the use of sentence_segmenter by @jasinliu in #1629
• feat: aspect critic with reference by @shahules786 in #1638
• docs: change setting llm by @shahules786 in #1639
• fixed wrong variable names in starter guide using azure openai by @MichaelPerger in #1641
• feat: improvements in test gen by @shahules786 in #1645
• Add tutorial for evaluating LangGraph agents by @sahusiddharth in #1636
• docs: add cost tracker back by @shahules786 in #1653
• docs: tracing and debugging metrics by @shahules786 in #1652
• feat: upload testset generation upload by @jjmachan in #1647
• docs: reorganise files by @shahules786 in #1662
• docs: fix _arize.md by @suekou in #1643
• docs: persona generator by @jjmachan in #1664
• Bug fixing recall in _factual_correctness.py by @GitMarco27 in #1650

New Contributors

• @MichaelPerger made their first contribution in #1641
• @GitMarco27 made their first contribution in #1650

Full Changelog: v0.2.4...v0.2.5

v0.2.4

What's Changed

• Fix : Filename in the path by @satodayo in #1605
• fix: remove abstract class in Metric by @jjmachan in #1603
• Added pandas as a method for reading datasets by @joaorura in #1595
• fix: treat anthropic-direct API the same as bedrock in llamaindex wrapper by @ahgraber in #1584
• feat: add support for llamaindex by @jjmachan in #1609
• docs: Add embedding_model attribute when creating TestsetGenerator object by @Youngrok123 in #1606
• Fix: Add instance check before casting ChatGeneration object by @Youngrok123 in #1607
• docs: add comment for claude-3-5-sonnet-20241022 by @SokiKawashima in #1611
• docs: add community docs by @Yunnglin in #1612
• Handle embeddings for empty strings in AnswerSimilarity class by @baptiste-pasquier in #994
• Feature: batched execution by @ahgraber in #1589
• docs: how to add custom metrics by @jjmachan in #1616
• docs: added azure openai to default docs by @jjmachan in #1610
• Introduce epsilon in denominators to avoid division by zero by @jltham in #1622
• feat: Add support for llamaIndex in evaluation by @suekou in #1619
• feat: added upload for EvaluationResult by @jjmachan in #1625
• feat: automatic persona generation by @shahules786 in #1618
• docs: Refactor Code for Syntax Highlighting and URL Updates by @suekou in #1634
• fix: remove . as a split condition in noise_sensitivity.py by @jasinliu in #1627
• feat: improvements in test synthesization by @shahules786 in #1621

New Contributors

• @satodayo made their first contribution in #1605
• @joaorura made their first contribution in #1595
• @SokiKawashima made their first contribution in #1611
• @jltham made their first contribution in #1622
• @jasinliu made their first contribution in #1627

Full Changelog: v0.2.3...v0.2.4

v0.2.3

What's Changed

• feat: measure cost of testset generator by @jjmachan in #1560
• docs: added luka's blogs by @jjmachan in #1554
• Fix: add LLMContextPrecisionWithReference to init.py by @licux in #1561
• chore: add verbose option (V=1) to make targets by @trevorbowen in #1556
• add embeddings to TestsetGenerator by @hunter-walden2113 in #1562
• fixed verb agreement @ available_metrics by @gabrielhomsi in #1574
• Fix: Limit number of retries for parse failures by @rskew in #1569
• Fix: reference key error in LLMContextPrecisionWithoutReference by @xizhou-vw in #1570
• Updated return type description for evaluate function by @taihim in #1579
• Feat: add multimodal eval support by @Yunnglin in #1559
• fix: add reference_topics as default required columns in TopicAdherenceScore #1564 by @luqmansen in #1566
• fix: add reference tool call to required cols by @shahules786 in #1580
• Improve efficiency in factual correctness for precision mode by @Jeff-67 in #1578
• This commit implements the F-beta score metric by @Yuri-Albuquerque in #1543
• fix: agent goal accuracy by @shahules786 in #1583
• chores: fix pypi rendering by @shahules786 in #1581
• fix: typo: ROUGE is a metric, ROGUE is a scoundrel by @ahgraber in #1585

New Contributors

• @trevorbowen made their first contribution in #1556
• @hunter-walden2113 made their first contribution in #1562
• @gabrielhomsi made their first contribution in #1574
• @rskew made their first contribution in #1569
• @xizhou-vw made their first contribution in #1570
• @taihim made their first contribution in #1579
• @luqmansen made their first contribution in #1566
• @Yuri-Albuquerque made their first contribution in #1543

Full Changelog: v0.2.2...v0.2.3

v0.2.2

What's Changed

• docs: update summarization calculation by @jCobbSK in #1510
• fix: use get_temperature in base LLM generate by @sapphire-arches in #1520
• doc: Update modifying prompts metrics doc by @FrankyCTY in #1525
• fixbug: Expand sentence-ending punctuation check in _create_statements method by @Jeff-67 in #1523
• fix: zero division error by @shahules786 in #1530
• feat: add traces to EvaluationResult by @jjmachan in #1531
• feat: translate instruction when adapting prompt by @Yunnglin in #1529
• chore: remove old prompt object by @jjmachan in #1535
• feat: improved the testset generation to_pandas and docs by @jjmachan in #1536
• fixed typo in "single_turn_ascore" by @minorun365 in #1542
• update AspectCritic docs examples by @licux in #1547
• Fix: Resolve Agentic failure loop with improved json parsing by @ahgraber in #1541
• feat: throw error when max_token limit is reached by @jjmachan in #1549
• docs: fix Amazon bedrock info by @jjmachan in #1551
• Added community blog posts in Japanese by @minorun365 in #1550
• docs: make llms and embeddings explicit by @jjmachan in #1553

New Contributors

• @jCobbSK made their first contribution in #1510
• @sapphire-arches made their first contribution in #1520
• @FrankyCTY made their first contribution in #1525
• @Jeff-67 made their first contribution in #1523
• @Yunnglin made their first contribution in #1529
• @licux made their first contribution in #1547
• @ahgraber made their first contribution in #1541

Full Changelog: v0.2.1...v0.2.2

v0.2.1

What's Changed

• Fix Discord link by @NirantK in #1498
• Fix Quickstart link by @NirantK in #1499
• Rewrite Open Analytics by @NirantK in #1500
• Suggestions for Introduction and Key Features by @NirantK in #1501
• docs: fix broken links in testset generation custimisation by @jjmachan in #1502
• Update Bedrock modules for LangChain v0.3.x by @minorun365 in #1487
• Fixes: Quickstart Ragas Metrics by @NirantK in #1504
• Add Contributors by @NirantK in #1505
• docs: Change test_size parameter name to testset_size by @Youngrok123 in #1507
• docs: readme add llm by @shahules786 in #1511
• docs: add faq by @shahules786 in #1503
• fix: propagate callbacks in RagasOutputParser by @dlyz in #1513
• docs: fix usage of eval dataset by @shahules786 in #1514
• feat: use ensure_ascii=False for KnowledgeGraph files by @dlyz in #1518
• fix: import error for TestsetGeneration and small fixes by @jjmachan in #1516

New Contributors

• @NirantK made their first contribution in #1498
• @minorun365 made their first contribution in #1487
• @dlyz made their first contribution in #1513

Full Changelog: v0.2.0...v0.2.1

v0.2.0

What's Changed

• feat: topic adherence metric by @shahules786 in #1327
• feat: tool call accuracy by @shahules786 in #1311
• feat: sql sementic equivalence by @shahules786 in #1323
• feat: Factual correctnes metric by @shahules786 in #1334
• docs: migrating over to mkdocs by @jjmachan in #1301
• feat: Adding transforms for testset generation by @jjmachan in #1344
• docs: traditional metrics by @shahules786 in #1346
• feat: transforms for Knowledge Graphs by @jjmachan in #1345
• docs: core concepts in metrics by @shahules786 in #1347
• chores: cleanup metrics by @shahules786 in #1348
• docs: refactor core concepts by @shahules786 in #1351
• wrap as_complete with async by @liupgd in #1350
• chores: add datasets to init by @shahules786 in #1356
• docs: new getting started by @shahules786 in #1355
• docs: metrics concepts by @shahules786 in #1361
• docs: add mindmap by @shahules786 in #1362
• Fixed Nav, search, and some styling issues for the docs by @joelk9895 in #1363
• feat: transform engine to execute transforms by @jjmachan in #1352
• chores: remove fuzzywuzzy by @shahules786 in #1369
• docs: component guides by @shahules786 in #1365
• chores: fix typo by @shahules786 in #1370
• fix:ensure correct sequence in tool calls by @shahules786 in #1371
• [howtos] update Langfuse cookbook by @jannikmaierhoefer in #1368
• feat: adding an implementation of abstractQA by @jjmachan in #1359
• feat: implement TestsetGenerator by @jjmachan in #1375
• docs: testset generation by @shahules786 in #1373
• feat: added more simulators by @jjmachan in #1376
• Update models in Ragas documentation: change gpt-3.5-turbo-16k to gpt-4o-mini and gpt-4 to gpt-4o by @suekou in #1380
• fix: aspect_critique documentation by @jjmachan in #1382
• docs: added google analytics to mkdocs by @jjmachan in #1384
• docs: fix index for metric/index.md by @jjmachan in #1388
• fix: fixed verdict comparison in evolutions.py by @MarkHmnv in #1366
• feat: output new Testset object for generation by @jjmachan in #1386
• Update answer_relevance.md: Minor spell correct by @soumendrak in #1390
• feat: support for callbacks and traces in testset generation by @jjmachan in #1392
• non-asii characters support in faithfulness by @gargewp in #1396
• feat: get and set prompts mixin by @shahules786 in #1391
• fixes: rename naming in testset generation by @shahules786 in #1393
• chore!: moving to pydantic2 by @jjmachan in #1394
• docs: improve getting started by @shahules786 in #1398
• Fix typo in metric error message by @suekou in #1404
• feat!: moving experimental.testset to replace old testset by @jjmachan in #1401
• feat: renamed sample operations and cleaned up test warnings by @jjmachan in #1407
• added better styling to reder jupyter notebooks nicely. by @zodwick in #1389
• docs: add core concepts for test generation by @shahules786 in #1402
• docs:Auto Gen of Reference Page by @zodwick in #1408
• fix key error for FaithfulnesswithHHEM by @linzzzzzz in #1410
• fix: remove usage of contexts in response relevancy by @shahules786 in #1409
• fix: score for jupyter by @shahules786 in #1411
• feat: NER extractor and Jaccard by @shahules786 in #1406
• feat(styles): refactor CSS to use custom variables by @zodwick in #1413
• docs : add customisation by @shahules786 in #1412
• feat(docs): add support for reusable snippets by @zodwick in #1414
• Feature/css variables by @zodwick in #1415
• docs: fix nav bar by @shahules786 in #1416
• feat: more features to prompt object by @jjmachan in #1418
• chore: add tracking for new testset generation by @jjmachan in #1419
• feat: adapt prompt to different languages by @jjmachan in #1429
• chores: add reference to gitignore by @shahules786 in #1428
• docs: redo intro by @shahules786 in #1427
• Add Custom Ragas Light and Dark Themes (v1) by @zodwick in #1431
• fix attribute error for faithfulness metric by @linzzzzzz in #1436
• docs: language adaptation by @shahules786 in #1446
• fix: data export by @shahules786 in #1461
• feat: save and load Prompts by @jjmachan in #1458
• refactor: convert faithfulness, answer_correctness and noise_sensitivity to new prompt object by @jjmachan in #1462
• Minor typo in _answer_relevance.py by @pjanowski in #1452
• feat: added repr for prompt objects by @jjmachan in #1459
• feat: change context recall and precision prompts to new by @jjmachan in #1467
• docs: fix invalid links and imports by @shahules786 in #1473
• feat: changed summary to new prompt by @jjmachan in #1469
• docs: redo readme by @shahules786 in #1480
• docs: new build process with rtd and API documentation by @jjmachan in #1474
• chore: added install for mkdocs insiders by @jjmachan in #1481
• chores: update gitignore by @shahules786 in #1483
• docs: improve testgen namings by @shahules786 in #1489
• Fixed an issue where non-ASCII characters were changed to Unicode characters within the prompt by @Youngrok123 in #1490
• docs: fix search by @shahules786 in #1491
• feat: added save and load to RagasDataset by @jjmachan in #1492
• docs: moved from ipnb files to md files by @jjmachan in #1482
• feat: set ensure_ascii to false by @jjmachan in #1494
• feat: small refactors and cleanups by @jjmachan in #1493

New Contributors

• @liupgd made their first contribution in #1350
• @joelk9895 made their first contribution in #1363
• @jannikmaierhoefer made their first contribution in #1368
• @suekou made their first contribution in #1380
• @MarkHmnv made their first contribution in #1366
• @soumendrak made their first contribution in #1390
• @gargewp made their first contribution in #1396
• @zodwick made their first contribution in #1389
• @linzzzzzz made their first contribution in https://git...

v0.2.0-beta

What's Changed

• feat: topic adherence metric by @shahules786 in #1327
• feat: tool call accuracy by @shahules786 in #1311
• feat: sql sementic equivalence by @shahules786 in #1323
• feat: Factual correctnes metric by @shahules786 in #1334
• docs: migrating over to mkdocs by @jjmachan in #1301
• feat: Adding transforms for testset generation by @jjmachan in #1344
• docs: traditional metrics by @shahules786 in #1346
• feat: transforms for Knowledge Graphs by @jjmachan in #1345
• docs: core concepts in metrics by @shahules786 in #1347
• chores: cleanup metrics by @shahules786 in #1348
• docs: refactor core concepts by @shahules786 in #1351
• wrap as_complete with async by @liupgd in #1350
• chores: add datasets to init by @shahules786 in #1356
• docs: new getting started by @shahules786 in #1355
• docs: metrics concepts by @shahules786 in #1361
• docs: add mindmap by @shahules786 in #1362
• Fixed Nav, search, and some styling issues for the docs by @joelk9895 in #1363
• feat: transform engine to execute transforms by @jjmachan in #1352
• chores: remove fuzzywuzzy by @shahules786 in #1369
• docs: component guides by @shahules786 in #1365
• chores: fix typo by @shahules786 in #1370
• fix:ensure correct sequence in tool calls by @shahules786 in #1371
• [howtos] update Langfuse cookbook by @jannikmaierhoefer in #1368
• feat: adding an implementation of abstractQA by @jjmachan in #1359
• feat: implement TestsetGenerator by @jjmachan in #1375
• docs: testset generation by @shahules786 in #1373
• feat: added more simulators by @jjmachan in #1376
• Update models in Ragas documentation: change gpt-3.5-turbo-16k to gpt-4o-mini and gpt-4 to gpt-4o by @suekou in #1380
• fix: aspect_critique documentation by @jjmachan in #1382
• docs: added google analytics to mkdocs by @jjmachan in #1384
• docs: fix index for metric/index.md by @jjmachan in #1388
• fix: fixed verdict comparison in evolutions.py by @MarkHmnv in #1366
• feat: output new Testset object for generation by @jjmachan in #1386
• Update answer_relevance.md: Minor spell correct by @soumendrak in #1390
• feat: support for callbacks and traces in testset generation by @jjmachan in #1392
• non-asii characters support in faithfulness by @gargewp in #1396
• feat: get and set prompts mixin by @shahules786 in #1391
• fixes: rename naming in testset generation by @shahules786 in #1393
• chore!: moving to pydantic2 by @jjmachan in #1394
• docs: improve getting started by @shahules786 in #1398
• Fix typo in metric error message by @suekou in #1404
• feat!: moving experimental.testset to replace old testset by @jjmachan in #1401
• feat: renamed sample operations and cleaned up test warnings by @jjmachan in #1407
• added better styling to reder jupyter notebooks nicely. by @zodwick in #1389
• docs: add core concepts for test generation by @shahules786 in #1402
• docs:Auto Gen of Reference Page by @zodwick in #1408
• fix key error for FaithfulnesswithHHEM by @linzzzzzz in #1410
• fix: remove usage of contexts in response relevancy by @shahules786 in #1409
• fix: score for jupyter by @shahules786 in #1411
• feat: NER extractor and Jaccard by @shahules786 in #1406
• feat(styles): refactor CSS to use custom variables by @zodwick in #1413
• docs : add customisation by @shahules786 in #1412
• feat(docs): add support for reusable snippets by @zodwick in #1414
• Feature/css variables by @zodwick in #1415
• docs: fix nav bar by @shahules786 in #1416
• feat: more features to prompt object by @jjmachan in #1418
• chore: add tracking for new testset generation by @jjmachan in #1419

New Contributors

• @liupgd made their first contribution in #1350
• @joelk9895 made their first contribution in #1363
• @jannikmaierhoefer made their first contribution in #1368
• @suekou made their first contribution in #1380
• @MarkHmnv made their first contribution in #1366
• @soumendrak made their first contribution in #1390
• @gargewp made their first contribution in #1396
• @zodwick made their first contribution in #1389
• @linzzzzzz made their first contribution in #1410

Full Changelog: v0.1.20...v0.2.0-beta