Skip to content

Commit

Permalink
fix(testset): for generating testset with new docs (#1001)
Browse files Browse the repository at this point in the history
fixes #848 

This pull request introduces updates to the TestsetGenerator and
ComplexEvolution classes within the generator.py and evolutions.py files
respectively. The primary focus of these changes is to improve the
handling of new documents during test dataset generation, ensuring that
data from previous documents does not interfere with the generation
process.

### Before
![Screenshot 2024-05-31 183725
before](https://github.com/explodinggradients/ragas/assets/77217074/dcc24e81-2e35-4e20-8638-2904d6779225)
Here row 2 and 4 are generated from the document used for the previous
generation
### After
![Screenshot 2024-05-31 184647
after](https://github.com/explodinggradients/ragas/assets/77217074/582afeeb-9ade-4d35-891c-af54af21c859)
Here all rows generated are based on the new document provided
  • Loading branch information
Manav916 committed Jul 2, 2024
1 parent 0fbd45f commit 3101ce8
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 11 deletions.
15 changes: 7 additions & 8 deletions src/ragas/testset/evolutions.py
Original file line number Diff line number Diff line change
Expand Up @@ -352,14 +352,13 @@ def init(self, is_async: bool = True, run_config: t.Optional[RunConfig] = None):
run_config = RunConfig()
super().init(is_async=is_async, run_config=run_config)

if self.se is None:
# init simple evolution to get seed question
self.se = SimpleEvolution(
generator_llm=self.generator_llm,
docstore=self.docstore,
node_filter=self.node_filter,
question_filter=self.question_filter,
)
# init simple evolution to get seed question
self.se = SimpleEvolution(
generator_llm=self.generator_llm,
docstore=self.docstore,
node_filter=self.node_filter,
question_filter=self.question_filter,
)
# init evolution filter with critic llm from another filter
assert self.node_filter is not None, "node filter cannot be None"
if self.evolution_filter is None:
Expand Down
6 changes: 3 additions & 3 deletions src/ragas/testset/generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -217,11 +217,11 @@ def generate_with_langchain_docs(
)

def init_evolution(self, evolution: Evolution) -> None:
evolution.docstore = self.docstore

if evolution.generator_llm is None:
evolution.generator_llm = self.generator_llm
if evolution.docstore is None:
evolution.docstore = self.docstore


if evolution.question_filter is None:
evolution.question_filter = QuestionFilter(llm=self.critic_llm)
if evolution.node_filter is None:
Expand Down

0 comments on commit 3101ce8

Please sign in to comment.