-
Notifications
You must be signed in to change notification settings - Fork 646
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation of Noise sensitivity metrics from RAGChecker #1190
Implementation of Noise sensitivity metrics from RAGChecker #1190
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR, great work. @sahusiddharth
Two more suggestions
- Can you fix the conflict?
- Can you also add a documentation associated with noise sensitivity?
TBH understanding the use and internals of this is little overwhelming, users will find it hard to adopt it if they can't understand it.
Could you please provide some suggestions on the types of documentation that might be required? Your input would be greatly appreciated. |
a5b2c28
to
dfe069f
Compare
Hey @sahusiddharth I did see this doc written by one of author of ragchecker, I have also emailed him asking for an intuitive explanation of noise sensitivity (let's wait for few hours and see if he replies). Otherwise, it would be nice to follow how the format in metrics as here https://docs.ragas.io/en/stable/concepts/metrics/context_recall.html |
Hi @shahules786, Got it. Please let me know when you hear back from the author. In the meantime, I’ll check the metrics format you mentioned. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @sahusiddharth The changes you did looks good. I also did some corrections on top of that. I think it's best to move forward and do the doc now - it the author reacts later we can get his feedback on it later. Can you prepare the docs to go with this?
It needs 3 sections:
A brief intuitive description of metrics
an example
how it's calculated using the example
Hi @shahules786, I wanted to clarify something regarding the noise sensitivity implementation. There are two types: one for when relevant context is retrieved and another for when irrelevant context is retrieved. The current implementation only addresses the relevant context. Would you prefer that I add the handling for irrelevant context first, or should I complete the documentation for the basic implementation before proceeding with the additional functionality? |
@sahusiddharth I did notice that, thought that using relevant might be more useful but now I think if the user has the ability to switch b/w both using an argument that would be better. Can you modify it to include that behavior? Then we can add documentation for both in same page. |
I’m happy to make the necessary modifications. I would appreciate some additional guidance on how best to return the results when asked for both. Since the output could be returned in multiple types of data, such as dictionaries, named tuples, or tuples, I’m considering the most appropriate format. {'noise_sensitivity_relevant': 0.0, 'noise_sensitivity_irrelevant': 0.0} I was thinking to return only the number when asked for a specific one. Could you please advise on the preferred format for returning these results? |
@sahusiddharth Thanks for asking that. I think both should not be an option - it would be one of them 'relevant' or 'irrelevant'. By default, it should stay as 'relevant'. Make sure that when you write the doc give credit to Ragchecker by citing the work. |
@shahules786, I'm almost done with the documentation, to properly show the power of noise sensitivity, the example is getting long, Do you have a problem with that? |
@sahusiddharth We can refine it later , but I also show some basic examples here |
@shahules786, Have gone through them, I didn't liked it that much the answers generated by llm is rarely using the information provided in the context and I didn't find it intuitive enough. |
@sahusiddharth I agree, their dataset generation is naive. |
@shahules786, tried returning tuple when given we want noise sensitivity for both relevant and irrelevant the make-ci was giving me error.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, I have made few changes to polish it up.
Solves:
[R-293] Noise sensitivity metrics from RAGChecker #1185
Took inspiration from RAGChecker from AWS Noise sensitivity noise sensitivity metrics.
Have tested it locally, it is working giving the results.
Input