{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":637924634,"defaultBranch":"main","name":"ragas","ownerLogin":"explodinggradients","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2023-05-08T17:48:04.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/122604797?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726068403.0","currentOid":""},"activityList":{"items":[{"before":"a4a3e5655665fc716b286c6590e1718a22edfd26","after":"ae50d459904759e08ad926a7b598d0d15efaf647","ref":"refs/heads/main","pushedAt":"2024-09-11T12:58:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"fix: make score nested if loop_is_running (#1276)","shortMessageHtmlLink":"fix: make score nested if loop_is_running (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2519638954\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1276\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1276/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1276\">#1276</a>)"}},{"before":"3076f509dc5fe2b0418b1de60ece7dc991da0e84","after":"a4a3e5655665fc716b286c6590e1718a22edfd26","ref":"refs/heads/main","pushedAt":"2024-09-11T10:45:06.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: v1 to v2 dataset (#1275)\n\nfixes: #1271","shortMessageHtmlLink":"fix: v1 to v2 dataset (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2519368972\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1275\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1275/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1275\">#1275</a>)"}},{"before":"c615a9f89258b9f3370bc9b9eb9c3a447e9c2792","after":"3076f509dc5fe2b0418b1de60ece7dc991da0e84","ref":"refs/heads/main","pushedAt":"2024-09-11T04:28:21.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: non llm based metrics  (#1268)\n\n1) rename metrics\r\n2) delay import of optional dependencies","shortMessageHtmlLink":"fix: non llm based metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2517352529\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1268\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1268/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1268\">#1268</a>)"}},{"before":"4e6a96a0ff3ac613f64e2d01199e42665cea643f","after":"c615a9f89258b9f3370bc9b9eb9c3a447e9c2792","ref":"refs/heads/main","pushedAt":"2024-09-10T17:29:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: inverse distance (#1267)","shortMessageHtmlLink":"fix: inverse distance (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2517035727\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1267\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1267/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1267\">#1267</a>)"}},{"before":"054c0e90ce98ddbee68010941f3385ae5e9b45ed","after":"4e6a96a0ff3ac613f64e2d01199e42665cea643f","ref":"refs/heads/main","pushedAt":"2024-09-10T13:27:10.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: set and get prompts for metrics (#1259)\n\n```python\r\nfrom ragas.experimental.metrics._faithfulness import FaithfulnessExperimental, LongFormAnswerPrompt\r\n\r\nfaithfulness = FaithfulnessExperimental() \r\nfaithfulness.get_prompts()\r\n\r\n#{'long_form_answer_prompt': <ragas.experimental.metrics._faithfulness.LongFormAnswerPrompt at 0x7fd7baa8efb0>,\r\n#'nli_statement_prompt': <ragas.experimental.metrics._faithfulness.NLIStatementPrompt at 0x7fd7baa8f010>}\r\n\r\nlong_form_prompt = LongFormAnswerPrompt()\r\nlong_form_prompt.instruction = \"my new instruction\"\r\n\r\nprompts = {\"long_form_answer_prompt\":long_form_prompt}\r\nfaithfulness.set_prompts(**prompts)\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Jithin James <jamesjithin97@gmail.com>","shortMessageHtmlLink":"feat: set and get prompts for metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2513095505\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1259\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1259/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1259\">#1259</a>)"}},{"before":"9709139f942770f2cc229bb48111042ec1262e1c","after":"054c0e90ce98ddbee68010941f3385ae5e9b45ed","ref":"refs/heads/main","pushedAt":"2024-09-10T12:27:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Non LLM based metrics (#1260)\n\nAdded support for \r\n1. BLEU SCORE\r\n2. ROGUE SCORE\r\n3. STRING MATCH\r\n4. EDIT DISTANCE","shortMessageHtmlLink":"Non LLM based metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2513630390\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1260\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1260/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1260\">#1260</a>)"}},{"before":"8cd081e96d3c0cc27a72785df1fd2f2b9f0fcaf6","after":"9709139f942770f2cc229bb48111042ec1262e1c","ref":"refs/heads/main","pushedAt":"2024-09-10T04:48:28.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: handle v0.1 dataset (#1262)\n\nfixes: #1257","shortMessageHtmlLink":"fix: handle v0.1 dataset (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2515383034\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1262\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1262/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1262\">#1262</a>)"}},{"before":"2e6ac350c14e4886e121b9136789e4c8776d7742","after":"8cd081e96d3c0cc27a72785df1fd2f2b9f0fcaf6","ref":"refs/heads/main","pushedAt":"2024-09-09T14:43:49.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: add single_turn_ascore (#1261)","shortMessageHtmlLink":"fix: add single_turn_ascore (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2514148812\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1261\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1261/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1261\">#1261</a>)"}},{"before":"193947a688250015ba18f43d404612213e551b7b","after":"2e6ac350c14e4886e121b9136789e4c8776d7742","ref":"refs/heads/main","pushedAt":"2024-09-09T13:00:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Add support to MultiTurn Metrics (#1249)\n\n```python\r\nfrom ragas import evaluate\r\nfrom ragas.metrics import rubrics_score_without_reference\r\nfrom ragas.dataset_schema import EvaluationDataset, MultiTurnSample\r\nfrom ragas.messages import HumanMessage\r\n\r\nsample1 = MultiTurnSample(user_input=[HumanMessage(content=\"What is X\")])\r\nsample2 = MultiTurnSample(user_input=[HumanMessage(content=\"What is X\")])\r\nds = EvaluationDataset(samples=[sample1, sample2])\r\nevaluate(ds,metrics=[rubrics_score_without_reference])\r\n```","shortMessageHtmlLink":"Add support to MultiTurn Metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2507554544\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1249\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1249/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1249\">#1249</a>)"}},{"before":"d5fe288405f35212e731ad2c4cd332c4595e6005","after":null,"ref":"refs/heads/ref/pull/1256/head","pushedAt":"2024-09-07T10:55:34.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"}},{"before":"d7df217a78276bace60de887b74d5f0d758ea112","after":"193947a688250015ba18f43d404612213e551b7b","ref":"refs/heads/main","pushedAt":"2024-09-07T10:55:13.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Feat: Opik Integration (#1256)\n\nThis PR adds an integration to the [Opik\r\nplatform](https://github.com/comet-ml/opik), in includes both a new\r\ntracer and some documentation updates.\r\n\r\n---------\r\n\r\nCo-authored-by: Sid Mehta <siddharthm2350@gmail.com>\r\nCo-authored-by: Abby Morgan <86856445+anmorgan24@users.noreply.github.com>","shortMessageHtmlLink":"Feat: Opik Integration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2510188208\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1256\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1256/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1256\">#1256</a>)"}},{"before":null,"after":"d5fe288405f35212e731ad2c4cd332c4595e6005","ref":"refs/heads/ref/pull/1256/head","pushedAt":"2024-09-07T10:46:16.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"added extra dependency to dev","shortMessageHtmlLink":"added extra dependency to dev"}},{"before":"ab5fc751b3344c7f01761491127e8519ecb61c0c","after":"d7df217a78276bace60de887b74d5f0d758ea112","ref":"refs/heads/main","pushedAt":"2024-09-05T15:25:45.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"New data representation (#1222)\n\n```python\r\nfrom ragas import evaluate\r\nfrom ragas.metrics import answer_correctness, answer_similarity, faithfulness, context_precision, context_recall\r\nfrom ragas.dataset_schema import SingleTurnSample,EvaluationDataset\r\n\r\ns1 = SingleTurnSample(user_input=\"What is X\", response=\"X is Y\", reference=\"Y is Y\",retrieved_contexts=[\"Nothing here\"])\r\ns2 = SingleTurnSample(user_input=\"What is X\", response=\"X is Y\", reference=\"Y is Y\",retrieved_contexts=[\"Nothing here\"])\r\neval_dataset = EvaluationDataset(samples=[s1,s2])\r\n\r\noutput = evaluate(eval_dataset, metrics=[faithfulness, answer_correctness,context_precision,context_recall,])\r\n\r\n```","shortMessageHtmlLink":"New data representation (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2486830285\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1222\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1222/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1222\">#1222</a>)"}},{"before":"fe379a1c97d18ce2c203d80432a3da6622337968","after":"ab5fc751b3344c7f01761491127e8519ecb61c0c","ref":"refs/heads/main","pushedAt":"2024-09-04T10:03:46.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"update documentation for faithfulness with HHEM (#1242)\n\ncc @forrestbao\r\n\r\n---------\r\n\r\nCo-authored-by: Miaoran <miaoran@vectara.com>\r\nCo-authored-by: jjmachan <jamesjithin97@gmail.com>\r\nCo-authored-by: Shahules786 <Shahules786@gmail.com>","shortMessageHtmlLink":"update documentation for faithfulness with HHEM (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2503914626\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1242\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1242/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1242\">#1242</a>)"}},{"before":"68d52b9133f382630b217300b529e6fd8d21460a","after":"fe379a1c97d18ce2c203d80432a3da6622337968","ref":"refs/heads/main","pushedAt":"2024-09-03T06:08:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat (experimental) added new prompt and metric into `ragas.experimental` (#1240)\n\nyou can use it like\r\n```py\r\nfrom ragas.experimental.metrics import FaithfulnessExperimental\r\nfrom ragas.metrics import faithfulness\r\nfrom ragas import evaluate\r\n\r\n\r\nf = FaithfulnessExperimental(llm=LangchainLLMWrapper(gpt4o))\r\nfaithfulness.llm = LangchainLLMWrapper(gpt4o)\r\n\r\n# row = amnesty_qa[\"eval\"][0]\r\n# await f.ascore(row)\r\n# await faithfulness.ascore(row)\r\n\r\nr = evaluate(\r\n    amnesty_qa[\"eval\"].select(range(10)),\r\n    metrics=[f, faithfulness],\r\n    raise_exceptions=True,\r\n    callbacks=[]\r\n)\r\n```","shortMessageHtmlLink":"feat (experimental) added new prompt and metric into `ragas.experimen…"}},{"before":"fa864a678effff4300323065d019cbd04e70de51","after":"68d52b9133f382630b217300b529e6fd8d21460a","ref":"refs/heads/main","pushedAt":"2024-09-02T17:57:28.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: new Prompt object and faithfulness metric (#1232)","shortMessageHtmlLink":"feat: new Prompt object and faithfulness metric (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2493964142\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1232\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1232/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1232\">#1232</a>)"}},{"before":"d58dc01001013683d04be8af7529264783934c51","after":"fa864a678effff4300323065d019cbd04e70de51","ref":"refs/heads/main","pushedAt":"2024-09-02T17:56:44.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"load HHEM on specified device (#1235)\n\nAllow users to specify the device to load HHEM and add `_create_batch`\r\nto avoid OOM\r\n\r\n---------\r\n\r\nCo-authored-by: Miaoran <miaoran@vectara.com>\r\nCo-authored-by: jjmachan <jamesjithin97@gmail.com>","shortMessageHtmlLink":"load HHEM on specified device (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2497775452\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1235\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1235/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1235\">#1235</a>)"}},{"before":"7f1073febc879066c6bc763d5a83cd2d1991d21d","after":"d58dc01001013683d04be8af7529264783934c51","ref":"refs/heads/main","pushedAt":"2024-08-27T09:39:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"[FIX] - Fix for summarization edge case (#1201)\n\nThis PR adds a fix for the issue mentioned in #1108 \r\n\r\nHowever I have a points to discuss @shahules786 :\r\n- I had added `conciseness_score` to penalize long summaries, but I also\r\ndo not want to promote very very short and skimpy summaries, need to\r\nfind a middle ground.\r\n- Is `averaging` a good way to combine `QA_score` and\r\n`conciseness_score`?\r\n- Ranking based metrics to measure quality of summarization (as\r\nmentioned by shahul in the above issue)\r\n\r\nGiven the conclusions we reach based on these discussion points, I will\r\npush more commits, let's keep this PR open till we resolve these points.\r\n\r\n---------\r\n\r\nCo-authored-by: Shahules786 <Shahules786@gmail.com>","shortMessageHtmlLink":"[FIX] - Fix for summarization edge case (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2468105254\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1201\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1201/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1201\">#1201</a>)"}},{"before":"9dc26e2ecb22fd19de017dc00a210b375a1b6d0f","after":"7f1073febc879066c6bc763d5a83cd2d1991d21d","ref":"refs/heads/main","pushedAt":"2024-08-27T09:31:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Bugfix of evaluate function in llama_index.py (#1223)\n\nDict in dataset of the evaluate function specified. Does not work ->\r\nDataset and then check of \"ground_truth\" in column_names.","shortMessageHtmlLink":"Bugfix of evaluate function in llama_index.py (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2487029743\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1223\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1223/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1223\">#1223</a>)"}},{"before":"a5b601cd21bc5cad9953963918f54ba6e53941b4","after":"9dc26e2ecb22fd19de017dc00a210b375a1b6d0f","ref":"refs/heads/main","pushedAt":"2024-08-27T09:31:25.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"chore: removed extra of requirements (#1225)","shortMessageHtmlLink":"chore: removed extra of requirements (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2488447119\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1225\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1225/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1225\">#1225</a>)"}},{"before":"5bd0408a6a70deb62729f7e262465aa53b79664b","after":"a5b601cd21bc5cad9953963918f54ba6e53941b4","ref":"refs/heads/main","pushedAt":"2024-08-27T09:19:25.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"fix : solves #1215 - Inconsistency in the Value of EVALMODE_TO_COLUMNS[EvaluationMode.ca] (#1216)\n\nSets `EVALMODE_TO_COLUMNS[EvaluationMode.ca] = [\"contexts\", \"answer\"]`\r\nin `src/ragas/validation.py` to fix #1215.\r\n\r\nCo-authored-by: Juan Gonzalez <jucgonzalezes@users.noreply.github.com>","shortMessageHtmlLink":"fix : solves <a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2482841971\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1215\" data-hovercard-type=\"issue\" data-hovercard-url=\"/explodinggradients/ragas/issues/1215/hovercard\" href=\"https://github.com/explodinggradients/ragas/issues/1215\">#1215</a> - Inconsistency in the Value of EVALMODE_TO_COLUMN…"}},{"before":"8da231dc2accaaf769a63e4d5ff42c1208f6630a","after":"5bd0408a6a70deb62729f7e262465aa53b79664b","ref":"refs/heads/main","pushedAt":"2024-08-26T05:52:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"Tiny typo fix: it's -> its (#1219)\n\nA non-urgent, non-critical fix from contraction to possessive.\r\n\r\nElsewhere there is also an \"it's\" in `Question: Where is France and what\r\nis it's capital?`, but since that's a prompt a human might reasonably\r\ntype, and in the answer \"its\" is used correctly, I thought it didn't\r\nneed to be fixed.","shortMessageHtmlLink":"Tiny typo fix: it's -&gt; its (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2484238260\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1219\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1219/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1219\">#1219</a>)"}},{"before":"d5b60bba475fb099b3af2544d80412ffc6e8abc8","after":"8da231dc2accaaf769a63e4d5ff42c1208f6630a","ref":"refs/heads/main","pushedAt":"2024-08-23T13:37:46.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"Implementation of Noise sensitivity metrics from RAGChecker (#1190)\n\nSolves: \r\n- #1185 \r\n\r\n- Took inspiration from RAGChecker from AWS Noise sensitivity [noise\r\nsensitivity](https://github.com/amazon-science/RAGChecker/tree/main/ragchecker)\r\nmetrics.\r\n- Have tested it locally, it is working giving the results.\r\n\r\n### Input\r\n```python\r\nfrom datasets import Dataset \r\nfrom ragas.metrics import noise_sensitivity_relevant, noise_sensitivity_irrelevant\r\nfrom ragas import evaluate\r\ndata_sample = {\r\n    \"question\": [\"What is the Life Insurance Corporation of India (LIC) known for?\"],\r\n    \"ground_truth\": [\"The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.\"],\r\n    \"answer\": [\"The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributs to the financial stability of the country.\"],\r\n    \"contexts\": [[\"The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.\",\r\n        \"LIC is the largest insurance company in India, with a vast network of policyholders and a huge investments.\",\r\n        \"As the largest institutional investor in India, LIC manages a substantial funds, contributing to the financial stability of the country.\",\r\n        \"The Indian economy is one of the fastest-growing major economies in the world, thanks to the secors like finance, technology, manufacturing etc\"]]\r\n}\r\n\r\n\r\ndataset = Dataset.from_dict(data_sample)\r\nmetrics = [noise_sensitivity_relevant, noise_sensitivity_irrelevant]\r\nscore = evaluate(dataset,metrics=metrics)\r\nscore.to_pandas()\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Shahules786 <Shahules786@gmail.com>","shortMessageHtmlLink":"Implementation of Noise sensitivity metrics from RAGChecker (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2461677788\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1190\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1190/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1190\">#1190</a>)"}},{"before":"37e985fe9444ebbe5693dcdbcf26709632e5912b","after":"d5b60bba475fb099b3af2544d80412ffc6e8abc8","ref":"refs/heads/main","pushedAt":"2024-08-15T15:13:32.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"Docs: fix typo in metrics (#1203)","shortMessageHtmlLink":"Docs: fix typo in metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2468265111\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1203\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1203/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1203\">#1203</a>)"}},{"before":"35688f816194b1c69ec20e05f7f37a03506b6aae","after":"37e985fe9444ebbe5693dcdbcf26709632e5912b","ref":"refs/heads/main","pushedAt":"2024-08-15T15:02:40.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"docs: add context utilisation to the index (#1202)","shortMessageHtmlLink":"docs: add context utilisation to the index (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2468249045\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1202\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1202/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1202\">#1202</a>)"}},{"before":"5c1f9a2afbd3681d3d923114d8430419736fcd06","after":"35688f816194b1c69ec20e05f7f37a03506b6aae","ref":"refs/heads/main","pushedAt":"2024-08-15T07:51:16.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"docs: rubrics based metrics (#1200)\n\ndocs for #1189","shortMessageHtmlLink":"docs: rubrics based metrics (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2467574793\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1200\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1200/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1200\">#1200</a>)"}},{"before":"2868a0517726ba67b2d986371f04532dae2b1d58","after":"5c1f9a2afbd3681d3d923114d8430419736fcd06","ref":"refs/heads/main","pushedAt":"2024-08-15T03:36:05.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"metrics: add domain specific rubrics based scoring (#1189)\n\nThis can be regarded as a next step refining on aspect critic metric. Ie\r\nLEVEL 1: user evaluated using just simple criteria\r\n**LEVEL 2: the user specifies criteria associated with each score for\r\nthe entire dataset**\r\nLEVEL 3: user specifies individual criteria associated with each score\r\nfor each sample in the dataset\r\n\r\n```python\r\nfrom ragas import evaluate\r\nfrom datasets import Dataset, DatasetDict\r\n\r\nfrom ragas.metrics import reference_free_rubrics_score, labelled_rubrics_score\r\nrows = {\r\n    \"question\": [\r\n        \"What's the longest river in the world?\",\r\n        \"What does the Democratic Republic of Congo flag represent?\"\r\n    ],\r\n    \"ground_truth\": [\r\n        \"The Nile is a major north-flowing river in northeastern Africa. It flows into the Mediterranean Sea. The Nile is the longest river in Africa and has historically been considered the longest river in the world, though this has been contested by research suggesting that the Amazon River is slightly longer. Of the world's major rivers, the Nile is one of the smallest, as measured by annual flow in cubic metres of water. About 6,650 km (4,130 mi) long, its drainage basin covers eleven countries: the Democratic Republic of the Congo, Tanzania, Burundi, Rwanda, Uganda, Kenya, Ethiopia, Eritrea, South Sudan, Sudan, and Egypt.\",\r\n        \"The national flag of the Democratic Republic of the Congo represents blue for peace, red for 'the blood of the country's martyrs', yellow for the country's wealth, and a star for a radiant future for the country.\"\r\n    ],\r\n    \"answer\": [\r\n        \"The longest river in the world is the Nile, stretching approximately 6,650 kilometers (4,130 miles) through northeastern Africa, flowing through countries such as Uganda, Sudan, and Egypt before emptying into the Mediterranean Sea. There is some debate about this title, as recent studies suggest the Amazon River could be longer if its longest tributaries are included, potentially extending its length to about 7,000 kilometers (4,350 miles).\",\r\n        \"The flag of the Democratic Republic of the Congo (DRC) features a sky blue field with a red diagonal stripe bordered by narrow yellow edges, and a yellow five-pointed star in the upper left corner. Each element on the flag carries specific symbolism: the blue represents peace, the red symbolizes the blood of the country's martyrs, the yellow denotes the nation's wealth, and the star stands for hope for a better future.\"\r\n    ],\r\n    \"contexts\": [\r\n        [\r\n            \"Scientists debate whether the Amazon or the Nile is the longest river in the world. Traditionally, the Nile is considered longer, but recent information suggests that the Amazon may be longer.\",\r\n            \"The Nile River was central to the Ancient Egyptians' rise to wealth and power. Since rainfall is almost non-existent in Egypt, the Nile River and its yearly floodwaters offered the people a fertile oasis for rich agriculture.\",\r\n            \"The world's longest rivers are defined as the longest natural streams whose water flows within a channel, or streambed, with defined banks.\",\r\n            \"The Amazon River could be considered longer if its longest tributaries are included, potentially extending its length to about 7,000 kilometers.\"\r\n        ],\r\n        [\r\n            \"The flag of the second Republic of Mobutu Sese Seko became the official banner after Mobutu established his dictatorship. This flag was used from 1966 to 1971 and consisted of the same yellow star, now made smaller, situated in the top corner of the hoist side, with a red, yellow-lined band running diagonally across the center. The red symbolized the people's blood; the yellow symbolized prosperity; the blue symbolized hope; and the star represented unity.\",\r\n            \"The current flag of the Democratic Republic of Congo, which has been adopted after the approval of a new constitution in 2006, is composed of a blue sheet, red diagonal stripe and a yellow five-pointed star at the top of the left part of the flag. Blue symbolizes peace, red stands for blood of martyrs, yellow color that frames the red stripe denotes prosperity and the star represents hope for a brighter future of the country.\",\r\n            \"The blue color in the flag symbolizes peace, the red should remind of the country’s martyrs, the yellow is for the country’s riches and the star represents the future.\"\r\n        ]\r\n    ]\r\n}\r\n\r\n\r\n\r\ndataset = Dataset.from_dict(rows)\r\n\r\nresult = evaluate(\r\n    dataset,\r\n    metrics=[\r\n        reference_free_rubrics_score,\r\n        labelled_rubrics_score\r\n    ],\r\n)\r\n\r\n\r\n\r\n```\r\n\r\n---------\r\n\r\nCo-authored-by: Shahules786 <Shahules786@gmail.com>","shortMessageHtmlLink":"metrics: add domain specific rubrics based scoring (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2461617735\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1189\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1189/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1189\">#1189</a>)"}},{"before":"b988f538b968b378604240d2b279f8bf4d8ecca9","after":"2868a0517726ba67b2d986371f04532dae2b1d58","ref":"refs/heads/main","pushedAt":"2024-08-14T14:37:32.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"shahules786","name":"ikka","path":"/shahules786","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/25312635?s=80&v=4"},"commit":{"message":"fix: typo in model name (#1198)","shortMessageHtmlLink":"fix: typo in model name (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2466031566\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1198\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1198/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1198\">#1198</a>)"}},{"before":"a77c4759747fa8c3782242c8fe3d7a8f74a826d6","after":"b988f538b968b378604240d2b279f8bf4d8ecca9","ref":"refs/heads/main","pushedAt":"2024-08-14T11:17:31.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"fix: added a test for prompts (#1197)","shortMessageHtmlLink":"fix: added a test for prompts (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2465539890\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1197\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1197/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1197\">#1197</a>)"}},{"before":"17718f2c91d13a674fd4c5b93ffb0501ad411db6","after":"a77c4759747fa8c3782242c8fe3d7a8f74a826d6","ref":"refs/heads/main","pushedAt":"2024-08-14T06:14:10.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jjmachan","name":"Jithin James","path":"/jjmachan","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/5261489?s=80&v=4"},"commit":{"message":"feat: deeper helicon integration (#1196)\n\nbased on #1182\r\n\r\n---------\r\n\r\nCo-authored-by: Justin <justintorre75@gmail.com>","shortMessageHtmlLink":"feat: deeper helicon integration (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2464965549\" data-permission-text=\"Title is private\" data-url=\"https://github.com/explodinggradients/ragas/issues/1196\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/explodinggradients/ragas/pull/1196/hovercard\" href=\"https://github.com/explodinggradients/ragas/pull/1196\">#1196</a>)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEszVh_QA","startCursor":null,"endCursor":null}},"title":"Activity · explodinggradients/ragas"}