Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: fix typo in metrics #1203

Merged
merged 2 commits into from
Aug 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/concepts/metrics/context_utilization.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Context utilization is like a reference free version of [context_precision](cont


```{math}
\text{Context Precision@K} = \frac{\sum_{k=1}^{K} \left( \text{Precision@k} \times v_k \right)}{\text{Total number of relevant items in the top } K \text{ results}}
\text{Context Utilization@K} = \frac{\sum_{k=1}^{K} \left( \text{Precision@k} \times v_k \right)}{\text{Total number of relevant items in the top } K \text{ results}}
````

```{math}
Expand All @@ -18,9 +18,9 @@ Where $K$ is the total number of chunks in `contexts` and $v_k \in \{0, 1\}$ is
Question: Where is France and what is it's capital?
Answer: France is in Western Europe and its capital is Paris.

High context precision: ["France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower", "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history."]
High context utilization: ["France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower", "The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and the vast Palace of Versailles attest to its rich history."]

Low context precision: ["The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and", "France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower",]
Low context utilization: ["The country is also renowned for its wines and sophisticated cuisine. Lascaux’s ancient cave drawings, Lyon’s Roman theater and", "France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower",]
```

## Example
Expand All @@ -44,7 +44,7 @@ score.to_pandas()

## Calculation

Let's examine how context utilization was calculated using the low context precision example:
Let's examine how context utilization was calculated using the low context utilization example:

**Step 1**: For each chunk in retrieved context, check if it is relevant or not relevant to arrive at the answer for the given question.

Expand All @@ -58,8 +58,8 @@ Let's examine how context utilization was calculated using the low context preci
\text{Precision@2} = {\text{1} \over \text{2}} = 0.5
````

**Step 3**: Calculate the mean of precision@k to arrive at the final context precision score.
**Step 3**: Calculate the mean of precision@k to arrive at the final context utilization score.

```{math}
\text{Context Precision} = {\text{(0+0.5)} \over \text{1}} = 0.5
\text{Context Utilization} = {\text{(0+0.5)} \over \text{1}} = 0.5
```
1 change: 1 addition & 0 deletions docs/concepts/metrics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ Evaluating the end-to-end performance of a pipeline is also crucial, as it direc
faithfulness
answer_relevance
context_precision
context_utilization
context_recall
context_entities_recall
semantic_similarity
Expand Down
Loading