Skip to content

Commit d2734ac

Browse files
authored
docs: update cleanlab integration description (#1080)
* Update cleanlab integration description Signed-off-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com> * address feedback Signed-off-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com> --------- Signed-off-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
1 parent 1f590fc commit d2734ac

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

docs/user-guides/community/cleanlab.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
# Cleanlab Integration
22

3-
The `cleanlab trustworthiness` flow uses trustworthiness score with a default threshold of 0.6 to determine if the output should be allowed or not (i.e., if the trustworthiness score is below the threshold, the response is considered "untrustworthy").
3+
Cleanlab's state-of-the-art [LLM uncertainty estimator](https://cleanlab.ai/blog/trustworthy-language-model/) scores the _trustworthiness_ of any LLM response, to detect incorrect outputs and hallucinations in real-time.
44

5-
A high trustworthiness score generally correlates with high-quality responses. In a question-answering application, high trustworthiness is indicative of correct responses, while in general open-ended applications, a high score corresponds to the response being helpful and informative. Trustworthiness scores are less useful for creative or open-ended requests.
5+
In question-answering or RAG applications: high trustworthiness is indicative of a correct response. In open-ended chat applications, a high score corresponds to the response being helpful and informative. Low trustworthiness scores indicate outputs that are likely bad or incorrect, or complex prompts where the LLM might have output the right response this time but might output the wrong response when run on the same prompt again (so it cannot be trusted).
66

7-
The mathematical derivation of the score is explained in [Cleanlab's documentation](https://help.cleanlab.ai/tutorials/tlm/#how-does-the-tlm-trustworthiness-score-work), and you can also access [trustworthiness score benchmarks](https://cleanlab.ai/blog/trustworthy-language-model/).
7+
The trustworthiness score is further explained and comprehensively benchmarked in [Cleanlab's documentation](https://help.cleanlab.ai/tlm/).
88

9-
You can easily change the cutoff value for the trustworthiness score by adjusting the threshold in the [config](https://github.com/NVIDIA/NeMo-Guardrails/tree/develop/nemoguardrails/library/cleanlab/flows.co). For example, to change the threshold to 0.7, you can add the following flow to your config:
9+
The `cleanlab trustworthiness` guardrail flow uses a default trustworthiness score threshold of 0.6 to determine if your LLM output should be allowed or not. When the trustworthiness score falls below the threshold, the corresponding LLM response is flagged as _unstrustworthy_. You can easily change the cutoff value for the trustworthiness score by adjusting the threshold in the [config](https://github.com/NVIDIA/NeMo-Guardrails/tree/develop/nemoguardrails/library/cleanlab/flows.co). For example, to change the threshold to 0.7, add the following flow to your config:
1010

1111
```colang
1212
define subflow cleanlab trustworthiness
@@ -23,12 +23,12 @@ define bot response untrustworthy
2323

2424
## Setup
2525

26-
Install `cleanlab-studio` to use Cleanlab's trustworthiness score:
26+
Install the Python client to use Cleanlab's trustworthiness score:
2727

2828
```
2929
pip install cleanlab-studio
3030
```
3131

32-
Then, you can get an API key for free by [creating a Cleanlab account](https://app.cleanlab.ai/?signup_origin=TLM) or experiment with TLM in the [playground](https://tlm.cleanlab.ai/). You can also [email Cleanlab](mailto:sales@cleanlab.ai) for any special requests or support.
32+
You can get an API key for free by [creating a Cleanlab account](https://tlm.cleanlab.ai/) or experiment with the trustworthiness scores in the [playground](https://chat.cleanlab.ai/chat). Feel free to [email Cleanlab](mailto:suport@cleanlab.ai) with any questions.
3333

3434
Lastly, set the `CLEANLAB_API_KEY` environment variable with the API key.

0 commit comments

Comments
 (0)