You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The expected input format is a list of dictionaries, where each dictionary has the following keys:
145
+
- name: Name of the evaluation criteria.
146
+
- criteria: Instructions specifying the evaluation criteria.
147
+
148
+
max_tokens (int, default = 512): the maximum number of tokens that can be generated in the response from `TLM.prompt()` as well as during internal trustworthiness scoring.
149
+
If you experience token/rate-limit errors, try lowering this number.
142
150
For OpenAI models, this parameter must be between 64 and 4096. For Claude models, this parameter must be between 64 and 512.
143
151
144
-
num_candidate_responses (int, default = 1): how many alternative candidate responses are internally generated in `TLM.prompt()`.
145
-
`TLM.prompt()` scores the trustworthiness of each candidate response, and then returns the most trustworthy one.
146
-
This parameter must be between 1 and 20. It has no effect on `TLM.score()`.
147
-
Higher values here can produce more accurate responses from `TLM.prompt()`, but at higher runtimes/costs.
148
-
When it is 1, `TLM.prompt()` simply returns a standard LLM response and does not attempt to auto-improve it.
152
+
reasoning_effort ({"none", "low", "medium", "high"}, default = "high"): how much internal LLM calls are allowed to reason (number of thinking tokens)
153
+
when generating alternative possible responses and reflecting on responses during trustworthiness scoring.
154
+
Reduce this value to reduce runtimes. Higher values may improve trust scoring.
155
+
156
+
num_self_reflections (int, default = 3): the number of different evaluations to perform where the LLM reflects on the response, a factor affecting trust scoring.
157
+
The maximum number currently supported is 3. Lower values can reduce runtimes.
158
+
Reflection helps quantify aleatoric uncertainty associated with challenging prompts and catches responses that are noticeably incorrect/bad upon further analysis.
159
+
This parameter has no effect when `disable_trustworthiness` is True.
149
160
150
-
num_consistency_samples (int, default = 8): the amount of internal sampling to measure LLM response consistency, a factor affecting trustworthiness scoring.
151
-
Must be between 0 and 20. Higher values produce more reliable TLM trustworthiness scores, but at higher runtimes/costs.
161
+
num_consistency_samples (int, default = 8): the amount of internal sampling to measure LLM response consistency, a factor affecting trust scoring.
162
+
Must be between 0 and 20. Lower values can reduce runtimes.
152
163
Measuring consistency helps quantify the epistemic uncertainty associated with
153
164
strange prompts or prompts that are too vague/open-ended to receive a clearly defined 'good' response.
154
165
TLM measures consistency via the degree of contradiction between sampled responses that the model considers plausible.
155
-
156
-
num_self_reflections(int, default = 3): the number of self-reflections to perform where the LLM is asked to reflect on the given response and directly evaluate correctness/confidence.
157
-
The maximum number of self-reflections currently supported is 3. Lower values will reduce runtimes/costs, but potentially also the reliability of trustworthiness scores.
158
-
Reflection helps quantify aleatoric uncertainty associated with challenging prompts and catches responses that are noticeably incorrect/bad upon further analysis.
166
+
This parameter has no effect when `disable_trustworthiness` is True.
159
167
160
168
similarity_measure ({"semantic", "string", "embedding", "embedding_large", "code", "discrepancy"}, default = "discrepancy"): how the
161
169
trustworthiness scoring's consistency algorithm measures similarity between alternative responses considered plausible by the model.
162
170
Supported similarity measures include - "semantic" (based on natural language inference),
163
171
"embedding" (based on vector embedding similarity), "embedding_large" (based on a larger embedding model),
164
172
"code" (based on model-based analysis designed to compare code), "discrepancy" (based on model-based analysis of possible discrepancies),
165
-
and "string" (based on character/word overlap). Set this to "string" for minimal runtimes/costs.
166
-
167
-
reasoning_effort ({"none", "low", "medium", "high"}, default = "high"): how much internal LLM calls are allowed to reason (number of thinking tokens)
168
-
when generating alternative possible responses and reflecting on responses during trustworthiness scoring.
169
-
Higher reasoning efforts may yield more reliable TLM trustworthiness scores. Reduce this value to reduce runtimes/costs.
170
-
171
-
log (list[str], default = []): optionally specify additional logs or metadata that TLM should return.
172
-
For instance, include "explanation" here to get explanations of why a response is scored with low trustworthiness.
173
+
and "string" (based on character/word overlap). Set this to "string" for minimal runtimes.
174
+
This parameter has no effect when `num_consistency_samples = 0`.
The expected input format is a list of dictionaries, where each dictionary has the following keys:
176
-
- name: Name of the evaluation criteria.
177
-
- criteria: Instructions specifying the evaluation criteria.
176
+
num_candidate_responses (int, default = 1): how many alternative candidate responses are internally generated in `TLM.prompt()`.
177
+
`TLM.prompt()` scores the trustworthiness of each candidate response, and then returns the most trustworthy one.
178
+
You can auto-improve responses by increasing this parameter, but at higher runtimes/costs.
179
+
This parameter must be between 1 and 20. It has no effect on `TLM.score()`.
180
+
When this parameter is 1, `TLM.prompt()` simply returns a standard LLM response and does not attempt to auto-improve it.
181
+
This parameter has no effect when `disable_trustworthiness` is True.
178
182
179
-
use_self_reflection (bool, default = `True`): deprecated. Use `num_self_reflections` instead.
183
+
disable_trustworthiness (bool, default = False): if True, trustworthiness scoring is disabled and TLM will not compute trust scores for responses.
184
+
This is useful when you only want to use custom evaluation criteria or when you want to minimize computational overhead and only need the base LLM response.
185
+
The following parameters will be ignored when `disable_trustworthiness` is True: `num_consistency_samples`, `num_self_reflections`, `num_candidate_responses`, `reasoning_effort`, `similarity_measure`.
180
186
"""
181
187
182
188
prompt: Optional[str]
@@ -647,6 +653,8 @@ class MessageChatCompletionDeveloperMessageParam(TypedDict, total=False):
The expected input format is a list of dictionaries, where each dictionary has the following keys:
67
+
- name: Name of the evaluation criteria.
68
+
- criteria: Instructions specifying the evaluation criteria.
69
+
70
+
max_tokens (int, default = 512): the maximum number of tokens that can be generated in the response from `TLM.prompt()` as well as during internal trustworthiness scoring.
71
+
If you experience token/rate-limit errors, try lowering this number.
64
72
For OpenAI models, this parameter must be between 64 and 4096. For Claude models, this parameter must be between 64 and 512.
65
73
66
-
num_candidate_responses (int, default = 1): how many alternative candidate responses are internally generated in `TLM.prompt()`.
67
-
`TLM.prompt()` scores the trustworthiness of each candidate response, and then returns the most trustworthy one.
68
-
This parameter must be between 1 and 20. It has no effect on `TLM.score()`.
69
-
Higher values here can produce more accurate responses from `TLM.prompt()`, but at higher runtimes/costs.
70
-
When it is 1, `TLM.prompt()` simply returns a standard LLM response and does not attempt to auto-improve it.
74
+
reasoning_effort ({"none", "low", "medium", "high"}, default = "high"): how much internal LLM calls are allowed to reason (number of thinking tokens)
75
+
when generating alternative possible responses and reflecting on responses during trustworthiness scoring.
76
+
Reduce this value to reduce runtimes. Higher values may improve trust scoring.
77
+
78
+
num_self_reflections (int, default = 3): the number of different evaluations to perform where the LLM reflects on the response, a factor affecting trust scoring.
79
+
The maximum number currently supported is 3. Lower values can reduce runtimes.
80
+
Reflection helps quantify aleatoric uncertainty associated with challenging prompts and catches responses that are noticeably incorrect/bad upon further analysis.
81
+
This parameter has no effect when `disable_trustworthiness` is True.
71
82
72
-
num_consistency_samples (int, default = 8): the amount of internal sampling to measure LLM response consistency, a factor affecting trustworthiness scoring.
73
-
Must be between 0 and 20. Higher values produce more reliable TLM trustworthiness scores, but at higher runtimes/costs.
83
+
num_consistency_samples (int, default = 8): the amount of internal sampling to measure LLM response consistency, a factor affecting trust scoring.
84
+
Must be between 0 and 20. Lower values can reduce runtimes.
74
85
Measuring consistency helps quantify the epistemic uncertainty associated with
75
86
strange prompts or prompts that are too vague/open-ended to receive a clearly defined 'good' response.
76
87
TLM measures consistency via the degree of contradiction between sampled responses that the model considers plausible.
77
-
78
-
num_self_reflections(int, default = 3): the number of self-reflections to perform where the LLM is asked to reflect on the given response and directly evaluate correctness/confidence.
79
-
The maximum number of self-reflections currently supported is 3. Lower values will reduce runtimes/costs, but potentially also the reliability of trustworthiness scores.
80
-
Reflection helps quantify aleatoric uncertainty associated with challenging prompts and catches responses that are noticeably incorrect/bad upon further analysis.
88
+
This parameter has no effect when `disable_trustworthiness` is True.
81
89
82
90
similarity_measure ({"semantic", "string", "embedding", "embedding_large", "code", "discrepancy"}, default = "discrepancy"): how the
83
91
trustworthiness scoring's consistency algorithm measures similarity between alternative responses considered plausible by the model.
84
92
Supported similarity measures include - "semantic" (based on natural language inference),
85
93
"embedding" (based on vector embedding similarity), "embedding_large" (based on a larger embedding model),
86
94
"code" (based on model-based analysis designed to compare code), "discrepancy" (based on model-based analysis of possible discrepancies),
87
-
and "string" (based on character/word overlap). Set this to "string" for minimal runtimes/costs.
88
-
89
-
reasoning_effort ({"none", "low", "medium", "high"}, default = "high"): how much internal LLM calls are allowed to reason (number of thinking tokens)
90
-
when generating alternative possible responses and reflecting on responses during trustworthiness scoring.
91
-
Higher reasoning efforts may yield more reliable TLM trustworthiness scores. Reduce this value to reduce runtimes/costs.
92
-
93
-
log (list[str], default = []): optionally specify additional logs or metadata that TLM should return.
94
-
For instance, include "explanation" here to get explanations of why a response is scored with low trustworthiness.
95
+
and "string" (based on character/word overlap). Set this to "string" for minimal runtimes.
96
+
This parameter has no effect when `num_consistency_samples = 0`.
The expected input format is a list of dictionaries, where each dictionary has the following keys:
98
-
- name: Name of the evaluation criteria.
99
-
- criteria: Instructions specifying the evaluation criteria.
98
+
num_candidate_responses (int, default = 1): how many alternative candidate responses are internally generated in `TLM.prompt()`.
99
+
`TLM.prompt()` scores the trustworthiness of each candidate response, and then returns the most trustworthy one.
100
+
You can auto-improve responses by increasing this parameter, but at higher runtimes/costs.
101
+
This parameter must be between 1 and 20. It has no effect on `TLM.score()`.
102
+
When this parameter is 1, `TLM.prompt()` simply returns a standard LLM response and does not attempt to auto-improve it.
103
+
This parameter has no effect when `disable_trustworthiness` is True.
100
104
101
-
use_self_reflection (bool, default = `True`): deprecated. Use `num_self_reflections` instead.
105
+
disable_trustworthiness (bool, default = False): if True, trustworthiness scoring is disabled and TLM will not compute trust scores for responses.
106
+
This is useful when you only want to use custom evaluation criteria or when you want to minimize computational overhead and only need the base LLM response.
107
+
The following parameters will be ignored when `disable_trustworthiness` is True: `num_consistency_samples`, `num_self_reflections`, `num_candidate_responses`, `reasoning_effort`, `similarity_measure`.
![stainless-app[bot]](https://avatars.githubusercontent.com/in/378072?v=4&size=40){kind=avatar}
0 commit comments