Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable analyzing evaluators/annotators on data without multiple generator models #293

Merged
merged 6 commits into from
Apr 28, 2024
Merged

Enable analyzing evaluators/annotators on data without multiple generator models #293

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7e38694
update if statement to only apply if no generator column
rdnfn Apr 17, 2024
4f5c2a6
Merge branch 'tatsu-lab:main' into main
rdnfn Apr 19, 2024
ecd86d5
Merge branch 'tatsu-lab:main' into main
rdnfn Apr 27, 2024
0402542
add try/except statement to correlation computation
rdnfn Apr 27, 2024
5243254
fix spelling
rdnfn Apr 27, 2024
9eb4277
add more informative logging message if correlation computation fails
rdnfn Apr 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 14 additions & 3 deletions src/alpaca_eval/analyze.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ def estimate_correlations(
corresponding gold annotations.

groupby: list[str], optional
Columns to groupby for computing the ldeaderboard.
Columns to groupby for computing the leaderboard.

Returns
-------
Expand Down Expand Up @@ -307,8 +307,19 @@ def estimate_correlations(
left_index=True,
right_index=True,
)
s = spearmanr(df["win_rate_2"], df["win_rate_1"]).statistic
r = pearsonr(df["win_rate_2"], df["win_rate_1"]).statistic
try:
s = spearmanr(df["win_rate_2"], df["win_rate_1"]).statistic
r = pearsonr(df["win_rate_2"], df["win_rate_1"]).statistic
except ValueError:
logging.warning(
(
"Could not compute correlations. This issue may be due to a lack of different "
f"values of the column data is grouped by (using {groupby} column of dataset). "
f"The computation failed for the following dataframe:\n{df}"
),
)
s = np.nan
r = np.nan

return dict(spearman=s, pearson=r)

Expand Down
Loading