Add customizable evaluation dimensions #256

bugsz · 2024-11-28T20:58:00Z

Provide a way for people to customize the evaluation dimensions they want.
Currently use CustomEvaluationDimension to specify a dimension, and CustomEvaluationDimensionList to group.
To create the dimension, one can directly use a dictionary, or compose the existing metrics by specifying their names.

TODOs:

Move the file to evaluator.py
Figure out the best practice to store and use these dimensions (is using the name to specify the best way?)

Closes #

📑 Description

✅ Checks

My pull request adheres to the code style of this project
My code requires changes to the documentation
I have updated the documentation as required
All the tests have passed
Branch name follows type/descript (e.g. feature/add-llm-agents)
Ready for code review

ℹ Additional Information

codecov · 2024-11-28T20:59:24Z

Codecov Report

Attention: Patch coverage is 38.09524% with 39 lines in your changes missing coverage. Please review.

Project coverage is 35.94%. Comparing base (dbd8294) to head (6155292).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
sotopia/database/evaluation_dimensions.py	37.09%	39 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (dbd8294) and HEAD (6155292). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (dbd8294) HEAD (6155292)

3 2

@@             Coverage Diff             @@
##             main     #256       +/-   ##
===========================================
- Coverage   74.47%   35.94%   -38.54%     
===========================================
  Files          61       50       -11     
  Lines        3162     2557      -605     
===========================================
- Hits         2355      919     -1436     
- Misses        807     1638      +831

Files with missing lines	Coverage Δ
sotopia/database/__init__.py	`95.45% <100.00%> (-4.55%)`	⬇️
sotopia/database/evaluation_dimensions.py	`37.09% <37.09%> (ø)`

... and 32 files with indirect coverage changes

XuhuiZhou

Great job, @bugsz can you also add relevant docs?

examples/experiment_eval.py

XuhuiZhou · 2024-11-30T22:22:53Z

sotopia/database/evaluation_dimensions.py

+        return validator
+
+    @staticmethod
+    def generate_dimension_model(dimension_ids: list[str]) -> Type[BaseModel]:


what is this function used for?

create an validator for the evaluation metric?

why name it as generate? then

Also consider add a docstring explanation here?

XuhuiZhou · 2024-11-30T22:24:49Z

sotopia/database/evaluation_dimensions.py

+        )
+
+    @staticmethod
+    def generate_dimension_model_from_name(


rename this? Also can we get rid of the printing here? And add the existing names to the doc?

sotopia/database/evaluation_dimensions.py

XuhuiZhou · 2024-11-30T22:31:23Z

sotopia/database/evaluation_dimensions.py

+    range_low: int = Field(index=True)
+
+
+class CustomEvaluationDimensionList(JsonModel):


can we use this as a model to save a set of dimensions? like same name sotopia, then it automatically retrieve all the sotopia original dimensions and would be ready to use

Yes that is what I am thinking. Do we want to allow different evaluation metrics to have the same name?

What do you mean? E.g., ?

For example, there is an original sotopia dimension and a refined one, say
[old] goal: provide a goal score of 1-10 where higher score indicates higher completion
[new] goal: provide a goal score of 1-10, where 1-3: xxx, 4-6: yyy

I think this is something we do not want to see, but sometimes we might need to have these two at the same time?

ProKil · 2024-12-04T02:09:47Z

@bugsz Could you fix the mypy tests first?

bugsz · 2024-12-04T05:21:23Z

@bugsz Could you fix the mypy tests first?

Fixed. Will update the doc

XuhuiZhou

Add pytests and be sure to only merge to /demo instead of /main
Make sure to update the API doc?
This change would break a lot of places!! please update them accordingly

(basically check anywhere mentioning ReachGoalLLMEvaluator?)

XuhuiZhou · 2024-12-03T22:59:12Z

sotopia/database/evaluation_dimensions.py

+    range_low: int = Field(index=True)
+
+
+class CustomEvaluationDimensionList(JsonModel):


What do you mean? E.g., ?

XuhuiZhou · 2024-12-03T23:00:05Z

sotopia/database/evaluation_dimensions.py

+        return validator
+
+    @staticmethod
+    def generate_dimension_model(dimension_ids: list[str]) -> Type[BaseModel]:


why name it as generate? then

Also consider add a docstring explanation here?

XuhuiZhou · 2024-12-06T01:31:34Z

docs/pages/concepts/evaluation.mdx

remove this then?

XuhuiZhou · 2024-12-06T01:35:08Z

examples/experiment_eval.py

@@ -108,6 +108,20 @@ def _iterate_env_agent_combo_not_in_db(
    env_ids: list[str] = [],
    tag: str | None = None,
 ) -> Generator[EnvAgentCombo[Observation, AgentAction], None, None]:
+    # method 1 for loading evaluation metric
+    evaluation_dimensions = (


It is not a good thing to write things like this? (remove method 1?) move it as an example to the doc?

XuhuiZhou · 2024-12-07T22:22:48Z

docs/pages/concepts/evaluation_dimension.md

@@ -0,0 +1,92 @@
+## Overview
+
+Evaluation dimensions are used to evaluate the quality of social interactions.


make sure mention they can use SotopiaDimension here as well, and people don't need to initialize the database when using that

XuhuiZhou · 2024-12-07T22:23:39Z

docs/pages/concepts/evaluation.mdx

Either move the content from docs/pages/concepts/evaluation_dimension.md to here or either remove this file?

And don't forget to update the API doc? you can use chatgpt to do that for ya, but it's good that we have it

XuhuiZhou · 2024-12-07T22:52:27Z

tests/database/test_database.py

+    )
+    custom_dimension.save()
+    pk = custom_dimension.pk
+    dimension = CustomEvaluationDimension(uuid_str=pk)


CustomEvaluationDimension(uuid_str=pk) this is not how you fetch data from the Redis database

…(pk) (#262) Co-authored-by: openhands <openhands@all-hands.dev>

* Fix test_create_custom_dimension to use CustomEvaluationDimension.get(pk) * Update documentation for SotopiaDimension and EvaluationDimensionBuilder * [autofix.ci] apply automated fixes * Add API documentation for evaluation dimensions * Refine API documentation for evaluation_dimensions.py to match style * [autofix.ci] apply automated fixes --------- Co-authored-by: openhands <openhands@all-hands.dev> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>

add customizable evaluation dimensions

4e78730

XuhuiZhou requested changes Nov 30, 2024

View reviewed changes

XuhuiZhou reviewed Nov 30, 2024

View reviewed changes

add docs

974cd88

fix mypy error & refactor examples

4355c79

add docs for evaluation dimensions

6e5754b

XuhuiZhou requested changes Dec 6, 2024

View reviewed changes

bugsz added 3 commits December 6, 2024 21:36

update docs and examples

0368a0c

add test cases and fix mypy issue

fd1e984

fix mypy issue

6155292

XuhuiZhou changed the base branch from main to demo December 7, 2024 22:14

XuhuiZhou requested changes Dec 7, 2024

View reviewed changes

XuhuiZhou and others added 3 commits December 8, 2024 15:02

Fix test_create_custom_dimension to use CustomEvaluationDimension.get…

e9f1cd9

…(pk) (#262) Co-authored-by: openhands <openhands@all-hands.dev>

add doc

24c1fb1

XuhuiZhou changed the base branch from demo to main December 8, 2024 20:43

XuhuiZhou changed the base branch from main to demo December 8, 2024 20:49

XuhuiZhou approved these changes Dec 8, 2024

View reviewed changes

XuhuiZhou merged commit 5a9f4b7 into demo Dec 8, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add customizable evaluation dimensions #256

Add customizable evaluation dimensions #256

bugsz commented Nov 28, 2024

codecov bot commented Nov 28, 2024 •

edited

Loading

XuhuiZhou left a comment

XuhuiZhou Nov 30, 2024

bugsz Dec 3, 2024

XuhuiZhou Dec 3, 2024

XuhuiZhou Nov 30, 2024

XuhuiZhou Nov 30, 2024

bugsz Dec 3, 2024

XuhuiZhou Dec 3, 2024

bugsz Dec 6, 2024

ProKil commented Dec 4, 2024

bugsz commented Dec 4, 2024

XuhuiZhou left a comment

XuhuiZhou Dec 3, 2024

XuhuiZhou Dec 3, 2024

XuhuiZhou Dec 6, 2024

XuhuiZhou Dec 6, 2024

XuhuiZhou Dec 7, 2024

XuhuiZhou Dec 7, 2024

XuhuiZhou Dec 7, 2024

XuhuiZhou Dec 7, 2024

		range_low: int = Field(index=True)


		class CustomEvaluationDimensionList(JsonModel):

		@@ -0,0 +1,92 @@
		## Overview

		Evaluation dimensions are used to evaluate the quality of social interactions.

Add customizable evaluation dimensions #256

Add customizable evaluation dimensions #256

Conversation

bugsz commented Nov 28, 2024

📑 Description

✅ Checks

ℹ Additional Information

codecov bot commented Nov 28, 2024 • edited Loading

Codecov Report

XuhuiZhou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ProKil commented Dec 4, 2024

bugsz commented Dec 4, 2024

XuhuiZhou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 28, 2024 •

edited

Loading