Allow custom evaluation data in Playground

README.md

@@ -11,3 +11,5 @@
 
 ## What is Athena?
 Athena is an advanced system designed to assist educators by providing (semi-)automated assessments for various types of academic exercises. Through its integration with learning management systems (LMS), Athena offers an efficient and innovative way to evaluate students' work in large courses. The system has been expanded from its original focus on textual exercises to now include support for programming exercises and has plans for future support of additional exercise types such as UML modeling and mathematics.
+
+**Documentation:** [ls1intum.github.io/Athena/](ls1intum.github.io/Athena/)

assessment_module_manager/modules.docker.ini

@@ -45,4 +45,5 @@ url = http://module-modeling-llm:5008
 type = modeling
 supports_evaluation = false
 supports_non_graded_feedback_requests = false
-supports_graded_feedback_requests = true
+supports_graded_feedback_requests = true
+

assessment_module_manager/modules.ini

@@ -45,4 +45,5 @@ url = http://localhost:5008
 type = modeling
 supports_evaluation = false
 supports_non_graded_feedback_requests = false
-supports_graded_feedback_requests = true
+supports_graded_feedback_requests = true
+

docs/Makefile

@@ -3,6 +3,24 @@
 
 # You can set these variables from the command line, and also
 # from the environment for the first two.
+SPHINXAUTOBUILD = sphinx-autobuild
+ALLSPHINXLIVEOPTS = $(ALLSPHINXOPTS) -q \
+   --port 0 \
+   --host 0.0.0.0 \
+   --open-browser \
+   --delay 1 \
+   --ignore "*.swp" \
+   --ignore "*.pdf" \
+   --ignore "*.log" \
+   --ignore "*.out" \
+   --ignore "*.toc" \
+   --ignore "*.aux" \
+   --ignore "*.idx" \
+   --ignore "*.ind" \
+   --ignore "*.ilg" \
+   --ignore "*.tex" \
+   --watch source
+
 SPHINXOPTS    ?=
 SPHINXBUILD   ?= sphinx-build
 SOURCEDIR     = .
@@ -13,11 +31,14 @@ help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
 livehtml:
-	sphinx-autobuild "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	$(SPHINXAUTOBUILD) -b html $(ALLSPHINXLIVEOPTS) $(SOURCEDIR) $(BUILDDIR)
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)."
 
 .PHONY: help Makefile
+# .PHONY: livehtml
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/index.rst

@@ -23,6 +23,13 @@ Athena will use the information it is given and provide the automatic suggestion
     overview/athena
     overview/playground
 
+.. toctree::
+    :caption: User Guide
+    :includehidden:
+    :maxdepth: 2
+
+    user_guide/index
+
 .. toctree::
     :caption: Setup
     :includehidden:

docs/requirements.txt

@@ -1,5 +1,5 @@
 Sphinx==6.2.1
-sphinx-rtd-theme==1.2.0
+sphinx-rtd-theme==2.0.0
 sphinx-autobuild==2021.3.14
 docutils==0.19
 sphinxcontrib-bibtex==2.5.0

docs/run/docker.rst

@@ -1,3 +1,5 @@
+.. _run_docker:
+
 From Docker
 ===========================================
 

docs/run/local.rst

@@ -1,3 +1,5 @@
+.. _run_local:
+
 From the Command Line
 ===========================================
 

docs/run/playground.rst

@@ -1,3 +1,5 @@
+.. _run_playground:
+
 Run the Playground
 ===========================================
 

docs/setup/install.rst

@@ -1,3 +1,5 @@
+.. _setup_install:
+
 Python and Poetry Setup
 ===========================================
 

docs/user_guide/conduct_experiment.rst

@@ -0,0 +1,52 @@
+.. _conduct_experiment_guide:
+
+=============================
+Conducting an Experiment
+=============================
+
+To conduct an experiment in the Athena Playground, follow these steps:
+
+1. **Define Experiment:**
+    - Scroll to the Evaluation Mode section.
+    - In "Define Experiments", choose execution modes, exercise types, and manage training and evaluation data.
+    - Alternatively, import an experiment configuration using the "Import" button.
+    - When done, press "Define Experiment".
+    - Export the experiment configuration using the "Export" button for future reference.
+
+    .. figure:: ../images/playground/evaluation_mode/define_experiment.png
+       :width: 500px
+       :alt: Define Experiment Interface of the Athena Playground
+
+       Evaluation Mode: Define Experiment Interface of the Athena Playground
+
+2. **Configure Modules:**
+    - Select and configure the modules you wish to include in your experiment.
+    - Ensure each module is set up with appropriate parameters for effective comparison.
+    - Import module configurations using the "Import" button, if needed.
+    - Export the module configurations using the "Export" button for future reference.
+
+    .. figure:: ../images/playground/evaluation_mode/configure_modules.png
+       :width: 500px
+       :alt: Configure Modules Interface of the Athena Playground
+
+       Evaluation Mode: Configure Modules Interface of the Athena Playground
+
+3. **Conduct Experiment:**
+    - Press "Start Experiment" to begin the experiment.
+    - The steps performed include sending submissions, sending feedback for training submissions, generating feedback suggestions, and running automatic evaluations.
+    - If training submissions are provided, you will need to manually continue the experiment by pressing "Continue".
+    - If automatic evaluations is enabled, for instance LLM-as-a-judge for text exercises, you will also need to manually confirm it.
+    - Export and import the experiment results as needed using the "Export" and "Import" buttons, respectively.
+
+    .. figure:: ../images/playground/evaluation_mode/conduct_experiment_text.png
+       :width: 500px
+       :alt: Conduct Experiment Interface for a Text Exercise of the Athena Playground
+
+       Evaluation Mode: Conduct Experiment Interface for a Text Exercise of the Athena Playground
+
+4. **Annotate Feedback Suggestions:**
+    - Annotate the generated feedback suggestions with "Accept" or "Reject" as a tutor would.
+
+5. **Export Results:**
+    - At the end of the experiment, or at any time during the experiment, export the results using the "Export" button.
+    - Make sure that you also exported the experiment configuration and module configurations to have a complete record of the experiment.