Fix: correctly order the ground truth and prediction for ARFF files in run.data_content

[LennartPurucker]

Bug - What does this PR fix?

The order of the ground truth and predictions is mixed up in the current implementation of how a run stores prediction data and how it uses the prediction data to build an ARFF file.

As a result, the ground truth is treated as the predictions, and the predictions are treated as the ground truth.
Consequently, publishing a run to uploads the wrong values for these columns to the OpenML server.

Impact

This is not validated on the server side (to my understanding). Hence, all ARFF files of predictions uploaded using the OpenML Python API are most likely wrong. Moreover, evaluations of such runs also report the wrong scores. This might have impacted the results of papers that used scores of runs uploaded by the Python Client for meta-analysis.

Reference Issues

Multiple issues exist as a result of this. The following issues are likely related to this problem: #1197, #559, openml/OpenML#1185

Fix

I changed the order in the appropiate places and added a test for the (IMO) expected/correct behavior. Additionally, I changed the tests that checked for the old order to the new order.

New Order

The order I used follows the order of ARFF files uploaded by the R Client API. I used the following code snippet to find the order.

from openml.runs import get_run

# 1875133 is a run from mlr 
print(get_run(1875133).predictions_url)

# After downloading and opening the file stored under the predictions_url, one can see that the file's order is:
# @attribute 'repeat' numeric
# @attribute 'fold' numeric
# @attribute 'row_id' numeric
# @attribute 'prediction' {'1','2'}
# @attribute 'truth' {'1','2'}
# @attribute 'confidence.1' numeric
# @attribute 'confidence.2' numeric

Open Questions

The contribution guidelines mention that I should change the progress.rst. I am unsure if this would be below 13.1 and what I should mention there. What do you think? I updated the progress.rst for the newest version.

Required (Server-Side) Follow-up Actions

This bug affects a lot of already published runs on OpenML.
We might need to change/adjust the uploaded ARFF files and re-evaluate all these runs.

[mfeurer]

Looks good to me, let's wait for the unit tests to work again and then merge this.

[PGijsbers]

Can you add that it is specifically about regression tasks? thanks

[LennartPurucker]

The switched order is a problem for regression and classification tasks (and maybe learning curve).