[Feature] Support computing mean scores in UniformConcatDataset

[gaotongxiao]

Motivation

Since text recognition models are usually evaluated on multiple datasets, it's hard to compare the model's performance across epochs without a unified indicator such as the mean scores. This PR supports get_mean in UniformConcatDataset. When it's on, the mean score of {metric_name} of concatenated datasets will be added to the evaluation results with the name mean_{metric_name}.

Modification

Modified mmdet.datasets.ConcatDataset.evaluate() to compute the mean scores when both self.separate_eval and self.get_mean are True. Also disabled evaluating the datasets as a whole as this feature was not verified on MMOCR. We don't want to make excessive changes in this PR.

Use case

To show the average results, set show_mean_scores=True in data.val and data.test of the model's config: (e.g. in

mmocr/configs/textrecog/crnn/crnn_academic_dataset.py

Lines 24 to 31 in 3188e53

	val=dict(
	type='UniformConcatDataset',
	datasets=test_list,
	pipeline=test_pipeline),
	test=dict(
	type='UniformConcatDataset',
	datasets=test_list,
	pipeline=test_pipeline))

)

 val=dict( 
     type='UniformConcatDataset', 
     datasets=test_list, 
     show_mean_scores=True,
     pipeline=test_pipeline), 
 test=dict( 
     type='UniformConcatDataset', 
     datasets=test_list, 
     show_mean_scores=True,
     pipeline=test_pipeline)) 

BC-breaking (Optional)

No

[gaotongxiao]

Any other comments? If not, I'll start to update the configs.

[xinke-wang]

Any other comments? If not, I'll start to update the configs.

Wait a moment. I am still testing this code.

[xinke-wang]

I have some minor concerns about the current evaluation process:

• Current log of recognition is not clear especially when the number of test datasets increases, for example, the index 0_, 1_, 2_ ... makes it difficult to find the specific subset. (Not sure if it is necessary to fix in this PR)

In comparison, the det log prints details of each dataset in a clearer way. So the users can check the performance of a specific dataset easily.

• Using different dataset type in detection task cannot trigger the NotImplementedError, instead, it raises AttributeError: UniformConcatDataset: 'IcdarDataset' object has no attribute 'flag'.

Using the following toy data config to reproduce the issue

root = 'tests/data/toy_dataset'

# dataset with type='TextDetDataset'
train1 = dict(
    type='TextDetDataset',
    img_prefix=f'{root}/imgs',
    ann_file=f'{root}/instances_test.txt',
    loader=dict(
        type='HardDiskLoader',
        repeat=4,
        parser=dict(
            type='LineJsonParser',
            keys=['file_name', 'height', 'width', 'annotations'])),
    pipeline=None,
    test_mode=False)

# dataset with type='IcdarDataset'
train2 = dict(
    type='IcdarDataset',
    ann_file=f'{root}/instances_test.json',
    img_prefix=f'{root}/imgs',
    pipeline=None)

test = dict(
    type='TextDetDataset',
    img_prefix=f'{root}/imgs',
    ann_file=f'{root}/instances_test.txt',
    loader=dict(
        type='HardDiskLoader',
        repeat=1,
        parser=dict(
            type='LineJsonParser',
            keys=['file_name', 'height', 'width', 'annotations'])),
    pipeline=None,
    test_mode=True)

train_list = [train1, train2]

test_list = [test, train2]

• Other things look good to me. A suggestion is how about use print_mean_scores or show_average_performance or get_mean_scores. Only using get_mean is a little bit confused, and the users may have to check the doc to understand what this param refers to.

[gaotongxiao]

@xinke-wang Unfortunately, even though we can print the results for each dataset, there is no way to get rid of the summary at the end of the evaluation, which can lead to duplicate outputs. To streamline the evaluation report, I can make another PR after which users can choose the evaluation metric(s) to report by customizing the config.