[Feature] Generating and plotting confusion matrix #1301

HJoonKwon · 2022-02-17T16:57:31Z

Thank you for the great repository. I've been inspired a lot by you guys!
Here is the PR that adds a python file to generate/plot a normalized confusion matrix (n_classes x n_classes). The most part of the file is from mmdetection repo., and I changed it a bit in a way to be able to work on segmentation tasks. I tested it on my local computer with some prediction results formatted in .pkl. The plot looks like this!

If you get interested, I will add UT for this feature!

Motivation

There was neither a function to generate n_classes x n_classes confusion matrix nor to draw it. (related Issue: #1203). I thought it's always better to visualize everything, so I decided to make it.

Modification

I referred to mmdetection repo. as mentioned above.
First of all, I changed this function below

def calculate_confusion_matrix(dataset, results):

Referring to

def get_confusion_matrix(pred_label, label, num_classes, ignore_index):

in test_metrics.py, I calculated the confusion matrix based on the dataset(defined in the configuration file) and the results(loaded from pkl file). Other irrelevant variables were removed here as well.

def calculate_confusion_matrix(dataset, results):
    """Calculate the confusion matrix.

    Args:
        dataset (Dataset): Test or val dataset.
        results (list[ndarray]): A list of segmentation results in each image.
    """
    n = len(dataset.CLASSES)
    confusion_matrix = np.zeros(shape=[n, n])
    assert len(dataset) == len(results)
    prog_bar = mmcv.ProgressBar(len(results))
    for idx, per_img_res in enumerate(results):
        res_segm = per_img_res
        gt_segm = dataset.get_gt_seg_map_by_idx(idx)
        inds = n * gt_segm + res_segm
        inds = inds.flatten()
        mat = np.bincount(inds, minlength=n**2).reshape(n, n)
        confusion_matrix += mat
        prog_bar.update()
    return confusion_matrix

And finally, I changed the default color theme of the figure because 'plasma' makes high values look too shiny. So maybe the default can be 'winter' instead.

parser.add_argument(
        '--color-theme',
        default='winter',
        help='theme of the matrix color map')

Furthermore, I can add ignore_index thing as well, which isn't included here yet.

The confusion matrix figure would be saved then to save_dir which is given from the input arguments. This python file requires a prediction result file in .pkl format as it does in mmdetection repository.

BC-breaking (Optional)

No.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

MengzhangLI · 2022-02-17T17:39:55Z

Hi, thanks for your contribution!

We would review it asap.

Junjun2016 · 2022-02-20T17:01:32Z

Hi @HJoonKwon
Nice feature, can add its usage to the doc.

Junjun2016 · 2022-02-20T17:02:23Z

Also, add some examples with mmseg.

HJoonKwon · 2022-02-21T12:07:02Z

@Junjun2016 Thank you! Should I add its usage and some examples to docs/en/useful_tools.md?

MengzhangLI · 2022-02-22T11:36:50Z

tools/confusion_matrix.py

+        description='Generate confusion matrix from segmentation results')
+    parser.add_argument('config', help='test config file path')
+    parser.add_argument(
+        'prediction_path', help='prediction path where test .pkl result')


Currently we do not generate .pkl result like MMDetection, neither do prediction result. ;(

Oh. I thought test.py generates either prediction result or eval result depending on args.eval argument. If .pkl format doesn't work, then I can make confusion_matrix.py take the numpy type of prediction result just from

results = multi_gpu_test( model, data_loader, args.tmpdir, args.gpu_collect, False, pre_eval=args.eval is not None and not eval_on_format_results, format_only=args.format_only or eval_on_format_results, format_args=eval_kwargs)

instead and it would work the same.

OK, let me try to test whether .pkl could work, actually I never used pickle format before.

Sure! The result(numpy array) can be stored in any kind of format tho :)

MeowZheng · 2022-02-25T05:44:22Z

docs/en/useful_tools.md

+### 1.Generate a prediction result in pkl format using `test.py`
+```shell


Need a blank line

Suggested change

### 1.Generate a prediction result in pkl format using `test.py`

```shell

### 1.Generate a prediction result in pkl format using `test.py`

```shell

MD022/blanks-around-headings/blanks-around-headers: Headings should be surrounded by blank lines

MeowZheng · 2022-02-25T05:46:26Z

docs/en/useful_tools.md

+```shell
+python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${PATH_TO_RESULT_FILE}]
+```
+Note that the argument for ```--eval``` should be  ```None``` so that the result file contains numpy type of prediction results. The usage for distribution test is just the same.


Suggested change

Note that the argument for ```--eval``` should be ```None``` so that the result file contains numpy type of prediction results. The usage for distribution test is just the same.

Note that the argument for ```--eval``` should be ```None``` so that the result file contains numpy type of prediction results. The usage for distribution test is just the same.

Need a blank line
MD031/blanks-around-fences: Fenced code blocks should be surrounded by blank lines

MeowZheng · 2022-02-25T05:46:40Z

docs/en/useful_tools.md

+Example:
+```shell


Suggested change

Example:

```shell

Example:

```shell

MeowZheng · 2022-02-25T05:46:50Z

docs/en/useful_tools.md

+### 2. Use ```confusion_matrix.py``` to generate and plot a confusion matrix
+```shell


Suggested change

### 2. Use ```confusion_matrix.py``` to generate and plot a confusion matrix

```shell

### 2. Use ```confusion_matrix.py``` to generate and plot a confusion matrix

```shell

MeowZheng · 2022-02-25T05:47:05Z

docs/en/useful_tools.md

+python tools/test.py \
+configs/fcn/fcn_r50-d8_512x1024_40k_cityscapes.py \
+checkpoint/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608-efe53f0d.pth \
+--out result/pred_result.pkl \


Suggested change

--out result/pred_result.pkl \

--out result/pred_result.pkl

MeowZheng · 2022-02-25T05:48:13Z

tools/confusion_matrix.py

+    return args
+
+
+def calculate_confusion_matrix(dataset, results):


I have a little concern that memory might be overloaded when loading all predicted results of a dataset

It would probably be predicted results from the test set that may be not that big(depending on applications tho). Do you have any suggestions? :)

MeowZheng · 2022-02-25T05:48:28Z

docs/en/useful_tools.md

+Example:
+```shell


Suggested change

Example:

```shell

Example:

```shell

MeowZheng · 2022-02-25T06:22:15Z

tools/confusion_matrix.py

+    if args.cfg_options is not None:
+        cfg.merge_from_dict(args.cfg_options)
+
+    results = mmcv.load(args.prediction_path)


From our experience, loading the result for a whole dataset always got stuck in segmentation task

Should I add some lines to check memory allocated to the results and give a warning to users to reduce the size of the test set used for test.py? Or do you have any suggenstions? :)

Might we merge this pr and then fix bugs of load and dump seg predicted results? @Junjun2016

I did pull-rebase based on the most recent commit of master branch in case :)

might fix format problem in docs/en/useful_tools.md, and we will merge this pr

dirtycomputer · 2022-05-22T05:25:08Z

python tools/confusion_matrix.py work_dirs/upernet_swin_tiny_patch4_window7_512x512_160k_CVC/upernet_swin_tiny_patch4_window7_512x512_160k_CVC.py work_dirs/upernet_swin_tiny_patch4_window7_512x512_160k_CVC/result.pkl work_dirs/upernet_swin_tiny_patch4_window7_512x512_160k_CVC/
Traceback (most recent call last):
File "tools/confusion_matrix.py", line 184, in
main()
File "tools/confusion_matrix.py", line 164, in main
raise TypeError('invalid type of prediction results')
TypeError: invalid type of prediction results

HJoonKwon · 2022-05-22T06:27:11Z

@dirtycomputer Did you turn off the Eval argument when you created .pkl result? The result should be prediction results, not the evaluation metrics.

dirtycomputer · 2022-05-22T07:49:42Z

THANKS IT WORKS

TWang1017 · 2022-06-19T02:46:43Z

tools/confusion_matrix.py does not work when reduce_zero_label=True, the class number in the prediction does not match that in the labels

* generate and plot confusion matrix * fix typo * add usage and examples for confusion matrix * deal with nan values(pick pr#7147 mmdet) * fix md format

mgsdqs · 2022-08-12T05:08:55Z

tools/confusion_matrix.py在 reduce_zero_label=True 时不起作用，预测中的类号与标签中的类号不匹配

I want to know how to solve this problem ?

shenxiangkei · 2023-04-16T12:17:43Z

May I ask you how to removed the Eval parameter? Thanks

HJoonKwon · 2023-04-16T13:01:46Z

May I ask you how to removed the Eval parameter? Thanks

Use --out argument instead of --eval. You can refer to here

shenxiangkei · 2023-04-16T13:03:50Z

Thank you very much for your reply, we found that setting the default of eval to None can achieve the same function. Thanks again for your reply!

…

------------------ 原始邮件 ------------------ 发件人: "Hyeokjoon ***@***.***>; 发送时间: 2023年4月16日(星期天) 晚上9:01 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [open-mmlab/mmsegmentation] [Feature] Generating and plotting confusion matrix (PR #1301) May I ask you how to removed the Eval parameter? Thanks Use --out argument instead of --eval. You can refer to here — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

HJoonKwon force-pushed the master branch from e7a6f04 to b2e7bb7 Compare February 21, 2022 13:19

MengzhangLI reviewed Feb 22, 2022

View reviewed changes

MengzhangLI self-assigned this Feb 23, 2022

HJoonKwon force-pushed the master branch 2 times, most recently from ab242ef to 5b45915 Compare February 24, 2022 12:09

MeowZheng reviewed Feb 25, 2022

View reviewed changes

HJoonKwon force-pushed the master branch from 5b45915 to 359c938 Compare February 26, 2022 06:18

Junjun2016 approved these changes Mar 1, 2022

View reviewed changes

HJoonKwon added 4 commits March 2, 2022 19:56

generate and plot confusion matrix

ede5567

fix typo

ff9049f

add usage and examples for confusion matrix

d417525

deal with nan values(pick pr#7147 mmdet)

67cddeb

HJoonKwon force-pushed the master branch from 359c938 to 67cddeb Compare March 2, 2022 10:57

fix md format

dd53260

MeowZheng approved these changes Mar 3, 2022

View reviewed changes

MeowZheng merged commit 369a2ee into open-mmlab:master Mar 3, 2022

MIMIWAWA mentioned this pull request Aug 6, 2022

Following your descriptions, could you tell me how to turn off the Eval argument entirely when creating the .pkl result? #1874

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Generating and plotting confusion matrix #1301

[Feature] Generating and plotting confusion matrix #1301

HJoonKwon commented Feb 17, 2022 •

edited

Loading

MengzhangLI commented Feb 17, 2022

Junjun2016 commented Feb 20, 2022

Junjun2016 commented Feb 20, 2022

HJoonKwon commented Feb 21, 2022

MengzhangLI Feb 22, 2022

HJoonKwon Feb 22, 2022 •

edited

Loading

MengzhangLI Feb 22, 2022 •

edited

Loading

HJoonKwon Feb 22, 2022

MeowZheng Feb 25, 2022

MeowZheng Feb 25, 2022

MeowZheng Feb 25, 2022

MeowZheng Feb 25, 2022

MeowZheng Feb 25, 2022

MeowZheng Feb 25, 2022

HJoonKwon Feb 26, 2022

MeowZheng Feb 25, 2022

MeowZheng Feb 25, 2022

HJoonKwon Feb 26, 2022

MeowZheng Mar 1, 2022

HJoonKwon Mar 2, 2022

MeowZheng Mar 3, 2022 •

edited

Loading

dirtycomputer commented May 22, 2022

HJoonKwon commented May 22, 2022

dirtycomputer commented May 22, 2022

TWang1017 commented Jun 19, 2022

mgsdqs commented Aug 12, 2022

shenxiangkei commented Apr 16, 2023

HJoonKwon commented Apr 16, 2023

shenxiangkei commented Apr 16, 2023 via email

		### 1.Generate a prediction result in pkl format using `test.py`
		```shell

	Note that the argument for ```--eval``` should be ```None``` so that the result file contains numpy type of prediction results. The usage for distribution test is just the same.

	Note that the argument for ```--eval``` should be ```None``` so that the result file contains numpy type of prediction results. The usage for distribution test is just the same.

		### 2. Use ```confusion_matrix.py``` to generate and plot a confusion matrix
		```shell

		return args


		def calculate_confusion_matrix(dataset, results):

[Feature] Generating and plotting confusion matrix #1301

[Feature] Generating and plotting confusion matrix #1301

Conversation

HJoonKwon commented Feb 17, 2022 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

MengzhangLI commented Feb 17, 2022

Junjun2016 commented Feb 20, 2022

Junjun2016 commented Feb 20, 2022

HJoonKwon commented Feb 21, 2022

Choose a reason for hiding this comment

HJoonKwon Feb 22, 2022 • edited Loading

Choose a reason for hiding this comment

MengzhangLI Feb 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MeowZheng Mar 3, 2022 • edited Loading

Choose a reason for hiding this comment

dirtycomputer commented May 22, 2022

HJoonKwon commented May 22, 2022

dirtycomputer commented May 22, 2022

TWang1017 commented Jun 19, 2022

mgsdqs commented Aug 12, 2022

shenxiangkei commented Apr 16, 2023

HJoonKwon commented Apr 16, 2023

shenxiangkei commented Apr 16, 2023 via email

HJoonKwon commented Feb 17, 2022 •

edited

Loading

HJoonKwon Feb 22, 2022 •

edited

Loading

MengzhangLI Feb 22, 2022 •

edited

Loading

MeowZheng Mar 3, 2022 •

edited

Loading