Dataset Preparation Guide.

[yustiks]

Hello!
Thank you for the amazing work done!
I would like to calculate MaP for object detection model and experiencing some problems.
First, I couldn't find the link in the page of accuracy_checker/ReadMe:
"If you want to evaluate models using prepared config files and well-known datasets, you need to organize folders with validation datasets in a certain way. More detailed information about dataset preparation you can find in Dataset Preparation Guide."

Where is this Dataset Preparation Guide?

[eaidova]

@yustiks thank you for your note. looks like link on web page was currepted, when we move this file. This issue has beed already fixed in develop branch. and targeted to openvino 2021.3 release..
For now, you can use this link: https://github.com/openvinotoolkit/open_model_zoo/blob/master/datasets.md

[yustiks]

@yustiks thank you for your note. looks like link on web page was currepted, when we move this file. This issue has beed already fixed in develop branch. and targeted to openvino 2021.3 release..
For now, you can use this link: https://github.com/openvinotoolkit/open_model_zoo/blob/master/datasets.md

Thank you!
Another question is related towards how to prepare data for object detection by PyTorch?
I trained PyTorch model for object detection, and my configuration_file looks like this:

models:
- name: keras_retinanet
    launchers:
      - framework: pytorch
        device: GPU
        model: magnit_digit_detector_25.pth
        adapter: pytorch_ssd_decoder
    datasets:
      - name: ms_coco_detection_91_classes
        annotation_conversion:
          converter: mscoco_detection
          annotation_file: instances_val2017.json
          has_background: True
          sort_annotations: True
          use_full_label_map: True
           - name: dataset_name
       annotation: mscoco_detection.pickle
       data_source: val2017

       preprocessing:
        - type: resize
          dst_width: 600
          dst_height: 1024

        - type: padding
          dst_height: 600
          dst_width: 1024
          pad_type: right_bottom

       postprocessing:
        - type: faster_rcnn_postprocessing_resize
          dst_height: 600
          dst_width: 1024

       metrics:
        - type: MaP

Then I run accuracy_checker with the following code:

accuracy_check -c /home/iivanova/scripts_/configuration_file -m /home/iivanova/weights/ -s /home/iivanova/coco_images/val2017/ -a /home/iivanova/coco_images/annotations/

and I have the next error:

yaml.scanner.ScannerError: mapping values are not allowed here
  in "/home/iivanova/scripts_/configuration_file", line 3, column 14

What could be the problem?

[eaidova]

@yustiks looks like you have wrong indention in yml file (launchers and datasets sections should be on the same indention level like name).
BTW specify only weight (pth file) is not enough for pytorch model running. You need also provide model class for import.
You can find info how to configure pytorch launcher here: https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/accuracy_checker/accuracy_checker/launcher/pytorch_launcher_readme.md

[yustiks]

@yustiks looks like you have wrong indention in yml file (launchers and datasets sections should be on the same indention level like name).
BTW specify only weight (pth file) is not enough for pytorch model running. You need also provide model class for import.
You can find info how to configure pytorch launcher here: https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/accuracy_checker/accuracy_checker/launcher/pytorch_launcher_readme.md

Thank you!
I changed parameters now according to your answer, and still have the same errors.
Therefore, I have a question on how to load MaskRCNN model?

[eaidova]

changed parameters now according to your answer, and still have the same errors.

this error is not from the tool, this error speaks that you have invalid yml file, line 3, column 14 it is the line and position which wrong. there is standard rules for yaml creating and looks like you do not follow them...
Please carefully look on your file, mapping values are not allowed here usually tell that you mix listing values and dictionary (often it happens when you have wrong alignment or forgot to add space after colons)
You can look on any config example in this repo, possibly it helps you to understand what went wrong.

Therefore, I have a question on how to load MaskRCNN model?

Not sure that I understand you question... Which model do you mean? Do you have specific questions related config writing or what?

[yustiks]

changed parameters now according to your answer, and still have the same errors.

this error is not from the tool, this error speaks that you have invalid yml file, line 3, column 14 it is the line and position which wrong. there is standard rules for yaml creating and looks like you do not follow them...
Please carefully look on your file, mapping values are not allowed here usually tell that you mix listing values and dictionary (often it happens when you have wrong alignment or forgot to add space after colons)
You can look on any config example in this repo, possibly it helps you to understand what went wrong.

Therefore, I have a question on how to load MaskRCNN model?

Not sure that I understand you question... Which model do you mean? Do you have specific questions related config writing or what?

You are right, thank you for pointing out this problem.
I changed the configuration file, and now I have the following error:

accuracy_checker.config.config_validator.ConfigError: Invalid config for Pytorch_Launcher: missing required fields: module

What could be the problem?

[eaidova]

As I said before you need to provide module for network architecture loading.
Please read launcher configuration instruction
It says:

• module pytorch network module for loading.
• checkpoint - pre-trained model checkpoint weigts (Optional).
• python_path - appendix for PYTHONPATH for making network module visible in current python environment (Optional).
• module_args - list of positional arguments for network module (Optional).
• module_kwargs - dictionary (key: value where key is argument name, value is argument value) which represent network module keyword arguments.

It does not look like your config format...

[yustiks]

As I said before you need to provide module for network architecture loading.
Please read launcher configuration instruction
It says:

• module pytorch network module for loading.
• checkpoint - pre-trained model checkpoint weigts (Optional).
• python_path - appendix for PYTHONPATH for making network module visible in current python environment (Optional).
• module_args - list of positional arguments for network module (Optional).
• module_kwargs - dictionary (key: value where key is argument name, value is argument value) which represent network module keyword arguments.

It does not look like your config format...

thank you for fast reply!
Yes, it is clear.
Where to find instructions on what module should be for mask-rcnn?
there is example with alexnet, but I couldn't find mask-rcnn and resnet50.

[yustiks]

I have found an answer.
thank you!
now the module is as following:

      module: torchvision.models.detection.maskrcnn_resnet50_fpn

but I have another error...

    raise ConfigError(reason)
accuracy_checker.config.config_validator.ConfigError: Invalid config for PyTorchSSDDecoder: missing required fields: scores_out, boxes_out

[eaidova]

Sorry, we provide examples only for Open Model Zoo models, and as I know there is no pytorch mask rcnn, so it is the reason why example is unavailable. I only can suggest to read docs and deeper analyze model code for preparing model configuration., At least, it should be clear how to import any other model.
Provided example is just show how to import model from torchvision.

• module: torchvision.models.alexnet is function for model creation (or can be class) like here https://pytorch.org/vision/0.8/models.html#torchvision.models.alexnet

•   module_kwargs:
      pretrained: True
  

is named arguments for this function. There is no checkpoint, because it loaded inside this function.
I believe it will work in the same way for all torchvision models.
For custom model you need do the simmilar thing - define model class or function for loading, provide argumets for its creation and possibly path to weights checkpoint.

I do not know about which model do you speak (yes, I know what is mask rcnn, but there are different implementations which can require different guiding), so I believe it is all what I can do for you...

[eaidova]

please read the docs, the error message is clear. Documentation contains info about required fields
https://docs.openvinotoolkit.org/2021.2/_tools_accuracy_checker_README.html

[yustiks]

Sorry, we provide examples only for Open Model Zoo models, and as I know there is no pytorch mask rcnn, so it is the reason why example is unavailable. I only can suggest to read docs and deeper analyze model code for preparing model configuration., At least, it should be clear how to import any other model.
Provided example is just show how to import model from torchvision.

• module: torchvision.models.alexnet is function for model creation (or can be class) like here https://pytorch.org/vision/0.8/models.html#torchvision.models.alexnet

•   module_kwargs:
      pretrained: True

is named arguments for this function. There is no checkpoint, because it loaded inside this function.
I believe it will work in the same way for all torchvision models.
For custom model you need do the simmilar thing - define model class or function for loading, provide argumets for its creation and possibly path to weights checkpoint.

I do not know about which model do you speak (yes, I know what is mask rcnn, but there are different implementations which can require different guiding), so I believe it is all what I can do for you...

yes, you are right! thank you! I was mistaken, there should be resent50...

[yustiks]

I changed the configuration_file:

models:
- name: pytorch
  launchers:
    - framework: pytorch
      model:   magnit_digit_detector_25.pth
      device: GPU
      adapter:
        type: pytorch_ssd_decoder
        scores_out: detection_output
        boxes_out: bbox_pred
      batch: 128
      module: torchvision.models.detection.maskrcnn_resnet50_fpn
      module_kwargs:
        pretrained: True
        
  datasets:
    - name: ms_coco_detection_91_classes
      annotation_conversion:
        converter: mscoco_detection
        annotation_file: instances_val2017.json
        has_background: False
        sort_annotations: True
        use_full_label_map: True

      annotation: mscoco_detection.pickle
      dataset_meta: mscoco_detection.json
      data_source: val2017
      preprocessing:
        - type: resize
          dst_width: 600
          dst_height: 1024

        - type: padding
          dst_height: 600
          dst_width: 1024
          pad_type: right_bottom

      postprocessing:
        - type: faster_rcnn_postprocessing_resize
          dst_height: 600
          dst_width: 1024
      metrics:
        - type: coco_precision

then I run accuracy_checker:

accuracy_check -c /home/iivanova/scripts_/configuration_file -m /home/iivanova/weights/ -s /home/iivanova/test_coco_magnit/ -a /home/iivanova/test_coco_magnit/

and I have the next error:

23:52:09 accuracy_checker ERROR: 'dict' object has no attribute 'data'
Traceback (most recent call last):
  File "/home/iivanova/miniconda3/envs/tf_gpu/lib/python3.7/site-packages/accuracy_checker-0.8.1-py3.7.egg/accuracy_checker/main.py", line 314, in main
    stored_predictions=args.stored_predictions, progress_reporter=progress_reporter, **evaluator_kwargs
  File "/home/iivanova/miniconda3/envs/tf_gpu/lib/python3.7/site-packages/accuracy_checker-0.8.1-py3.7.egg/accuracy_checker/evaluators/model_evaluator.py", line 233, in process_dataset_sync
    batch_predictions = self.launcher.predict(filled_inputs, batch_meta, **kwargs)
  File "/home/iivanova/miniconda3/envs/tf_gpu/lib/python3.7/site-packages/accuracy_checker-0.8.1-py3.7.egg/accuracy_checker/launcher/pytorch_launcher.py", line 141, in predict
    for output_name, res in zip(self.output_names, outputs)
  File "/home/iivanova/miniconda3/envs/tf_gpu/lib/python3.7/site-packages/accuracy_checker-0.8.1-py3.7.egg/accuracy_checker/launcher/pytorch_launcher.py", line 141, in <dictcomp>
    for output_name, res in zip(self.output_names, outputs)
AttributeError: 'dict' object has no attribute 'data'

what could be the reason?

[vladimir-dudnik]

@yustiks have you solved your issue?

[yustiks]

@yustiks have you solved your issue?

no =(