Refactor operations

[vinnamkim]

Summary

• This file has too many lines (2007 lines) in it: https://github.com/openvinotoolkit/datumaro/blob/develop/datumaro/components/operations.py
• Move annotation matchers and mergers to the seperate directory
• Move get_merger to the seperate directory (it is a pre-work for refactoring mergers.)

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added the description of my changes into CHANGELOG.​
• I have updated the documentation accordingly

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

[codecov-commenter]

Codecov Report

Patch coverage: 87.84% and project coverage change: +0.01 🎉

Comparison is base (508c573) 78.88% compared to head (b3f72d2) 78.89%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #901      +/-   ##
===========================================
+ Coverage    78.88%   78.89%   +0.01%     
===========================================
  Files          204      208       +4     
  Lines        25020    25037      +17     
  Branches      5031     5031              
===========================================
+ Hits         19736    19753      +17     
  Misses        4152     4152              
  Partials      1132     1132              

Flag	Coverage Δ
macos-11_Python-3.8	77.87% <87.84%> (+0.01%)	⬆️
ubuntu-20.04_Python-3.8	78.88% <87.84%> (+0.01%)	⬆️
windows-2019_Python-3.8	78.82% <87.84%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
datumaro/components/merge/base.py	93.93% <ø> (ø)
datumaro/components/merge/__init__.py	38.46% <38.46%> (ø)
datumaro/components/annotations/merger.py	85.29% <85.29%> (ø)
datumaro/components/annotations/matcher.py	92.68% <92.68%> (ø)
datumaro/components/annotations/__init__.py	100.00% <100.00%> (ø)
datumaro/components/dataset.py	82.25% <100.00%> (+0.10%)	⬆️
datumaro/components/operations.py	82.64% <100.00%> (-1.34%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[sooahleex]

Looks good to me.

[wonjuleee]

Why did you introduce annotations directory and what stands for? We may have prior understandings to make the same consensus. Could you explain?

[wonjuleee]

Are you thinking about introducing merge directory as a python API refactoring manner?

[vinnamkim]

Why did you introduce annotations directory and what stands for? We may have prior understandings to make the same consensus. Could you explain?

First of all, this file has too many lines (2007 lines) in it: https://github.com/openvinotoolkit/datumaro/blob/develop/datumaro/components/operations.py. Therefore, many classes and functions for different purposes are defined in this file. This is the full list of definitions in this file:

My motivation is to refactor merge related things to somewhere but not in operations.py.
To achieve it, I firstly try to refactor operations.py for Matchers and Mergers.

[wonjuleee]

Why did you introduce annotations directory and what stands for? We may have prior understandings to make the same consensus. Could you explain?

First of all, this file has too many lines (2007 lines) in it: https://github.com/openvinotoolkit/datumaro/blob/develop/datumaro/components/operations.py. Therefore, many classes and functions for different purposes are defined in this file. This is the full list of definitions in this file:

My motivation is to refactor merge related things to somewhere but not in operations.py. To achieve it, I firstly try to refactor operations.py for Matchers and Mergers.

I agree with separating Matcher and Merger from operations. But my question is about annotation directory. I would like to know what annotation directory means for Matcher and Merger.

[bonhunko]

LGTM

[vinnamkim]

I agree with separating Matcher and Merger from operations. But my question is about annotation directory. I would like to know what annotation directory means for Matcher and Merger.

Since Matcher and Merger correspond to each Annotation 1:1, it seems appropriate move them under annotations directory. So ultimately, all Annotations should be moved to annotations directory as well.

[wonjuleee]

I agree with separating Matcher and Merger from operations. But my question is about annotation directory. I would like to know what annotation directory means for Matcher and Merger.

Since Matcher and Merger correspond to each Annotation 1:1, it seems appropriate move them under annotations directory. So ultimately, all Annotations should be moved to annotations directory as well.

alright, I agree that matcher should be closely related to the annotation-level operation. But I think that merger is more about dataset-level operation. What do you think? Do you have any plan to create dataset directory in a similar vein?
What about merge directory? Could you share your plan or opinion about refactoring components in Datumaro?

[vinnamkim]

But I think that merger is more about dataset-level operation. What do you think? Do you have any plan to create dataset directory in a similar vein? What about merge directory? Could you share your plan or opinion about refactoring components in Datumaro?

1. Mergers corresponding to each Annotation (e.g., LabelMerger) which are in charge of element-wise merge operations are moved to ./datumaro/components/annotations directory.
2. Merge policies (for dataset level, e.g., ExactMerge or IntersectMerge) are moved to ./datumaro/components/merge directory.

My plan was like this, and in this PR, 1) is completed and 2) is partially completed.