Misalignment of the Hard Example Mining interface with Sedna

[FuryMartin]

What happened:

The current implementation of Hard Example Mining in Incremental Learning is incorrect, causing the PCB-AoI/IncrementalLearning to fail to run.

What you expected to happen:

Fix this bug to make the hard example mining interface functional in Ianvs. #122 will also require this feature.

How to reproduce it (as minimally and precisely as possible):

1. Build an environment in which PCB-AoI is runnable.
2. Fix path errors in yamls.

You need to check the following files:
examples/pcb-aoi/incremental_learning_bench/fault detection/benchmarkingjob.yaml
examples/pcb-aoi/incremental_learning_bench/fault detection/testalgorithms/fpn/fpn_algorithm.yaml
examples/pcb-aoi/incremental_learning_bench/fault detection/testenv/testenv.yaml.

The main issue is that the fault detection folder needs to be added to the path.

For example: The following line needs to be modified to ./examples/pcb-aoi/incremental_learning_bench/fault detection/testenv/testenv.yaml

ianvs/examples/pcb-aoi/incremental_learning_bench/fault detection/benchmarkingjob.yaml

Line 9 in f2352ce

testenv: "./examples/pcb-aoi/incremental_learning_bench/testenv/testenv.yaml"

You may also need to fix paths for workspace, train_url, test_url, initial_model_url according to your fold structure

4. Run PCB-AoI/IncrementalLearning by

ianvs -f examples/pcb-aoi/incremental_learning_bench/fault detection/benchmarkingjob.yaml

Then you will see errors like below:

Traceback (most recent call last):
  File "./ianvs/core/testcasecontroller/testcase/testcase.py", line 74, in run
    res, system_metric_info = paradigm.run()
  File "./ianvs/core/testcasecontroller/algorithm/paradigm/incremental_learning/incremental_learning.py", line 91, in run
    r)
  File "./ianvs/core/testcasecontroller/algorithm/paradigm/incremental_learning/incremental_learning.py", line 132, in _inference
    job = self.build_paradigm_job(ParadigmType.INCREMENTAL_LEARNING.value)
  File "./ianvs/core/testcasecontroller/algorithm/paradigm/base.py", line 100, in build_paradigm_job
    ModuleType.HARD_EXAMPLE_MINING.value))
  File "/home/XXX/miniconda3/envs/ianvs-py36/lib/python3.6/site-packages/sedna-0.4.1-py3.6.egg/sedna/core/incremental_learning/incremental_learning.py", line 75, in __init__
    hem = hard_example_mining.get("method", "IBT")
AttributeError: 'IBTFilter' object has no attribute 'get'

Anything else we need to know?:

The reason for this error is that Ianvs passed incorrect parameters for hard_example_mining when calling Sedna's IncrementalLearning interface.

Ianvs's IncrementalLearning is implemented by introducing Sedna's IncrementalLearning, as shown below.

ianvs/core/testcasecontroller/algorithm/paradigm/base.py

Lines 96 to 100 in f2352ce

	if paradigm_type == ParadigmType.INCREMENTAL_LEARNING.value:
	return IncrementalLearning(
	estimator=self.module_instances.get(ModuleType.BASEMODEL.value),
	hard_example_mining=self.module_instances.get(
	ModuleType.HARD_EXAMPLE_MINING.value))

Ianvs passes an instance of the user-defined hard_example_mining module (PCB-AoI/hard_example_mining.py is an example) into the hard_example_mining keyword argument of Sedna's IncrementalLearning.

However, hard_example_mining in Sedna is defined as a dictionary with method and param fields, and the correct parameters for instantiating IncrementalLearning are as bellow.

il_model = IncrementalLearning(
            estimator=Estimator,
            hard_example_mining={
                "method": "IBT",
                "param": {
                    "threshold_img": 0.9
                }
            }
        )

Inside Sedna's IncrementalLearning, an instance of the hard_example_mining algorithm will be created using the ClassFactory.get_cls() method based on this dictionary. See details in Sedna's Source Code

        self.hard_example_mining_algorithm = None
        if not hard_example_mining:
            hard_example_mining = self.get_hem_algorithm_from_config()
        if hard_example_mining:
            hem = hard_example_mining.get("method", "IBT")
            hem_parameters = hard_example_mining.get("param", {})
            self.hard_example_mining_algorithm = ClassFactory.get_cls(
                ClassType.HEM, hem
            )(**hem_parameters)

According to the current parameter passing method of Ianvs, the hard_example_mining in the above code will not be None, thus triggering hem = hard_example_mining.get("method", "IBT").

However, we are passing a module that lacks the get method, which is typical of a dict. Additionally, the example does not provide any further definitions, leading to an AttributeError.

[MooreZheng]

Will this PR #133 fix this issue?

[FuryMartin]

Will this PR #133 fix this issue?

No, #133 only fixes YAML path errors by adding /fault detection/ to the path links.

The essence of the issue is that Sedna's IncrementalLearning requires a dict containing the algorithm name, which is then used to load the module in Sedna; however, Ianvs's current implementation directly transmits a module.

The incorrect parameter passing problem caused an AttributeError when Sedna IncrementalLearning attempted to call the get method.

[FuryMartin]

I have figure this problem out.

All we need to do is to replace the code at

ianvs/core/testcasecontroller/algorithm/module/module.py

Lines 110 to 117 in f2352ce

	if self.url:
	try:
	utils.load_module(self.url)
	# pylint: disable=E1134
	func = ClassFactory.get_cls(
	type_name=class_factory_type, t_cls_name=self.name)(**self.hyperparameters)
	return func

with the code below

        if self.url:
            try:
                utils.load_module(self.url)
                # pylint: disable=E1134
                if class_factory_type == ClassType.HEM:
                    func = {"method": self.name, "param":self.hyperparameters}
                else:
                    func = ClassFactory.get_cls(
                        type_name=class_factory_type, t_cls_name=self.name)(**self.hyperparameters)
                return func

This will prevent HardExampleMining from being loaded as a class instance during Ianvs's module loading process, allowing the actual loading to be handled by Sedna's IncrementalLearning or JointInference.

[FuryMartin]

Maybe kind/bug is more suitable for this issue than kind/cleanup.