Add ignite event filters to monai handlers (StatsHandler in particular)

[jak0bw]

I am sorry if I missed an already existing feature but I could not find an option to use the event filter option (https://pytorch.org/ignite/generated/ignite.engine.events.Events.html paragraphs starts with Since v0.3.0) for monai.handler.StatsHandler.

Originally I only wanted to log the stats for every Xth iteration but wanted to use the default logger. I solved this with a lambda function which refers to to the created StatsHandler:
stats_handler = StatsHandler(iteration_print_logger=lambda engine: stats_handler._default_iteration_print(engine) if engine.state.iteration % iteration_log_interval == 0 else None, tag_name='Mean Squared Error', output_transform=from_engine(['loss'], first=True))

But this feels a little clunky to me. As a solution I had the idea to expose the ignite.event_filter feature to the user by adding an iteration_filter and an epoch_filter argument to the StatsHandler Class. These would enable setting specifying event filters for both the ITERATION_COMPLETED and EPOCH_COMPLETED events in the attach function

MONAI/monai/handlers/stats_handler.py

Lines 122 to 142 in cdac033

	def attach(self, engine: Engine) -> None:
	"""
	Register a set of Ignite Event-Handlers to a specified Ignite engine.
	Args:
	engine: Ignite Engine, it can be a trainer, validator or evaluator.
	"""
	if self.name is None:
	self.logger = engine.logger
	if self.logger.getEffectiveLevel() > logging.INFO or logging.root.getEffectiveLevel() > logging.INFO:
	warnings.warn(
	"the effective log level of engine logger or RootLogger is higher than INFO, may not record log,"
	" please call `logging.basicConfig(stream=sys.stdout, level=logging.INFO)` to enable it."
	)
	if self.iteration_log and not engine.has_event_handler(self.iteration_completed, Events.ITERATION_COMPLETED):
	engine.add_event_handler(Events.ITERATION_COMPLETED, self.iteration_completed)
	if self.epoch_log and not engine.has_event_handler(self.epoch_completed, Events.EPOCH_COMPLETED):
	engine.add_event_handler(Events.EPOCH_COMPLETED, self.epoch_completed)
	if not engine.has_event_handler(self.exception_raised, Events.EXCEPTION_RAISED):
	engine.add_event_handler(Events.EXCEPTION_RAISED, self.exception_raised)

(if iteration_log or epoch_log is 'True'). It is currently possible to solve this problem by writing a new logger function (more or less what I did above) and handle the event_filtering there but to me it feels like exposing the ignite.event_filter feature is the better solution to keep the logging and filter functionality separated.

This idea of exposing the ignite.event_filter feature to the user might also apply to other monai.handlers.

[Nic-Ma]

Thanks for raising it.
CC @vfdev-5 , any suggestion for this feature request?

Thanks in advance.

[jak0bw]

By the way, we haven't yet released that in stable but on master we have more options: every, once, before, after. https://github.com/pytorch/ignite/blob/d2020e4e253ac1455a757c2db895c68ccfd2b958/ignite/engine/events.py#L62-L68

Ah interesting, I didn't know that but sounds quite interesting and I think we would want to support it fully...

iteration_log: bool | Callable | Dict[str, int]

Such that Callable is directly corresponds to event_filter and dicts can be: {"every": N} or {"once", N} or {"before", N} or {"after", N} and also {"after": M, "before": N}...

This then looks like quite a sensible solution, I think.

Just for my understanding after this feature is merged to stable, it will be possible in poytorch-ignite to mix multiple arguments for example after=9, before=21 and every=5, to only execute for iterations [10, 15, 20]?

[vfdev-5]

Just for my understanding after this feature is merged to stable, it will be possible in poytorch-ignite to mix multiple arguments for example after=9, before=21 and every=5, to only execute for iterations [10, 15, 20]?

Right now, onlybefore and after can work together but not with every or once. For complex filtering user should use event_filter function.
However, your example "after=9, before=21 and every=5, to only execute for iterations [10, 15, 20]" looks good to support as well. We have to think about.

[jak0bw]

Just my two cents on this topic. I think being able to use before, after, and every together (and keeping once separately) makes sense as it probably covers most instances and ist not that complex (in my opinion at least). The custom event filter could then be mostly used to filter on engine state and only in obscure instances to filter on the iteration counter, if this makes sense?

I only could think of one other (maybe) useful option in extending once to int | list[int] to enable giving a list of [5,8,23] to only execute once for iterations 5, 8 and 23 (if this isn't already supported).

[vfdev-5]

Just my two cents on this topic. I think being able to use before, after, and every together (and keeping once separately) makes sense as it probably covers most instances and ist not that complex (in my opinion at least).
The custom event filter could then be mostly used to filter on engine state and only in obscure instances to filter on the iteration counter, if this makes sense?

I agree

I only could think of one other (maybe) useful option in extending once to int | list[int] to enable giving a list of [5,8,23] to only execute once for iterations 5, 8 and 23 (if this isn't already supported).

Thanks for the suggestion, seems interesting to support !

(Created new issue for that : pytorch/ignite#2853)

[jak0bw]

Thanks for listening, love what you do 🤗