Duplicate StorageCrawl jobs

[bugsyb]

Hi Guys,

Not sure if observation is correct though got me having some doubts.
I've noticed having duplicate entries for StorageCrawl jobs in Jobs table for same Storage and same type of net, please see below:

Faces:

postgres@postgres:/$ echo "SELECT (argument,last_run,last_checked,reserved_at,execution_duration,class) FROM public.oc_jobs WHERE argument like '%faces%'"|psql pchmura

                                                                                         row                                                                                          
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ("{""storage_id"":10,""root_id"":31946,""override_root"":31947,""last_file_id"":497814,""models"":[""faces""]}",0,1670928161,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":10,""root_id"":31946,""override_root"":31947,""last_file_id"":500716,""models"":[""faces""]}",0,1670928255,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":11,""root_id"":31957,""override_root"":31958,""last_file_id"":773474,""models"":[""faces""]}",0,1670928017,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":11,""root_id"":31957,""override_root"":31958,""last_file_id"":773357,""models"":[""faces""]}",0,1670928018,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":11,""root_id"":31957,""override_root"":31958,""last_file_id"":1964206,""models"":[""faces""]}",0,1670928020,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":10,""root_id"":31946,""override_root"":31947,""last_file_id"":497484,""models"":[""faces""]}",0,1670928045,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
(6 rows)

Imagenet:

postgres@postgres:/$ echo "SELECT (argument,last_run,last_checked,reserved_at,execution_duration,class) FROM public.oc_jobs WHERE argument like '%imagenet%'"|psql pchmura
                                                                                           row                                                                                           
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 ("{""storage_id"":10,""root_id"":31946,""override_root"":31947,""last_file_id"":500848,""models"":[""imagenet""]}",0,1670928015,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":10,""root_id"":31946,""override_root"":31947,""last_file_id"":500602,""models"":[""imagenet""]}",0,1670928257,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":11,""root_id"":31957,""override_root"":31958,""last_file_id"":1964427,""models"":[""imagenet""]}",0,1670928253,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
 ("{""storage_id"":11,""root_id"":31957,""override_root"":31958,""last_file_id"":1963949,""models"":[""imagenet""]}",0,1670928254,0,0,"OCA\\Recognize\\BackgroundJobs\\StorageCrawlJob")
(4 rows)

Is this correct and expected? For the context, am currently running Recognize on two nodes (with GPU though GPU portion is probably irrelevant), earlier was running on four CPU based and worry is that it might be due to this (see last_file_id being really far from each other, almost like the second crawl would be a dup?)

[marcelklehr]

Is this correct and expected?

This is not expected. You can remove the StorageCrawlJob instances with the lower last_file_id for the same storage.

[marcelklehr]

This can happen when you disable and enable a classification option multiple times in a row. I'll see what I can do about this

[bugsyb]

Thank you!