[bug]: Unexpected image loss and deletion failure

[MylesCroft]

Is there an existing issue for this?

• I have searched the existing issues

OS

Windows

GPU

cuda

VRAM

8Gb

What version did you experience this issue on?

3.6 RC 5/6

What happened?

Upgraded to RC6 from RC 5 which resulted in broken thumbnails and missing images. Upon investigation I found:

I have had around 1,500 images in the images folder that should have been deleted some dating back as November, 2023. These are not intermediate files as the majority was generated with no adaptors/canvas/etc and I clear the intermediaries regularly.

Images from a working gallery was missing ~200 images alongside their thumbnails.

My workflow is to

1. Create a scratch gallery for generated images
2. Create a final project or organised folder on theme

I will move selected individual images from the scratch folder to the final folder and then delete the scratch gallery to reduce drive clutter and free up space. For some reason Invoke has failed to do this since last November for whatever reason.

The missing images are from December and the only difference is that I upgrade to RC 5 at this point as I did not do much rendering of images during this period.

I have attached an image of the broken galleries (with two boards showing My Board) and also of the workflow I normally use.

Upon initial discovery I reported the issue and was advised to run the db repair script, and when the images was not restored I investigated the images folder which is when I discovered the above issues.

Screenshots

Additional context

No response

Contact Details

No response

[psychedelicious]

I understand you also ran a db maintenance script. I assume that's the one that is bundled with the app. At what point in this process did you run that?

---

I want to be very clear - this issue involves an observation of the state of your data after you did a number of things:

• Cloned your disk
• Upgraded to the RC
• Ran a db migration script

There are two major assumptions here:

1. That "Invoke has failed to [delete ~1500 images]"
2. That "[Invoke unexpectedly deleted] ~200 images alongside their thumbnails"

The issue not "invoke messed up my images". It's just that "my images are messed up after these things occurred". There's a big difference between those two statements.

I'm being a stickler on this because I don't want to give anyone the idea that invoke doesn't handle user data carefully. That's not the case.

The behaviour you are asserting occurred is very unexpected and this is the first report of anything like this happening (to my knowledge). For these reasons, I suspect other confounding factors contributed to your observation, and the Invoke app didn't go rogue on you.

[psychedelicious]

I will test deleting a board and review that code to see if I can find a spot where the delete could fail. We use a utility called send2trash which is a cross-platform wrapper for deleting files by sending them to the OS's trash area. It's possible that this didn't do its job.

---

I don't have a way to test for "invoke unexpectedly deleted 200 images", because there's no code that could do that. The only time image deletion logic is called is when the API routes are hit, and that doesn't just happen randomly.

The methods that delete images are called in 4 places, in API route handlers:

• The handler to delete an individual image (by image name)
• The handler to delete a list of images (by list of image names)
• The handler to delete intermediate images (which first queries the db for all intermediate image names, then deletes them one by one)
• The handler to delete all images in a board (which gets all image names on a board, then deletes them one by one)

All of these areas require a correctly formed payload to be included in a network request. Explicit user action is required to call any of those routes. I don't think Invoke went and deleted any images without your direct user action.

[MylesCroft]

I understand you also ran a db maintenance script. I assume that's the one that is bundled with the app. At what point in this process did you run that?

I want to be very clear - this issue involves an observation of the state of your data after you did a number of things:

• Cloned your disk
• Upgraded to the RC
• Ran a db migration script

There are two major assumptions here:

1. That "Invoke has failed to [delete ~1500 images]"
2. That "[Invoke unexpectedly deleted] ~200 images alongside their thumbnails"

The issue not "invoke messed up my images". It's just that "my images are messed up after these things occurred". There's a big difference between those two statements.

I'm being a stickler on this because I don't want to give anyone the idea that invoke doesn't handle user data carefully. That's not the case.

The behaviour you are asserting occurred is very unexpected and this is the first report of anything like this happening (to my knowledge). For these reasons, I suspect other confounding factors contributed to your observation, and the Invoke app didn't go rogue on you.

I have four disks in this server, OS, Data, Scratch and AI. All are separate media drives 3 SSD and 1 HDD (Data) with the AI SSD being 512Gb and has 300Gb free space. The AI media is further disconnected from Windows OS in that I do not run any processes such as back up or indexing on this drive instead doing backups manually using a simple copy batch file to a NAS. The OS disk was flagged as a TRIM warning indicating the drive was reaching EoL which was replaced as a precautionary measure and was a one-to-one clone with no integrity errors during the process or a chkdsk validation post-cloning. I also had not run the back up script since November, which is why the images was lost so that is on me slacking here and to stress I do not hold Invoke to account for images being lost.

The disk was replaced after RC5, but many of the images was rendered with RC5 but since they are missing I can't confirm that.

I ran the database repair script after upgrading RC6 as per advice on Discord, as I had completely forgotten it existed. The upgrade to RC6 highlighted the missing images as noted.

So in essence:

Updated to RC5, and continued using Invoke when I had limited time to do so.
A week or so after this I got a warning that the OS drive was reaching EoL and was replaced that day using Macrium to clone the drives.
The images gallery was still available after this, as I posted some on the Discord group and certainly up to last week when I was referring to their metadata in the interface.
A few days ago I updated to RC6 and noticed the broken thumbnails, I raised this as an issue on Discord thinking it was a similar issue I experienced in the with the UI which a web reset would fix it but didn't. I was advised to run the DB script to fix things and that's why I noticed there was no images in the galleries.

I proceeded to to a further investigation of the Invoke images folder to recover the images to find that all images from mid-December so probably RC5 images was missing from the gallery (skipped most of the 3.6 RC due to time restrictions) that was created in this period which is roughly around 200 test images.
I also found that around 1,500 orphaned images from early November onwards had not been deleted from several galleries despite the galleries or individual files being deleted.

I do not want to cause unnecessary alarm on this and understand your stance especially as I am an RC tester so could be an side effect of being part of this. However it should be noted I also take data integrity very seriously on my rigs and this is the first major data loss I have experienced in nearly 20 years and extensively looked over any tasks or procedures during this time that could have influenced data loss, the reality is that the server was offline for much of December other than the days I needed to do some renders. I had initially thought it was just a simple case of broken links in the database due to the upgrade but it turned into a larger issue which is where we are at now.

At this point I might just move all images out from the Invoke folder and rebuild the database from scratch, just would be a be a pain to pull required reference images back into Invoke.

** Robocopy command used for reference in case you think it's due to /MIR

Robocopy "Source" "Target" /E /COPYALL /SEC /MAXAGE:45 /R:5 /ZB /mt:16