Medusa deletes every file

[VexingHex]

Describe the bug
For some reason, Medusa has decided to suddenly delete every single TV show and Anime that I had on my server that I have been collecting for the last 3 years. I have not changed any configs but it seemed it happened shortly after my server automatically updated the docker container for Medusa. There was no configuration change or any external action taken aside from docker pulling an automatic update.

To Reproduce
Steps to reproduce the behavior:

1. Update Medusa
2. Lose all media

Expected behavior
Medusa does not delete every TV Show and Anime in the media folder

Medusa (please complete the following information):

• OS: Linux-5.10.28-Unraid-x86_64-with (Docker Container)
• Branch: master
• Commit: unknown
• Python version: 3.8.10 (default, May 6 2021, 06:30:44) [GCC 9.3.0]
• Database version: 44.18

Additional context
I have been using Medusa for nearly 2 years and I'm honestly considering dropping it after this. I have never once had a single program cause this much damage, let alone one that has worked reliably up until this point. How is it even possible for something like this to get past the tests and the Devs and be put on the master branch? This is literally a breaking change that has cost every show I had, downloaded and watched by medusa or not, and I don't think I can recover them, even with a parity disk in unraid. Everything is gone...

[p0psicles]

Medusa should never remove shows.
There has been made a small change in the previous release 0.5.19 that impacted users with postprocessing and the move method. But that would only result in deleting left over files from postprocessing.

So to make it clear. We don't even have code that can delete episodes in the show folder.

Maybe you have a divergent config that caused this?

[VexingHex]

It is the same config that I have been using for nearly a year with no changes made to it recently. I have my post-processing set to move but there was nothing that was meant to be deleted. It appears that whatever changes were made caused it to delete things, even if that was not the intended effect. There are no "leftover files" left on my server. The unraid notifications and logs show that at approximately 8:09 am CST yesterday medusa was automatically updated, and shortly after every single file within my media directory. Not just the shows watched by medusa or downloaded by medusa, but every single file in all of the media folders medusa had access to. This has never happened before and only happened after the update was pulled in automatically. These are the exact settings I have on post processing that I have always been using and never once have they caused anything like this.

Nothing about them indicates that medusa should ever purge every file in the directory so there is something in the update that was pushed that caused this. I can not find any other explanation.

[p0psicles]

Wait? /shows isn't your final show store right? You do have show root folders setup?

[VexingHex]

/shows is a mapping through the docker container in unraid, it maps to where the finalized downloads of shows from qbittorent go to with 2 sub directories in it /shows/TV and /shows/Anime where shows are put acordingly. This is the way I have had it set for nearly a year with no issues until yesterday when suddenly everything disappeared.

[p0psicles]

I'm linking to this other issue. As you probably have the same issue.
#10023 (comment)

But I'll summarize.
The postprocessing folder should be totally separated from your shows root dir.

Postprocessing folder is a temporary location. Files are put there and after moved on to the shows folder.

And altough it might work somehow before, it was not a setup that was supported by us. So therefor this change we made, could not have been tested for this. As we not even knew users could do this.

So I feel very bad for your loss. But the only thing we could have done better here, is educated you guys better, or maybe added checks to not make this setup possible.

But I think we even have this mentioned in our wiki. If not maybe we should improve that

[p0psicles]

We reverted the change in a new release 0.5.20 today. I understand this will not give you back your data. Again sorry for that

[VexingHex]

That does seem to be a relatively similar directory structure to mine, which is how I would imagine most people have theirs laid out. I have no memory of ever reading in any of the documentation or the wiki that the configuration I have is "unsupported" and all it says in the wiki is that it is meant to take the files that are finished within the download client which is what I have set up. The directory was only containing files that were no longer seeding or downloading. There is nothing I have seen in the setting pages when setting up post-processing that ever indicated that.

From the Post Processing page:

"The folder where your download client puts the completed TV downloads.
Note: Please use separate downloading and completed folders in your download client if possible.
The Post processing dir is also used when your download client is running on a different machine. It will try to map a postprocessed folder to PP dir."

and

"What method should be used to put files into the library?
Note: If you keep seeding torrents after they finish, please avoid the 'move' processing method to prevent errors."

From the General > Misc Page:

"Show root directories
where the files of shows are located
These changes are automatically saved!"

I also have not seen anything in the wiki that said otherwise. My understanding was that it was meant to watch the "finished downloading directory" and then look for the shows that it recognizes the names of and rename them to the proper name, and that is the experience I have had up until yesterday. I am not sure how the changes made turned that into "delete every file that is found". I would hope that the update was reverted as I can only imagine how many others have their setups the same way since there is no clear indication of needing to do it differently, and while I am not sure why that would necessarily be something you could not have tested for I legitimately do hope that you work to prevent this from happening to others. I never had issues aside from maybe not being able to find an obscure show. I have had good experiences with this software and have stuck with and defended it when told I should use alternatives like sonarr, and I can no longer say that is the case. I have recommended it to many friends and now have to make sure they are using a different layout before their entire media library is deleted. I apologize if I sound upset, I know you didnt personally come over and press the delete key on my entire library, but the reality is I am moderately infuriated by this entire ordeal, not only because of the loss of nearly 4 tb of data, but because there was no clear warning or indication and that the setup I had has worked for my entire time using medusa with no issue. I still don't fully understand what specific aspect caused the shows to be completely deleted but you seem to know and have reverted it, and I hope that you take the time and effort to improve the documentation and checks to make sure something like this doesn't happen again.

[jameejay]

I have the same issue 10023, but with TrueNas > jails > Medusa. Woke up and all video files were gone. Luckily I just used the rollback feature to roll back my dataset and Medusa. All my show files reappeared.

I'm using mounting points from Medusa to a folder on my dataset.

[jameejay]

I since updated to 0.5.20 and the shows remain intact.

[jameejay]

Unfortunately, I am unable to use a mounting point that connects to a folder outside /root. Although, that could just be user error on my part. Fairly new to this stuff.

[OmgImAlexis]

Unfortunately, I am unable to use a mounting point that connects to a folder outside /root. Although, that could just be user error on my part. Fairly new to this stuff.

What do you mean?

[OmgImAlexis]

@jameejay you should have another directory passed through for the post processing directory.

I genuinely don't understand why any of you had this setup incorrectly like this. All downloaders down the same way so this isn't medusa specific. The place where your downloads go should never ever be the place you store your media. That's just common sense.

[jameejay]

@jameejay you should have another directory passed through for the post processing directory.

I genuinely don't understand why any of you had this setup incorrectly like this. All downloaders down the same way so this isn't medusa specific. The place where your downloads go should never ever be the place you store your media. That's just common sense.

I could be wrong, but Medusa sends it to Sabnzb which handles the download and processing?

Over on Sab, I have separate folders for incomplete/downloaded. From those temporary folders, the shows get moved to my PlexMedia dataset where /Shows exist.

That said, I'm pretty sure I have things set up incorrectly over at my TrueNas jails...

[jameejay]

I'm trying to understand/learn...

What is supposed to be in Post-Processing Dir where it says "The folder where your download client puts the completed TV downloads."? I assumed it's where the final processed shows live?

Side note, my Sabnzb has temp folders. I'm assuming "completed" means my main repository for all shows.

[OmgImAlexis]

Your downloader should have a working directory for itself where it stores the complete and incomplete they can either be in the same or seperate directories. The downloader shouldn’t move it to the final directory. That’s for medusa todo.

The place where plex, etc finds your media is where medusa should move it.

[nydave69]

Your downloader should have a working directory for itself where it stores the complete and incomplete they can either be in the same or seperate directories. The downloader shouldn’t move it to the final directory. That’s for medusa todo.

The place where plex, etc finds your media is where medusa should move it.

The downloaded would not want to have it's working files in the same directory as completed.
because when the downloaded finishes it move it to the completed folder that can be separated by type
I have a completed_downloads/MU and completed_downloads/CP
Medusa looks in completed_downloads/MU for files to process.

If you check the utorrent docs you will find where they specificily state that the working and completed folders need to be different. If they were the same then Medusa would try to process incomplete files.

[OmgImAlexis]

@nydave69 there are loads of different downloaders, deluge for example works fine with a single directory for processing/downloading and a single for completed. It all depends on the downloader and your setup.

[VexingHex]

I have spent the last 4 days completely re-working the structure and trying to recover and get things downloading and functioning properly again, I just want to confirm that this is what you claim would be a "supported" configuration.

The Mappings in Unraid:

The General Settings in Medusa:

The Post-Processing Page in Medusa:

(I am using "Copy" temporarily, I plan to change the "Processing Method" to "Move" to avoid using more space than needed once I know things are working and it delete every file again.)

Does this look like the setup that you are saying I should have been using for the last few years, and are there any issues that would come from this? A question that comes to mind would be if the processor is unsure how to handle a file or what its proper name would be, or I manually torrented something medusa was unable to find, would that cause issues, and would they still be moved to the "final" directory for plex to see, even if medusa doesn't know what they are?

[flick1999]

I just lost about 30TB of TV shows. I am beyond livid.

3294 Shows (2012 Active) | 56062 +74 Snatched / 229487 Episodes Downloaded - gone, all gone.

My Post Process set up is the same it's been for several years - grab from the /downloads, move to the /media/tv/<showname? folder.

I did notice that when I ran a manual post-process and selected "delete folders" as I always did, that it deleted all folders in the /downloads, even the ones that it didn't process. I thought that was odd, but didn't bloody think to go look into the show folders. Not until the wife said "uh, what happened to my show?" and I went and looked. And wept. And started drinking.

Seriously, man, WTF?

[p0psicles]

@flick1999 As already mentioned a couple of timtes. Medusa does not and has never deleted shows from the show root dirs, unless the user has configured their shows root folder in their PP folder (and that also has been resolved a release ago).

So if your PP folder is /downloads . And your shows root folder is /media/tv, then something else -unrelated- to medusa removed the files.

[flick1999]

So, I'll simply call bullshit:

• The server logs indicate the deletion happened after that update of Medusa that's discussed above
• I have three instances of Medusa running against three libraries. Only one is configured to run the post-processor; that's the one that deleted the files and that was the only library that was lost.
• Medusa runs out of dockers, so perhaps there's a mapping at that level that caused the issue
• I understand that correlation is not causation, but it's pretty damned suspicious that these files were deleted at the same time you introduced this change, particularly as I've not lost files like this in the 12+ years I've been doing this hobby, and this particular server system has been steady-state for the last 4-5 years

Regardless, it is what it is. I stubbornly clung to Medusa when my peers moved to Sonarrr, was happy I did so when y'all picked up what appeared to be an abandoned project and started making it better. I now regret that decision.

Thank you for attending my Ted Talk.

[p0psicles]

I'm just stating the facts. All user reported issues until now where (proven) to be user misconfigurations. That's why we only got a few reports.

As you haven't got into much detail or posted any logs, I can't pinpoint to your misconfiguration. But Medusa is not capable of removing show data except for this specific issue (which has been fixed for a few weeks).

So the sumarize, you could only have this issue when having your show final destination folders in the same location as your postprocessing folder and where using move as the postprocessing method while running that specific release.

I do feel for your loss. I can imagine it sucks to lose so much data, you saved over time. So I do really feel sorry for that

[OmgImAlexis]

Just going to mention the fact we have 100k+ people using this and 4 people here have had an issue. That's not even 1% of users.

This genuinely seems like a user error otherwise we'd be seeing this way more widespread.

[OmgImAlexis]

Also we're not trying to downplay this, yes it sucks to lose the media but no matter what we say it's not going to bring the media back.

[flick1999]

I get it, OmgImAlexis. We're all interested in figuring out exactly what happened. I've been quiet for a few days as I've had to dig out all of my old forensics gear to see if I can do a recovery operation, plus juggle my day job.

I'm pleased to say that it looks like I'll be able to recover most of the data. I'll know for sure when I'm done, which I project to take about 32 days. I have to do 22 drives one by one, and carefully move the recovery files to safe spots without jeopardizing the overall effort. (Let's just say thank god Best Buy has 14TB drives on sale for $199.99 right now for Black Friday!)

I was able to bring the system up to a point where I can check the docker mappings:

/tv - /mnt/user/TV/
/downloads - /mnt/user/FilingQueue/
/backups - /mnt/user/Backups/Medusa
/FilingQueue - /mnt/user/FilingQueue/
/unraid/ - /mnt/
/config - /mnt/cache/.appdata/medusa

Based on the discussion above, I currently suspect the issue lies with that next to bottom mapping, where the docker maps to the root of the mnt systems. That's definitely not standard, and my notes don't indicate why I put that in there. Perhaps a workaround for an issue in the distant past? I've been using this docker for Medusa for almost five years now, but I was using a different one prior to that before I migrated to LinuxServer.IO's, so it could be a setting that the prior docker required.

Oh, I should provide specifics, whoops. From https://forums.unraid.net/topic/53499-support-linuxserverio-medusa/, the support forum for this Docker and its Unraid template:

• Application Name: Medusa-
• Application Site: https://github.com/pymedusa/Medusa-
• Docker Hub: https://hub.docker.com/r/linuxserver/medusa/-
• Github: https://github.com/linuxserver/docker-medusa

Anyway, that's all I have for now. I'll continue to review logs but the focus right now is on data recovery. That has obvious priority over forensics. Pains me to say it, as I've spent a portion of my career doing deep disk forensics, but I'm having to stomp on evidence as part of my recovery efforts due to hardware limitations. I no longer have access to a full forensics lab where I could just do it all in parallel. I'm quite happy I still have my write blocker, though!

Oh, and yes, p0psicles, I use the "Move" method. The other methods didn't work properly in my configuration.

Thanks. Hopefully back with a good update in a few weeks.

[flick1999]

Just to close the loop on this - while I wouldn't say it was a misconfiguration, it was definitely a unique one. I have a second instance of Medusa running, where it takes Ended shows and moves them to a separate storage area called /TV.Archives. I do this to pull Ended shows out of the actively monitored for changes folders, and to allow Medusa to pull the latest show/episode information and subtitles.

What makes that pertinent to this issue is that I had the download folders on that instance set to the active TV folder structure. So, while it wasn't the same instance of Medusa pointing to the TV folders as Downloads, as warned against above, it was indeed an instance of Medusa pointing at an active TV folder structure. I can see that that instance did a periodic scan after it updated to the version with the "new method of deleting files" and that that is what resulted in /TV being deleted.

The data recovery operation is going reasonably well. I anticipate getting about 90% of /TV back in place, along with many lessons learned. For starters, I'm now leveraging the immutable attribute (chattr +i) to ensure that nothing can modify/delete/whatever any files without my express approval. That'll stop issues of this nature from occurring in the future, as the only thing I hadn't protected against was a "local to the server" utility running rampant.

I'm still rather curious as to what problem #9950 was trying to solve, but I'm glad that #10020 swiftly addressed it.

Medusa still meets my needs better than the other options out there, so I'll continue to be around.

Thanks.