Skip to content

History

Jump to bottom
Ben Adrian Sarmiento edited this page Feb 1, 2024 · 28 revisions

31.01.2024

well here we go again v0.9.3-hotfix.6 but this time I'm pretty sure I've squashed the fucking refresh bug. it was an edge case with repairs and the race condition it introduces. i've added semaphore locks here and there and that caused some lockups on a condition you only get with repaired torrents.

i'm gonna focus on dmm for now... i'm really sick of looking at people's zurg logs...

What's fixed

  • refresh!

What's removed

  • none

What's new

  • downloads_every_mins to configure how often downloads mount should be refreshed (default is 60 mins) this is only relevant if you've set enable_download_mount to true

30.01.2024

v0.9.3-hotfix.4 I was hoping we won't reach this high hotfix# but here we are

What's fixed

  • hopefully refresh as i rework repairs again

What's removed

  • expose_full_path because it was causing ID issues

What's new

  • none

Re v0.9.5-hotfix.5 Hey, just to explain the differences:

Hotfix4 isolated worker pool for refresh and getting info and repairs

Hotfix5 unifies all worker pools into 1

Thats the only thing i can think that is affecting users not getting new content

28.01.2024

After 18 hrs of zurg work, a couple of DMM downtimes in between, 1,5k line changes, we got ourselves v0.9.3-hotfix.3! i'm not ignoring your messages people, just wanted to send this out before the weekend is over. i'll attend to your messages tomorrow

What's fixed

  • repairs, ofc, the main reason why we got this hotfix
  • plex scanner being stuck! i had them but i no longer experience them so report if you still do
  • speed ? try just setting up a new library and benchmark. i had mine scan 100 movies in under 10 mins

What's removed

  • preferred_hosts in the config: causing more issues than it should
  • use_download_cache in the config: also causing issues
  • realdebrid_timeout_secs in the config: replaced, see below

What's new

  • added easy way to share your logs (when reporting issues on Github)
  • api_timeout_secs in the config: API requests timeout, changed default from 60s to 4s (increase this if you're having API timeout issues)
  • download_timeout_secs in the config: download requests timeout, default is set to 2s
  • repair_every_mins in the config: repair is now triggered on demand by broken torrents or by schedule as set here. Default is 60 mins.
  • expose_full_path in the config: this helps plex or jellyfin or infuse identify content better. Might break *Arrs if enabled.

24.01.2024

zurg v0.9.3-hotfix.2 release:

It fixes some connection issues caused by tcp keepalive connections and Real-Debrid throttling zurg users. I set a limit there to conform with Real-Debrid's imposed limits. Also proactive repair is fixed so when it fails to check for certain torrents, zurg won't consider the torrent uncached and attempt to repair it even if it was an api failure. The logging bit of repair is also more descriptive now so you don't assume what's happening. Debug level logging has also been set as default and will stay until zurg reaches 1.0.

A lot of zurg issues are being reported on all directions and it has been hard to track them. Different discord channels and DMs, reddit posts and chats, and also on Github - it has been challenging trying to correlate all of them and the speed on fixing these issues is affected. I hope it's not a big ask to file everything on Github https://github.com/debridmediamanager/zurg-testing/issues/new/choose

Thanks all!

20.01.2024

zurg update: v0.9.3-hotfix.1

  • fix repair issues
  • fix issues on marking torrents as unplayable (seems to be a glitch on realdebrid api but this is now handled)
  • fix routing issues with Infuse (please use :9999/infuse/ instead of :9999/dav/)

16.01.2024

v0.9.2 was received tremendously by the community and I'm often seeing my work on places I didn't expect it to be. Today, after making sure all of the added functionalities work as it should be, I'm releasing v0.9.3! More than a month's worth of work has been poured into this and a lot of feedback from many, many Plex server admins have been incorporated. I promised renaming files/moving files/artificial sorting to come to this version but the refactor needed to pull it off might cause too much instability for this version. Hence, I decided to release now rather than later. Enough blabbering! Here are the changes:

Proactive cache status checking. Every torrent in your library is continuously checked if it is still cached in RD or not, enhancing availability for all your files. If you have "repair" functionality enabled, it means your torrents will always be playable as long you keep zurg running.

"Unplayable" directory. If you're annoyed by the many rar-related log messages, or you also use Real-Debrid to download other non-video content like games or books, then expect those torrents to be found on __unplayable__ (instead of movies) from now on. Just to clarify, "rar'ed" torrents are no longer hidden, they're now sent to this directory.

CDN (real-debrid.cloud) streaming fix. If you've experienced frequent "playback restarts" when using RD's CDN hosts, then this should now be fixed.

Fix only_show_the_biggest_file. I claimed in many occassions that this feature is working only to find out recently that I am using an unreleased version. It should now work as expected. Also, when deleting the only file shown in this torrent, it will also delete it in RD.

Add more file filters only_show_files_with_size_lte and only_show_files_with_size_gte. "lte" and "gte" means "less than or equal" (greater). In a certain directory, you can add >= number_of_bytes or <= number_of_bytes to get rid of the noise.

The on_library_update hook now works on Windows. It's using Powershell. Please send me your working Powershell scripts so I can add them in the repo.

Support for a network proxy has been added. In your config.yml you can add proxy: "socks5://[username:password@]host:port". HTTP, HTTPS and SOCKS5 proxies.

Downloads directory. All your unrestricted links can now be found on a new directory __downloads__. A lot of users requested this but honestly, I do not see the point. But hey, it's there now!

version.txt. You'll find a new file when you mount zurg that contains version information. Now, you can check if the mount is ready or not by checking for that file's existence.

New zurg utility commands: clear-downloads and clear-torrents. Only usable from the command line, it clears all your unrestricted links and your whole library in seconds (it doesn't send thousands of request to RD so you don't need to worry about rate limits).

Handle EOF errors, empty RD accounts. zurg now supports the neophytes!

Handle non-premium users and add warnings when your premium is expiring. zurg won't start anymore if it detects that you are not a premium user (instead of displaying confusing errors). zurg tells you 48 hours before expiry if you need to renew to prevent risk of losing your library. Backup often in DMM to be safe!

Username+Password protection. A lot of people are mindlessly exposing their zurg servers on public IPs so in order to prevent leaking stuffs, you can add username: and password: on your config.yml.

No need to delete data folder when changing values of retain_rd_torrent_name and retain_folder_name_extension. I also suggest the *Arr stack people to read the config.yml.example.

Network tests for ipv4, ipv6 and both. Run zurg network-test -h to check. I actually like this feature in particular because it load balances RD better.

Did I already mention that the repair functionality was reworked again for the Nth time? I recently found out that reinserting != repairing torrents in Real-Debrid. It works but only when another Real-Debrid user downloads that one broken file (e.g. an episode in a season pack) in RD which refreshes that particular file in RD's cache and so now when you reinsert the torrent, it picks up the newly cached file. Now, zurg does this all the time. Note that when zurg is repairing files, and you restart zurg, it loses track of the torrents it needs to delete and reinsert for repair.

Well... that's it! You've reached the end of the release notes. It's a long one this time but there's still more to come on future versions (renames and also a feature that will no longer need you to do renames). Happy 2024!

09.12.2023

rdt-client users: I have found a fix to handle waiting_files_selection better! It should work now in v0.9.2-hotfix.4

07.12.2023

Released v0.9.2-hotfix.3 to fix the repair issues once and for all!

Also added a way to specify log level with LOG_LEVEL env var e.g. LOG_LEVEL=WARN will only output warnings and errors, etc. It's INFO by default, set to LOG_LEVEL=DEBUG when debugging issues

06.12.2023

v0.9.2-hotfix.2 is now available

  • Fix the rar repair issue (if you have experienced torrents that are being auto-rar'ed by Real-Debrid no matter what files you select)
  • Added auto_delete_rar_torrents to auto-delete these auto-rar'ed torrents (since there's no method to fix them as of yet)
  • Added /logs endpoint and logs are also kept in files

I've released v0.9.2-hotfix.1

Docker image: docker pull ghcr.io/debridmediamanager/zurg-testing:v0.9.2-hotfix.1 Binary download: https://github.com/debridmediamanager/zurg-testing/tree/main/releases/v0.9.2-hotfix.1

  • file listing update issue fixes
  • file delete issue fixes
  • added size_gte size_gte any_file_inside_size_gte and any_file_inside_size_lte filters
  • debug endpoint now contains loaded config
  • on library update hook shows the output again

Just try it and lmk

02.12.2023

This is it! v0.9.2 is now available. It's a 2.7k line update which comes with breaking changes (read the instructions carefully in order to not lose your library). I just really wanted to move on from this memory leak issue so that we will have a peaceful binge watch for the holidays with the fam.

Enjoy!! :snowman2:

Step 1: stop Plex server Step 2: stop zurg rclone mount Step 3: download the new version

if you are using docker:

docker pull ghcr.io/debridmediamanager/zurg-testing:v0.9.2
docker compose down

if you are using the binary:

wget https://github.com/debridmediamanager/zurg-testing/raw/main/releases/v0.9.2/zurg-v0.9.2-linux-amd64.zip
# or just get them from the repo

# and then remember to stop zurg!

Step 4: delete the contents of your zurg's data/ directory Step 5: if you are using webdav, change webdav rclone config (rclone.conf) and point it to http://localhost:9999/dav/ Step 6: start zurg Step 7: Go to http://localhost:9999/ to confirm if zurg is running Step 8: start rclone mount again Step 9: Check if the mount is working first ls -1R /pathtoyourzurgmount Step 10: If everything works, start Plex server again

So as of v0.9.2 ...

Here are the only relevant configs for zurg, the rest have been removed:

  • host
  • port
  • concurrent_workers
  • check_for_changes_every_secs
  • enable_repair
  • on_library_update
  • retain_folder_name_extension
  • retain_rd_torrent_name
  • preferred_hosts
  • network_buffer_size
  • serve_from_rclone
  • force_ipv6
  • realdebrid_timeout_secs
  • retries_until_failed
  • use_download_cache

Yes use_download_cache is now disabled by default

Output logging has been remove from on_library_update hook Deleting torrents also trigger on_library_update hook

Fixed the issue on deletes not working Also apparently the file deletes aren't retained properly across zurg restarts, it should be working now BUT you will see the file you have deleted previously unless that triggered the whole torrent being deleted (by deleting all the files inside)

The router has been completely reworked to handle more concurrent requests, that's why I had to change the dav route

Torrent manager is the god class in zurg and has been completely reworked

You can now see the version if you run zurg version Or you can also pass the config file path now with flags, check zurg -h

A lot of performance tweaks here and there...

Please please please report issues here with all the details you can provide! https://github.com/debridmediamanager/zurg-testing/issues

28.11.2023

v0.9.1-hotfix.2 Reliability hotfix also on :latest

As usual docker pull ghcr.io/debridmediamanager/zurg-testing:latest

  • Added new configs (optional since there are sensible defaults set)
# reliability configs
unrestrict_workers: 10 # since unrestricting has a different rate limit, use a different worker pool. decrease this if you are getting 429s
release_unrestrict_after_ms: 100 # wait time for every unrestrict worker to be released. increase this if you are getting 429s
rate_limit_sleep_secs: 6 # wait time after getting a 429 from Real-Debrid API
  • Fix __all__ not loading properly, containers should not go unhealthy anymore
  • Fix memory leak issue and optimize general memory profile
  • Fix Plex scanner being stuck issue (run this to check your system for stuck scanners ps aux --sort=-time | grep "Plex Media Scanner")

if you pull the latest image (i just pushed a couple mins ago) i added these 2 configs to handle those unusual cases of realdebrid api acting up

realdebrid_timeout_secs: 60 # api timeout
retries_until_failed: 5 # api failures until considered failed

these are the default values when you dont set them

Anyone having issues on Real-debrid CDN hosts (ends with .cloud) should pull the new hotfix.2 update because the previous hotfix.2 images had issues with Cloudflare connections.

26.11.2023

Just released v0.9.1-hotfix.1 (also latest) because I saw that Plex won't play error videos because the error video file size and original content file size is different and then it gets "stuck" for about a minute trying to read the faulty file (also rclone showing unexpected EOF errors because of the file size mismatch). It should no longer happen and Plex can scan expired files faster.

Also fixed repairs not triggering on unrestrictable links when I reworked the code there to fix the issue where zurg gets stuck when there are more expired torrents than concurrent_workers. It should now trigger properly.

No changes in configs necessary, just upgrade the image

25.11.2023

v0.9.1 released! docker pull ghcr.io/debridmediamanager/zurg-testing:latest or :v0.9.1

Changelog:

  • Fixed health check issue (if you still see zurg as unhealthy after upgrading, restart it)
  • Before it throws you an error video, zurg's data fetcher will retry up to 5 times now
  • zurg now runs on the main CPU thread to maximize performance
  • Added retain_rd_torrent_name in config.yml, set to true to make it truly compatible with rdt-client
  • Repairs should now prevent torrents from being renamed
  • The config on_library_update gets passed the updated directories now as arguments `
  • Added serve_from_rclone in config.yml, set to true if you want zurg to just pass the file URL to rclone for downloading
  • Added force_ipv6 in config.yml, set to true if you want zurg to connect to Real-Debrid's IPv6 address
  • In progress torrents no longer gets listed in directories
  • Adjusted the modification times displayed for torrent directories and files. Files will now show ended property while directories show added property.
  • Timezone issue fixed for modification times
  • The rclone http mount now displays modification times for files

Moving forward, unless a hotfix is needed for a critical bug in zurg, I am adjusting the release schedule to be every 2 weeks to give me more time to work on other projects. Enjoy!

21.11.2023

🌈 IT'S BEEN A MONTH SINCE ZURG'S FIRST RELEASE! 🥳

You guys remember zurg-1000? 😅 That's exactly a month ago!

Today's special so here is a special release!

Binaries for all platforms

Check out the latest releases on https://github.com/debridmediamanager/zurg-testing/tree/main/releases/

Deletes now work on webdav

You can delete a torrent , and...

Deleting individual files also work Just make sure you retain your data/ directory and delete info_cache_time_hours in your configs, it's not needed If all files are deleted inside a torrent, the torrent is also deleted

Webdav now works as fast as http Lots of optimizations were done and also, the key update: Just add pacer_min_sleep = 0 in your rclone.conf

[zurg]
type = webdav
url = http://zurg:9999
vendor = other
pacer_min_sleep = 0

Note: be careful on editing rclone.conf as your mount might not work properly, best to stop Plex server first if you dont know what you are doing or your library might reset itself

Several new filters are added!

not_regex
any_file_inside_not_regex
any_file_inside_not_contains
any_file_inside_not_contains_strict

and possibly the best filter ever (can only be topped by anime: true)...

has_episodes: true

which is a detection algorithm of episodes found inside a torrent

my config.yml looks like this now

directories:
  anime:
    group_order: 10
    group: media
    filters:
      - and:
        - has_episodes: true
        - any_file_inside_regex: /^\[/ # e.g. [SubsPlease]
        - any_file_inside_not_regex: /s\d\de\d\d/i
  shows:
    group_order: 20
    group: media
    filters:
      - has_episodes: true
  movies:
    group_order: 30
    group: media
    filters:
      - regex: /.*/

Sponsors

THANK YOU!!! you know who you are Thanks to vanderruss for sponsoring me on github My patreon page is also live 🎇

19.11.2023

Hey all! I had several fuckups on the releases recently. The latest tag was pointing to an unstable code 😨 for a long time and I think a lot of people tried it and obviously got a bad experience with setting up.

I've fixed this issue and did a ton of rework on v0-defuse to again resolve the very difficult programming problem of race conditions and hopefully the latest image should be at par if not better than the previous versions. ⛅

I've also complete reworked the repair functionality :tools: and it shouldn't recreate gazillions of torrents. The repairs happen on the hash level now (for maximum efficiency). If one file is broken inside a group of torrents with the same hash, it will check the whole directory for any other broken files and it will only redownload what is broken (it doesn't reinsert torrents like what others do because that is not 100% guaranteed to repair broken files). The health check has been removed because it was also causing issues for libraries that takes a long time to get complete information for.

FYI http is faster than webdav by a factor of 4.5 (i compare zurg by itself now, lol)

p.s

(I really need to start focusing on the fuse work or else zurg will never get to v1 aka climax)

I'm sleepy & tired but cannot forget to thank @godver3 @nometaisbeta @yeungdrizzy for being amazing project sponsors! Also first time I saw zurg being put on a guide written by @thebropuks which is just mind blowing for a month-old software to make that much impact. 🤯

Enjoy the rest of the weekend y'all!

11.11.2023

New release! latest or v0-defuse doesn't mattery for now

I've reworked every part of the architecture

the repair function is fully rewritten, it should be behaving properly now :slight_smile: Added a proper http client with retry/unrestrict

this setting is now fixed retain_folder_name_extension , it is now working to make it more compatible with rdt-client

and I added preferred_hosts! As to how to determine what to put on preferred hosts list, check my script here https://github.com/debridmediamanager/real-debrid-network-test/blob/main/real-debrid-network-test.sh (you can put as many as you want)

I can finally work back again on the fuse branch. Time to sleep!

07.11.2023

^ I've created a hotfix for this, please reconfigure your docker-compose files and use this image instead

docker pull ghcr.io/debridmediamanager/zurg-testing:v0-defuse
  • It contains the logging improvements (no way to configure logging level yet)
  • race condition fixes making it more stable
  • the repair functionality apparently can't be disabled entirely so I've fixed it here as well

(no fuse mount yet, hence defuse because i made an effort to remove it as it's not yet ready)

06.11.2023

Hi all, i've recovered and am back working on the project again :slight_smile:

Some updates before the weekend: i haven't reached the point of working on the renames and deletes yet, but the mounting works, and can see the directories and files. it is again magnitudes faster because no translation of data is happening between protocols as we don't need rclone as a middle layer anymore.

I've also improved the way logging works and log level should be configurable now.

There are also several race conditions reported by different users that I've resolved. Again the directory mapping for torrents should be more stable on the next release.

(the http and webdav ports were untouched so you could still use that as a "stable" fallback)

The problem I'm having now is the way bytes are read inside a file. If using http or webdav, large chunks are requested at a time. This is great because when the connection closes, zurg can immediately stop requesting the file from RD servers. No bytes wasted. If using a filesystem mount, the OS controls the read buffer and it seems to be just 1kb chunks? It's super inefficient so I'm currently working on my own implementation of a cache layer that divides the whole file into parts and parts are loaded as needed. This is the main blocker of an early release.

Anw, hope everyone had a nice weekend!

01.11.2023

:atom: zurg progress update

eta on the rename/delete function is by end of this week it will only support linux for now, and you will no longer have to use rclone

i'm collecting non critical bugs that will also be included in the release

zurg will also be released as a binary apart from the docker image

29.10.2023

Update for today:exclamation: I have fixed this issue on the latest release that's available to be pulled now

  • any_file_inside_regex and other any_file_inside_* filters works now
  • added network_buffer_size in config.yml to let you configure the best buffer size that's most compatible with your internet speed. there's a perfect setting here that depends on MTU, connection speed etc. so I made this configurable.
  • the issues affecting playability of the files (the files being unplayable, etc.) should now be fixed

it should make things a lot more stable!

28.10.2023

A new release should be available in the next 10 mins

  • repaired files creating a different folder should not happen again (no need to delete anything here for this fix to work)
  • if same filename and size, then it deduplicates the files inside a folder (shows you the latest file)
  • directory mapping issue fixed, thanks @Deep-value !

I tried making an improvement on the responsiveness of the playback. If this is something you're interested on testing, pull the latest image :slight_smile:

This helps even if you're using filebot

New release ⚠️ Pull the latest image, it now supports hooks to library updates! (bash scripts)

Add this on your config.yml if you want your Plex libraries to always be updated :slight_smile:

on_library_update: |
  token="Your X-Plex-Token Here"
  plex_url="http://plex.box"
  sections_xml=$(curl -s "$plex_url/library/sections" -H "X-Plex-Token: $token")
  section_ids=$(echo "$sections_xml" | grep -o 'key="[0-9]*"' | awk -F'"' '{print $2}')
  for id in $section_ids; do
      curl -s -X POST "$plex_url/library/sections/$id/refresh" -H "X-Plex-Token: $token"
  done
  echo "Plex library refresh triggered!"

26.10.2023

please pull the latest release that will be available in 10 minutes

docker image inspect ghcr.io/debridmediamanager/zurg-testing:latest | grep revision

should be 0fd2fa691e1c32aec9f7762274fc9941e9313925

auto repair is done :slight_smile:

25.10.2023

https://github.com/debridmediamanager/zurg-testing new release available in a few mins

24.10.2023

please test the latest release. unfortuntely i haven't created a proper docker release pipeline so every release uses the latest tag for now

another change again for rclone.conf https://github.com/debridmediamanager/zurg-testing/blob/main/rclone.conf no_slash = false

the new release includes repair functionality, tested on a few use cases

23.10.2023

ok i think i have fixed the issues on the http mount mainly

  • file metadata - file type and file sizes are now properly set (you need to update rclone config again and set no_head to false
  • when updating the torrent list, all directories goes empty and then back to normal again (or sometimes the directory mapping just fails)

The docker image is building now and will be available in a few minutes

change from zurg-1000 to zurg-testing

it's built now, docker pull ghcr.io/debridmediamanager/zurg-testing:main https://github.com/debridmediamanager/zurg-testing/blob/main/README.md

22.10.2023

pushed a new release, i recommend everyone to pull that image and update your rclone configs

if you are using a dockerized rclone, the new config should look like this

[zurg]
type = http
url = http://zurg:9999/http
no_head = true
no_slash = true

i have now heavily tested this with the rclone usecase

benchmark based on https://github.com/debridmediamanager/zurg-testing/blob/main/experiment.md

it's around 5-6x faster

it seems davfs2 is crazy fast though based on @kakasanti 's tests :slight_smile: thank you!

21.10.2023

@here https://github.com/debridmediamanager/zurg-1000

I think i will remove the stupid limit, it was supposed to send the message that this is a beta/test version so people will provide feedback, but i think it will make the test comparisons look silly

(c) 2023 Debrid Media Manager

Clone this wiki locally