Additional Parsers

[bun-dev]

I added a parser for mixtape moe. It works identical to webmshare.
Tested it on my tumblr and it seems to work.
@"https://my.mixtape.moe/" + mixtapeId + ".mp4";

[bun-dev]

I attempted to add a parser for MEGA. Please note this still needs work. Since mega handles urls differently, I found it best to use an existing api to handle it. MegaApiClient.

Currently, it can handle downloading shared folders and single links.

To-do:

-Pipe the progress bar into the GUI.
Right now there's no progress indicator for when a file downloads from Mega. The api does however allow for a progress hook which I have tested and works. I just need to make it work for TumblThree.
-Bandwidth notification. Mega only allows users to download 2GB per x hours. I need a way for TumblThree to let the user know they've used up their bandwidth.
-Setup async downloading.

I changed a few methods around to accommodate for mega. For example, an optional UrlType parameter was added to "ExternalVideoPost(string url, string id, string date, UrlType utype=UrlType.none) : base(url, id, date,utype)". I needed a way to differentiate normal direct links and mega links. There's probably a better way to handle it, but I was mostly just focusing on getting it to work.

[bun-dev]

update#2

Changes

-Removed urlType. Instead of having a urltype parameter, a simple check for mega urls in DownloadBinaryPost is made
-Async downloading for Mega.Reworked the mega parser to work with TumblThree's DownloadFileWithResumeAsync method by using the Stream from client.Download(link)
-Removed ProgressionStream.cs.

-Possible Bug

-If a file is deleted from the tumblr site's download folder, it doesn't seem to redownload it because CheckIfFileExistsInDB(url) returns it as true. I''m not sure if I introduced this or not.

To-do:

-wildcard selector for webmshare,mixtape moe and MEGA.
-bandwidth notification
-improve check for existing file for mega
-add mega urls to DB

[johanneszab]

-Possible Bug

-If a file is deleted from the tumblr site's download folder, it doesn't seem to redownload it because CheckIfFileExistsInDB(url) returns it as true. I''m not sure if I introduced this or not.

No, it's wanted behavior. Now you can move your files (to a larger disk/NAS), leave the index in place and it will not download the same files twice. My initial thoughts implementing TumblThree was to safe download bandwidth, not so much disk space.

Why would you delete a file that you just downloaded, and then want to re-download it anyways? That doesn't make any sense for me. Of course, you're testing, there it's okay, but some people already complained about it and wanted a "re-download"-function. For that, you can simple remove the blog (index), and re-add it, but that seems to be too complicate for some people. If all the downloaded files are still in place, nothing will be re-downloaded, except for the missing files.

[bun-dev]

Now you can move your files (to a larger disk/NAS), leave the index in place and it will not download the same files twice.

That makes sense. Since I was testing, I got confused on whether I broke existing parsers or not. It's good to see this was intended behavior.

Update3
GoogleDrive Parser added, using Google.Apis v3. It will download nested folder structures, files and so on. Should handle just about every way a drive link can be posted.(/file/,/folders/open?id=,etc). The code is very ugly for it right now, and does not currently use TumblThee's async method, nor is it added to the DB. I'll integrate that when I have some more free time.

The following additional parsers were added:
Uguu
SafeMoe
LoliTemel
Catbox
These all work identical to mixtape.moe(direct links), so no further changes should be necessary for them.

Other changes

• Added wildcard filter, so that any file type can be downloaded from file hosts which support multiple formats.

To-do

• Previous Mega issues.
• GoogleDrive code clean-up.
• General better way of handling non-direct link parsers, such as Mega, GoogleDrive,Onedrive,DropBox,etc.

[johanneszab]

Okay, so, I'm not sure about this patch since it becomes larger and larger. At least it looks like you've some real interest in this project and I don't want to scare you away, since I'm currently writing a paper at work (and thus at home too) and am really low in time. It has to be somewhat finished in 4 weeks, and we're no point close to that.

• So, first I'm really glad that someone has interest in help programming, and not just opening new issues.
• I can give you full access to the repo, then we're at least two people contributing code. And as I've said above, I probably won't do much for the next month as well. Thus you can clean up your implementation and prepare a new release If you feel like the code is good. Or add something else to the code.
• I never really liked adding more and more external websites/parser. I do see that it makes sense feature wise and to make people happy, but it always felt like an endless story to me, since there will always be someone that wants a different site to be added. So, if you want to do it, that's great. I think It doesn't make much sense me interfering with your code here anyways while you're still implementing things (mega, google, ..). I mean, it's not that there isn't anything else to do ;). I can do one of the several other issues/request then like the file renamer.
• Just make sure the code isn't getting too messy.
• How big is the zip now with all the external libraries?
• Can make your resharper code guidelines team-wide and add then to the solution? I actually used our work template here shortly, but didn't add it to TumblThree. Maybe it's better to add one otherwise someone else will come and force down it's template and the commits getting full with code style changes.

So, thanks again for your already made contribution!

[johanneszab]

I've made a new branch for the mega and google drive parser as there was still some work to do and I wanted a new release that fixes the proxy issue (#204).