Features: multithread and tqdm progress bar

[darodi]

• contains PR Fix of mangaFox to work with parallel downloads & added the missing sites to have parallel download #161
• mulithread added for mangaEden
• tqdm progress bar for chapters
• tqdm progress bar for files (+ requirements for windows) (removed, better display without it)
• tqdm progress bar for auto download (indicates the comic/manga being processed)

tqdm changes based on #159
fixes #123
fixes #153

[dsanchezseco]

@darodi same as I said in #153 but this is kinda more psychedelic when printing the page progress. I don't know if it's just on linux, on wide monitors or what.

Edit: the missing link to the output

[darodi]

Thanks for the feedback.
Did you install colorama?

pip install colorama

I had this kind of behavior when I didn't use it with Windows.
I'll try some tests using Linux this evening.

[dsanchezseco]

@darodi i've installed and there's no difference. I'll check this afternoon on Mac too

[darodi]

@dsanchezseco

Here is the result using ubuntu 18.04 wsl
with my last commit

https://asciinema.org/a/KKPjATHIAbCOpVb8PITz76itW

trying you example I get

https://asciinema.org/a/dBP34HkoxHvxIq5vw2Na8v1KC

It seems better.

However, doing some tests I noticed a few new problems with the multithread introduced in PR #161

I ran 2 multithread downloads on the same host for 2 different mangas.
Both downloads where running with 4 threads. The host could not handle 8 threads.
Each manga was getting 503 errors on the host.
Threads where continuing instead of stopping. Eventually, I got 2 mangas with missing pages.
I think the error handling should be changed. The chapter download should be stopped.

Another consequence of the multi thread is the fact that the abort "ctrl-c" isn't working anymore.

Regards,

darodi

[dsanchezseco]

@darodi Hi, nice.

Maybe leaving only the overall chapter bar instead of showing too the page bar could look cleaner and avoid the strange printing.

About the run you did, did you run two python main.py -.... ? If so, yeah, for now the amount of threads that it uses is fixed to 4, I was waiting to merge to add it as a param for the different computers that don't support so many threads or can support more.

Keep in mind that having more threads can potentially trigger the bot countermeasures the pages could have(as cloudfare for example) and give you the 503. Is open on issue #134 .

Regarding the ctrl+C i'll try to enable to cut the execution when that happens.

Thanks for the work!

[darodi]

@dsanchezseco

new commit. I disabled the progress bars for each file.

The issue I was pointing out is the error handling.

Let's say I'm using comic-dl in auto download mode for 10 comics.
Each comic has 5 chapters to download with 10 images each.

Currently, if there is any error downloading the 3rd image of the 4th chapter of the 5th comic,
other threads will continue downloading the remaining 7 images.
The parent process, for the 4th chapter, isn't notified of the error.
comic_dl.globalFunctions.GlobalFunctions#addOne will be called
and the 5th chapter will be processed, even though there is an error and a missing image in the 4th chapter.

In that case, it would be better not to call comic_dl.globalFunctions.GlobalFunctions#addOne, stop the download before the 5th chapter and proceed with the 6th comic.

[dsanchezseco]

Awesome!!! https://asciinema.org/a/3L8vAgrYGKdl8Pcw86MYen4tN

I'll try to work on the variable thread number and the error handling ASAP

[dsanchezseco]

Nice refactor of the multithread, I did it on a hurry changing all the files d'oh!

I would remove the commented code though

[darodi]

@dsanchezseco , @Xonshiz

For the moment, for each chapter, 4 threads are created to download the queue containing images.
For each chapter, there is a small overhead for creating those threads.
I was wondering if we couldn't improve that.

i.e.

• Create the 4 threads on the first chapter to download.
• feed a new queue on each chapter.
• wait for the end of the queue on the main process
• don't destroy the threads in the end of the chapter but continue with the next chapter and feed a new queue to the existing threads.
• the threads will be destroyed in the end, after the last chapter of the last comic

I must admit I haven't checked the time taken to create the threads.
That proposal would also prevent to specify a different number of threads for each host/site.
I don't know if it's worth it.