Hotfixes for YouTube and improve unavailable cases

[mauriciocolli]

Fix the issues that I mentioned in #262, and some more that I found along the way.

---

YouTube

• Take into account videos that have their views hidden.
• Handle videos with no views or with "Recommended to you" text.
• Handle video premiere's date and duration.
• Avoid crashing by letting exceptions bubble up.
  • reCAPTCHAS, for example, were not reaching where they should, resulting in a crash instead of opening it for the user to solve.
  • Maybe there's a better way/approach to how the exceptions are handled in the extractor (e.g. communicate error to front end, error propagation), I'll take a look some time.
• Fix bug when url isn't present in the browseEndpoint object.
• Detect simple 404s in the standard fetch method.
• Detect deleted/nonexistent/invalid channels and playlists.
• Detect when a stream is deleted or doesn't exist.

Downloader

• Make test downloader return a response instead of throwing an exception.
• Add latest url to the response to make detection of a redirect possible.
• Ignore null-keyed entries when iterating through the response headers.

[wb9688]

Thanks a lot for your PR! I wanted to implement the things you wrote about, but now that's not needed anymore. I'll take a closer look in a few hours.

[wb9688]

I just found another bug, which could be fixed by looping over all badges in getStreamType() in YoutubeStreamInfoItemExtractor. Could you please do that?

[B0pol]

Are there no other ways than relying on English?

[TobiGr]

At the moment there is no other way. We might need to check whether the value contains specific words like "no" or "you", which should be quite similar in most languages. In that way, we do not need to make a list for all translations later, but can reduce that to a minimum of items to check. But that ca wait for a later PR.

[TobiGr]

I just found another bug, which could be fixed by looping over all badges in getStreamType() in YoutubeStreamInfoItemExtractor. Could you please do that?

@wb9688 Do you mean something like

JsonArray badges = videoInfo.getArray("badges");
for (Object badge : badges) {
    if (((JsonObject) badge).getObject("metadataBadgeRenderer").getString("label").equals("LIVE NOW")) {
        return StreamType.LIVE_STREAM;
    }
}

[wb9688]

@TobiGr: That's exactly what I meant.

[TobiGr]

@wb9688 I could not find any StreamInfoItem which has two badges while running the tests from #266. Do you have a link to a document which contains such a case? I found one.

[B0pol]

@wb9688 I could not find any StreamInfoItem which has two badges while running the tests from #266. Do you have a link to a document which contains such a case? I found one.

can you give me a link please?

[TobiGr]

@B0pol It is a search querry for cirque du soleil

[mauriciocolli]

One more issue though: when opening a premiere video, it's not throwing the right exception.

We should add more status checks here.

[mauriciocolli]

Also, @TobiGr what about we disable these protected branches shenanigans?

In a lot of pull requests I see, it's infested with "merge branch dev into <branch>", because people keep pressing the update button.

For example, this one and #265 should be perfectly able to perform a simple merge. Then you don't need to overwrite the commits and rewrite history (I like seeing those verified commits).

[TobiGr]

@mauriciocolli
Should I add request.url() to https://github.com/TeamNewPipe/NewPipe/blob/b1eaf5616ad25abf2907d3283abae04334cbcdb2/app/src/main/java/org/schabi/newpipe/DownloaderImpl.java#L174? I think that is what you intended to have there. Is that correct?

[TobiGr]

In a lot of pull requests I see, it's infested with "merge branch dev into ", because people keep pressing the update button.

Yes, I hate that, too. I'd like people to rebase their commits to a have a readable history. We need to find a solution for this.

[wb9688]

@TobiGr: Does GitHub not have a rebase and merge button?

[mauriciocolli]

@TobiGr no, it should be latest url that response was produced to. In case of a redirection, the response would be made to the redirected url.

Here's a diff of what should be done:

diff --git a/app/src/main/java/org/schabi/newpipe/DownloaderImpl.java b/app/src/main/java/org/schabi/newpipe/DownloaderImpl.java
index 0ae072c66..6591fcbda 100644
--- a/app/src/main/java/org/schabi/newpipe/DownloaderImpl.java
+++ b/app/src/main/java/org/schabi/newpipe/DownloaderImpl.java
@@ -171,7 +171,9 @@ public class DownloaderImpl extends Downloader {
             responseBodyToReturn = body.string();
         }
 
-        return new Response(response.code(), response.message(), response.headers().toMultimap(), responseBodyToReturn);
+        final String latestUrl = response.request().url().toString();
+
+        return new Response(response.code(), response.message(), response.headers().toMultimap(), responseBodyToReturn, latestUrl);
     }
 
     /**

The Response#request returns the request that it originated from, then we can get the url.

Also, there's the other issue that I mentioned, so please wait a little bit longer.

Yes, I hate that, too. I'd like people to rebase their commits to a have a readable history. We need to find a solution for this.

Edit: I was thinking of the merge situation when I wrote below, but yes, would be nice.

I think this is about the protected branch:

I think if we disable that, we can merge requests without that merge hell. Imagine 10 pull requests and having to update each one. I think the update button would work, but then even more merge commits would be produced.

[TobiGr]

Here's a diff of what should be done:

Thanks. I'll change this when updating the extractor version.

One more issue though: when opening a premiere video, it's not throwing the right exception.

Ok. Are you on it, or should I take a look at it?

I think if we disable that, we can merge requests without that merge hell. Imagine 10 pull requests and having to update each one. I think the update button would work, but then even more merge commits would be produced.

ok.

[mauriciocolli]

@wb9688: I just found another bug, which could be fixed by looping over all badges in getStreamType() in YoutubeStreamInfoItemExtractor. Could you please do that?

TobiGr pushed this 3525223.

@TobiGr: Ok. Are you on it, or should I take a look at it?

I'm currently testing it., I'll open another pull request soon.

[mauriciocolli]

Oh, and @TobiGr, sorry if it caused a misunderstanding, but you would have to disable it, as I don't have permissions to change that setting (I don't even see here, screenshot from another repo).