[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels #944

Theta-Dev · 2022-10-12T13:34:47Z

YouTube is currently A/B testing a new layout on their channel pages, which uses a RichGridRenderer. The current version of NewPipe cant extract any videos from it, resulting in an empty channel page. This PR adds support for it.

Here is a response file with the new layout: https://code.thetadev.de/ThetaDev/rustypipe/src/branch/main/testfiles/channel/channel_videos_20221011_richgrid2.json

I could not reproduce the error on my own without tools (the A/B test seems to be pretty rare for me). But there are already people having issues with the channel pages in NewPipe. See https://libreddit.kavin.rocks/r/NewPipe/comments/y1zg5m/why_are_certain_educative_channels_invisible_from/

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API (no API changes).

Fixes TeamNewPipe/NewPipe#9223

Theta-Dev · 2022-10-12T14:24:39Z

Here is a debug build for testing:

app-debug.zip

Theta-Dev · 2022-10-18T22:01:20Z

Note that this new channel layout includes up to 3 tabs for videos, livestreams and shorts. So we need a way to fetch up to 3 tabs at once to fully support this.

Lets hope YouTube is not going to make this permanent...

AudricV · 2022-10-19T06:38:34Z

Lets hope YouTube is not going to make this permanent...

It will be, as it has been announced by YouTube. Check out this video of the Creator Insights official YouTube channel.

So we need a way to fetch up to 3 tabs at once to fully support this.

I don't think that's a good idea, we should just implement channel tabs in the extractor instead.

Theta-Dev · 2022-10-20T13:38:08Z

I guess you are right. I was thinking about having to use up to 3 requests go get subscription feed data, but that wont work anyway. YouTube does not show the upload date of shorts, so it wont be possible to merge the 3 tabs into a chronological feed.

So I guess we should stick to using the RSS feed for fetching the subscription feed (maybe make that the default option once channel tabs are permanently established).

For the actual channel views we would need the 3tab layout in NewPipe (and by that occasion, add channel playlists, too).

AudricV

Thanks, I just have a question.

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java

AudricV · 2022-10-29T16:21:54Z

@Theta-Dev Could you answer our questions, please? We'd like to merge this fix ASAP.

Theta-Dev · 2022-10-29T20:35:59Z

Okay, I added the setVisitorData method from my channel tabs PR to reproduce the A/B test that way.

Also redownloaded the mock files and switched the test to a channel (Jeff Geerling) that has all 3 tabs.

AudricV

Almost good!

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java

acrodemocide · 2022-10-31T21:31:01Z

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java

+
+        @Test
+        public void testDescription() throws Exception {
+            ExtractorAsserts.assertContains("K8s, Ansible, Raspberry Pi, photography, tech, DIY, tools", extractor.getDescription());


Are we worried about whether the description changes at all? I figure they usually don't, but I wasn't sure how these tests need to be maintained when the description on this channel is changed. (Unless we're working with the owner of this channel and he's agreed not to change the description).

YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.

AudricV · 2022-11-02T18:32:36Z

As the change seems to be globally rolled out, I removed the visitorData changes and the addition of your channel test (see why here: #944 (comment)). I also updated all the channel mocks, which contains these richGridRenderers.

This doesn't mean that I didn't like your code, please don't take this personally (especially because this is the third time that I do this in your PRs): I find the visitorData changes useful, but as they are not needed to fix this issue, I think it is better to open a separate PR for them!

Stypox

Thank you! I could not find anything to say about the code

Theta-Dev · 2023-01-19T13:21:11Z

@yauhen-l The error parsing the subscriber counts on the search page is unrelated to this PR and is already fixed by #978.

The missing livestreams issue will be fixed by #951. Since that requires new tabs to be added to the UI, implementing it might take some more time.

You can download a test build with channel tabs here: TeamNewPipe/NewPipe#9182

AudricV added bug Issue is related to a bug youtube service, https://www.youtube.com/ labels Oct 19, 2022

Theta-Dev mentioned this pull request Oct 23, 2022

Add support for channel tabs #951

Closed

3 tasks

opusforlife2 mentioned this pull request Oct 27, 2022

[STOP CREATING DUPLICATES] [YouTube] Channel page doesn't contain any video/no new video when not using dedicated feed TeamNewPipe/NewPipe#9223

Closed

6 tasks

AudricV reviewed Oct 28, 2022

View reviewed changes

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java Outdated Show resolved Hide resolved

AudricV changed the title ~~[YouTube] fix: support richGridRenderer on channel page~~ [YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels Oct 28, 2022

TobiGr reviewed Oct 29, 2022

View reviewed changes

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java Outdated Show resolved Hide resolved

AudricV added the ASAP Issue needs to be fixed as soon as possible label Oct 29, 2022

AudricV requested changes Oct 30, 2022

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java Outdated Show resolved Hide resolved

acrodemocide reviewed Oct 31, 2022

View reviewed changes

acrodemocide approved these changes Oct 31, 2022

View reviewed changes

steventrouble mentioned this pull request Nov 1, 2022

YT channels show no videos TeamNewPipe/NewPipe#9262

Closed

6 tasks

This comment was marked as off-topic.

Sign in to view

NoPlagiarism mentioned this pull request Nov 2, 2022

New videos failing to be found polymorphicshade/NewPipe#278

Closed

7 tasks

ShareASmile mentioned this pull request Nov 2, 2022

crash when trying to view PR slapperwan/gh4a#1169

Closed

Theta-Dev and others added 2 commits November 2, 2022 19:01

[YouTube] Support richGridRenderer on channel pages

20e4a35

YouTube is deploying a new layout on their channel pages, which uses richGridRenderer JSON objects.

[YouTube] Regenerate supported channels mocks

e4c24d4

AudricV force-pushed the fix-rich-grid-renderer branch from 5e528eb to e4c24d4 Compare November 2, 2022 18:31

AudricV approved these changes Nov 2, 2022

View reviewed changes

Stypox approved these changes Nov 2, 2022

View reviewed changes

AudricV merged commit 31bf704 into TeamNewPipe:dev Nov 2, 2022

This comment was marked as off-topic.

Sign in to view

ShareASmile mentioned this pull request Nov 3, 2022

crash when trying to view this particular PR slapperwan/gh4a#1253

Closed

qwerty287 mentioned this pull request Nov 3, 2022

Crash if opening PR and scrolling down to end slapperwan/gh4a#1256

Closed

This was referenced Nov 4, 2022

Release v0.24.1 TeamNewPipe/NewPipe#9290

Merged

Release v0.24.1 (PLEASE TEST!) TeamNewPipe/NewPipe#9292

Closed

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels #944

[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels #944

Theta-Dev commented Oct 12, 2022 •

edited by AudricV

Loading

Theta-Dev commented Oct 12, 2022 •

edited

Loading

Theta-Dev commented Oct 18, 2022

AudricV commented Oct 19, 2022

Theta-Dev commented Oct 20, 2022

AudricV left a comment

AudricV commented Oct 29, 2022

Theta-Dev commented Oct 29, 2022

AudricV left a comment

acrodemocide Oct 31, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

AudricV commented Nov 2, 2022

Stypox left a comment

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as resolved.

Theta-Dev commented Jan 19, 2023 •

edited

Loading

[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels #944

[YouTube] Support richGridRenderer on channel page to fix extraction of videos in channels #944

Conversation

Theta-Dev commented Oct 12, 2022 • edited by AudricV Loading

Theta-Dev commented Oct 12, 2022 • edited Loading

Theta-Dev commented Oct 18, 2022

AudricV commented Oct 19, 2022

Theta-Dev commented Oct 20, 2022

AudricV left a comment

Choose a reason for hiding this comment

AudricV commented Oct 29, 2022

Theta-Dev commented Oct 29, 2022

AudricV left a comment

Choose a reason for hiding this comment

acrodemocide Oct 31, 2022

Choose a reason for hiding this comment

This comment was marked as off-topic.

This comment was marked as off-topic.

AudricV commented Nov 2, 2022

Stypox left a comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

This comment was marked as outdated.

This comment was marked as resolved.

Theta-Dev commented Jan 19, 2023 • edited Loading

Theta-Dev commented Oct 12, 2022 •

edited by AudricV

Loading

Theta-Dev commented Oct 12, 2022 •

edited

Loading

Theta-Dev commented Jan 19, 2023 •

edited

Loading