Add more decryption function regexes #329

wb9688 · 2020-05-09T18:32:02Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

See https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/youtube.py#L1369. I didn't test this yet.

Edit: it looks like there are some differences in Python and Java regexes. I'll look into it tomorrow.

Edit 2: I think I fixed it now, but I need to test it more.

Stypox · 2020-05-09T19:51:30Z

Exception

User Action: requested stream
Request: https://www.youtube.com/watch?v=AGL7ealeXpw
Content Language: en_GB
Service: YouTube
Version: 0.19.3
OS: Linux Android 7.0 - 24

Crash log

org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor$DecryptException: Could not parse decrypt function 
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor.loadDecryptionCode(YoutubeStreamExtractor.java:753)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor.onFetchPage(YoutubeStreamExtractor.java:637)
	at org.schabi.newpipe.extractor.Extractor.fetchPage(Extractor.java:56)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:66)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:62)
	at org.schabi.newpipe.util.ExtractorHelper.lambda$getStreamInfo$3(ExtractorHelper.java:116)
	at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$5fJcha6Sq5APJBLdG6osaJby-mc.call(lambda)
	at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
	at io.reactivex.Single.subscribe(Single.java:3438)
	at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)
	at io.reactivex.Single.subscribe(Single.java:3438)
	at io.reactivex.internal.operators.maybe.MaybeFromSingle.subscribeActual(MaybeFromSingle.java:41)
	at io.reactivex.Maybe.subscribe(Maybe.java:4154)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.drain(MaybeConcatArray.java:153)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.request(MaybeConcatArray.java:78)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onSubscribe(FlowableElementAtMaybe.java:66)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray.subscribeActual(MaybeConcatArray.java:42)
	at io.reactivex.Flowable.subscribe(Flowable.java:14479)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
	at io.reactivex.Maybe.subscribe(Maybe.java:4154)
	at io.reactivex.internal.operators.maybe.MaybeToSingle.subscribeActual(MaybeToSingle.java:46)
	at io.reactivex.Single.subscribe(Single.java:3438)
	at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:237)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:272)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
	at java.lang.Thread.run(Thread.java:776)
Caused by: java.lang.UnsupportedOperationException
	at java.util.regex.Matcher.group(Matcher.java:383)
	at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:88)
	at org.schabi.newpipe.extractor.utils.Parser.matchGroup(Parser.java:81)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor.getDecryptionFuncName(YoutubeStreamExtractor.java:778)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor.loadDecryptionCode(YoutubeStreamExtractor.java:732)
	... 30 more

wb9688 · 2020-05-09T20:58:02Z

@Stypox: That's because named-capturing groups are only supported by Android since API 26… :/

wb9688 · 2020-05-10T06:55:02Z

@Stypox: It should be fixed now. Could you test it again?

Stypox · 2020-05-10T18:59:02Z

On my phone with Android 7.0 and on on AVD Android 10 everything works fine (video info loads and all resolutions play, tested on multiple videos).
On AVD Android 4.4 I keep getting "Recovering from player error" on some resolutions (always the same ones), but I guess that's an issue with the emulator since it also happens in other NewPipes. So I'd say there are no issues even on API 19.

I only got this exception, but I don't think it is related because sometimes I get it in other NewPipes. Closing and reopening the video solved the problem.

Exception

User Action: requested stream
Request: https://www.youtube.com/watch?v=86p9_ycUyq8
Content Language: en_US
Service: YouTube
Version: 0.19.3
OS: Linux Android 10 - 29

org.schabi.newpipe.extractor.stream.StreamInfo$StreamExtractException: Could not get any stream. See error variable to get further details.
	at org.schabi.newpipe.extractor.stream.StreamInfo.extractStreams(StreamInfo.java:192)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:70)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:62)
	at org.schabi.newpipe.util.ExtractorHelper.lambda$getStreamInfo$3(ExtractorHelper.java:116)
	at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$5fJcha6Sq5APJBLdG6osaJby-mc.call(Unknown Source:4)
	at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
	at io.reactivex.Single.subscribe(Single.java:3438)
	at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)
	at io.reactivex.Single.subscribe(Single.java:3438)
	at io.reactivex.internal.operators.maybe.MaybeFromSingle.subscribeActual(MaybeFromSingle.java:41)
	at io.reactivex.Maybe.subscribe(Maybe.java:4154)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.drain(MaybeConcatArray.java:153)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.request(MaybeConcatArray.java:78)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onSubscribe(FlowableElementAtMaybe.java:66)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray.subscribeActual(MaybeConcatArray.java:42)
	at io.reactivex.Flowable.subscribe(Flowable.java:14479)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
	at io.reactivex.Maybe.subscribe(Maybe.java:4154)
	at io.reactivex.internal.operators.maybe.MaybeToSingle.subscribeActual(MaybeToSingle.java:46)
	at io.reactivex.Single.subscribe(Single.java:3438)
	at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:919)

for some reason now my emulators are able to play videos and do not block at the first frame

wb9688 · 2020-05-10T19:02:10Z

@Stypox: Could you send me the page you got it on? Or could you not reproduce it? :/

Stypox · 2020-05-10T19:30:50Z

It's not reproducible, reopening the same video does not create any problem. As I said I also get it on my phone on 0.19.3, so it shouldn't be related to this pr

wb9688 · 2020-05-10T19:47:45Z

@Stypox: It could be related to this PR though. I've seen it once while running the tests (I think it was the Adele one that failed), but it was in combination with it not being able to extract I think the title and something else, so I disregarded it as not related to this PR.

B0pol · 2020-05-11T07:31:56Z

It's not related. I often get this error in regular version of NewPipe, like 1/20 videos.

Stypox

I tried with many videos on different network configurations and different phones and could not reproduce any crash. So this is ready.

TobiGr added the youtube service, https://www.youtube.com/ label May 9, 2020

wb9688 marked this pull request as draft May 9, 2020 18:35

wb9688 marked this pull request as ready for review May 11, 2020 07:43

wb9688 added this to the 0.19.4 milestone May 12, 2020

B0pol mentioned this pull request May 12, 2020

Couldn't get any stream / Could not play this stream (FIXED) TeamNewPipe/NewPipe#3576

Closed

Add more decryption function regexes

384d6ac

Stypox approved these changes May 29, 2020

View reviewed changes

TobiGr merged commit cf18cdb into TeamNewPipe:dev May 29, 2020

This was referenced May 29, 2020

Release 0.19.4 TeamNewPipe/NewPipe#3669

Merged

Release 0.19.4 TeamNewPipe/NewPipe#3670

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add more decryption function regexes #329

Add more decryption function regexes #329

wb9688 commented May 9, 2020 •

edited

Loading

Stypox commented May 9, 2020

wb9688 commented May 9, 2020

wb9688 commented May 10, 2020

Stypox commented May 10, 2020 •

edited

Loading

wb9688 commented May 10, 2020

Stypox commented May 10, 2020

wb9688 commented May 10, 2020

B0pol commented May 11, 2020 •

edited

Loading

Stypox left a comment

Add more decryption function regexes #329

Add more decryption function regexes #329

Conversation

wb9688 commented May 9, 2020 • edited Loading

Stypox commented May 9, 2020

Exception

wb9688 commented May 9, 2020

wb9688 commented May 10, 2020

Stypox commented May 10, 2020 • edited Loading

wb9688 commented May 10, 2020

Stypox commented May 10, 2020

wb9688 commented May 10, 2020

B0pol commented May 11, 2020 • edited Loading

Stypox left a comment

Choose a reason for hiding this comment

wb9688 commented May 9, 2020 •

edited

Loading

Stypox commented May 10, 2020 •

edited

Loading

B0pol commented May 11, 2020 •

edited

Loading