[YouTube] Fix hashtags links extraction and escape HTML links #1032
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AudricV
added
bug
…descriptions + HTML links webCommandMetadata object is contained inside a commandMetadata one, so it is not accessible from the root of the navigationEndpoint object. The corresponding statement has been moved at the bottom of the specific endpoints parsing, as the webCommandMetadata object is present almost everywhere, otherwise URLs of some endpoints would have be changed, such as uploader URLs (from channel IDs to handles). As no ParsingException is now thrown by getUrlFromNavigationEndpoint, and so by getTextFromObject, getUrlFromObject and getTextAtKey, the methods which were catching ParsingExceptions thrown by these methods had to be updated. URLs got in the HTML version of getTextFromObject are now escaped properly to provide valid HTML to clients. This has been also done for attribute descriptions, with the description text for this type of descriptions. As YouTube descriptions are in HTML format (except for the fallback on the JSON player response, which is plain text and only happens when there is no visual metadata or a breaking change), all URLs returned are escaped, so tests which are testing presence of URLs with escaped characters had to be updated (it was only the case for YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).
AudricV
force-pushed
the
yt_fix-comments-hashtags-links-extraction
branch
from
801f767 to
a63f289
Compare
Stypox
approved these changes
Feb 26, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
webCommandMetadataobject is contained inside acommandMetadataone, so it is not accessible from the root of thenavigationEndpointobject.The corresponding statement has been moved at the bottom of the specific endpoints parsing, as the
webCommandMetadataobject is present almost everywhere, otherwise URLs of some endpoints would have be changed, such as uploader URLs (from channel IDs to handles).As no
ParsingExceptionis now thrown bygetUrlFromNavigationEndpoint, and so bygetTextFromObject,getUrlFromObjectandgetTextAtKey, the methods which were catchingParsingExceptionsthrown by these methods had to be updated.URLs got in the HTML version of
getTextFromObjectare now escaped properly to provide valid HTML to clients. This has been also done for attribute descriptions, with the description text for this type of descriptions.As YouTube descriptions are in HTML format (except for the fallback on the JSON player response, which is plain text and only happens when there is no visual metadata or a breaking change), URLs returned are escaped, so tests which are testing presence of URLs with escaped characters had to be updated (it was only the case for
YoutubeStreamExtractorDefaultTest.DescriptionTestUnboxing).I've also updated the mocks of two tests classes of
YoutubeCommentsExtractorTestwhich were missing, in order to test completely my changes:RepliesTestandFormattingTest.Fixes #1019 (for real this time)
Related issue: TeamNewPipe/NewPipe#9774