[SoundCloud] Add support for on.soundcloud.com urls #1179
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TobiGr
added
enhancement
TobiGr
suggested changes
May 19, 2024
.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java
Outdated
Show resolved
Hide resolved
.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java
Outdated
Show resolved
Hide resolved
...bi/newpipe/extractor/services/soundcloud/linkHandler/SoundcloudStreamLinkHandlerFactory.java
Outdated
Show resolved
Hide resolved
|
I'm not sure what's going on with the failing tests. |
AudricV
previously requested changes
May 20, 2024
.../src/main/java/org/schabi/newpipe/extractor/services/soundcloud/SoundcloudParsingHelper.java
Show resolved
Hide resolved
TobiGr
approved these changes
Jul 22, 2024
TobiGr
changed the title
This comment was marked as off-topic.
This comment was marked as off-topic.
3 tasks
ShareASmile
added a commit
to ShareASmile/NewPipeExtractor
that referenced
this pull request
Jan 26, 2025
* [MediaCCC] Allow obtaining channel tab extractor from scratch i.e. without needing to pass through the conference/channel extractor This was needed because clients (like NewPipe) might rely on link handlers to hold as little data as possible, since they might be kept around for long or passed around in system transactions, so this commit allows obtaining a standalone link handler that does not hold a JsonObject within itself. * [MediaCCC] Add ChannelTabExtractorTest * [MediaCCC] Fix lambda link handler keeping reference to extractor This caused problems in NewPipe, because extractors are not serializable, and well, keeping references to them is a bad idea anyway. * LocaleCompat.forLanguageTag: return Optional if parsing fails It’s not obvious that the function will fail in some cases and throw an `IllegalArgumentException`. So instead of just failing if parsing fails, return an Optional that all callers have to decide what to do (e.g. the YoutubeExtractor can just ignore the locale in that case, like it does with most other fields in the json if they are unexpected). * Bump actions/cache from 3 to 4 Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@v3...v4) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * [PeerTube] Add support for stream frames/storyboards extraction Implement PeerTubeStreamExtractor.getFrames() * [PeerTube] Add test for segments and framesets * Bump org.junit:junit-bom from 5.10.0 to 5.10.2 Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.0 to 5.10.2. - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.10.0...r5.10.2) --- updated-dependencies: - dependency-name: org.junit:junit-bom dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump actions/upload-artifact from 3 to 4 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * [PeerTube] Fix tests for "What is PeerTube?" video * [YouTube] Detect new account termination messages * Update mocks for YoutubeChannelExtractorTest.NotAvailable * [SoundCloud] Fix SoundcloudSearchExtractorTest.NoNextPage Search did not return no item at all, causing a NothingFoundException. New search query yields three items on first page * Make getCommentText @nonnull * Throw ParsingException instead of IllegalArg * [YouTube] Support new A/B tested comments data Also improve current comments code by removing outdated comment renderer data. * [YouTube] Add support for styles in attributed descriptions Also refactor descriptions parsing. * [YouTube] Replace link text with accessibility label * [YouTube] Cleanup description helper Remove unneeded isClose field, and make constants private * [YouTube] Add YoutubeDescriptionHelperTest * [YouTube] Avoid XSS attacks in description or comments * [YouTube] Add missing prettyPrint query parameter to mixes continuations * [YouTube] Remove usage of API keys for InnerTube requests, bump versions The API keys are not used anymore by official clients in almost all cases (still used by the Android app until it gets a configuration) for all requests we made. Clients and device OS versions have been bumped to their latest stable version known. Methods and fields related to API keys have been renamed or deleted if they're no longer relevant. * [YouTube] Update Android client player parameters YouTube disabled the effectiveness of the parameters which were used (the player response we get redirects to another video), but new parameters which work around Android's client integrity checks have been found. * [YouTube] Fix YoutubeStreamExtractorDefaultTest tests * [YouTube] Fix YoutubeStreamExtractorRelatedMixTest * [YouTube] Add support for new crisis meta info action data The new action data can return multiple contact actions instead of only one, which will be concatenated by a new line return. This commit fixes tests of the CrisisResources test class of YoutubeSearchExtractorTest. * [YouTube] Fix InteractiveTabbedHeader.testTabs test YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels, which contains Shorts about the game of the topic channel but are not uploaded on the game topic channel. As this tab is already supported by the extractor, fetching a gaming topic channel now returns a tab instead of none. Channel name and channel URL of these Shorts needs to be set to null in a separate commit, as Shorts on this tab do not have the topic channel as their uploader. * [YouTube] Update mocks * [YouTube] Improve meta info code for review * [YouTube] Create constants for client names/versions * Improve assertTabsContain() to also check size * Fix JDoc Add Checkstyle exception for LineLength in JDoc links. See https://github.com/TeamNewPipe/NewPipeExtractor/actions/runs/8804403691/job/24164703883 * [PeerTube] Fix test * Bump com.github.spotbugs:spotbugs-annotations from 4.8.3 to 4.8.5 Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.3 to 4.8.5. - [Release notes](https://github.com/spotbugs/spotbugs/releases) - [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md) - [Commits](spotbugs/spotbugs@4.8.3...4.8.5) --- updated-dependencies: - dependency-name: com.github.spotbugs:spotbugs-annotations dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump org.mozilla:rhino from 1.7.13 to 1.7.15 Bumps [org.mozilla:rhino](https://github.com/mozilla/rhino) from 1.7.13 to 1.7.15. - [Release notes](https://github.com/mozilla/rhino/releases) - [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md) - [Commits](https://github.com/mozilla/rhino/commits) --- updated-dependencies: - dependency-name: org.mozilla:rhino dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fix JDoc YoutubePostLiveStreamDvrDashManifestCreator.java:90: error: unexpected end tag: </p> * version 0.24.0 * [PeerTube] Fix test UploaderName was changed by user * Bump peaceiris/actions-gh-pages from 3 to 4 Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4. - [Release notes](https://github.com/peaceiris/actions-gh-pages/releases) - [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md) - [Commits](peaceiris/actions-gh-pages@v3...v4) --- updated-dependencies: - dependency-name: peaceiris/actions-gh-pages dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * [PeerTube] Fix testing comment content The comment is not available anymore. * [PeerTube] Fix test for like count Number changed * Bump com.github.spotbugs:spotbugs-annotations from 4.8.5 to 4.8.6 Bumps [com.github.spotbugs:spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.8.5 to 4.8.6. - [Release notes](https://github.com/spotbugs/spotbugs/releases) - [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md) - [Commits](spotbugs/spotbugs@4.8.5...4.8.6) --- updated-dependencies: - dependency-name: com.github.spotbugs:spotbugs-annotations dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump org.junit:junit-bom from 5.10.2 to 5.10.3 Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.2 to 5.10.3. - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.10.2...r5.10.3) --- updated-dependencies: - dependency-name: org.junit:junit-bom dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump com.google.code.gson:gson from 2.10.1 to 2.11.0 Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.10.1 to 2.11.0. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.10.1...gson-parent-2.11.0) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * [Youtube] Adjust throttling function extraction to changes (TeamNewPipe#1191) * [Youtube] Adjust throttling function extraction to changes --------- Co-authored-by: Stypox <stypox@pm.me> * [Youtube] Adjust throttling function extraction to changes (TeamNewPipe#1191) * [Youtube] Adjust throttling function extraction to changes --------- Co-authored-by: Stypox <stypox@pm.me> * [Bandcamp] Update artist page detection Bandcamp changed the way the footer is rendered. Therefore, we check for a link to the cart page instead. * Use the new URL encode/decode methods introduced in Java 10 (TeamNewPipe#1196) * Use Java 10 URLDecoder/URLEncoder methods * Simplify compatParseMap * [Bandcamp] Null-safe url catenation in track playlist Previously, if no URL was provided due to the track being a preorder or unpaid track that doesn't have its own public page, we would catenate the artist URL and null to a nonsensical string. This invalid URL would also be used to try fetching a thumbnail URL. The downstream behavior of the NewPipe client is only marginally improved, as it now no longer throws visible exceptions due to the missing thumbnails, but still gives an unfriendly error upon clicking the item that has a `null` URL. * [Bandcamp] Show additional info in radio kiosk A newer version of the radio list API delivers additional fields. Thanks @crisp5. * Add support for on.soundcloud.com urls (TeamNewPipe#1179) Add support for on.soundcloud.com urls Fixes TeamNewPipe#1178 Co-authored-by: TobiGr <tobigr@users.noreply.github.com> * [Bandcamp] Upgrade incoming links to HTTPS * [YouTube] Workaround 403s on streaming URLs of WEB client after some time These changes work around an anti-bot token, for which its requirement is A/B tested on the WEB client. In this test, streaming URLs of this client return HTTP errors 403 if the token is not provided after some time. It also allows to not fetch the JavaScript player for non-age restricted videos, reducing data usage. The TVHTML5 embed client is now only fetched in the case of age-restricted videos. The methods forceFetchAndroidClient and forceFetchIosClient of YoutubeStreamExtractor have been removed, as they are now not needed anymore. These changes also break the extraction of appropriate error states for private and deleted videos and invalid video IDs. * [YouTube] Update clients info * [YouTube] Fix YoutubeSearchExtractorTest.CrisisResources The "blue whale" search query does not return a crisis resource panel anymore, so it was changed to a different word, "suicide". * [YouTube] Fix some YoutubeStreamExtractorDefaultTest tests - Fix typo in folder name of DescriptionTestPewdiepie test; - Fix constant usage of DownloaderTestImpl as download implementation for UnlistedTest and CCLicensed tests. * [YouTube] Update mocks * [YouTube] Fix JavaScript n parameter decoding function name extraction This commits fixes extraction of the function name decoding the n parameter for HTML5 clients' streaming URLs for YouTube base JavaScript player 3400486c. Two new regexes have been added to the existing ones. All regexes and what they extract has been documented. * [YouTube] Don't provide streaming URLs which have an non-decoded n param This param used to throttle bandwidth of streaming URLs which have this parameter when the correct value is not provided but it is not the case anymore, as the streaming URLs return now an HTTP response code 403 in this case. * [YouTube] Do not get twice runs array in YoutubeParsingHelper The runs object was computed twice in getTextFromObject and getUrlFromObject methods, leading to unneeded search costs. This has been avoided by storing the array in method variables. * [YouTube] Support playlists as URL navigation endpoints * [YouTube] Move channel header's verified status code to YoutubeChannelHelper Also throw an exception when we cannot get the verified status of a channel in YoutubeChannelExtractor due to a missing channelHeader, if the channel has no channelAgeGateRenderer. * [YouTube] Add base implementation for show InfoItems As there are multiple show UI elements which share a lot of common data, a base implementation, an abstract class named YoutubeBaseShowInfoItemExtractor, has been created to handle common cases. * [YouTube] Add support for showRenderers in search results * [YouTube] Support shows in channels and provide verified status to items Also fix naming of info items' collection methods. * [YouTube] Add common methods to get ID, name and age gate object of channels Also move duplicate strings into constants and support pageHeader channel header in user channels on YoutubeChannelHelper methods. * [YouTube] Support pageHeader on user channels Also move duplicate strings into constants and add a missing default switch case. * [YouTube] Fix NPE when getting channel header for videos channel tab * [YouTube] Fix VSauce test of YoutubeChannelExtractorTest test class The channel description has been changed and some expected words have been removed. * [YouTube] Update mocks of YoutubeChannelTabExtractorTest.Shorts test For some reason, mocks of the continuation were not parsed. All mocks of the test have been updated to fix the issue. * Release v0.24.2 * Add issue templates (TeamNewPipe#1204) * Create bug report template * Create feature request template * Create config.yml --------- Co-authored-by: TobiGr <tobigr@users.noreply.github.com> * Fix issue template dir name * [PeerTube] Fix parsing id for instances whose domain ends with a or c The pattern to detect the channel ID was faulty and e.g. the ID detected for "https://kolektiva.media/video-channels/documentary_channel" was "a/video-channels" which is wrong. A new pattern is added to distinguish between URLs and potential IDs because IDs must not start with a "/" while IDs inside an URL must. Fixes TeamNewPipe/NewPipe#11369 * Bump org.junit:junit-bom from 5.10.3 to 5.11.0 Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.10.3 to 5.11.0. - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.10.3...r5.11.0) --- updated-dependencies: - dependency-name: org.junit:junit-bom dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * [SoundCloud] Fix test: title changed * Update dependency coordinates due to case sensitivity in JitPack * [YouTube] Add support for shortsLockupViewModels This new UI data type is replacing the reelItemRenderer one. * [YouTube] Remove visitorData usage for shorts continuations It isn't required anymore and not used by extractor anymore since commit 5a6da5f, as the wrong page ID is used as a visitor data (the VerifiedStatus value as a string). * [YouTube] Update YoutubeChannelTabExtractorTest.Shorts test class mocks * [YouTube] Fix serialization of Videos channel tab when already fetched Also remove usage of Optional as fields as it is not a good practice. This simplifies in some places channel info extraction code. * [YouTube] Fix extraction of n param deobfuscation function name This commit adds two new regular expressions to parse the n parameter function. It also improves existing regular expressions by using the constant representing multiple characters instead of adding the one or multiple characters token manually in each regex for everything and not only function names. * Bump org.junit:junit-bom from 5.11.0 to 5.11.2 Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.0 to 5.11.2. - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.11.0...r5.11.2) --- updated-dependencies: - dependency-name: org.junit:junit-bom dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump org.junit:junit-bom from 5.11.2 to 5.11.3 Bumps [org.junit:junit-bom](https://github.com/junit-team/junit5) from 5.11.2 to 5.11.3. - [Release notes](https://github.com/junit-team/junit5/releases) - [Commits](junit-team/junit5@r5.11.2...r5.11.3) --- updated-dependencies: - dependency-name: org.junit:junit-bom dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Add documentation for faulty framesets * [YouTube] Add support for automatic dubbed and secondary tracks * Use JDoc and inherit doc * [YouTube] Fix Shorts' thumbnails extraction in their channel tab Wrong methods were used to access and extract the thumbnails' data. This has been fixed with this commit. * Update user agent to Firefox ESR 128 * [YouTube] Move channel verified status check from badges to a method This method will be used in more places such as the new playlist item data. Support for a new icon used in artist channels has been also added. * [YouTube] Add support for playlists lockupViewModels This new data type, A/B tested or rolled out at the time the changes are commited, is present on multiple surfaces. * Refactored Identifiers (TeamNewPipe#1205) Extractor.pageFetched -> Extractor.isPageFetched Stream.equalStats(Stream) is renamed to Stream.areStatsEqual(Stream) Stream.getBitrate() is renamed to Stream.getBitRate() * Revert "Refactored Identifiers (TeamNewPipe#1205)" This reverts commit 0de2241. * Make RecordingDownloader more resillient against ReCaptchaExceptions Implement a max number of requests per minute to prevent hitting reate limits and triggering ReCaptchaExceptions. This slows down the RecordingDownloader significantly and can be adjusted if needed. A request ist retried once when facing a ReCaptchaException. * [YouTube] Update mocks * Fix tests * Release v0.24.3 * Fix JDoc * fix: update iOS client, add visitor data to YouTube requests * move ProtoBuilder, add tests, fix incompatible rand call * update comment * iPhone 15 Pro Max instead of 16 in comments * Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Co-authored-by: Stypox <stypox@pm.me> * Release v0.24.4 * Make it really old Android compatible * Update ci.yml Remove the scheduled run * Bump dependencies Bump [com.puppycrawl.tools:checkstyle](https://github.com/checkstyle/checkstyle) from 10.4 to 10.12.2. * Fix line length * Make it really old Android compatible - again Base64 is not available on such Androids too --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Stypox <stypox@pm.me> Co-authored-by: Profpatsch <mail@profpatsch.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: TobiGr <tobigr@users.noreply.github.com> Co-authored-by: AudricV <74829229+AudricV@users.noreply.github.com> Co-authored-by: XiangRongLin <41164160+XiangRongLin@users.noreply.github.com> Co-authored-by: Fynn Godau <fynngodau@mailbox.org> Co-authored-by: Isira Seneviratne <31027858+Isira-Seneviratne@users.noreply.github.com> Co-authored-by: Marco Sirabella <marco@sirabella.org> Co-authored-by: opusforlife2 <53176348+opusforlife2@users.noreply.github.com> Co-authored-by: floriegl <22279483+floriegl@users.noreply.github.com> Co-authored-by: Thompson3142 <115718208+Thompson3142@users.noreply.github.com> Co-authored-by: congyuluo <78986434+congyuluo@users.noreply.github.com> Co-authored-by: ThetaDev <thetadev@magenta.de> Co-authored-by: Zsombor Gegesy <gzsombor@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1178