[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests. #239
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thumbnail: quality before: https://peertube.cpy.re/static/thumbnails/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg quality after: https://peertube.cpy.re/static/previews/d2a5ec78-5f85-4090-8ec5-dc1102e022ea.jpg description: we were getting about the first 260 characters, we now get full description (with fallback to first 260 chars if the get request for full description fails) test: updated tests to match description, also changed some test: it was assertEquals(extracted, expected), but the proper way to do it is assertEquals(expected, extracted) metadata: got host, privacy (public, private, unlisted), licence, language, tags
TobiGr
reviewed
Jan 20, 2020
.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java
Outdated
Show resolved
Hide resolved
if the description length is above 254, and ends with ..., it means the description we got from the first request is shortened. why above 254: because in fact, shortened descriptions length are between 255 : https://framatube.org/videos/watch/24efbe1b-73c0-4d72-a3ff-77c8b32d3fcd https://framatube.org/videos/watch/1ca99f40-fb5b-4fa4-abe5-4d54325df7fc and 269: https://framatube.org/videos/watch/4d661d5f-a884-4510-bca8-15cb19aa3fe5 also fixed a typo in StreamExtractor.java
actually, the max description length is 250 after request with our extractor. during my tests, I made API requests with Firefox, copy/pasted into echo "insert description" | wc, and it was giving a wrong length, maybe due to the escapers, I have no idea anyway, it's now fixed
TobiGr
reviewed
Jan 20, 2020
.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java
Outdated
Show resolved
Hide resolved
Co-Authored-By: Tobias Groza <TobiGr@users.noreply.github.com>
+ created getBoolean method in JsonUtils.java
on peertube.co.uk: https://i.imgur.com/8Pai1rb.png on newpipe, before this commit: https://i.imgur.com/NIRbs4Z.png on newpipe, with this commit: https://i.ibb.co/mhKKtRD/Screenshot-20200123-185422-New-Pipe-Debug.png
B0pol
changed the title
https://peertube.cpy.re/videos/watch/d2a5ec78-5f85-4090-8ec5-dc1102e022ea anonyme zirbeldrüse is his displayName, what was displayed in NewPipe. but on the website, it's shown 777@mastodon.xyz
TobiGr
reviewed
Jan 24, 2020
extractor/src/main/java/org/schabi/newpipe/extractor/utils/JsonUtils.java
Outdated
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/utils/JsonUtils.java
Outdated
Show resolved
Hide resolved
.../java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamExtractor.java
Outdated
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/stream/StreamExtractor.java
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/stream/StreamExtractor.java
Show resolved
Hide resolved
|
About PR in NewPipe repository: I plan to make a first & quick PR right after this PR get merged, to update extractor version, and fix description formatting, because currently, all descriptions are formatted from HTML, but PeerTube gives plain Markdown, so I will change that, see this commit, |
so that java can automatically translate with Locale.getDisplayLanguage(), instead of always having English name of the language
1 task
|
Are there other changes needed? |
|
Sorry, there are just too many PRs :D I forgot about this one. |
B0pol
added a commit
to B0pol/NewPipe
that referenced
this pull request
Feb 2, 2020
description: - PeerTube: it's now full description (it cut at 250 characters before), and it displays ok (newlines are ok, but markdown isn't) - MediaCCC: descriptions are now displayed well (newlines added) - YouTube: timestamps in descriptions are clickable and work more PeerTube fixes: thumbnail is now high quality age limit is now handled upload date in «recently added» feed is good now (it was one hour delayed) all fixes come from TeamNewPipe/NewPipeExtractor#239, so it need to be merged before this PR
1 task
yausername
suggested changes
Feb 6, 2020
...ctor/src/main/java/org/schabi/newpipe/extractor/services/peertube/PeertubeParsingHelper.java
Outdated
Show resolved
Hide resolved
B0pol
added a commit
to B0pol/NewPipe
that referenced
this pull request
Feb 6, 2020
description: - PeerTube: it's now full description (it cut at 250 characters before), and it displays ok (newlines are ok, but markdown isn't) - MediaCCC: descriptions are now displayed well (newlines added) - YouTube: timestamps in descriptions are clickable and work more PeerTube fixes: thumbnail is now high quality age limit is now handled upload date in «recently added» feed is good now (it was one hour delayed) all fixes come from TeamNewPipe/NewPipeExtractor#239, so it need to be merged before this PR
yausername
suggested changes
Feb 7, 2020
extractor/src/main/java/org/schabi/newpipe/extractor/stream/Description.java
Outdated
Show resolved
Hide resolved
yausername
approved these changes
Feb 8, 2020
|
Just FYI, I've succeed to extract some metadata for YouTube:
I will extract for YouTube
I won't extract for YouTube:
But this will be in another extractor PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
I fixed some issues with PeerTube that we have now.
Also, I extracted metadata: host, privacy, licence, language, tags.
For YouTube, someone could extract privacy, category (and maybe tags) after this PR, that would be a nice thing.
Fixes TeamNewPipe/NewPipe#2201 (comment) 1. 2. and 5.
3 is already fixed, and I plan to fix 4 (default instance only) and 6 with my future Frontend PR.
TeamNewPipe/NewPipe#2201 (comment) I aimed to do that.
Some in-app before/after screenshots:
Screenshot from PeerTube website
Screenshot from NewPipe (with my changes)
Thumbnail and description before
Thumbnail now
Description now
Btw, for the fronted I already handled the fact we are not getting info:
If no metadata is extracted, it stays as it was before
If only some metadata are extracted, no problem