Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PeerTube] added metadata, fix descriptions, fix thumbnail, fix upload date, fix age limit, update tests. #239

Merged
merged 21 commits into from
Feb 8, 2020
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a281519
added metadata, fix descriptions, fix thumbnail, update tests
B0pol Jan 19, 2020
b382416
changed the way to extract peertube description
B0pol Jan 20, 2020
ad7f97a
fix PeerTube description and add more description tests
B0pol Jan 20, 2020
f403490
Refactoring
B0pol Jan 20, 2020
bcfe7be
Merge branch 'dev' into peertube
B0pol Jan 22, 2020
1a15c0e
agelimit now returns 18 if the video is marked as nsfw, 0 otherwise
B0pol Jan 23, 2020
74439f6
add extraction for support info and rename getLanguageInfo function
B0pol Jan 23, 2020
20da475
empty support returns "", same for empty description
B0pol Jan 23, 2020
a691d6d
fix upload date: there was a one hour delay
B0pol Jan 23, 2020
c790261
update test
B0pol Jan 23, 2020
812c8e0
authorName in comments now follow PeerTube website
B0pol Jan 23, 2020
b816e48
Merge branch 'dev' into peertube
B0pol Jan 24, 2020
341372c
reindenting (ctrl alt l) on JsonUtils and PeertubeStreamExtractor
B0pol Jan 24, 2020
e392b6c
getLanguageInfo returns Locale instead of String
B0pol Jan 25, 2020
b671a4b
Merge branch 'dev' into peertube
B0pol Feb 1, 2020
5756df8
Use GMT as base time (actually fix upload date)
B0pol Feb 6, 2020
26c65b2
Create class Description
B0pol Feb 6, 2020
70a40e7
Description: rm constructor by serviceId
B0pol Feb 7, 2020
0f8a7f9
fix testGetUploadDate for PeerTubeStreamExtractor
B0pol Feb 7, 2020
11bcc78
Description implements Serializable. fix NotSerializableException
B0pol Feb 7, 2020
0e33249
Fix SoundCloud description test
TobiGr Feb 8, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ To test changes quickly you can build the library locally. Using the local Maven
2. It's _recommended_ that you change the `version` of this library (e.g. `LOCAL_SNAPSHOT`).
3. Run gradle's `ìnstall` task to deploy this library to your local repository (using the wrapper, present in the root of this project: `./gradlew install`)
4. Change the dependency version used in your project to match the one you chose in step 2 (`implementation 'com.github.TeamNewPipe:NewPipeExtractor:LOCAL_SNAPSHOT'`)

> Tip for Android Studio users: After you make changes and run the `install` task, use the menu option `File → "Sync with File System"` to refresh the library in your project.

## Supported sites
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

public class MediaCCCStreamExtractor extends StreamExtractor {

Expand Down Expand Up @@ -47,8 +48,8 @@ public String getThumbnailUrl() throws ParsingException {

@Nonnull
@Override
public String getDescription() throws ParsingException {
return data.getString("description");
public Description getDescription() throws ParsingException {
return new Description(getServiceId(), data.getString("description"));
}

@Override
Expand Down Expand Up @@ -225,4 +226,41 @@ public String getName() throws ParsingException {
public String getOriginalUrl() throws ParsingException {
return data.getString("frontend_link");
}

@Override
public String getHost() throws ParsingException {
return "";
}

@Override
public String getPrivacy() throws ParsingException {
return "";
}

@Override
public String getCategory() throws ParsingException {
return "";
}

@Override
public String getLicence() throws ParsingException {
return "";
}

@Override
public Locale getLanguageInfo() throws ParsingException {
return null;
}

@Nonnull
@Override
public List<String> getTags() throws ParsingException {
return new ArrayList<>();
}

@Nonnull
@Override
public String getSupportInfo() throws ParsingException {
return "";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import java.text.SimpleDateFormat;
import java.util.Calendar;
import java.util.Date;
import java.util.TimeZone;

import org.jsoup.helper.StringUtil;
import org.schabi.newpipe.extractor.exceptions.ContentNotAvailableException;
Expand All @@ -12,21 +13,23 @@
import com.grack.nanojson.JsonObject;

public class PeertubeParsingHelper {

private PeertubeParsingHelper() {
}

public static void validate(JsonObject json) throws ContentNotAvailableException {
String error = json.getString("error");
if(!StringUtil.isBlank(error)) {
if (!StringUtil.isBlank(error)) {
throw new ContentNotAvailableException(error);
}
}

public static Calendar parseDateFrom(String textualUploadDate) throws ParsingException {
Date date;
try {
date = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.S'Z'").parse(textualUploadDate);
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.S'Z'");
sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
date = sdf.parse(textualUploadDate);
} catch (ParseException e) {
throw new ParsingException("Could not parse date: \"" + textualUploadDate + "\"", e);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ public String getCommentText() throws ParsingException {
try {
Document doc = Jsoup.parse(htmlText);
return doc.body().text();
}catch(Exception e) {
} catch(Exception e) {
return htmlText.replaceAll("(?s)<[^>]*>(\\s*<[^>]*>)*", "");
}
}
Expand All @@ -83,15 +83,15 @@ public String getAuthorThumbnail() throws ParsingException {
String value;
try {
value = JsonUtils.getString(item, "account.avatar.path");
}catch(Exception e) {
} catch(Exception e) {
value = "/client/assets/images/default-avatar.png";
}
return baseUrl + value;
}

@Override
public String getAuthorName() throws ParsingException {
return JsonUtils.getString(item, "account.displayName");
return JsonUtils.getString(item, "account.name") + "@" + JsonUtils.getString(item, "account.host");
}

@Override
Expand Down
Loading