[archiveorg] Fix extraction #23827
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -40,9 +40,12 @@ def _real_extract(self, url): | |
| video_id = self._match_id(url) | ||
| webpage = self._download_webpage( | ||
| 'http://archive.org/embed/' + video_id, video_id) | ||
| input_element_with_playlist = self._search_regex( | ||
| r'(<\s*input.*\s*class\s*=\s*[\'"].*\s*js-play8-playlist\s*.*[\'"]\s*.*>)', | ||
| webpage, 'jwplayer playlist') | ||
| jwplayer_playlist = self._parse_json(self._search_regex( | ||
| r'.*\s+value\s*=\s*[\'"](.+)[\'"][\s/]', | ||
| input_element_with_playlist, 'playlist data'), video_id) | ||
| info = self._parse_jwplayer_data( | ||
| {'playlist': jwplayer_playlist}, video_id, base_url=url) | ||
|
|
||
|
|
@@ -52,7 +55,7 @@ def get_optional(metadata, field): | |
| metadata = self._download_json( | ||
| 'http://archive.org/details/' + video_id, video_id, query={ | ||
| 'output': 'json', | ||
| }).get('metadata', {}) | ||
|
There was a problem hiding this comment. The idea is to make the extraction more resilient against website changes. With metadata being optional for downloading videos, it's probably better not to crash if the metadata element changes its name. There was a problem hiding this comment. This is not enough. If you reconsider this optional then you must also:
There was a problem hiding this comment. You are right. Thanks. Commit 1326a5a fixes this by setting |
||
| info.update({ | ||
| 'title': get_optional(metadata, 'title') or info.get('title'), | ||
| 'description': clean_html(get_optional(metadata, 'description')), | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extract_attributes.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the pointer. I changed the code to use
extract_attributes()in commit e910f49.It was still bugging me that I couldn't use
get_element_by_class()in the line before as it only returns the content of the element and not the element (as the name of the function indicates). So I changedget_element_by_class()(and it's associated functions) in commit b98d1c0 to accept an optionalinclude_tagparameter. By default it's set toFalse. So no existing code will be affected by it.This should prove useful in the future and make those functions more intuitive to use.
I also added tests to
test_utils.py. While doing so I noticed thatget_element_by_class()didn't work for class names starting with a hyphen. I fixed that, too.