Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[instagram] Fix for missing edge_media_to_comment' field and add date' metadata #250

Merged
merged 2 commits into from
May 8, 2019
Merged

[instagram] Fix for missing edge_media_to_comment' field and add date' metadata #250

merged 2 commits into from
May 8, 2019

Conversation

iamleot
Copy link
Contributor

@iamleot iamleot commented May 8, 2019

Probably since last hours/days (at least the CI has not spotted
that problem yet) the edge_media_to_comment' field is not always present in the sharedData JSON. Sometimes it is present in the response, other times not, also for the same URL. Previously, the count' field of edge_media_to_comment' was used to populate comments' metadata with the number of comments.
For completeness a complete test showing such error will be added
at the end of this pull request.

Add date' metadata using taken_at_timestamp' field.

Here the complete output of the failing tests:

% env PYTHONPATH=. python3.7 test/test_results.py instagram

https://www.instagram.com/p/BqvsDleB3lV/
.
https://www.instagram.com/p/BoHk1haB5tM/
E
https://www.instagram.com/p/Bqxp0VSBgJg/
.
https://www.instagram.com/p/BtOvDOfhvRr/
.
https://www.instagram.com/explore/tags/instagram/
E
https://www.instagram.com/instagram/
.
======================================================================
ERROR: test_InstagramImageExtractor_2 (__main__.TestExtractorResults)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_results.py", line 258, in test
    self._run_test(extr, url, result)
  File "test/test_results.py", line 61, in _run_test
    tjob.run()
  File "test/test_results.py", line 152, in run
    for msg in self.extractor:
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 32, in items
    for data in self.instagrams():
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 232, in instagrams
    return self._extract_postpage(url)
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 53, in _extract_postpage
    'comments': text.parse_int(media['edge_media_to_comment']['count']),
KeyError: 'edge_media_to_comment'

======================================================================
ERROR: test_InstagramTagExtractor_1 (__main__.TestExtractorResults)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_results.py", line 258, in test
    self._run_test(extr, url, result)
  File "test/test_results.py", line 61, in _run_test
    tjob.run()
  File "test/test_results.py", line 152, in run
    for msg in self.extractor:
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 32, in items
    for data in self.instagrams():
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 162, in _extract_tagpage
    yield from self._extract_page(url, 'TagPage')
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 135, in _extract_page
    yield from self._extract_postpage(url)
  File ".../gallery-dl/gallery_dl/extractor/instagram.py", line 53, in _extract_postpage
    'comments': text.parse_int(media['edge_media_to_comment']['count']),
KeyError: 'edge_media_to_comment'

----------------------------------------------------------------------
Ran 6 tests in 11.946s

FAILED (errors=2)

iamleot added 2 commits May 8, 2019 13:58
`edge_media_to_comment' is no longer always present in the response
(also for the same media sometimes is present and sometimes is not present).
@mikf
Copy link
Owner

mikf commented May 8, 2019

Thank you!

@mikf mikf merged commit f347d2d into mikf:master May 8, 2019
@iamleot
Copy link
Contributor Author

iamleot commented May 8, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants