Add cbc as first canadian publisher #583
Conversation
src/fundus/publishers/ca/cbc_news.py
Outdated
|
|
||
| @attribute | ||
| def topics(self) -> List[str]: | ||
| return generic_topic_parsing(self.precomputed.ld.bf_search("ReportageNewsArticle")[0].get("articleSection")) |
There was a problem hiding this comment.
There seems to be a lot more information about the topics looking through the HTML using the term keyword.
Take this article for example.
src/fundus/publishers/ca/cbc_news.py
Outdated
| doc = document_fromstring(self.precomputed.html) | ||
| ld_nodes = self._author_ld_selector(doc) | ||
| try: | ||
| author_ld = json.loads(re.sub(r"(window\.__INITIAL_STATE__ = |;$)", "", ld_nodes[0].text_content())) | ||
| except json.JSONDecodeError: | ||
| return [] | ||
| if not (details := author_ld.get("detail")): | ||
| return [] | ||
| if not (content := details.get("content")): | ||
| return [] | ||
| return generic_author_parsing(content.get("authorList")) |
There was a problem hiding this comment.
Is there a reason we don't use the author given in the LD.
There was a problem hiding this comment.
Yes, it isn't accessible properly. This JSON is not wrapped within the usual ld json type and contains lists, which causes the bf_search to not work properly. I tried fixing it by replacing new.extend(v for v in node.values() if isinstance(v, dict) or isinstance(v, list)) this line in the function, but that ended up breaking the bf_search completely, so I figured this might be the straightforward option. But since we also need it in the topics, I did now implement a local fix.
There was a problem hiding this comment.
I think this should work now with the changes made here #592
There was a problem hiding this comment.
Unfortunately even with the changes made there, it still does not work, since the keywords are in a script block that is not classified as ld+json, so it is ignored by the default _base_setup.
There was a problem hiding this comment.
There is a really clean way to do this, but unfortunately, we need #588 for this. So let's finish this one after.
MaxDall
added
the
DON'T MERGE
MaxDall
removed
the
DON'T MERGE
No description provided.