Update crypto newsletter substack to RSS. fixes #4496 #4497
Conversation
| @@ -18,8 +18,8 @@ | |||
|
|
|||
|
|
|||
| @log_start_end(log=logger) | |||
| def scrape_substack(url: str) -> List[List[str]]: | |||
| """Helper method to scrape newsletters from substack. | |||
| def scrape_substack_rss(url: str, limit: int = 10) -> List[List[str]]: | |||
There was a problem hiding this comment.
Updated to Substack RSS naming for less confusion.
| ) | ||
| rss = soup.find("rss") | ||
| if rss: | ||
| posts = rss.find_all("item")[:limit] |
There was a problem hiding this comment.
RSS XML feeds use "item"
| time: str = post.find("time").get("datetime") | ||
| title: str = post.title.text | ||
| post_url: str = post.link.text | ||
| time_str = post.pubDate.text.split(" (")[0] |
There was a problem hiding this comment.
RSS items use "pubDate" as the date. This also splits since substack returns extra text in the pubdate that has to be parsed out.
| "https://thedefiant.io/api/feed", | ||
| "https://thedailygwei.substack.com/feed", | ||
| "https://todayindefi.substack.com/feed", | ||
| "https://defislate.substack.com/feed", |
There was a problem hiding this comment.
All the same substacks, just the RSS feed. Also removed Bankless.
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## develop #4497 +/- ##
==========================================
Coverage ? 55.37%
==========================================
Files ? 585
Lines ? 53216
Branches ? 0
==========================================
Hits ? 29470
Misses ? 23746
Partials ? 0 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
Description
Substack must have updated their HTML and UI, so this PR migrates to RSS feeds to fetch newsletter updates.
This PR:
Issue
#Fixes 4496
Screenshots
Before
After
How has this been tested?
Checklist:
feature/feature-nameorhotfix/hotfix-name.Update our documentation following these guidelines. Update any user guides that are affected by the changes.If a feature was added make sure to add it to the corresponding integration test script.Others
I have commented my code, particularly in hard-to-understand areas.