Enhance Scraper

[urvishp80]

Bitcoin Transcripts:

This update will account for edits to transcripts and will handle this issue. Handling Edits and AI-Generated Transcripts in Bitcoin Transcripts Repository

Includes:

• Unique custom 'id' (based on file path and filename) for each document to avoid duplication of data
• Script that updates the document with the provided fields, keeping the other fields as it is, ensuring any changes in the file are updated in the ES index.
• Convert code from JavaScript to Python
• Remove redundant code

Mailing Lists:

• update custom URL for domain field
• change cron job to be run manually for an inactive mailing lists

Bitcoin Talk:

• migrate from Node.js to Python

Bitcoin Optech:

• migrate from Node.js to Python

[kouloumos]

I reviewed the new logic for Bitcoin Transcripts, and it looks good overall! I left a couple of comments inline.

Unfortunately, I still can't test this locally due to the "ModuleNotFoundError: No module named 'common'" error I mentioned in our DMs.

Feature idea:
It would be extremely useful if all the scripts had a test mode to execute the logic without involving the Elasticsearch index. Perhaps exporting the docs in JSON could make it easier to test without the Elasticsearch dependency. What do you think?

[kouloumos]

This looks good to me and I think it can be merged iff it has been tested by @urvishp80.

As I previously said, I have difficulty testing this locally. The difficulty of testing locally because of coupling it so tightly with ES index has also been raised by @Emmanuel-Develops on discord. It's okay to merge this, and I will open an issue for dealing with locally running the scraper, what do you think @urvishp80 ?

Also, there are additional changes that needs to be done on the bitcoin transcripts scraper to allow filtering for AI-generated transcripts. As this PR contains a lot of changes and refactors, I think it's better to do that later alongside some other changes that I have in mind (I'll open an issue for these also).

[kouloumos]

Nice additions!

[urvishp80]

closes #67

[urvishp80]

closes #68