feat: trending queries

[grantdfoster]

Description

This PR fixes #297, #298

1. Refactoring and Code Organization:

   • Added TODO comments suggesting the refactoring of certain classes to a synapses directory for better organization, as they are used by both validator and miner components.

2. Enhancements in masa/base/miner.py:

   • Introduced a new method forward_tweets_synapse in BaseMinerNeuron to handle the forwarding of recent tweets using the RecentTweetsSynapse class.
   • Updated the axon attachment to use the new forward_tweets_synapse method.

3. Updates in masa/miner/masa_protocol_request.py:

   • Added an optional timeout parameter to the get and post methods to allow customizable request timeouts.

4. Improvements in masa/miner/twitter/tweets.py:

   • Modified the RecentTweetsSynapse class to include an optional timeout attribute.
   • Updated the forward_recent_tweets function to accept a max parameter, which is used to limit the number of tweets fetched.
   • Adjusted the TwitterTweetsRequest class to use the max_tweets parameter from the miner configuration.

5. Configuration Changes in masa/utils/config.py:

   • Added a new command-line argument --twitter.max_tweets_per_request to define the maximum number of tweets to scrape per request.

6. Enhancements in masa/validator/forwarder.py:

   • Replaced the fetch_twitter_config method with fetch_twitter_queries to fetch trending queries instead of a static configuration.
   • Updated the logic to check tweets since a specific date rather than just the current day.
   • Introduced a TweetValidator instance for validating tweets.

7. Dependency Update in pyproject.toml:

   • Updated the masa-ai dependency version from 0.2.3 to 0.2.5.

These changes aim to improve the flexibility, organization, and functionality of the codebase, particularly in handling Twitter data and configuration management.

Notes for Reviewers

None

Signed commits

• Yes, I signed my commits.

[codecov]

Codecov Report

Attention: Patch coverage is 74.89362% with 59 lines in your changes missing coverage. Please review.

Project coverage is 65.47%. Comparing base (59f0f18) to head (14fa81e).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
masa/validator/forwarder.py	59.37%	26 Missing ⚠️
masa/miner/twitter/tweets.py	29.41%	12 Missing ⚠️
masa/api/server.py	22.22%	7 Missing ⚠️
masa/base/validator.py	85.29%	5 Missing ⚠️
masa/base/miner.py	72.72%	3 Missing ⚠️
masa/utils/uids.py	85.00%	3 Missing ⚠️
masa/synapses/__init__.py	92.30%	2 Missing ⚠️
masa/validator/scorer.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #301       +/-   ##
===========================================
+ Coverage   53.18%   65.47%   +12.28%     
===========================================
  Files          23       23               
  Lines        1286     1318       +32     
===========================================
+ Hits          684      863      +179     
+ Misses        602      455      -147     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[hide-on-bush-x]

LGTM, tested, seems to be working as expected ( only on testnet )

[hide-on-bush-x]

would that make sense that the sample size of the organic is defined by the requester instead of the miner?

[grantdfoster]

we can definitely make a ticket for that - my concern is that there would need to be a hard-coded limit on the organic sample size, otherwise someone could spam the entire network and request 240 miners / call

[grantdfoster]

waiting for tests to pass / codcov report, and community input on mip2 before merging!

[hide-on-bush-x]

LGTM

One comment: I am facing some weird errors when getting 429 or similar from the protocol ( They are not clear enough I guess )

2024-11-20 10:30:07.382 |      ERROR       | bittensor:loggingmachine.py:457 |  - ConnectionError#af67eeea-81d9-429e-88a6-1c44fbee9f1b: HTTPConnectionPool(host='x.x.x.x', port=8080): Max retries exceeded with url: /api/v1/data/twitter/tweets/recent (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10f5ca210>: Failed to establish a new connection: [Errno 61] Connection refused')) -

Maybe we could improve this error handling and exposing something like "4XX error from protocol" to make that easier to understand

[grantdfoster]

@hide-on-bush-x added improved error handling in recent tweets request!