Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twitter searches fail with blocked (403) #846

Closed
SapiensSuwan opened this issue Apr 20, 2023 · 206 comments
Closed

Twitter searches fail with blocked (403) #846

SapiensSuwan opened this issue Apr 20, 2023 · 206 comments
Labels
bug Something isn't working module:twitter upstream

Comments

@SapiensSuwan
Copy link

SapiensSuwan commented Apr 20, 2023

OS: macOS Venture 13.3.1

Python Version: Python 3.8.3

snscrape Version: snscrape 0.6.2.20230320

snscrape module: snscrape.modules.twitter -> TwitterSearchScraper

Bug log:

Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=bitcoin+since%3A2021-07-01+until%3A2022-12-31&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe: blocked (403)
4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=bitcoin+since%3A2021-07-01+until%3A2022-12-31&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.
Errors: blocked (403), blocked (403), blocked (403), blocked (403)
Traceback (most recent call last):
  File "bitcoin21-22-01.py", line 11, in <module>
    for i, tweet in enumerate(sntwitter.TwitterSearchScraper('bitcoin since:2021-07-01 until:2022-12-31').get_items()):
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 1661, in get_items
    for obj in self._iter_api_data('https://api.twitter.com/2/search/adaptive.json', _TwitterAPIType.V2, params, paginationParams, cursor = self._cursor):
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 761, in _iter_api_data
    obj = self._get_api_data(endpoint, apiType, reqParams)
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/modules/twitter.py", line 727, in _get_api_data
    r = self._get(endpoint, params = params, headers = self._apiHeaders, responseOkCallback = self._check_api_response)
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/base.py", line 251, in _get
    return self._request('GET', *args, **kwargs)
  File "/Users/opt/anaconda3/lib/python3.8/site-packages/snscrape/base.py", line 247, in _request
    raise ScraperException(msg)
snscrape.base.ScraperException: 4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&include_ext_is_blue_verified=1&include_ext_verified_type=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_ext_limited_action_results=false&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_ext_collab_control=true&include_ext_views=true&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=bitcoin+since%3A2021-07-01+until%3A2022-12-31&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&include_ext_edit_control=true&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo%2CeditControl%2Ccollab_control%2Cvibe failed, giving up.

Original Code:

import snscrape.modules.twitter as sntwitter
import pandas as pd

# Setting variables to be used below
# maxTweets = 5000

# Creating list to append tweet data to
tweets_list1 = []

# Using TwitterSearchScraper to scrape data and append tweets to list
for i, tweet in enumerate(sntwitter.TwitterSearchScraper('bitcoin since:2021-07-01 until:2022-12-31').get_items()):
    # if i>maxTweets:
    #     break

    tweets_list1.append([tweet.date, tweet.id, tweet.rawContent, tweet.user.username])

    if (i + 1) % 5000 == 0:

        # Creating a dataframe from the tweets list above
        tweet_df1 = pd.DataFrame(tweets_list1, columns=['Datetime', 'Tweet Id', 'Text', 'Username'])
        print(tweet_df1)

        # Export dataframe into a csv
        tweet_df1.to_csv(f'bitcoin-{(i+1)/5000}.csv', sep=',', index=False)
        tweets_list1 = []

Note: The code was working a couple days ago and now it gives me this error again. I did try to search bitcoin as keyword in twitter and it worked fine. Sorry in advance if I made any duplicated issue report. I haven't seen a 403 error here.

@JustAnotherArchivist
Copy link
Owner

JustAnotherArchivist commented Apr 20, 2023

Yeah, the exact error is indeed new, but this is effectively #834.
However, the 403 affects all searches, not only the 'latest' one, so I'll leave it open anyway.

@JustAnotherArchivist JustAnotherArchivist changed the title sntwitter - Blocked(403) error Twitter searches fail with blocked (403) Apr 20, 2023
@emmanuelva
Copy link

The Search page from twitter seems to be gone

@nicolegorton

This comment was marked as spam.

@JustAnotherArchivist
Copy link
Owner

There is no need to repeat 'me too!!1!'. Yes, everyone is affected at the moment.

@Wouze
Copy link

Wouze commented Apr 20, 2023

The Search page from twitter seems to be gone

yes for some reason
I've seen tweets from elon about how twitter data is getting ripped of
idk if there is any relation
https://twitter.com/elonmusk/status/1648793947806871556?s=20

@Wouze
Copy link

Wouze commented Apr 20, 2023

so basically twitter search is no longer available, only available to logged in users. Is there a way of doing the same searches while being logged in? maybe by providing tokens/cookies/email&pass?

@JustAnotherArchivist
Copy link
Owner

@Wouze #270

@AlekseiShkurin
Copy link

Are there any solutions being explored at the moment? Or any temp fixes anyone discovered?

@cunninghamx
Copy link

:f: in chat for twitter search. Search now redirects to login. @JustAnotherArchivist i hope we as a community find another way in, but thank you so much for the hard work you have put into this

@Fa5g
Copy link

Fa5g commented Apr 21, 2023

I think it would be a really good idea to fix snscrape to work once the user logs in and improve it to post auto messages, add follow/unfollow features, etc. taking care of the limits of Twitter, in this way the tool would not be forgotten.

@TheTechRobo
Copy link
Contributor

@Fa5g As several people have mentioned in this thread, snscrape can't and won't support authentication, although you are welcome to make a fork of it: #270

@dengkefeng

This comment was marked as off-topic.

@JustAnotherArchivist

This comment was marked as off-topic.

@Hesko123
Copy link

Hesko123 commented Apr 21, 2023

It seems that it's now coming to an end. Fck elon musk ngl, fcked my journey.

https://nitter.net/search?f=tweets&q=testing&since=&until=&near=

Nitter seems working for searches ! (maybe a hope for a future workaround)

@dengkefeng

This comment was marked as off-topic.

@dengkefeng

This comment was marked as off-topic.

@ccaballes

This comment was marked as spam.

@ccaballes

This comment was marked as off-topic.

@iGrumpyMonk

This comment was marked as off-topic.

@Isid28

This comment was marked as off-topic.

@katesanalyst

This comment was marked as off-topic.

@ccaballes

This comment was marked as off-topic.

@katesanalyst

This comment was marked as off-topic.

@katesanalyst

This comment was marked as off-topic.

@vladkens

This comment was marked as off-topic.

@leetfin

This comment was marked as resolved.

@JustAnotherArchivist
Copy link
Owner

JustAnotherArchivist commented May 10, 2023

The eagle has landed.
User searches (that is, searching for users by keyword, not searching for a user's tweets) are currently not supported as they use a completely different API.
List searches (including the twitter-list-posts alias) are not possible with the new API.
For any other problems, please file a complete bug report.

@jannat5134

This comment was marked as off-topic.

@jannat5134

This comment was marked as off-topic.

@JustAnotherArchivist

This comment was marked as off-topic.

@engrCoy
Copy link

engrCoy commented May 10, 2023

The eagle has landed. User searches (that is, searching for users by keyword, not searching for a user's tweets) are currently not supported as they use a completely different API. List searches (including the twitter-list-posts alias) are not possible with the new API. For any other problems, please file a complete bug report.

Sorry just wanted to as a question. I might just be confused, but what I understand is that we're not able to search by keyword with the new API right? Is there a different way to extract from the API or Twitter isn't allowing any of the search functions?

@JustAnotherArchivist
Copy link
Owner

@engrCoy Yeah, you misread that. Searching for tweets works now except for a few unsupported filters like list:...; there isn't currently a list of what works and doesn't, but keywords are no problem. It's only the search for users (the 'Users' tab on the web search) that's broken because the new API endpoint does not support it.

One general word of caution: the new API is even more unreliable than the old one; you will regularly see scrapes terminate early for no apparent reason.

@engrCoy
Copy link

engrCoy commented May 10, 2023

@JustAnotherArchivist I appreciate the clarification on that, and also the precaution about the unreliable output due to an abrupt termination.

As for the use of keywords I tried using that on a sample code that I made(please refer to the code attached below) but I think it's still blocked or something. Oh, please let me know if this is a separate problem that I need a bug report. Thanks in advance, and I love your work!

import snscrape.modules.twitter as sntwitter
import pandas as pd

tweets = []

for i,tweet in enumerate(sntwitter.TwitterSearchScraper(query = "#opportunity").get_items()):
    if i>100:
        break
    tweets.append([tweet.user.username, tweet.date, tweet.likeCount, tweet.retweetCount, tweet.replyCount, tweet.lang,
                   tweet.sourceLabel, tweet.content])

@JustAnotherArchivist
Copy link
Owner

That search works fine for me with the latest dev version. If that's what you're using, yes, please file a complete bug report.

@engrCoy
Copy link

engrCoy commented May 10, 2023

Will do, thank you so much!

@anchitshrivastava
Copy link

Can you elaborate a bit about the Dev version?
How do I install it?

That search works fine for me with the latest dev version. If that's what you're using, yes, please file a complete bug report.

@defaultsettings
Copy link

Can you elaborate a bit about the Dev version? How do I install it?

That search works fine for me with the latest dev version. If that's what you're using, yes, please file a complete bug report.

you can see following post - https://stackoverflow.com/questions/19042389/conda-installing-upgrading-directly-from-github

@JustAnotherArchivist
Copy link
Owner

@anchitshrivastava It's explained in the README.

@ihabpalamino
Copy link

helloo i am sorry but i am using snscrape version 0.6.2.20230320 and still does not work for me the search should i do an update?

@myt-christine

This comment was marked as off-topic.

@kkedji

This comment was marked as off-topic.

@myt-christine

This comment was marked as off-topic.

@oguzhari
Copy link

helloo i am sorry but i am using snscrape version 0.6.2.20230320 and still does not work for me the search should i do an update?

Use git version.

pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git

@moral-drip
Copy link

helloo i am sorry but i am using snscrape version 0.6.2.20230320 and still does not work for me the search should i do an update?

Use git version.

pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git

That worked for me! Issue resolved.

@ihabpalamino
Copy link

thank you so much guys it works for me now @JustAnotherArchivist Thank you for your work

@nv78

This comment was marked as resolved.

@ihabpalamino

This comment was marked as resolved.

@ihabpalamino

This comment was marked as resolved.

@JustAnotherArchivist
Copy link
Owner

As already mentioned, the fix is currently only available in the development version. A new release will be made soon if it turns out there are not too many sharp edges. There are also a few unrelated things I want to implement/fix first.

To upgrade your existing installation to the latest development version until then, use:

pip3 install --upgrade git+https://github.com/JustAnotherArchivist/snscrape.git

I will lock this issue now since the bug is fixed. Please file a complete bug report if you still experience problems not described above.

Repository owner locked as resolved and limited conversation to collaborators May 10, 2023
@JustAnotherArchivist JustAnotherArchivist unpinned this issue Jun 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working module:twitter upstream
Projects
None yet
Development

No branches or pull requests