No more data! Scraping will stop now. - Cannot fetch more tweets using twint

[Logenleedev]

Make sure you've checked the following:

• [] Python version is 3.6;
• [] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• [] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

Please provide the exact command ran including the username/search/code so I may reproduce the issue.

In jupyter notebook, I enter the following config code:

c = twint.Config()
c.Limit = 3200
c.Username = "jerallaire"
c.Pandas = True
c.Retweets = True
c.Output = "Tweets.csv"

twint.run.Search(c)

Description of Issue

Please use as much detail as possible.

Based on twint official documentation, it should not be hard to fetch the most recent 3200 tweets from a specific user. However, my problem is: after I run the config code, I only get the most recent 20-60 tweets. Something must be wrong and hope to get some.

Environment Details

Using Windows, Linux? What OS version? Running this in Anaconda? Jupyter Notebook? Terminal?

Mac. I am running this code on Jupyter Notebook

[dominikbenk]

Same issue.

[christineeeeee]

same issue here. i could only fetch ~90 tweets including "apple" and "$aapl" for one day, which is not possible...

[Dentordev]

Have we tried selenium or requests HTML? They have the ability to scroll down and render more of what is hidden

[johnguirgis]

I am also facing the same issue.

[iamthelabhras]

Same issue here. Running Twint on MacOS (Big Sur 11.4) from the command line.

[gansanay]

Same issue here with Python 3.9.6 on Windows 10 with roughly the same code (get all tweets for one username), having just ran pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint makes things even worse.

[MarionJacquet]

Same issue here - using Python 3.10.0 on MacOS (Big Sur 11.6) from the command line
Twint only retrieves the last ≈2050 tweets, published in the past 10 days... I modified url.py (lines 70+92) according to instructions : problem remains...

++ I also tried fetching tweets that I know exist from 2018 = Twint immediately returns "no more data" message

[richardpickett]

+1

Ubuntu, python 3.8.10, both pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint and normal checkout + install, always gets about 200 tweets.

I've tried --resume, doesn't matter.

cmd I'm running: twint -u balajis -o t.csv --csv --count

[19barsav]

Seems like it's only pulling the last 7 days for me, so I believe I have the same issue.

[CharlesXu123]

same issue here, checked out the latest version and it only give 20-80 tweets running on linux.

[queerterpreter]

I was having the same issue, where a search with the exact same parameters would pull a random number of tweets between 0 and 60 (same top tweets each time, so what was random was where it stopped)

As per #1319, I deleted the hashtag in line 92 of url.py and it now works with no problem.

[2spoopy4me]

I'm only getting a certain amount of tweets as well...any ideas?

[weirdyang]

#1266 (comment)

[eminekahveci]

Friends, I changed the typed_query on lines 135 and 136 in url.py and the twint started to work, but only the last week's data is coming. How can we remove this limit? How can I get more history and more

[eminekahveci]

[sarahdewolf]

Did anyone find a solution or work around?

[2spoopy4me]

I'd like to spend some time doing a deep-dive on the code to figure out why it's not working. I don't have much info on how twint works; maybe the nature by which is interacts with Twitter, twitter has limited it's functionality so twint can't work properly?