No more data! Scraping will stop now. - Cannot fetch more tweets using twint

Question

Based on twint official documentation, it should not be hard to fetch 3200 tweets from a specific user. However, my problem is: after I run the config code, I only get the most recent 20-60 tweets. Something must be wrong and hope to get some.

I installed the latest version of twint and here is my config code

c = twint.Config()
c.Limit = 3200
c.Username = "jerallaire"
c.Pandas = True
c.Retweets = True
c.Output = "Tweets.csv"

twint.run.Search(c)

Answer 1

'''There is a workaround, but it has a limitation to 20 tweets, at least for me. It works to retrieve tweets beyond 22nd of August, but you have to set a small interval for 'c.Since' and 'c.Until'.

e.g.:

c.Since = '2021-03-21'
c.Until = '2021-03-22'

Be aware that even with this one, it fails somethimes. If you set 'c.Pandas' to True, you could check if your dataframe is empty and if so, run again the configuration (twint.run.Search(c))'''

seems that everyone is having the same problem with the limitation tweets on the github repo.

Try to put the date you want to receive the tweets. maybe you can make a script that change the date every single day.

Answer 2

There is a fork repository for this project that worked for me.

https://github.com/woluxwolu/twint

And use: twint.run.Profile(c)

Answer 3

Twint seems to be archived by the author. The support for it is also getting reduced. I suggest using some other library that still has support.

I used snscrape for tweet extraction and its serving my purpose.

In your case you can try like this:

import snscrape.modules.twitter as sntwitter
import pandas as pd

search_query = "(from:@jerallaire) until:2022-09-01 since:2017-08-01"
tweets = []
tweet_limit = 3200

for tweet in sntwitter.TwitterSearchScraper(search_query).get_items():
    if len(tweets) == tweet_limit:
        break
    else:
        tweets.append([tweet.date, tweet.username, tweet.content])
        
df = pd.DataFrame(tweets, columns=['Date', 'User', 'Tweet'])

df.to_csv('tweetsjerallaire.csv')

Have a look https://github.com/JustAnotherArchivist/snscrape