Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying some machine learning with Twitter, and I was wondering if there were any pre-collected just massive tweet data that anyone knew about before I went and just used the API (which has a request limit anyway)?

share|improve this question

closed as off-topic by gnat, Ixrec, Scant Roger, MichaelT, Matthieu M. Dec 21 '15 at 12:32

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to find or recommend tools, libraries, programming languages, resources (including books, blogs, tutorials, and examples), or projects to undertake are off-topic on Programmers as they attract opinionated answers that won't have lasting value to others. You may be able to get help in The Whiteboard, our chat room." – gnat, Ixrec, Scant Roger, Community, Matthieu M.
If this question can be reworded to fit the rules in the help center, please edit the question.

How many is "massive"? – Morons Dec 8 '11 at 3:15
Really hoping there is an answer to this. – Kyle Hodgson Dec 8 '11 at 3:16
Must it be twitter? Wikipedia provides all its data as a free download: Wikipedia dataset – edwardw Dec 8 '11 at 3:48

You can't. There were some available, such as 476 million tweets from SNAP. But Twitter has changed its Terms of Service to make such distribution of tweets impossible.

share|improve this answer

Check out InfoChimps. It looks like they have both free and paid datasets. You could also take a look at this similar question on Quora, which mentions some older datasets that are no longer public, but may be available by contacting the sources. Another question from GetTheData lists other possibilities.


All that said, it looks like Twitter's API TOS prevents the sharing of datasets with tweet texts, so that might be why it's difficult to find something. Still, it looks like someone found a way around that by distributing a dataset with tweet IDs and then reassembling the dataset with a Python script. I haven't tested it, but you take a look at it here.

share|improve this answer

The API is has a request limit, but the querying directly on the site is is not limited. You just need to do a bit of Browser automation\scraping.* (As others have stated, you can't distribute the data, but you should be fine for your own personal use)

I did not verify if this is acceptable with in the Twitter TOS. But you don't need to agree to the TOS to search, so...

share|improve this answer
Screen scraping would fall under the site TOS which you implicitly agree to by visiting the site. Launching a screenscraper counts as visiting the site. Had to deal with such things in the past when we needed some financial data from a stock exchange who didn't provide an API, we ended up getting a contract with them explicitly permitting us to scrape. – jwenting Dec 8 '11 at 7:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.