Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm trying some machine learning with Twitter, and I was wondering if there were any pre-collected just massive tweet data that anyone knew about before I went and just used the API (which has a request limit anyway)?

share|improve this question
    
How many is "massive"? –  Morons Dec 8 '11 at 3:15
    
Really hoping there is an answer to this. –  Kyle Hodgson Dec 8 '11 at 3:16
1  
Must it be twitter? Wikipedia provides all its data as a free download: Wikipedia dataset –  edwardw Dec 8 '11 at 3:48

3 Answers 3

You can't. There were some available, such as 476 million tweets from SNAP. But Twitter has changed its Terms of Service to make such distribution of tweets impossible.

share|improve this answer

Check out InfoChimps. It looks like they have both free and paid datasets. You could also take a look at this similar question on Quora, which mentions some older datasets that are no longer public, but may be available by contacting the sources. Another question from GetTheData lists other possibilities.

Edit

All that said, it looks like Twitter's API TOS prevents the sharing of datasets with tweet texts, so that might be why it's difficult to find something. Still, it looks like someone found a way around that by distributing a dataset with tweet IDs and then reassembling the dataset with a Python script. I haven't tested it, but you take a look at it here.

share|improve this answer

The API is has a request limit, but the querying directly on the site is is not limited. You just need to do a bit of Browser automation\scraping.* (As others have stated, you can't distribute the data, but you should be fine for your own personal use)

I did not verify if this is acceptable with in the Twitter TOS. But you don't need to agree to the TOS to search, so...

share|improve this answer
    
Screen scraping would fall under the site TOS which you implicitly agree to by visiting the site. Launching a screenscraper counts as visiting the site. Had to deal with such things in the past when we needed some financial data from a stock exchange who didn't provide an API, we ended up getting a contract with them explicitly permitting us to scrape. –  jwenting Dec 8 '11 at 7:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.