Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

After 1 months research I basically give up on getting all URL's from a search results programmatically,

I looked at Google Search API to find a way to get millions of search results "URL's" to be specific to a text file or something relative but no success, but I am 100% there must be a way or trick of doing it.

Real Question :

Is there anyway programmatically or manually I can get 1000+ search results (URLs using search query e.g. "Apple" returns million of results on google and I want as much as possible URLs of them results in a text file)

Note :

Don't care for any specific search engine or programming language or technique or software or just point me to right direction, but yeah I tried it with google API i can't get more then 100 results at all.

share|improve this question
What exactly are you trying to achieve? Maybe there is a better way to achieve it. – Loki Astari Sep 11 '12 at 21:21
See my answer to :… – Morons Sep 12 '12 at 2:50
@Morons sorry but your answer ain't helpful – tereško Sep 12 '12 at 7:47
@LokiAstari I want to get all URLs for a specific query for whole web :) – tereško Sep 12 '12 at 7:48
@tereško: No. That is a step in what you are trying to achieve. What are you actually trying to achieve. Maybe there is a better way to achieve your end result. – Loki Astari Sep 12 '12 at 16:21

Here is an open source project that you could use to model what you want. It is in Java, but the source looks well commented.

share|improve this answer
it won't gonna return thousands of results back or will it :) – tereško Sep 12 '12 at 7:49
After reading through the methods and such, it looks like you can page through the results. So you will get x results at a time, and then you can ask for page 2, page 3, etc. – david.tanner Sep 12 '12 at 14:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.