Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I grasp that it can fetch internet resources, but I'm not really fully appreciating what that means. I was doing the challenges at, and I got past the one that requires urllib to fetch the page and read the source, but I don't see how this is useful quite yet. And examples are kind of lacking in diversity.

What else can it do, or how can I use this to my advantage?

(Besides building a web browser...)

share|improve this question
up vote 1 down vote accepted

you seem to be lacking use-cases: some examples:

  • access public APIs: google, twitter, etc.
  • write a custom web-crawler

or in the context of a web-application:

  • use http-based services (e.g. uploading to AWS)
  • test your own site / webapp
  • authenticate your webapp users with a third party service

... and many more, you can access any service exposed on the web.

share|improve this answer

Screen Scraping is one possible application

It is a technique that is commonly used to automate tasks that would normally involve a person opening a browser, browsing to a page and manually analysing a page.

Other more common uses might be to access and use web services, or any sort of communication where the server is accessible via a URL

share|improve this answer
Any reason for the downvote? – Karthik T Dec 19 '12 at 3:33
Wasn't me...... – Aerovistae Dec 19 '12 at 7:59

The internet and the world wide web is much, much bigger than just a bunch of web pages. There is a lot of information held behind web apis. For example, you can get stock quotes, weather, the price of tea in china, and so on.

Urllib, and libraries built on top of it, make it remarkably easy to gain access to this data. Much of it is in the form of JSON, which can easily be converted to native python data structures such as dictionaries and lists.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.