Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

When I want to learn a vast topic on Wikipedia, I don't know where to start. For instance say I want to learn about Binary Stars, I then have to know other things linked on that pages and linked pages on all the linked pages and so on for the specified number of levels. I want to write a web crawler like HTTracker or something similar, that will display a hierarchy of the links on a certain page and the links on those linked pages.I wish to use as much prewritten code as possible. Here is an example:

Pretending we are bending the rules by grabbing links from only the first sentence of each page

The example archives and "processes" two levels deep

The page is Ternary operation

The First Level

In mathematics a ternary operation is an N-ary operation

The Second Level

Under Mathematics:

Mathematics (from Greek μάθημα máthēma, “knowledge, study, learning”) is the abstract study of topics encompassing quantity, structure, space, change and others; it has no generally accepted definition.

Under N-ary

In logic,mathematics, and computer science, the arity i/ˈærɨti/ of a function or operation is the number of arguments or operands that the function takes

Under Operation

In its simplest meaning in mathematics and logic, an operation is an action or procedure which produces a new value from one or more input values


I need some way to determine what order to approach all these wiki pages to learn the concept ( in this case ternary operations )... Following along with this example, one way to show the path to read would a printout flowout like so:

enter image description here

This shows that the first sentence of the Mathematics page doesn't link to the first sentence of pages linked on ternary page two levels deep. (Please tell me how I should explain this ) ---> In other words, the child node of the top pages first sentence, ternary_operation, does not have any child nodes that reference the children of the top pages other children nodes- N-ary and operation. Thus it is safe to read this first. Since N-ary has a link to operations we should read the operation page second and finally read the N-ary page last.

Again, I wish to use as much prewritten code as possible, and was wondering what language to use and what would be the simplest way to go about doing this if there isn't already something out there?

share|improve this question

closed as too broad by gnat, GlenH7, MichaelT, user61852, Dynamic Jun 30 '14 at 18:02

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs. If this question can be reworded to fit the rules in the help center, please edit the question.

2 Answers 2

If I would do it, I would crawl everything with Selenium, even if it is slow it works behind a browser so it will not be seen as a crawler, it has JavaScript, you can do logins, you can press buttons, you can use Xpath to select what you want, you can save the pages containing images, and many more.

On the other hand if you just want an offline Wikipedia you can check out WikiTaxi project.

share|improve this answer

Wikipedia on Focused Crawler,

A focused crawler ideally would like to download only web pages that are relevant to a particular topic and avoid downloading all others. Therefore a focused crawler may predict the probability that a link to a particular page is relevant before actually downloading the page.

Check out Focused Crawling Using Context Graphs by M. Diligenti et. al. It presents a nice model. Also, section 3.3 discusses a crawling strategy,

The crawler utilizes the classifiers trained during the context graph generation stage to organize pages into a sequence of M = N + 2 queues, where N is the maximum depth of the context graphs. The i-th class (layer) is associated to the i-th queue i = 0, 1,..., N. Queue number N + 1 is not associated with any class, but reflects assignments to “other”. The 0-th queue will ultimately store all the retrieved topically relevant documents.

Initially all the queues are empty except for the dummy queue N + 1, which is initialized with the starting URL of the crawl. The crawler retrieves the page pointed to by the URL, computes the reduced vector representation and extracts all the hyperlinks. The crawler then downloads all the children of the current page. All downloaded pages are classified individually and assigned to the queue corresponding to the winning layer, or the class “other”. Each queue is maintained in a sorted state according to the likelihood score associated with its constituent documents. When the crawler needs the next document to move to, it pops from the first non-empty queue. The documents that are expected to rapidly lead to targets are therefore followed before documents that will in probability require more steps to yield relevant pages. However, depending on the relative queue thresholds, frequently high-confidence pages from queues representing longer download paths are retrieved.

The setting of the classifier thresholds that determine whether a document gets assigned to the class denoted “other” determines the retrieval strategy. In our default implementation the likelihood function for each layer is applied to all the patterns in the training set for that layer. The confidence threshold is then set equal to the minimum likelihood obtained on the training set for the corresponding layer.

During the crawling phase, new context graphs can periodically be built for every topically relevant element found in queue 0. However, our focused crawler can also be configured to ask for the immediate parents of every document as it appears in queue 0, and simply insert these into the appropriate queue without re-computing the merged context graph and classifiers. In this way it is possible to continually exploit back-crawling at a reasonable computational cost.

This strategy may not be exactly what you need. But this should come to help. Also check out,

share|improve this answer
@Eduard Florinescu and Shuvo Naser I appreciate you input. However, crawling is maybe less needed than I specified or maybe not exactly what I meant. I want the program to finally show me in which order I should read the wiki pages based on what pages reference/link what... Maybe after doing this I could crawl the pages in a web archive sort where I could click next and it would bring me to the next page ( whether it is a referenced link or back to the page that reference the link )... –  Chris Okyen Oct 8 '12 at 17:52
@Eduard Florinescu I want to have it so it will be "smart" in picking the order of pages I need to read to understand a page or group of pages "concept"; doing so by partly AI in a sense and partly by the user giving feedback.... –  Chris Okyen Oct 8 '12 at 20:16
@Chris Okyen, Check out the rewritten the answer. –  IAmTheDude Oct 9 '12 at 4:55
@ChrisOkyen: I think I understand what you want, but I am afraid you need to work out your own solution, I don't know of any prepackaged solution for that, also I think it will take a while to make that solution, and you probably will get better with reviewing and editing the articles manually in the first place. –  Eduard Florinescu Oct 9 '12 at 8:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.