I think there is a lot of information to gain, but most of the time we don't know how. From personal experience:
Huge amount of data
We did not work in the super big data space, but we had about 30GB of data extracted. First we didn't think to much about having a good database schema, because we didn't know yet which information would be useful and used at all. It has become a problem. Having a good idea of how to store the data and set up the database nicely for big data is in my opinion very important. Don't underestimate that.
The most important thing seems to be the queries. Using only the ones with offer the most performance and not requiring table lookups (sql) is necessary. NoSql types like Neo4j's graphdatabase offer themselves to store huge networks. As the web can be seen as one huge network, it may be something to think about...
Preparing the data
Only to retrieve data from the web does not really help. It has to be preprocessed to unfold the knowledge we want to acquire. I can provide two examples.
- We gathered a reasonable amount of anonymized reviews. The goal was to predict the rating based on the text. Now you have a lot of unstructured text, which is unusable information for any machine learning technique. We need features, vectors (etc.) to train a model for prediction. Bringing the text into structured information is one whole chapter itself. You may find a sizeable amount of papers in the internet about text mining. I can recommend having a look at opennlp.apache.org.
- You can also extract pretty simple information, which is very obvious in the first place, but may unveil unexpected information. Analyzing networks like Facebook or Mail traffic can give information about ones person. Twitter is also such a source. How many followers does one have, how many is he following. The point here is, accessing these information is not easy. Most data 'stores' like Google, Facebook or Amazon are slowly but surely closing their API. You will need keys to access the data. These keys are sold by the companies, so you basically have to buy the information you get. Some of them still have limited public API for free, but that gives you only very limited possibilities.
What is the information you want to find?
Both examples have one in common. We didn't know what information can be gained at all. I mean we already know that Lady Gaga has a LOT of followers. But what about John Doe. Does his network tell you something about his future? There is a theory that people with higher betweenness centrality would have potentially more success with their start-up. But where does such a theory come from? Only by investigation of past data over time.
If we go back to the web mining and take an example, that we would like to know which job offers the most vacancies at the moment. We have a concrete goal here. We fetch webcontent and parse the text. In a first step we may only look for words like 'job', 'salary' etc. But how to find the 'word' in the text which equals the job ('software engineer'). Lets assume we have an ontology for that. We are able to fetch all the data, parse it and out comes the amount of vacancies per job. It requires a lot of advanced stuff to get decent results. It is nowhere near good. If someone has other approaches or better ones he is welcome to share them :)
But in contrast to this example, most of the time you don't know what you are looking for. It is like testing and seeing what comes out. Maybe you have studies, researches or papers which tell you what you might look for. I don't know where the possibilities end. I think the information of the web contains a lot more information in between than is visible.
I know, it's a lot of text. But I wanted to give you some insight, because your 2 points are very hard to answer. Based on your tag "Career Development" I guess you want to predict a persons 'fitting job'?
What is the pros and cons of the information extraction and web mining
It's hard to extract and find what you are looking for. Its also hard to obtain the information you need, because a lot of 'nice-to-have' datasources are to pay or simply not accessible. But if you have them, you can extract A LOT of knowledge and predictions. For careers for example you might want to see what most programmes have in common to get a 'native programmers profile'. But first - find a way to access person profiles... :)
What is the present significance of them ?
The significance ranges from millions of dollars to none. If you can read German (or let Google translate it) https://www.web-analyzer.com/ is worth a look. They extract some information transform it into money. It's hard to say if they have a lot of customers. I doubt it, because it's a vague business, you can't prove anything.
People claim that it is possible to predict stock exchange prices. If it was... you decide what value this information would have. As I said, I don't know how far the potential information from the web is reaching. But I believe it predicts a lot sales of products.
If you were able to predict a job of a person based on their profile or behaviour in the Internet, then find a business model to sell this information. Isn't it quite similar to the adds in Facebook. They show you only the adds which are most promising for you, based on your profile (if you are single or engaged). You may then show vacancies or courses for education, job information, etc.