Web mining and Information extraction

Question

There is a path to build your future but there is no detail information about these path. So, I am asking these questions to here ;

• What is the pros and cons of the information extraction and web mining ?
• What is the present significance of them ?

NOTE: If you have other question related these path, feel free to reedit my question.

Answer 1

I think there is a lot of information to gain, but most of the time we don't know how. From personal experience:

Huge amount of data

We did not work in the super big data space, but we had about 30GB of data extracted. First we didn't think to much about having a good database schema, because we didn't know yet which information would be useful and used at all. It has become a problem. Having a good idea of how to store the data and set up the database nicely for big data is in my opinion very important. Don't underestimate that.

The most important thing seems to be the queries. Using only the ones with offer the most performance and not requiring table lookups (sql) is necessary. NoSql types like Neo4j's graphdatabase offer themselves to store huge networks. As the web can be seen as one huge network, it may be something to think about...

Preparing the data

Only to retrieve data from the web does not really help. It has to be preprocessed to unfold the knowledge we want to acquire. I can provide two examples.

1. We gathered a reasonable amount of anonymized reviews. The goal was to predict the rating based on the text. Now you have a lot of unstructured text, which is unusable information for any machine learning technique. We need features, vectors (etc.) to train a model for prediction. Bringing the text into structured information is one whole chapter itself. You may find a sizeable amount of papers in the internet about text mining. I can recommend having a look at opennlp.apache.org.
2. You can also extract pretty simple information, which is very obvious in the first place, but may unveil unexpected information. Analyzing networks like Facebook or Mail traffic can give information about ones person. Twitter is also such a source. How many followers does one have, how many is he following. The point here is, accessing these information is not easy. Most data 'stores' like Google, Facebook or Amazon are slowly but surely closing their API. You will need keys to access the data. These keys are sold by the companies, so you basically have to buy the information you get. Some of them still have limited public API for free, but that gives you only very limited possibilities.

What is the information you want to find?

Both examples have one in common. We didn't know what information can be gained at all. I mean we already know that Lady Gaga has a LOT of followers. But what about John Doe. Does his network tell you something about his future? There is a theory that people with higher betweenness centrality would have potentially more success with their start-up. But where does such a theory come from? Only by investigation of past data over time.

If we go back to the web mining and take an example, that we would like to know which job offers the most vacancies at the moment. We have a concrete goal here. We fetch webcontent and parse the text. In a first step we may only look for words like 'job', 'salary' etc. But how to find the 'word' in the text which equals the job ('software engineer'). Lets assume we have an ontology for that. We are able to fetch all the data, parse it and out comes the amount of vacancies per job. It requires a lot of advanced stuff to get decent results. It is nowhere near good. If someone has other approaches or better ones he is welcome to share them :)

But in contrast to this example, most of the time you don't know what you are looking for. It is like testing and seeing what comes out. Maybe you have studies, researches or papers which tell you what you might look for. I don't know where the possibilities end. I think the information of the web contains a lot more information in between than is visible.

Conclusion

I know, it's a lot of text. But I wanted to give you some insight, because your 2 points are very hard to answer. Based on your tag "Career Development" I guess you want to predict a persons 'fitting job'?

What is the pros and cons of the information extraction and web mining ?

It's hard to extract and find what you are looking for. Its also hard to obtain the information you need, because a lot of 'nice-to-have' datasources are to pay or simply not accessible. But if you have them, you can extract A LOT of knowledge and predictions. For careers for example you might want to see what most programmes have in common to get a 'native programmers profile'. But first - find a way to access person profiles... :)

What is the present significance of them ?

The significance ranges from millions of dollars to none. If you can read German (or let Google translate it) https://www.web-analyzer.com/ is worth a look. They extract some information transform it into money. It's hard to say if they have a lot of customers. I doubt it, because it's a vague business, you can't prove anything.

People claim that it is possible to predict stock exchange prices. If it was... you decide what value this information would have. As I said, I don't know how far the potential information from the web is reaching. But I believe it predicts a lot sales of products.

If you were able to predict a job of a person based on their profile or behaviour in the Internet, then find a business model to sell this information. Isn't it quite similar to the adds in Facebook. They show you only the adds which are most promising for you, based on your profile (if you are single or engaged). You may then show vacancies or courses for education, job information, etc.

Answer 2

Slomo has done a pretty good job of answering your question from a technical point of view, although from your recent comments, it seems that he too misunderstood your question. You seem to be asking whether a career in web data mining is a good choice, and how to get started on that career path. I do not work in this field directly, but have a few insights.

First, as you can see from Slomo's answer, this is a technically very challenging field field. Your task (in general) is essentially to extract human-readable content from a jumble of often badly structured HTML, layout code, adverts, etc. This in itself is a challenge, but having obtained the raw data, you have an even greater challenge of making sense of it. Sometimes it's fairly straightforward, such as basic formatting and cleanup of data and normalising it, and sometimes its hugely difficult, such as determining how positive or negative a plain text review is (as in Slomo's post).

The above field, in general, is referred to as data mining, and you can get plenty of information by searching for terms such as "data mining career". The various technical difficulties mentioned above mean that you need to have a very good understanding of the field in order to do well or excel in it. As always with programming, I am sure you could just find your way into a position somehow, but most data mining specialists have some formal education. For a traditional approach, a typical B.Sc. in computer science will include some data mining and artificial intelligence education, but you'd probably want to have at least an M.Sc. specialising in data mining to even start standing out, especially if you want to work at the various large companies (Google, Facebook, Yahoo, MS, etc) who typically invest heavily in data mining. The challenges and scale of the problem typically mean that smaller companies do not typically attempt data mining the internet on a large scale, leading to a narrow, but rewarding field.

To summarise, this is a very difficult field, but offers great value to the companies who can invest in it. Being a narrow field, you will probably find a high barrier of entry, but if you can work your way to a good position in it, you will do well for yourself.