Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I don't have a great background in information organisation / retrieval, but I know of a few ways of dealing with the problem. For structured information, it's possible to go OOish - everything "has-a" or "has-many" something else, and you navigate the graph to find relationships between things. For unstructured information, you have techniques like text search and tagging. I know about basic CS data structures and algorithms for structured data, but what I'm interested in goes beyond that. I want to know about how unstructured data can be related to structured data "intelligently", i.e. without users having to explicitly understand the object hierarchy.

What resources - articles or books - are there that summarise the CS theory behind these techniques or could introduce me to others? Since so far it's been difficult to communicate exactly what kind of problem I'm dealing with, here's an example scenario. It is my own wording and the domain has been (radically) obfuscated to protect the client.

A wine enthusiast wants to build a website for her wine society. Wine farms could capture many wine prices and enter their locations. Users would search for wine tasting venues in their area. Or all wines which cost below X per bottle. Nothing difficult about that so far, because we're dealing with structured information in a straightforward graph of objects or database tables.

However, they also have advice articles (unstructured info), such as "10 things to look out for in a good Red" (sorry, we're stretching the limits of my knowledge about wine :). As a user, you'd want the website to display a link to the article when you're viewing wine farms. If the enthusiast decides to have a section where people can list "recipes for meals which go well with wine" on her website, she might also want the article to appear next to those recipes which go well with red wine. Now, you don't want the enthusiast who captures the data to have to link every new article to every meal (because it won't just be this one article that is relevant to recipes or wines); and you don't want meal authors to be bogged down by exactly which one, two or three articles are most relevant out of a library of 50. You certainly don't want to introduce a "red wine advice" field on every wine or meal. So that's where you might use tagging. Wines, recipes and the red wine advice article could all be tagged, "red". Great solution that really works for, right? Well yes, as long as the data capturers know that the "red" tag exists. Maybe they decide to use "Claret" instead. Doh! If there are only 5 tags, they probably get it right, but if there are 50 tags, or even just 20 they might not. Worse still you get the spam scenario where the author of a new article just applies every tag - "red", "white", "shiraz", "chenin blanc" etc.

At this point we need to consider other techniques, like text search. The wine is marked as a red, the article is chock full of the word "red wine". Done. Well, you might want to apply some of the data clustering techniques that can help you to better identify what a piece of text's main topics is. Or create an indexing process that drops out common words and punctuation. No point matching two items because they both use the word "but" a lot. Actually, the best results of all would probably be to use a combination of everything I've mentioned.

You may notice that in this example my tone moves from, "I know exactly what I'm doing" to "try do some stuff like...". So, I'm looking for resources that will teach me about dealing with the combination of structured and unstructured info - tagging, text search with smart indexing and clustering as well as any other techniques which I don't know about yet. Some discussion of strengths and weaknesses of each would also be appropriate.

share|improve this question

closed as off-topic by MichaelT, durron597, GlenH7, gnat, Ixrec Oct 18 '15 at 18:37

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – Community, durron597, GlenH7, gnat, Ixrec
If this question can be reworded to fit the rules in the help center, please edit the question.

It's not clear to me if you're looking for really basic information on things like graph representation and traversal, in which case almost any good data structures book will do, or if you're trying to figure out a way to infer a model of structure from information that is relatively unstructured, as one might do in data mining projects.

If it's the second case, and you understand the basics of graphs, you might take a look at Probabilistic Graphical Models (Koller and Friedman) and Introduction to Statistical Relational Learning (Getoor and Taskar). I've not gotten very far in these books, but they have a style that makes them more usable than average to industry programmers like me who never spent much time in math classes in college. Also Data Minding, Second Edition by Ian Witten is fairly pragmatic.

For more superficial, but also more hands-on, coverage, look at Algorithms of the Intelligent Web, Haralambos Marmanis and Dmitry Babenko, and Collective Intelligence in Action, by Satnam Alag.

share|improve this answer
Algorithms of the Intelligent Web looks promising. – Fritz Meissner Mar 15 '11 at 12:22

Two books as this sounds like you're looking for data structures and algorithms (more or less)

  1. Data Structures and Algorithm Analysis in C++ (code is in c++ principles are universal.)
  2. Introduction to Algorithms (some data structures but more focus on the algorithms used to manipulate them to do things such as sorting, searching etc.)
share|improve this answer
I'm not sure how it came across this way, I'll reword - I've read these – Fritz Meissner Mar 15 '11 at 7:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.