Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 2 down vote favorite

I work at a large company, and we are stuck with a crappy home-baked bug tracking tool, which does too many things half-decently, but it search functionality is next-to-impossible to use (has about 80 fields that one has to / can fill out) and the search needs to be precise. We have many duplicate bugs filed that are never consolidated, plus pulling up all bugs filed against some product in some version just will not be easy AT ALL.

Getting rid of the current system is a political and strategical impossibility. Hundreds of people are tied into it. What I am hoping to do it iterate over bugs 1 through however many (there is a way, I believe) and then save the resulting HTML pages named 'bug000001.html', 'bug000002.html', etc. After that I would want some sort of Google-Style full text indexing going. What we have is so bad, that doing this would be worthwhile. Bugs do not change that much over time, so I am not that worried about being out-of-sync with the original - it will not be great.

So, my question is: suppose I have downloaded the html files in the same folder. What free and open source tool can I use then that would allow fuzzy search on the content? I realize that I can concoct a specific regex every time I search that would probably do the task, but I would like to make this even easier, where mis-spellings do not hurt. I realize that Google's full power of personalized, history-based search will probably not be accessible to me, but I will gladly settle for something close, that is free and easy to use.

If you have suggestions for a good free and open source tool/Python library that I can use to automate simple browser actions, such as entering text into a textbox, clicking the button and then saving the content into an HTML file, then please list that as well, although that probably should be a separate question.

EDIT I do not have access to the source code for PhP front end or back end; this is all locked away by socialists on another continent.

share|improve this question
1  
Point Google Desktop at the folder of downloaded items. – iterationx Jun 7 '11 at 21:39
I'm not sure that you're going to be able to find a technical solution to what appears to be a social problem. – Carson63000 Jun 8 '11 at 5:41
You might want to consider parsing your HTML output, and putting the results in a MySQL database. MySQL has text searching built in, although I don't think it's fuzzy searching. – Gilbert Le Blanc Jun 8 '11 at 12:55
Sounds like an absolutely horrific place to work if you are forced into using a home built tool for bug tracking that you have no say over and you are part of the user community and no management support. Leave a copy of "Managing Humans" on your bosses desk sometime. – Bill Leeper Jun 9 '11 at 13:55
@Bill, this is just a reality of 1000+ organization with 10+ locations and headquarters abroad. The department that I work at is pretty cool and my boss and his boss are quite cool; but their reach is limited. Once I get this thing to work, my colleagues will start using it - hence progress. We will have to foil the Nazis, but that's a good learning experience that will come handy later in life too. – Job Jun 9 '11 at 18:17

3 Answers

up vote 2 down vote

I've used a system called Xapian before, it was really rather good for full-text searching, and according to its website ( http://xapian.org/features ) it can index html just fine.

It even has python bindings in the shape of the xappy extensions ( http://code.google.com/p/xappy/ ).

I'd suggest giving that a try (although I think you might be digging yourself into a deeper hole with your "custom" bug tracker by going this route. Good luck!).

EDIT

Oh, also, if you need the "automated text+button" thingy you could take a look at Selenium ( http://pypi.python.org/pypi/selenium ) which can automate browser tasks via python (link is the python bindings, not selenium itself, which is http://seleniumhq.org/ ). It's not completely reliable but it should be good enough to get the job done.

share|improve this answer
+1 for "deeper hole". Be wary of adding on to an already broken system. It's just php; can't you tweak the code to offer a list of possible bugs when they're keying in their reports? Or hell, just make the fields you need, required. – Satanicpuppy Jun 7 '11 at 21:43
@Satanicpuppy, more details: the "just PhP" part has been developed by a team abroad who value job security over producing good product. Only they can make changes, and they do not always listen. It is pretty hard to describe how we want this to function. Well ... I mean it is easy to say that we want the search capability to be as easy as FogBugz, but I do not know what percentage of our engineers are capable of something like that algorithmically (or, if using tools/libraries, just designing a decent product), not to mention the GUI, the time and other considerations. – Job Jun 7 '11 at 21:49
1  
@job: Do you have database-level access to the data, at least? Might be able to do something with that... – Satanicpuppy Jun 7 '11 at 21:53
@Satanicpuppy, Nope :) Having that would make matters easier. In fact, I am thinking of turning the html pages back into table records or at least stripped-down text files, because the html itself will contain a bunch of garbage text, including labels that might divert attention from content. – Job Jun 7 '11 at 22:06
up vote 2 down vote

You should look into the search engine called SOLR. It is document based and has some examples where they indexed web pages. There is a sample front end that will probably do what you need in terms of search.

The back end of SOLR is Lucene which is well documented and has great full text search capabilities.

You should be able to set it up so that SOLR scrapes your pages (it might even be able to scrape the live system directly), then using the HTML tags parse out the document into things like subject, user etc.

share|improve this answer
Please elaborate on "then using the HTML tags parse out the document into things like subject, user etc.". I am pretty sure that I will need to clean up my html pages with another script; currently they are too heavy. However, what does SOLR do with html exactly? Turns it into a table schema somehow? That sounds a bit far-fetched because one html file can differ in its format from another. – Job Jun 8 '11 at 19:27
1  
I haven't done much with SOLR specifically, but when I did I thought the tutorials for setting it up had a web scraping example where it scraped a web page. I have also used the underlying technology Lucene to do web scraping and it's a matter of setting up and using the appropriate xpath patterns to capture your data. – Bill Leeper Jun 9 '11 at 13:53
up vote 0 down vote

One thing you might want to consider is installing a bug tracking system yourself, and re-typing all of the bugs from the in-house system to your bug tracking system.

I realize that would be a lot of data entry. You might be able to automate the initial loading somehow.

Once people see what types of summaries you can provide, they'll be curious. At that point, you can introduce them to the bug tracking system.

share|improve this answer
I like this idea, but the old system still will not go away any time soon. Several hundred of people are tied into it. I am looking for something with very good search capabilities. – Job Jun 8 '11 at 14:24
@Job: The old system never has to go away. Hopefully, your bug tracking system has an export function that allows you to load the old system, either manually or automatically. Worst case, you type the bug report twice. – Gilbert Le Blanc Jun 8 '11 at 14:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.