Tell me more ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 6 down vote favorite
2

I have to work with a lot of logs daily: appreport, message, debug, systemout, syserr, etc.

To do so, I login to one of many different servers through putty and use grep with many different patterns, which takes a lot of time.

Is there a better way to do this? I'm looking for:

  1. An application which searches for a string in selected files connecting to a Unix server.
  2. Any ideas on how to develop a webapp which would do dynamic log searches and show results.

I have knowledge of Java, so libraries that could be used in a Java webapp would be ideal.

share|improve this question

5 Answers

up vote 8 down vote accepted

Splunk :

Splunk collects, indexes and harnesses the massive volumes of valuable machine data generated by your complex IT infrastructure, whether physical, virtual or in the cloud. Use Splunk and put your machine data to work. Troubleshoot problems and investigate security incidents in minutes, not hours or days. Monitor your end-to-end infrastructure to avoid service degradation or outages. Meet compliance mandates at lower cost. Correlate and analyze complex events spanning multiple systems. Gain real-time operational visibility and insights for IT and Business users.

Use this. It probably does everything you want.

share|improve this answer
+1, splunk is great – smp7d Dec 2 '11 at 14:12
Splunk does everything you want and a lot more. The free version is pretty limited though and the enterprise version can be pricey. – Qwerky Dec 2 '11 at 14:18
My sample work Ex: 1)login to example_server from putty 2)ssh to ssh_server 3)cd to req dir 4)perform cat etc., BTW ssh_server will not allow directly to be logged in from putty, i have to first login to example_server and then there to ssh_server. now how can i configure these log files to be used by splunk to search for a string as i use grep for. i've installed splunk on my laptop, and by clicking add data >files and dir > add new it show full path to your data field, wht path should i fill it with ? – cypronmaya Dec 8 '11 at 6:52
@cypronmaya: That's a separate question. Please ask that separately. – S.Lott Dec 8 '11 at 11:27
up vote 3 down vote

You should try logstash. If you click Learn on the homepage there you'll find an excellent video introduction to it.

share|improve this answer
thanks :) i will try it, BTW i do not have complete access to logs, i can perform normal grepping only nothing more than that – cypronmaya Dec 2 '11 at 9:57
up vote 2 down vote

If you are willing to learn some python (it's easy especially with basic programming knowledge) try fabric.

http://pypi.python.org/pypi/Fabric/1.3.3

http://docs.fabfile.org/en/1.3.3/index.html

Originaly started as python (web) deployment tool. But it realy is a lib to streamline unix commands through ssh to a remote host (e.g.: to grep logs).

share|improve this answer
up vote 0 down vote

XpoLog is a Java base log viewer, search and analysis software. We use it both in testing and production as log management platform.

It is cost effective and cost 50% less than Splunk, better alternative.

share|improve this answer
Why is it a better alternative besides of the price ? – Matthieu Dec 12 '11 at 14:21
up vote 0 down vote

Graylog2 is another option. It has a web interface, is mostly written in Java and uses ElasticSearch as back-end. So you get search for free.

Rackspace cloud monitoring uses it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.