I'm writing a log parser that continuously looks for specific patterns and then fires off events to another system when it finds these matches. The firing of the event is time critical to catch errors or ensuing problems. This part is non-negotiable in terms of design.
The log files belong to Applications and the Operating System; thus they can be rapid in logging, large in size and rotate based on time, or size.
What is a better approach to reading these log files, detecting when they change, and reading from their last read location as I wrote code to do this using a RandomAccessFile which kept the last read location for every file.
For dealing with large files and rapid logging I did this:
- DataProducer - would get last read point in file and then find any changes then add to a Queue
- DataConsumer - any changes to the Queue by way of size, would then cause this thread to wake and start parsing all the items in the Queue for that log file. Deleting what it has parsed.
I am looking for some suggestions (in light of what I have done) that may improve the current way I manage reading a file, and picking up when there are changes, and parsing the file efficiently (considering size and rotation). Code or an API will do.