I have a very big file delimited by some sequence of characters '*L*I*N*E'. The file will be of the order of 250G. And each line comes around 600bytes to 1000 bytes. I will be performing the following operations on the file,
Read the file line by line and for each line, I will be giving it to a parser which will do some calculation on the line and update some stats. The parser will take roughly 15 micros per line.
As of now I am using the
BufferedReader to read the lines and pass it over to the parser in a single thread. My question is if I have a separate reader thread which only reads the file and dump everything in a queue (in memory) and have my parser act on the in-memory queue(in a separate consumer thread), can I achieve better throughput?
Nothing changes except that my parser acts on the inmemory data and another thread takes only IO operation(read the file and dump in a queue).
It is a complex piece of software in which the above is one part, so I am trying to speed up the above part. Hence I can't able to post the actual code.