I have FILE_A which has over 300,000 lines and FILE_B which has over 30 million lines. I created a Bash script that greps each line in FILE_A over in FILE_B and writes the result of the grep to a new file.
This whole process is taking over 5 hours.
How can I improve the performance of my script?
grep -F -m 1 as the grep command. FILE_A looks like this:
and FILE_B is like this:
So with Bash I have a
while loop that picks the next line in FILE_A and greps it over in FILE_B. When the pattern is found in FILE_B, I write it to file result.txt.
while read -r line; do grep -F -m1 $line 30MFile done < 300KFile