MD5
is said to by quite slow (see below). The MD5
digest is rather long. I'd recommend modification time, size and CRC-32
checksum for file comparisons. A discussion on CRC-32
is here. As its name implies, CRC-32
has 32-bit hash values. A Java implementation is available in java.util.zip.CRC32
Edit:
The speed advantage of CRC-32 vs. MD5 is smaller than I've expected. CRC-32 needs some 20% less time compared to MD5.
I used the following Java code to find the difference (and demo the usage of both methods):
import java.security.*;
import java.util.Random;
import java.util.zip.CRC32;
public class HashBench {
@SuppressWarnings("unused")
public static void main(String[] args) throws Exception {
int noOfLoopIterations = 100 * 1000;
int bytesInMessageBuffer = 100 * 1024;
byte randomByteBuffer[] = new byte[bytesInMessageBuffer];
byte md5Digest[];
MessageDigest md5;
CRC32 crc;
long crcValue;
long startTime;
long elapsedTime;
new Random().nextBytes(randomByteBuffer);
// MD5 benchmark
o("Starting MD5 benchmark ...(" + bytesInMessageBuffer/1024 + "KByte messages)");
md5 = MessageDigest.getInstance("MD5");
startTime = System.nanoTime();
for (int i = 1; i < noOfLoopIterations; i++)
{
md5Digest = md5.digest(randomByteBuffer);
}
showElapsed(noOfLoopIterations, startTime);
// CRC-32 benchmark
o("Starting CRC-32 benchmark ... (" + bytesInMessageBuffer/1024 + "KByte messages)");
crc = new CRC32();
startTime = System.nanoTime();
for (int i = 1; i < noOfLoopIterations; i++)
{
crc.reset();
crc.update(randomByteBuffer);
crcValue = crc.getValue();
}
showElapsed(noOfLoopIterations, startTime);
o("Ciao!");
}
private static void showElapsed(int noOfLoopIterations, long startTime) {
long elapsedTime;
elapsedTime = System.nanoTime() - startTime;
o("Elapsed time: " + num(elapsedTime / 1000000000.0) + "s for " + String.format("%1$,.0f", 1.0 * noOfLoopIterations) + " loops");
o("Time per digest: " + num(elapsedTime / (1000000.0 * noOfLoopIterations)) + "ms");
o("");
}
private static void o(String s) {
System.out.println(s);
}
private static String num(double x) {
return String.format("%1$,.2f", x);
}
}
The result:
Starting MD5 benchmark ...(100KByte messages)
Elapsed time: 28,94s for 100.000 loops
Time per digest: 0,29ms
Starting CRC-32 benchmark ... (100KByte messages)
Elapsed time: 23,89s for 100.000 loops
Time per digest: 0,24ms
To avoid the influence of disk caching and other external effects, I just fill a byte array with random values. The benchmark executes the hash/checksum calculation repeatedly.
Conclusion: Calculation speed is not a convincing reason in this case.