For every build of my software I run it through a series of tests that execute a number of commands on a command line and captures the results.
Most of the time the output is a combination of: - Static information that is, and should be, the same between every run. - Dynamic information that is not easy to predict.
For the purpose of testing this software I desire to only verify that the static information is the same between each test.
Currently I use sed on the captured output of my automated tests to filter out the dynamic information. Any differences between test runs are then identified using the filtered data resulting from applying the sed rules.
Because of the large variety of command output that is processed, maintaining the sed rules has become an onerous job.
Is there a system, that with a large sample of executed test results, can show uncommon changes between two files contained in that large sample of test results? To clarify, if I were to diff two files, I would only want to see differences for items that changed 5% of the time between all the files involved. Any other data that changed 95+% of the time would be excluded as differences when comparing two of the files within the large set of test results.
In this example, among the large set of test runs, the first column is almost always consistent:
1/*/1 6643 2433 12343 232424 112 2/1/1 2311 455 1342 60233 5231 2/1/2 2355 1223 93 12389 1342
1/*/1 3121 112 98832 451 1233 2/1/1 6342 2345 84421 3456 8362 2/1/2 72453 5 421 64321 7634
1/*/1 12 65 312 653 973 2/1/1 12442 54231 46012 6 734 2/1/3 734 28 623 76 1834
On top of the three runs above, there are hundreds of samples of this same commands output generated for each incremental build of the software. When performing a difference between each of the three examples above I would like to have the following results:
No Differences between Run #1 and Run #2. Difference showing that "2/1/2" was changed to "2/1/3" when asked to show the difference between Run #3 and either of the other two runs.
Currently I have a sed script that replaces all the "random looking" numbers above (counts, timestamps, etc) with hashmarks or other constant indicators so that all that is left to compare is the starting column.
What options, if any, exist in this area? Is there a name for this kind of text comparison?