I'm working on a project that involves loading batches of 3000+ files to several dozen tables. There is no user interface, and the tables are simply available for querying. What are the best practices for testing this type of process?
- Loading a small set of data and validating each and every member?
- Loading all the data and validating a subset?
- Loading all the data and validating counts, averages, and other metrics?
Are there other types of testing that can be done, or is some combination best?