Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

I'm working on a data migration of several hundred nodes from a Drupal 6 to a Drupal 7 site. I've got the data exported to the new site and I want to check it. Harkening back to my statistics classes, I recall that there is some way to figure out a random number of nodes to check to give me some percentage of confidence that the whole process was correct. Can anyone enlighten me as to this practical application of statistics? For any given number of units, how big must the sample be to have a given confidence interval?

share|improve this question
5  
I probably wouldn't want to use statistics as a measure of success. This isn't dart throwing. Its a migration. You know what you started with so you should see that at the end –  Rig Sep 12 '12 at 13:16
    
This migration isn't a simple 1:1 mapping. I'm splitting fields, combining them, and creating more complex nested structures out of several fields. To verify all migrated data to validate my existing code, I would have to write new, other code that does the same thing. And then, if both codebases have different bugs, how would I know which was correct? –  user1936 Apr 23 '13 at 17:55
add comment

2 Answers 2

up vote 2 down vote accepted

I found this sample size calculator. For my population of 215 items, if I want a 95% confidence with +/- 5% confidence interval, I'll need to randomly sample 138 items.

Edit: Here's the actual formula that I was looking for.

share|improve this answer
add comment

Computers are really good at doing repetitive tasks and comparisons, it wouldn't take long to write a small app that verifies all the data, and unless the amount of data you transferred was huge it could be done overnight or over the weekend and get 100% certainty. Stats rely on getting a true random sample of the entire population, and failing to get a valid set makes that confidence worthless and both humans and computers are really bad at true random. If something does come up its a lot better to be 100% certain than 95%.

share|improve this answer
1  
It would take significant effort to write a script comparing Drupal 6 to 7 data with any custom fields. A small enough sample size would make it more practical to compare by hand as long as 100% confidence isn't necessary. It's what interns were made for. –  Matt S Jan 10 '13 at 19:29
    
See my comment on the section. This isn't a 1:1 migration; I have complex transfomations of the data. That "small app that verifies data" would have to have all the functionality that the real app does -- otherwise it wouldn't know if the transformations happened correctly. It would seem quite a nightmare to troubleshoot bugs between two codebases of identical functionality. –  user1936 Apr 23 '13 at 17:58
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.