Response rates needed for results to have statistical significance
As a general guideline, MTM recommends the following response rates for making decisions based on evaluation data:
- For less than 1,000 learners: 30 evaluations and 60% response rate
- For 1,000 learners: 500 evaluations
The attached table can help to determine the response rate required to achieve statistical significance within a 2.5% or 5% margin of error and a confidence interval of 95%. That said, business decisions can and should sometimes be made on data that is not statistically significant. This depends upon what the data is telling you, the culture of your organization, and the magnitude of the decisions you would be making based on it.
The table can also be used to help determine an appropriate sample size by considering the number of evaluations needed based on the population size and also allowing for the client's typical response rate. For example, if the number of evaluations needed based on the population size taking the training is 200 and the client's typical response rate is 50%, the sample size should be at least 400 and the class or course survey % should be set at a level that will allow for at least 400 surveys to be sent out.
Statistical Significance Tables
2.5 % Margin of Error
| Population Size | Sample Size | Response Rate |
|---|---|---|
| 10 | 10 | 100% |
| 20 | 20 | 100% |
| 30 | 29 | 97% |
| 50 | 48 | 96% |
| 75 | 72 | 96% |
| 100 | 94 | 94% |
| 150 | 137 | 91% |
| 200 | 177 | 89% |
| 250 | 215 | 86% |
| 300 | 251 | 84% |
| 400 | 318 | 80% |
| 500 | 377 | 75% |
| 600 | 432 | 72% |
| 700 | 481 | 69% |
| 800 | 526 | 66% |
| 1,000 | 606 | 61% |
| 1,200 | 674 | 56% |
| 1,500 | 759 | 51% |
| 2,000 | 869 | 44% |
| 2,500 | 952 | 38% |
| 3,500 | 1,068 | 31% |
| 5,000 | 1,176 | 24% |
| 7,500 | 1,275 | 17% |
| 10,000 | 1,332 | 13% |
| 25,000 | 1,448 | 6% |
| 50,000 | 1,491 | 3% |
| 75,000 | 1,506 | 2% |
| 100,000 | 1,513 | 2% |
| 250,000 | 1,527 | 1% |
| 500,000 | 1,532 | 0.3% |
5% Margin of Error
| Population Size | Sample Size | Response Rate |
|---|---|---|
| 10 | 10 | 100% |
| 20 | 19 | 95% |
| 30 | 28 | 93% |
| 50 | 44 | 88% |
| 75 | 63 | 84% |
| 100 | 80 | 80% |
| 150 | 108 | 72% |
| 200 | 132 | 66% |
| 250 | 152 | 61% |
| 300 | 169 | 56% |
| 400 | 196 | 49% |
| 500 | 217 | 43% |
| 600 | 234 | 39% |
| 700 | 248 | 35% |
| 800 | 260 | 33% |
| 1,000 | 278 | 28% |
| 1,200 | 291 | 24% |
| 1,500 | 306 | 20% |
| 2,000 | 322 | 16% |
| 2,500 | 333 | 13% |
| 3,500 | 346 | 10% |
| 5,000 | 357 | 7% |
| 7,500 | 365 | 5% |
| 10,000 | 370 | 4% |
| 25,000 | 378 | 2% |
| 50,000 | 381 | 1% |
| 75,000 | 382 | 1% |
| 100,000 | 383 | 0.4% |
| 250,000 | 384 | 0.2% |
| 500,000 | 384 | 0.1% |