Finding Your Confidence Interval
By Perry D. Drake
The following article, written by Perry D. Drake, appeared in Inside
Direct Mail, September 1999. It provides direct marketers with a means
of better interpreting their testing results.
When conducting marketing tests using samples, measures such as the percent responding or the payment rate are the "measures of concern."
Depending on the results of the sample estimates, in conjunction with the marketing cost and revenue figures, you will make a business decision of whether or not to rollout with a particular list of names, product offering or promotion.
Confidence in your sample estimates is critical to making correct marketing decisions. Unfortunately, regardless of how good your sample is or how little things change from test to rollout, with almost 100 percent certainty, you will never receive the exact same result in rollout that you did in test.
The problem is an error associated with sample estimates.
If, for example, the test reveals a fourpercent response rate to a particular offer, you will not receive that response rate in rollout regardless of having a large, and properly drawn sample. The reason: A sample is just thata sample. Sample estimates have a certain level of associated error variance. The higher the error variance associated with your sample estimates, the more likely your test results will be "off" from what you can expect in rollout.
But, this will not prevent you from making accurate marketing decisions based on these sample estimates.
In this article, you will learn how to place bounds around a test response rate, allowing you to determine a range in which the true response rate can lie in a rollout situation. Once calculated, you can run various profit calculations using the upper and lower bounds of the interval to determine a best and worse case profit scenario for the rollout.
Constructing the Confidence Interval
A confidence interval constructed around a single response rate is primarily used to assess a new list test in terms of its potential as a "new name" generator or to assess a new product in terms of its potential for a large scale rollout.
To calculate a confidence interval around a test proportion, the following information is required:
 The test response rate, which we'll label p.
This is response rate obtained for the test (i.e. percent responders, payment rate, percent telemarketing hits.).
 The sample size of the test, which we'll label n.
This is the number of names tested. NOTE: In order to use this formula, the sample size when multiplied by the sample response rate and when multiplied by one minus the sample response rate, must be greater than or equal to five.
 The desired confidence level.
A confidence interval constructed around the test response rate "p" will guarantee, with your specified level of confidence, that the true population proportion obtained in rollout will fall within those bounds. For example, constructing an interval around your test response rate with a 90percent confidence level will guarantee that the true response rate you can expect in rollout will fall within the bounds of the confidence interval with a 90percent probability.
NOTE: It is strongly advised that you never construct a confidence interval with less than a 90percent confidence level. Keep in mind, that with a 90percent confidence level there is a 10percent chance (100 percent minus 90 percent) that the true response rate you can expect in the rollout will not fall within the bounds of the constructed interval. Do you really want to be subjected to more than a 10percent risk of this happening? Dropping the confidence level below 90percent will cause you to assume more risk than you should be willing to accept in being misled by the resulting bounds of your confidence interval. As a result, you may make an incorrect marketing decision.
The formula used to calculate the lower and upper bounds of a confidence interval is:
Lower bound = "p" minus "z" times the standard deviation of the test response
Upper bound = "p" plus "z" times the standard deviation of the test response
Where the standard deviation of the test response rate is equal to the square root of the following calculation: the test response rate times one minus the test response rate divided by the sample size
And, where z equals 1.645, 1.96 or 2.575 for a 90, 95 or 99percent confidence level, respectively.
To illustrate, assume the marketing director at ACME Direct has conducted a new product test to 10,000 names from the primary customer segment of the ACME Direct database. The test results for this new product test yielded an order rate of 3.42 percent.
To assess the potential for this new product in a rollout, the marketing director decides to construct a confidence interval that is guaranteed to contain the true response rate he can expect in rollout with 95percent confidence.
Using the previously mentioned formula we first calculate the standard deviation of the test response rate as follows:

The resulting 95percent confidence interval is 3.06 percent to 3.78 percent. The marketing director can be 95percent confident, should he decide to rollout with the new product, the response rate will be no less than 3.06 percent, and no higher than 3.78 percent. He can use the lower bound to determine the worse case scenario in terms of profitability. Upon examination of this profit scenario, the marketing director will make his decision.
The confidence level selected depends on the risk you are willing to take in the resulting confidence interval not containing the response rate to expect in a rollout. If the risk is quite high, set the confidence level at 99 percent. If the risk is minimal to none, use industry standard levels of 90 percent to 95 percent. Base the "risk" on the cost figures associated with marketing the new product or offer. When in doubt, choose a 95 percent confidence level.
Interpreting the Confidence Interval
Assume the marketing director had previously determined that he needed at least a 3.25percent response rate to generate 10percent profit after overhead for this new producthis objective. The lower bound of the 90percent confidence interval was 3.06 percent. Therefore, if the marketing director rolls out with this new product offering, there is a chance that the response rate achieved will be lower than the minimum he requires. What should he do? He will:
 Calculate the profit assuming he receives only a 3.06percent response rate (the lower bound of the confidence interval) in rollout.
 Take into consideration the fact that the upside potential (3.78 percent) far outweighs the downside potential (3.06 percent) given he needs only a 3.25percent response rate to meet his goal of 10percent profit after overhead.
 Calculate a more conservative estimate of the worse case scenario, in terms of the response rate to expect in rollout, by constructing a 99percent confidence interval. Then assess the lower bound of this interval in terms of profitability.
Using this information, the marketing director can make a wise marketing decision. Just remember, confidence intervals will not give you a definitive answer to your question but will help you in assessing the test results by providing you with various scenarios based on the amount of error associated with your sample test results.
Another option available to direct marketers in making a more informed decision is testing more names. Testing more names minimizes the error associated with the testing results. While most list and format test quantities begin at 5,000 names, testing a larger sample can often give you a better picture of the responsiveness of the list or format. Testing more names minimizes the error associated with the testing results.
If your testing budget won't allow you to test more names, cut back on the number of test panels, not on the number of names tested per panel. Five test panels, for example each with an appropriate test panel size, will benefit your company far greater that ten test panels with too few names promoted in each.
Return to Listing of All Articles
Return to Home Page
