Statistical Reliability and Limitations

Reliability is the degree to which survey sample data reflects the actual population and the true parameters of that population. It is dependent primarily upon survey sample size. The precise statistical interpretation of a randomly selected survey sample, such as this one, is based on other factors as well. These factors include sample selection, types of questions asked, answers received, interviewer proficiency, and respondent quality. However, a general discussion of sample size only is pertinent at this point.

As a generalization, a sample of 400 randomly selected respondents, such as we have for our total completed interviews, will generate data reliable with 95% confidence and a ±4.9% sampling error. That is to say, if a similar survey were conducted repeatedly, results within ±4.9% would occur for any one question 95 out of 100 times. Looking at it another way, if a question received a "yes" answer by 60% of the 400 respondents, the chances are 95 out of 100 that between 55.1% and 64.9% of the total population would lodge a similar "yes" response, if asked. Sampling error such as this is applied to each cross-tabulation market cell as well as the total survey sample.

To judge significance on the total responses given for a particular question, find the applicable sampling error for the sample size under examination. Then add and subtract this sampling error to the percentages under examination. If the two percentage ranges overlap, one must judge there is no significant difference between the ranges. However, should the two ranges not overlap, one can deduce with the selected level of confidence that the variation is due to real differences in opinion and not due to chance. This discussion relates, in general, to a random sample survey where one is extracting a portion of a large population and hypothesizing that the attitudes of that portion are reflective, to a statistically measurable point, of the total population.

It is also important to point out, first of all, that surveys should never be viewed as 100% reliable. A small difference between two statistics or findings cannot be considered necessarily meaningful; however, as the sample size increases, the margin of error (sampling error) decreases, thereby providing more conclusive and reliable data.

In parts of this report we may refer to different statistical measures. A brief explanation of these will facilitate individual usage and analysis.

The arithmetic Mean is a measure of central tendency or the average. The mean is the most common measure of central tendency for variables measured at the interval level. Often referred to as the "average," it is merely the sum of the individual values for each case divided by the number of cases. The mean is a valuable tool for data analysis; however, it is a fixed point and does not indicate the range of responses.

The Standard Deviation (STD DEV) is a measure of the dispersion about the mean of an interval-level variable. More plainly stated, it is a measure of how close or how far all the answers are from the mean. The wider the spread in the response, the larger the standard deviation. It is used in comparing the variability of different groups. It is possible to have the same mean but differ in variability. The advantage of the standard deviation is that it has a more intuitive interpretation, being based on the same units as the original variable.

The Standard Error in the Mean (STD ERR) helps to determine the potential degree of discrepancy between the sample mean and the (usually) unknown population mean. Simply stated, it is a measure of how close or how far all of the sample means are from the true population mean. The standard error is part of the formula used to calculate confidence intervals on the total responses and gauge statistical difference.

The Median is the numerical value of the middle case or the case lying exactly on the 50^th percentile, once all the cases have been ranked ordered from highest to lowest. For example, if the answers were from 1 to 51, 26 would be the median, therefore there would be 25 answers above and 25 answers below the median.

The Median can be a better measure of central tendency than the mean when the sample is very small or when values are highly dispersed (extreme outlying values).