INTENTIONAL MUSEUM BLOG

Sample Size: How many questionnaires is enough?

RKA Blog_Evaluation Design Series_LogoSample size is a standard question we are asked, particularly for questionnaires since we will be using statistical analyses. For most audience research projects, we recommend collecting 400 questionnaires.  We are not alone in this general rule of thumb—400 is considered by some researchers (and market researchers in particular) to be the “magic number” in the world of sample sizes.  What makes 400 magical is that it is the most economical number of questionnaires to collect (from most populations) while keeping the margin of error at ± 5% (and the confidence level at 95%).  A sample size of 400 questionnaires keeps the cost of the research down while still allowing us to have high confidence in the results. 

 

To dive into this issue deeper, let’s talk about the three primary factors necessary to think about when deciding on a sample size: (1) population; (2) confidence level; and (3) margin of error.  Population is the number of people in the group from which you are sampling.  For instance, your population may be the number of annual visitors to the Museum, members, or visitors to a specific exhibition or program.  A fact that is often enlightening and counter-intuitive is that population does not have a proportional relationship to sample size.  To demonstrate this, follow my calculations by trying out one of the many sample size calculators available on the web, such as this one or this one.  Let’s start by determining a sample size for surveying the National Gallery of Art, which reported nearly 4 million visits in 2014 (3,892,459 to be exact).   Using the margin of error ± 5% and 95% confidence level, the sample size suggested is 385.  By comparison, the sample size suggested for The Phillips Collection, which welcomed 106,154 exhibition visitors in 2014, is 383.  Despite vastly different sized visiting populations, the recommended sample size for each museum differs by just two!  Again, this example demonstrates that sample size is not proportional to the population, but also, having an estimate of your population is often sufficient to determine a sample size (unless you are determining a sample size for a program with small attendance or other small populations).

 

Confidence level and margin of error (or confidence interval), as you might expect, indicate the level of confidence or how “sure” you are about the results of the questionnaires.  Here, the researcher has to make a choice about an appropriate confidence level and margin of error based on how the data will be used.  At RK&A, we generally plan for the margin of error at ± 5% and a confidence level at either 90 or 95% because it provides enough confidence in the data given how our museum clients use the data to make institutional decisions.  If we were working with a medical professional making life-or-death decisions, we would want to be more confident in the results (thus, a lower margin of error and higher confidence level).  So why not plan to be as confident in the results as possible (regardless of how they are used)?  Money.  Confidence comes at a cost because, like population and sample size, the relationship between sample size and margin of error is not proportional.  For instance, see the graph below based on the population reported above for the National Gallery of Art.  Notice that the slope of the line is steepest on the left side of the graph and more gradual on the right side.  This shows the law of diminishing returns at play.  There are great benefits when moving from a sample of 200 to 400 (margin of error diminishes by about 2 percent), but the benefits are not nearly as great when moving from a sample of 400 to 600 (the margin of error diminishes by less than 1 percent).  Thus returning to our initial point, collecting more than 400 questionnaires is rarely prudent since the cost of data collection will be going up disproportionate to the reduction of the margin of error.  For our museum clients, we do not think that increase in confidence justifies the extra costs.

blog chart

I would be remorse to end this post without a footnote.   While 400 is our rule of thumb for audience research data being collected through a standardized questionnaire, there are certainly many considerations and reasons why 400 might not be the magic number in every case.  We joke that the response to any methodological question is the often frustrating retort: “It depends.”  Sample size is no different—it depends.

3 Responses
  1. Ted Myers

    Very helpful article. Question: Do you use the same rule of thumb even if portions of the population may have very different responses? For example, in a school district with 15,000 teachers, 6,000 teach elementary and 8,000 secondary. Does the rule-of-thumb 400 cover all of them, as long as the sample is representative of the population, or would we need a larger total sample to reflect differing views that we expect from different segments?

    1. Amanda Krantz

      It will depend on the objectives of the study. If you have very specific questions about each audience segment (elementary versus secondary), I would aim for a higher sample size than just 400 teachers total. For instance, if we are conducting an audience research project where we are looking for seasonal differences in museum visitors, we aim to collect 400 per season (400 for the fall/winter and 400 for the spring/summer). However, we regularly run analyses on a sample of 400 that looks for differences by variables, such as gender, which is essentially splitting our sample in two like in the scenario you mention. This is fine in many cases.

Leave a Reply