Pilot-testing: How can I improve this instrument?

Happy new year! We are back with another post in the Evaluation Design Series on the topic of pilot-testing data collection instruments, particularly for open-ended interview guides or standardized surveys.  In short, pilot-testing (also called pre-testing) is administering an instrument a few times prior to data collection.  This includes all parts of the instrument, from the intercept script to the wording of each individual question.  Information from pilot-testing can tell you a number of things:

  • Is the recruitment protocol (e.g., script) working?
  • Does the instrument actually measure what it intends to measure?
  • Does the subject understand the questions?
  • Does the subject interpret any of the questions as biased?
  • Which topics/questions need to be treated with more sensitivity?

While pilot-testing is a valuable step in the evaluation process, it is sometimes skipped (due to time constraints or otherwise). Most evaluators, including experienced evaluators, can remember a time when they wished they had pilot-tested the instrument, so consider these quick tips:

  1. Write notes directly on the script/instrument during the pilot-test so you can quickly identify trouble spots to address.
  2. Rework your script/instrument based on your findings and test the materials again with a few new subjects.
  3. If you are still having trouble, administer the script/instrument to a colleague who is unfamiliar with your study and ask for feedback.

And don’t hesitate to write about your pilot-testing experiences, as doing so will provide an ongoing record of what works and what does not work, which will come in handy the next time you design data collection materials. Or it may provide insights that help explain the eventual results from data collection. For example, we’ve found pilot-testing interview instruments for children particularly helpful to ensure they understand the questions through age-appropriate language.

Leave a Reply