Want to Conduct a Survey?

There are a few things to consider before you even begin writing your survey:

1. First, make sure you know what you intend to accomplish with the survey and that your working group is clear and in agreement. In other words, it is important to have a well-defined purpose for your survey.
2. With this purpose in mind, identify the population you want to survey. Please note that under current U of I email policies the ability to send emails to large portions of the U of I community is granted to a limited number of managers who are responsible for email lists. Institutional Research can help you obtain permission to email your survey to these lists. However, you might also consider using a representative sample, rather than surveying all U of I students, for example.
3. Consider how your respondents will be receiving your survey, and what device or devices will they likely be using to complete it. Institutional Research recommends you become familiar with the devices used by your survey population and design the survey to accommodate them. Qualtrics can easily format the survey to display on phones as well as computers, for example.
4. Do you want to capture information about your respondents, such as name and University of Idaho number (Vandal number)? Or do you want to keep survey responses confidential? Both options are available, but you must decide which is needed.

Once you have a plan for your survey, it is time to start thinking about structuring the survey.

Recommended Reading

• How to Create an Effective Survey | Qualtrics
• How to Write a Good Survey | UC Merced

Keep Your Survey Short

Research consistently indicates that the longer a survey is, the more likely it is that recipients will not take the survey, or, if they start it, will not complete the survey.

The ideal is that respondents should be able to complete the survey in 10 minutes or less.

If it is necessary to make the survey longer, you need to make sure that your potential respondents are highly motivated to complete the survey. You might also see the information we’ve compiled on Improving Response Rates.

Test Your Survey

It’s recommended that you have at least five people take your survey to test it before you send it out. This way, you can hopefully catch errors and get feedback on whether it is understandable and simple to fill out.

Best Practices to Structure Your Survey

1. Start the survey with an easy, and if possible, fun question. You want to draw your respondents into the survey, rather than putting them off by asking difficult and/or sensitive questions right off the bat.
2. Place sensitive and difficult questions toward the end of the survey, rather than at the beginning. This includes demographic questions about gender, ethnicity, and so forth.
3. Also, only ask these demographic questions if they are relevant. Otherwise, omit them.
4. Limit the number of questions per page. Consider placing as few as one question per page. In this way, respondents don’t feel daunted by seeing a long page filled with questions.
5. Place open-ended (text response) questions near the end of the survey. The advantage of this is that, even if some respondents choose not to answer these questions, there will still be responses from them for the earlier questions.
6. Be aware that question order matters. Qualtrics recommends randomizing some or all question order if possible; read this article on how to randomize question order.
7. Add a progress bar. Although not universally recommended, progress bars help respondents see how they are progressing through the survey. Be aware, however, that if your survey contains branching logic, the progress bar may not show progress correctly, so you should probably omit it in that case.

Recommended Reading

• Survey Question Sequence, Flow, & Style Tips | Qualtrics
• Tips for Reducing Survey Question Order Bias | Qualtrics
• https://provost.tufts.edu/institutionalresearch/files/Survey-Best-Practices-Webinar.pdf

Here are some of the most common tips found in survey question-writing advice:

1. Limit the number of open-ended text questions. These are taxing for respondents and difficult to process when results are analyzed. They can be valuable when used properly, however. Recommended viewing: Qualtrics video re: text answers: Using Open Text Questions (qualtrics.com)
2. Avoid using Matrix questions. These are also taxing for respondents. In addition, they are not mobile-friendly, a consideration when targeting surveys to students.
3. Avoid asking more than one question at once. An example would be: “Customer service is responsive and helpful.” It is entirely possible for Customer Service to be responsive but not helpful, or vice versa. Break these into separate questions.
4. Avoid using jargon. Unless you are targeting your survey to a very specific audience, avoid using technical jargon or terms used in a particular field. Also avoid slang. Follow the rules of grammar and use ordinary English words.
5. Keep your questions concise. Despite this being a university, it’s best to keep your questions as short and easy to read as possible. Keep #3 in mind, however.
6. Avoid Leading Questions. Be careful how you word your questions to avoid suggesting to respondents that one answer is preferable (“which ice cream flavor is best?: Chocolate, boring vanilla, some other flavor). Also avoid strategies such as giving respondents an uneven scale (‘dissatisfied’, ‘satisfied’, ‘very satisfied’, ‘blown away’).
7. Make sure that your answer options are mutually exclusive. For example, if your question is: “What is your age?” and your choices are: a) 0-30, b) 30+, how is someone 30 supposed to respond? The ranges need to be adjusted so that they don’t overlap, for example: a) 0-30, b) 31+.

Recommended Reading

• Survey Questions: Tips & Examples in 2022 | Qualtrics
• Your Guide to Rating Scale Questions in 2022 | Qualtrics
• psr.iq.harvard.edu/files/psr/files/PSRQuestionnaireTipSheet_0.pdf
• 10 Simple Tips for Writing Good Survey Questions } cmoresearch.com
• Survey Design and Administration Best Practices | lsu.edu
• web.stanford.edu/dept/communication/faculty/krosnick/docs/2010/2010 Handbook of Survey Research.pdf

It is important to have valid, unbiased survey data, so that you can draw valid conclusions.

Types of Bias

Sampling Bias: You can introduce bias right at the beginning, when selecting your sample population. If your sample is not representative of the population you hope to survey, your data may be skewed and lead you to inaccurate conclusions.

Consult with Institutional Research if you have any questions about what constitutes an appropriate sample population for your survey, or about how to obtain contact information for your population.

Non-Response Bias: Even if your sample population is free of bias, it’s unlikely that you will get every member to respond to the survey. Non-response potentially introduces bias into the survey data if something about the survey itself causes certain demographics to be less likely to respond.

Examples:

1. Distributing a survey about familiarity with technology in marginalized populations via social media will receive responses from those who are conversant with social media. Those who are not, or lack access to computers or cell phones, will be unlikely to see the survey, much less respond. This could result in data that suggests a greater level of expertise with technology than actually exists in the sample population.
2. A university distributing a survey regarding student engagement to all students is an example of selecting an unbiased sample population. However, engaged students are probably more likely to respond than those with low engagement. This is likely to give false data suggesting that almost all students are highly engaged.

However, if it can be determined that the non-response is truly random, then the data the survey has gathered is unbiased. Another way to put this is that if there are no important differences between respondents and non-respondents, then there is no bias.

Note, though, that you have no way to know whether that’s true or not, unless you can obtain, (or already have,) data on your non-respondents, as well as your respondents (meaning the survey probably cannot be anonymous).

NOTE: Although a higher response rate is generally associated with a reduction in bias, there is no percentage (other than 100%) where it can always be assumed that your data is without bias.

Even if you have a 70% response rate, you could have bias, if the 30% who didn’t respond have important differences from the respondents.

Since a better response usually gives you better data, however, you may want to see some tips for Improving Response Rates.

Item Non-Response: This is when respondents take the survey but skip questions. Generally, there are 3 reasons why respondents skip a question:

1. They don’t understand the question.
2. They don’t know (or think they don’t know) the answer to the question.
3. They don’t want to answer the question. It may be annoying, offensive, or too much work.

See Tips on Writing Survey Questions for help in avoiding these problems. A skipped question here and there probably isn’t much of a problem, but if one or more questions have been consistently skipped by your respondents, you may be lacking data you hoped to obtain, or only receiving answers from respondents with a certain point of view, which will introduce bias into your data.

NOTE: It is NOT recommended that you force respondents to answer a question before proceeding with the survey, although Qualtrics does give you this option. You run the risk of the respondent simply quitting the survey.

Better options:

• Offer respondents the option to finish the survey later.
• Give respondents a reminder at the end of the survey if there are questions that aren’t completed and allow them to go back to them and complete them.
• Qualtrics also gives you the option to give them a reminder before they proceed if they have skipped a question, without preventing them from proceeding.

Question Order Bias: It turns out that the order in which questions are asked can also introduce significant bias into your survey results. A common way to reduce this bias is to randomize the order of your survey questions.

You might also see our tips on writing survey questions.

Recommended Reading

• Survey Bias: Common Types of Bias and How to Avoid Them | Qualtrics
• The Ultimate Guide to Survey Bias Types | Qualtrics
• Different Types of Survey Bias and How to Avoid Them | surveylegend.com
• Survey Question Sequence, Flow, & Style Tips | Qualtrics
• Tips for Reducing Survey Question Order Bias | Qualtrics.
• How to Order Your Survey Questions the Right Way | alchemer.com
• How to Minimize Question Order Bias in Your Surveys | Trustmary
• Question Randomization | Qualtrics.com
• Avoiding Biased Questions: Seven Examples of Bad Survey Questions | delighted.com)

Here are some general ways that research suggests you may be able to improve your response rates:

• Understand the technological capabilities and preferences of your target audience and take this into account when designing and distributing your survey. For example, research shows that young people (college-age and younger) prefer to use phones rather than computers for many tasks.
• Use a different mode to notify your target audience in advance. For example, if you plan to distribute your survey via email, you might want to send out a text message in advance or use social media to announce the distribution of your survey.
• Send reminders to your target audience. The recommended number of reminders is at least three, possibly up to six. The recommendation is to send a reminder 48 to 72 hours after the initial survey goes out. However, don’t send rapid-fire emails over a day or two.

Email Tips for Limiting Non-Response

Here are a few suggestions for the email that you send out with your survey. Research suggests these may increase your response rates.

• Do not use the word “Survey” in the subject line. However, do mention the University of Idaho. For example, you might say something like, “University of Idaho Institutional Research needs your feedback”.
• Personalize the email if possible. If you use a mailing list in Qualtrics, this can easily be done.
• Place the hyperlink to the survey near the top of the email.
• Keep your email short.
• Don’t include images. They slow downloading and get your email marked as spam.
• Put a name at the bottom. Ideally, your email should come (at least nominally) from someone in the university administration; your dean, if the survey is within your college, or a unit director, vice-president or provost as appropriate.
• Provide contact information for someone that people can contact if they have questions.
• Tell your respondents they are part of a select group. For example, “you are one of the students who have been chosen to represent seniors in the College of Business.”
• Give a deadline for completion.
• Tell your audience that you need their help.
• Tell your audience that their feedback is important.
• Give them an honest estimate of survey completion time — if possible, this should be less than 10 minutes.
• Promise confidentiality if applicable. The standard wording is “Results are confidential All data will be reviewed only in aggregate.”
• Proofread your email — several times! Also have someone else proofread it, preferably more than one person

Suggestions for Avoiding Spam Filters

• Someone cannot respond to your survey if they don’t receive it, which is likely to happen if your email with the survey link gets caught in their spam filter.
• Some spam filter algorithms are difficult or impossible to avoid.
• Avoid including these things in your email, as these are commonly used by scammers:
  • Capital letters.
  • Colored fonts.
  • A “From” name that looks strange. Anti-phishing advice (including the U of I’s own) advises readers to check the ‘From’ line to see if it matches the purported sender or is instead from an unknown address.
  • Images or attachments.
  • Phrases such as “Click here,” “Click below” or "This isn’t spam.”

The Best (and Most Expensive) Option for Increasing Response Rate: Incentives

As it implies, this means paying (or rewarding in some material way) people to take the survey.

There are three types of Incentives:

• Pre-Paid: Respondents receive the incentive before they even take the survey.
• Post-Paid: Respondents receive the incentive after they complete the survey. Historically, this has not been as effective as pre-Paid.
• Lottery: This usually has the weakest response and has only been shown to have a significant positive effect on response rate when the reward is valuable and highly desirable to the target audience, and the chance of winning is reasonably good. However, if you can get something donated, it can’t hurt.

Recommended Reading

• How to Increase Survey Response Rates | Qualtrics
• Looking to Improve Response Rates? Use These Levers | XM Institute

Once you have obtained survey data, you need to analyze it to find out what it tells you. This might seem straightforward, but you will want to understand any surprises that show up in your data.

If there are no surprises, or at least, no new information, then what was the point of the survey? Presumably, you were hoping to home in on critical data that will influence decision making. Therefore, it is important that you correctly interpret your results. View the Qualtrics (Xmbasecamp) video course about this topic.

The first step is to get your results into a form where you can see how respondents answered, rather than trying to sort through dozens or hundreds of individual survey responses.

Qualtrics itself offers handy ways to visualize your data via the Results tab. You can also download your data in several formats. Using the csv format, it is a simple matter to get your data into Excel, where it can be manipulated with that program.

Qualtrics Results Dashboard

Read an overview of the Qualtrics Results Dashboard

Clicking the Results tab will generate a basic bar chart for each question. These can be customized in many ways to provide a simple and easy to understand way to visualize your data. You can also use filters to focus on aspects of the data that you are interested in. Watch a short video on how to use the Results Dashboard to format your data.

Qualtrics Data and Analysis

Read an overview of the Data & Analysis Tab.

The Data and Analysis tab does allow you to do some statistical analysis of your survey data, although it is not as powerful as statistical processing programs. Its handiest uses, however, may be to make sure that your responses are in the form you want them before exporting them.

This is also where you can export your data to allow you to manipulate it in Excel or other programs.

You can also import responses to combine responses from multiple surveys.

You can delete responses; for instance, you can delete test responses, so you don’t have to worry about these affecting the final data.

You can also edit responses; for example, if a respondent tells you that they accidentally chose an answer they hadn’t intended and want it changed.

You also have the ability to allow a respondent to retake a survey if necessary.

Understanding the Data

Now that you have your data in a form that lets you see how your respondents answered your questions, you can start interpreting those answers — actual Data Analysis.

Institutional Research can help you with data analysis — it’s what we do, after all!

Recommended Reading

• Survey Data Analysis: Best Practices for Reporting - Qualtrics

• Survey Statistical Analysis Methods in 2023 - Qualtrics

• Cross-Tabulation Analysis: The Ultimate Guide - Qualtrics

• Survey Analysis in 2023: How to Analyze Results| getthematic.com

• A Beginner’s Guide to Survey Data Analysis and Data Collection - Chattermill

• How To Interpret Survey Results | Constant Contact