When Research Goes Wrong - Top 3 Fails and How to Avoid Them
Photo by Andrea Piacquadio from Pexels
Survey tools like SurveyMonkey or Typeform make research easier and more accessible for everyone. Whether you work with a self-service platform, or an established research provider, there are points of failure in any survey. Often they are only found when looking at the data. At this point, it is too late to fix them and re-fielding will be required, which impacts your budgets and timings.
Here are the top 3 data collection errors a researcher can encounter and how to avoid them.
Programming error
If you are running a survey on a self-service platform, take your time checking your survey thoroughly. It is tempting to rush to launch your survey quickly, but haste makes waste. Make sure you check, and double-check, and triple-check, your programmed survey, including correct spelling, grammar and filtering. Filtering errors will have a significant impact on your data, if questions were not asked of the right sub-group in your sample.
If you work with a research provider, then they will do these programming checks for you to make sure the online survey matches your questionnaire. Mistakes can still happen, especially if your questionnaire is complex, e.g. looped questions or piped-in text.
Research providers will typically launch with 5% of the sample, which is called a ‘soft launch’. Always check your soft launch data! This allows you not only to efficiently check the correct bases for all of your questions, but also do a ‘common sense check’ of the data. It helps you see how real people respond to your questionnaire and you can spot issues early and fix it before more data is collected. If you discover an issue during your soft launch, do a second soft launch and re-check the data to see if the issue has been fixed.
Translation error
If you are running a multi-market/ multi-language project, differences in translation between languages could be at fault. The main culprits are:
Wrong or literal translations: This can happen when translations are done using online translation services, such as Google Translate. The quality of these services can differ wildly by language and should never be used for questionnaire translations.
Wrong nuance: This can happen for challenging translations, e.g. of emotions. The translator may choose a right word, but not the right word for the context of the questionnaire. Always have these reviewed by a native speaker against the master language to ensure the two languages match.
Wrong overlay: Rather than programming a survey in each language, a research provider will create a master script and overlay local language translations on top of it. During this copy/paste process, labels on a Likert scale (e.g. agree/ disagree) could be swapped, the same translation could have been pasted into two different places or missed altogether. These types of errors can be difficult to spot, especially for longer or more complex questionnaires, or if you do not speak the language. Soft launch data checks may the first place you spot these, as the data will come out differently than expected.
These do not just apply to translations, but also localisations. For example, if you run an English language survey in the UK and the US, you need to change the word ‘football’ to ‘soccer’, ‘mobile phone’ to ‘cell phone’ and ‘petrol to ‘gas’.
Questionnaire design error
If you have already eliminated programming and translation errors, then a questionnaire design error could be at fault. During the design process, you can become very close to the questionnaire and you may not spot that things that are obvious to you, do not necessarily make sense to someone completing the survey.
Ambiguous or leading questions: You may be testing a specific hypothesis, but questions that subtly guide respondents towards a certain answer can introduce bias and compromise the validity of your data.
Using jargon: If you are surveying the general population or your costumers, using overly complex words or jargon can lead to confusion and misunderstanding.
Missing response options: Failing to provide relevant list response options can limit the accuracy of responses and force respondents to choose answers that do not accurately reflect their opinions or experiences.
Order effects: The order in which questions or list items are presented can influence respondents' answers and introduce bias into the results.
Poor formatting and layout: Poorly designed surveys with inconsistent formatting, cramped layout, or unclear instructions can deter respondents and compromise data quality. Grid questions should always be avoided.
A key step in avoiding questionnaire design errors is piloting your questionnaire. This can be done internally with colleagues who have not seen the questionnaire before. Talk to your testers to get their view on the survey experience and the cognitive processes they use to answer the questions. Consider your soft launch as an extended pilot before a full sample roll-out.
