“Sir, I want to ask you some questions if you agree?”
With that one sentence, our enumerator summarized the 120-word script provided to secure the informed consent of our survey participants – a script designed, in particular, to emphasize that participation would not result in any direct assistance. Humanitarian organizations, research institutes and think tanks around the world are conducting thousands of surveys every year. How many suffer from similar ethical challenges? And how many substandard survey results fall under the radar due to lack of effective quality assurance?
We were conducting a survey on the relationship between internal displacement, cross-border movement, and durable solutions in Borno, a linguistically diverse state in northeast Nigeria. Before data collection began, Translators without Borders (TWB) translated the survey into Hausa and Kanuri to limit the risk of mistranslations due to poor understanding of terminology. Even with this effort, however, not all the enumerators could read Hausa or Kanuri. Although enumerators spent a full day in training going through the translations as a group, there is still a risk that language barriers may have undermined the quality of the research. Humanitarian terminology is often complex, nuanced, and difficult to translate precisely into other languages. A previous study by Translators without Borders in northeastern Nigeria, for example, found that only 57% of enumerators understood the word ‘insurgency’.
We only know the exact phrasing of this interview because we decided to record some of our surveys using an audio recorder. In total, 96 survey interviews were recorded. Fifteen percent of these files were later transcribed into Hausa or Kanuri and translated into English by TWB. Those English transcripts were compared to the enumerator-coded responses, allowing us to analyze the accuracy of our results. While the process was helpful, the findings raise some important concerns.
Consent was not always fully informed
Efforts to obtain informed consent were limited, despite the script provided. According to the consultant, enumerators felt rushed due to the large numbers of people waiting to participate in the survey – but people were interested in participating precisely due to the misbelief that participation could result in assistance, which underlines the need for informed consent.
Alongside these ethical challenges, the failure to inform participants about the objectives of the research increases the risk of bias in the findings, prompting people to tailor responses to increase their chances of receiving assistance. Problems related to capacity, language, or questionnaire design can also negatively impact survey results, undermining the validity of the findings.
The enumerator-coded answers did not always match the transcripts
During data quality assurance, we also identified important discrepancies between the interview transcripts and the survey data. In some cases, enumerators had guessed the most likely response rather than properly asking the question, jumping to conclusions based on their understanding of the context rather than respondents’ lived experiences. If the response was unclear, random response options were selected without seeking clarification. Some questions were skipped entirely, but responses still entered into the surveys. The following example, comparing an extract of an interview transcript with the recorded survey data, illustrates these discrepancies.
| Interview transcript | Survey data |
| Interviewer: Do you want to go back to Khaddamari?
Respondent: Yes, I want to. Interviewer: When do you want to go back? Respondent: At any time when the peace reigns. You know we are displaced here. Interviewer: If the place become peaceful, will you go back? Respondent: If it becomes peaceful, I will go back. |
Do you want to return to Khaddamari in the future? Yes
When do you think you are likely to return? Within the next month What is the main reason that motivates you to return? Improved safety What is the second most important reason? Missing home What is the main issue which currently prevents return to Khaddamari? Food insecurity What is the second most important issue preventing return? Financial cost of return |
At no point in the interview did the respondent mention that he or she was likely to return in the next month. Food insecurity or financial costs were also not cited as factors preventing return. Without audio recordings, we would never have become aware of these issues. Transcribing even just a sample of our audio recordings drew attention to significant problems with the data. Instead of blindly relying on poor quality data, we were able to triangulate information from other sources, and use the interview transcripts as qualitative data. We also included a strongly worded limitations section in the report, acknowledging the data quality issues.
We suspect such data quality issues are common. Surveys, quite simply, are perhaps not the most appropriate tool for data collection in the contexts within which we operate. Certainly, there is a need to be more aware of, and more transparent about, survey limitations.
Despite these limitations, there is no doubt that surveys will continue to be widely used in the humanitarian community and beyond. Surveys are ingrained in the structure and processes of the humanitarian industry. Despite the challenges we faced in Nigeria, we will continue to use surveys ourselves. We know now, however, that audio recordings are invaluable for quality assurance purposes.
A manual audio recording strategy is difficult to replicate at scale
In an ideal world, all survey interviews would be recorded, transcribed, and translated. This would not only enhance quality assurance processes, but also complement survey data with rich qualitative narratives and quotes. Translating and transcribing recordings, however, requires a huge amount of technical and human resources.
From a technical standpoint, recording audio files of surveys is not straightforward. Common cell phone data collection tools, such as Kobo, do not offer full-length audio recordings as standard features within surveys. There are also storage issues, as audio files take up significant space on cell phones and stretch the limits of offline survey tools or browser caching. Audio recorders are easy to find and fairly reliable, but they require setting up a parallel workflow and a careful process of coding to ensure that each audio file is appropriately connected to the corresponding survey.
From a time standpoint, this process is slow and involved. As a general rule, it takes roughly six hours to transcribe one hour of audio content. In Hausa and Kanuri – two low resource languages that lack experienced translators – one hour of transcription often took closer to eight hours to complete. The Hausa or Kanuri transcripts then had to be translated into English, a process that took an additional 8 hours. Therefore, each 30-minute recorded survey required about one day of additional work in order to fully process. To put that into perspective, one person would have to work full time every day for close to a year to transcribe and translate a survey involving 350 people.
Language technology can offer some support
In languages such as English or French, solutions already exist to drastically speed up this process. Speech to text technologies – the same technologies used to send SMS messages by voice – have improved dramatically in recent years with the adoption of machine learning approaches. This makes it possible to transcribe and translate audio recordings in a matter of seconds, not days. The error rates of these automated tools are low, and in some cases are even close to rivaling human output. For humanitarians working in contexts with well resourced languages like Spanish, French, or even some dialects of Arabic, these language technologies are already able to offer significant support that makes an audio survey workflow more feasible.
For low-resource languages such as Hausa, Kanuri, Swahili, or Rohingya, these technologies do not exist or are too unreliable. That is because these languages lack the commercial viability to be priority languages for technology companies, and there is often insufficient data to train the machine translation technologies. In an attempt to close the digital language divide, Translators without Borders has recently rolled out an ambitious effort called Gamayun: the language equality initiative. This initiative is working to develop datasets and language technology in low-resource languages relevant to humanitarian and development contexts. The goal is to develop fit-for-purpose solutions that can help break down language barriers and make language solutions such as this more accessible and feasible. Still, this is a long term vision and many of the tools will take months or even years to develop fully.
In the meantime, there are four things you can do now to incorporate audio workflows into your data collection efforts
- Record your surveys using tape recorders. It is a valuable process, even if you are limited in how you are able to use the recordings right now. In our experience, enumerators are less likely to intentionally skip entire questions or sections if they know they are being recorded. Work is underway to integrate audio workflows directly into Kobo and other surveying tools, but for now, a tape recorder is an accessible and affordable tool.
- Transcribe and translate a small sample of your recordings. Even a handful of transcripts can prove to be useful verification and training tools. We recommend you complete the translations in the pilot stage of your survey, to give you time to adjust trainings or survey design if necessary. This can help to at least provide spot checks of enumerators that you are concerned about, or simply verify one key question, such as the question about informed consent.
- Run your recordings through automated transcription and translation tools. This will only be possible if you are working in major languages such as Spanish or French. Technology is rapidly developing, and every month more languages become available and the quality of these technologies improve. Commercially available services are available through Microsoft, Google, and Amazon amongst others, but these services often have a cost, especially at scale.
- Partner with TWB to improve technology for low-resource languages. TWB is actively looking for partners to pilot audio recording and transcription processes, to help gather voice and text data to build language technologies for low resource languages. TWB is also seeking partners interested in actively integrating these automated or semi-automated solutions into existing workflows. Get in touch if you are interested in partnering: [email protected]
Written by: Chloe Sydney, Research Associate at IDMC Eric DeLuca, Monitoring, Evaluation, and Learning Manager at Translators without Borders