Abstract
Background Our research team is conducting phenomenological interviews in Kenya with people who have not been able to access community eye health services, aiming to explore the barriers and ideas for potential service modifications. We conducted an embedded study that compared in-person and telephone interview modalities in terms of time requirements, costs, and data richness.
Methods A team of six interviewers conducted 31 in-person interviews and 31 telephone interviews using the same recruitment strategy, topic guide, and analytic matrix for each interview. We compared the mean duration; mean number of themes reported by each participant; total number of themes reported; interviewer rating of perceived richness; interviewer rating of perceived ease of building rapport; number of days taken by the team to complete all interviews; and all costs associated with conducting the interviews in each modality.
Findings In-person interviews were 44% more expensive and took 60% longer to complete than our telephone interviews (requiring 5 days and 3 days respectively). The average in-person interview lasted 110 seconds longer than the average telephone interview (p=0.05) and generated more words and themes. However, the full set of interviews from both approaches identified similar numbers of barriers (p=0.14) and the same number of solutions (p=0.03). Interviewers universally felt that the in-person approach was associated with better rapport and higher quality data (p=0.01). Triangulation of themes revealed good agreement, with 88% of all solutions occurring in both sets, and no areas of thematic dissonance.
Discussion The in-person approach required more time and financial resources, but generated more words and themes per person, and was perceived to afford richer data by interviewers. However, this additional richness did not translate into a greater number of themes that our team can act upon to improve services.
Introduction
Background
Qualitative interviews – especially those grounded in the phenomenological approach - are designed to elicit rich data about participants’ lived experiences and perceptions of a given phenomenon.1 Our research team has been using interviews to explore barriers to health service access and potential solutions that might improve accessibility by engaging with people who were referred to local eye clinics but were not able to access care in Kenya.2
Our project is embedded within the ‘Vision Impact Project’ (VIP) eye screening programme that operates in ten counties across the country. Over a million people have been screened in the past year, and over 150,000 of these people have received care in free local outreach clinics 3. However, internal data from the screening programme suggest that up to half of all those referred to local treatment clinics are not accessing care. Furthermore, early data from a related study suggests that certain sociodemographic groups have much lower odds of accessing care than others. In Meru County we have found that younger adults (aged 18-44 years) are the least likely to receive the care they need.4 We wanted to explore these peoples’ experiences and perceptions of specific barriers to accessing care, as well as their ideas around any changes we could make to the eye care services to make it easier to access care.
The VIP screening budget is limited, and programme implementers are keen that our interviews can be conducted quickly and as inexpensively as possible whilst still delivering robust findings. The incentive to deliver timely and affordable findings is further underlined by our desire to see embedded qualitative research adopted more widely across routine programmatic quality improvement initiatives, so that the voices of intended service beneficiaries can be included in decision making. Based on the findings of a recent scoping review on rapid qualitative research methods5 we have developed a rapid ‘abductive’ interview approach that uses a deductive analytic matrix to facilitate rapid iterative analysis of data whilst “making space for inductive identification of themes and issues not predicted at the outset”.1 Our work employs a phenomenological approach, grounded in a pragmatist philosophical paradigm.6
The work in Kenya is part of a broader overall project to develop equity-driven and evidence-based approaches to improve access to community-based services across Kenya, Botswana, India and Nepal.7 Hundreds of thousands of people are being screened and referred to local services each year, however only around half are able to access care. We wanted to develop an interview approach that could be taken to scale across these four countries – and potentially beyond – to deliver timely insights into how these programmes can be made more accessible, especially for ‘left behind’ groups. Given the scale of the project, telephone interviews are likely to offer the most pragmatic means of obtaining timely insights on how to improve services, however it is not clear what – if anything - would be lost from using telephone interviews as opposed to in-person interviews which generally represent the ‘gold standard’ in qualitative research.8
In-person interviews are commonly perceived the ‘best’ way of obtaining rich phenomenological data due to the fact that the interviewer can observe visual cues and quickly build rapport.9–11 However, telephone interviews offer unique advantages: the increased social distance can make it easier for participants to discuss sensitive topics; travel time and interviewer safety concerns are eliminated; power imbalances are partially concealed; and overall costs can be greatly reduced – depending on the specific study design and population.9,12,13 For our projects, participants can be spread across vast distances, meaning that the risks, costs, and time-requirements for in-person interviews are likely to compare poorly with telephone interviewing. However, we were unable to accurately quantify the trade-off between data quality and resource requirements between the two approaches. As such, we decided to conduct this study to assess which modality offers the best balance of richness, duration, and costs in the context of our work to explore barriers to access and potential solutions in Meru County, Kenya.
Mode comparison
A number of previous studies have sought to compare telephone and in-person interview modalities.10,12–17 In qualitative research, quality is conceptually linked to the ‘richness’ of the data obtained, described by Charmaz in terms of revealing participants’ true feelings, intentions and actions, and accessing their “otherwise inaccessible thoughts”.18 Many different proxies have been used to approximate richness in mode comparison studies. A crude but relatively common approach is to measure the duration or wordcount of each interview, working from the assumption that longer interviews, with more words spoken, are more likely to provide deeper insights into people’s lived experiences.12,13,15,16 Interview duration is often used in the same way, based on the assumption that longer interviews generate richer data, with some studies also reporting ‘interviewer dominance’ i.e. the proportion of the talking that is done by the interviewer as opposed to the participant.16 Surprisingly few qualitative mode effect studies compare the actual content of the interviews, despite the fact that this is a more nuanced way of assessing the amount of topic-related data that are generated.16,17 This approach is also relatively straightforward, requiring the reporting of the total number or unique themes that arise from each set of interviews and/or the mean number of themes identified by each interview.
A further approach entails having researchers subjectively rate their experience of each interview in terms of the perceived richness of the data obtained, as done by Abrams et al using a three-point Likert scale.19 Other reported measures include quantifying the word count of associated field notes for each interview and counting the amount of conversational turn-taking that occurs in each interview.15,16
In this study, we aimed to compare the data richness obtained from two sets of in-person and telephone interviews, electing to use a broad range of proxies: interview duration and wordcount; number of themes identified; and subjective interviewer rating of richness and rapport. We aimed to gather additional data on the time taken to complete each set of interviews, and the associated costs. We hypothesised that telephone interviews would be faster and less expensive to complete than in-person interviews, but offer less-rich data across all metrics of comparison.
Methods
Interviews
A previous equity analysis conducted by our team in Meru had found that younger adults (aged 18-44 years old) were the least likely to receive eye care in the county’s community-based screening programme.4 We obtained a list of all of the younger adults who did not receive care from Peek Vision, a partner organisation that provides the screening and patient flow management software for the programme.3 Peek also record contact numbers for all participants. We used computer-generated random numbers to identify interviewees from this target population, and to determine interview modality.
We performed the telephone interviews first. We performed 36 interviews but a retrospective saturation analysis found that saturation was reached after approximately 30 interviews. Our protocol for this study20 stipulated that we would compare an equal number of interviews from both modalities, with a minimum of 20 v 20. We decided to conduct 31 additional in-person interviews and compare these against the first 31 telephone interviews.
The same team of six data collectors conducted all interviews using the same semi-structured interview guide. The same process for audio recording data and directly transcribing quotes into the deductive analytic matrix was used for both modalities, and the same process of iterative review and analysis across all cases within each modality was used to generate the final themes. Data collectors received two days of training before conducting the interviews in September 2023.
Comparison domains
We gathered data on six domains of richness. We then collated data on the time taken to complete both sets of 31 interviews and their associated costs, following the approach set out in our protocol.20
Interview duration: We measured the duration of each interview in minutes from the start of the consenting process until the researcher concluded the interview e.g. by thanking the participant for answering all of their questions. In line with previous studies discussed above, this metric was used as a proxy for richness, based on the assumption that longer interviews capture richer data than shorter interviews. Note that we did not use interviewer dominance measures this is only possible with typed transcripts, and our approach is based around direct-from-audio entry of verbatim quotes.
Matrix wordcount: We counted the total number of words entered into the analytic matrix for each set of interviews. These were verbatim quotes directly transcribed from the audio by the data collectors. In line with previous research, we assumed assume that a higher wordcount was associated with richer data.
Total number of themes: We counted the total number of unique themes for barriers and solutions that were reported across all interviews with each modality. We assumed that the modality that captured the largest number of unique themes was capturing richer data. From an operational standpoint, our underlying study is primarily concerned with generating potential solutions that will improve equitable access, so the number of unique solutions that emerged from each set of interviews is a particularly important metric.
Number of themes reported by each participant: We also reported the range and mean number of unique themes (barriers and solutions) identified by each participant for each modality. This was to hedge against a situation where one modality generated a greater number of themes than the other, but only because of one or two prolific interviews.
Interviewer subjective rating of richness: After all of the interviews were complete, each of the six data collectors were asked to provide a single global summary rating of the perceived richness obtained from all in-person and all telephone interviews. Following the approach used by previous researchers, we used a simple Likert scale: low = 1, moderate = 2, high = 3.
Interviewer subjective rating of rapport: We supplemented the subjective rating of richness with a second question that asked data collectors to provide a global summary rating of the perceived ease of building rapport across all in-person and all telephone interviews. Again, we used a simple Likert scale: low = 1, moderate = 2, high = 3.
Time taken to plan and complete all interviews: We documented the total amount of time taken to plan and complete all interviews in each modality to the nearest half-day. This was recorded by the Kenyan research manager in charge of scheduling, supervision, and logistics for the local research activities.
Costs: Working with a health economist, we recorded costs from the payer’s perspective. Both modalities use the same sampling and analytic approach, so we only compared costs that were unique to each approach i.e. those associated with data collection. For telephone interviews these included airtime and staff daily salaries multiplied by the number of days required to complete data collection, starting with the first phone call to recruit the first participant, and ending with the conclusion of the final interview.
For in-person interviews we included the costs of printing consent forms, transport for researchers, transport reimbursement offered to participants; payments for local Community Health Promoters and sub-county health officials to assist with setting up the interviews (mobilisation/sensitisation), and staff daily salaries multiplied by the number of days taken for data collection.
The costs of voice recorders were not included in the comparison because they were used for both sets of interviews. Similarly, the same two-day training covered skills required for both interview modalities so this was not included in the comparison. We did not compare overhead costs unless they differed for the modalities. The local research manager also recorded any unforeseen additional costs associated with each modality.
Statistical approach
We used sign tests for the paired interviewer ratings. For the unpaired mean testing comparisons we used histograms to check the data for normality and then used T-tests or Mann-Whitney-U tests, as appropriate, to provide evidence as to whether the two modalities differed across the domains.
Triangulation of themes
Finally, we compared the barriers and solutions that emerged from both modalities. We identified themes that were identified in both sets of interviews (agreement); themes that emerged from one set of interviews but not the other (silence); and any areas of dissonance i.e. where themes from one set conflicted with those from the other.
Ethics
Ethical approval was granted by the Kenya Medical Research Institute (KEMRI), the Kenyan National Commission For Science, Technology & Innovation (NACOSTI), and the London School of Hygiene & Tropical Medicine research ethics committee. Each participant provided informed consent.
Findings
For our comparison we used 31 telephone interviews and 31 in-person interviews that were conducted by our team of six researchers in September 2023 across four sites in Meru County. Table 1 summarises our main findings.
Richness
The average in-person interview lasted 110 seconds longer than the average telephone interview (P=0.05) and generated 33% more words in the analytic matrix. On average, face-to-face interviews identified a greater number of barriers(p=0.14) and solutions (0.03), however the entire in-person interview set only identified one additional barrier and the same number of unique solutions as telephone interviews. All six data collectors were unanimous in their ratings of data richness and ease of building rapport, rating in-person interviews as ‘high’ and telephone interviews as ‘moderate’ for both measures (p=0.01).
Time requirements
It took two days to prepare for the in-person interviews and then three days to complete them. Preparation time included phoning potential participants to invite them to participate, and then scheduling meeting times and places, and organising transport and local logistics. This included working with local Community Health Promoters (CHPs) and sub-county health officials to sensitise and locate interviewees. This is a vital element in building trust and legitimising our work with participants: the CHPs visited each person to discuss the project and answer any questions, and then supported the researchers to connect them with the interviewees in the field.
Costs
Telephone interviews required three days of our data collectors’ time, plus the airtime used to complete the calls. Higher airtime costs for the telephone modality reflects the fact that phone calls were used for recruitment, consenting, and data collection, whereas the in-person approach only used calls for recruitment. Spending on data collectors’ salaries was the same for the in-person interviews – which were also completed in three days - however this modality incurred a number of additional costs. We paid two local county officials to assist with scheduling the in-person interviews, and for sensitization and mobilisation on the days of the interviews. We paid nine Community Health Promoters to build trust, explain the project, and physically locate interviewees. We printed physical consent forms, reimbursed travel to a convenient interview location for our interviewees and paid to transport our data collectors to the same location. Whilst the VIP programme operates across the entire county, at the time that our study was running the programme was operating in Meru town, meaning that all of the interviews were conducted within 15-30 minutes away from our offices. As such, we estimate that the transport costs for data collectors could easily rise by a factor of ten or more for in-person interviews conducted in other parts of the county.
Triangulation of themes
Table 3 presents the 21 unique barriers that were identified across all 62 interviews. There was agreement between in-person and telephone modalities on nine of these barriers (42.3%). There was silence on the remaining 12 (57.7%) with each modality identifying six unique barriers that did not emerge from the other set of interviews. We found no evidence of thematic dissonance.
Table 4 presents the 25 unique solutions that were identified across all 62 interviews. There was agreement between in-person and telephone modalities on 22 of these barriers (88.0%). There was silence on the remaining three (12.0%) with each modality identifying three unique barriers that did not emerge from the other set of interviews. We found no evidence of thematic dissonance.
Discussion
In this study we examined the quality, costs and time-requirements of in-person vs telephone modes, based on 62 interviews conducted with young adults who had not been able to access eye care services in Meru, Kenya. Even with serendipitously low transport costs, in-person interviews were almost twice as expensive as telephone interviews and took 1.7 times longer to complete. They delivered longer interviews with more words transcribed into the analytic matrix and more themes identified per interview. Our data collectors universally ascribed higher ratings of richness and ease of building rapport to in-person interviews. However, across both modalities, exactly the same number of unique solutions were identified.
Our findings align with the wider literature. Irvine et al. found that telephone interviews tended to be shorter than in-person interviews, although their study only included 11 interviews in total.15 Novick’s review of the literature found evidence that telephone interviews are generally less expensive and shorter than in-person interviews.9 In their retrospective mode-effect analysis of 300 interviews, Johnson et al. found that in-person interviews produced longer transcripts and more word-dense field notes, but generated the same themes as telephone and videocall-based interviews.16 Interestingly, subjective interviewer ratings were also similar across the approaches. Krouwel and colleagues also compared in-person interviews to those conducted using video-calling software. They found that in-person interviews generated more data but the overall number of themes derived from each approach was similar.17 Vogl’s triangulation of the themes that emerged from two sets of interviews with 56 children found negligible differences.13 Finally, in his systematic review comparing telephone and in-person approaches, Rahman concludes that both telephone and in-person modalities can generate comparably rich data, with telephone interviews tending to be less time consuming and less expensive.10
Given that empirical mode comparisons consistently find that remote interviews are able to generate similar qualitive themes at lower costs and in shorter time periods than in-person interviews, irrespective of research question and population studied, Rahman has argued that the in-person modality should only be used if the specific research question demands it.10 The relationship between depth of detail, number of themes, and agreement between themes is intriguing. Participants tend to provide much more detail about a given phenomena during in-person interviews, as indicated by longer transcripts, interview durations, and analytic matrix wordcounts. However, this additional detail rarely translates into identification of novel codes or themes when compared to remote approaches.
There were differences in the themes that emerged from both sets of interviews. Whilst the differences between the solutions was fairly minor, several of the barriers that were raised during the telephone interviews were potentially more candid than those derived from in-person interviews. A form of social desirability bias might have been at play, with interviewees feeling more comfortable disclosing potentially embarrassing or taboo issues when the interviewer was not physically sat in front of them.21,22 Some of the barriers that emerged exclusively from the phone interviews included forgetting about the appointment, assuming that the service would not meet their needs, and perceiving the mixing of men and women in a single queue as ‘shameful’.
In terms of strengths and limitations, whereas most research in this field tends to employ one or two metrics, our study compared eight different dimensions of performance, including proxies for richness (duration and wordcount), mean and aggregate themes, and subjective interviewer ratings, supplemented with an assessment of costs and time requirements. We conducted a relatively large number of interviews on a topic that is central to global efforts to extend Universal Health Coverage as part of the Sustainable Development Goals.23–25
Our sample size was based on a post-hoc saturation analysis, but the decision to compare 31 interviews with each modality rather than 30 vs 30 was essentially arbitrary. The generalisability of our findings is limited by our relatively focused research question and the homogeneity of our population (younger adults in Meru who were found to have an eye problem during screening but did not manage to access further care). Ultimately, whilst our study presents multiple measures we are not able to definitely say which approach is best, as there is no single ‘right’ way to balance differences in richness, costs, and time requirements.
Previous research has documented that the impact of qualitative research findings on real-world programmes is influenced by the timeliness of the findings,26,27 and our broader embedded qualitative work places a premium on rapidly identifying barriers and potential solutions to improve equitable access to care within a live, ongoing screening programme. Given our focus on identifying solutions and service modifications that can be rapidly tested, the lack of dissonance between the modes, lower costs, lower time requirements, and additional researcher safety benefits associated with telephone interviews means that we are very likely to continue using this approach.
Conclusions
Our set of 31 telephone interviews was completed in less time and at less expense than the 31 in-person interviews. Whilst the in-person modality generated longer interviews and more data, the ultimate number of themes that derived from both sets was nearly identical. For our purposes, telephone interviews offer clear operational advantages with no meaningful reduction in data quality.
Data Availability
Data produced in the present study are available upon reasonable request to the authors
Declarations
Conflict of interest
The authors declare no competing interests.
Funding
This study is supported by the National Institute for Health Research (NIHR) (using the UK’s Official Development Assistance (ODA) Funding) and Wellcome [215633/Z/19/Z] under the NIHR-Wellcome Partnership for Global Health Research. The views expressed are those of the authors and not necessarily those of Wellcome, the NIHR or the Department of Health and Social Care.
Author contributions
LA conceived the study with SK. LA and SK performed the data collection and analyses. LA wrote the initial draft. SK and MT provided additional written content. JT provided health economics input and reviewed the manuscript. DM provided statistical input and reviewed the manuscript. AB reviewed the protocol and provided additional methodological comments. All authors reviewed and approved the final version of the manuscript.
Ethical approval
The study was granted ethical approval by KEMRI scientific and ethics review unit on November 30th 2022, NACOSTI on 12th January 2023, and the LSHTM ethics committee on 5th May 2023.
References
