Screening for Poverty And Related Social determinants to improve Knowledge of and links to resources (SPARK): development and cognitive testing of a tool for primary care

Background to the SPARK Tool

An initial set of questions on social determinants for use in healthcare settings was developed by a collaboration of hospitals and a local public health organization in Toronto, Canada during 2011-2012.¹ Questions on language, immigration, race, ethnicity, religion, disability, gender identity, sexual orientation, income, and housing were constructed based on a review of the literature, meetings and consultations. These questions were pilot tested with approximately 1600 individuals across four organizations.^12,13 Subsequent research on how patients interpreted the questions, and how they performed in primary care^14–17 and other settings¹⁸, led to revisions and the inclusion of additional domains including employment, medication access and social isolation. Following pilot testing of these revised questions in five primary care clinics in Ontario, the questions were further refined, resulting in the Screening for Poverty And Related determinants to improve Knowledge of and links to local resources (SPARK) Tool. The SPARK Tool includes 16 questions that cover demographics (language, immigration status, Indigenous identity, race, disability status, sex at birth and gender identity, sexual orientation) and social needs (education, income, medication access, housing status, social isolation, transportation, cost of utilities, and precarious employment) [Appendix 1].

Data collection

To further refine the questions and complete cognitive testing, we administered the SPARK Tool and conducted in-depth interviews with a diverse set of individuals. Participants were recruited from four provinces in Canada: Saskatchewan, Manitoba, Ontario, and Newfoundland and Labrador. Study advertisements were circulated using social media, through classified advertising websites, using collaborator distribution lists, and with posters placed in waiting rooms in health centers and community spaces. Study advertisements were in English, French and the top three non-Indigenous languages spoken in participating provinces according to the latest national census¹⁹: Arabic, Farsi, Punjabi, simplified Chinese, and Tagalog. All interviews were conducted virtually and consent was obtained from participants once they had read and understood the detailed information sheet provided. Interviews were multi-part (completing the SPARK Tool using a think aloud process, additional comparator questions, and open-ended questions), and lasted approximately 1.5 hours. Data on Indigenous identity and race were not collected in Manitoba, based on feedback from the local research ethics board on the potential negative impact on relationships with Indigenous communities. In this paper, we report on the results of cognitive testing.

Cognitive interviews were conducted by five members of the study team (ADP, DH, IA, LK, ZS) and two research assistants, who all identified as female. All interviewers had moderate to extensive experience with conducting qualitative interviews but most were new to cognitive interviewing. Cognitive interviewing is a technique used to gain insight into learners’ perceptions in which individuals are invited to verbalize thoughts and feelings as they examine information.²⁰ In this study, surveys were self-administered in the presence of the interviewer.

Participants were asked to read the survey question and options provided aloud and think aloud, by sharing their immediate thoughts on each question. Participants then selected their answer choice. Minimal unscripted concurrent probing was done by interviewers, except to prompt participants to continue to think aloud throughout the survey. Interviewers recorded the thoughts of participants for each question. Interviewers were trained to look out for: issues related to question wording, issues related to unclear question objective, questions with redundancy or repetitiveness, issues of burden or length of the survey, and questions that lacked options participants frequently want to select.²¹

Data analysis

The analysis was conducted by the five members of the study team who had conducted the interviews. Informal analysis of the cognitive interviews was conducted as described by Willis.²¹ Using the interview notes, we thematically grouped unique topics or concerns that emerged for each question and assessed the frequency of each theme. Results were summarized by each question in a table with columns describing the topic or concern, and the frequency of each theme. Study team members reviewed the themes question-by-question and updated the SPARK Tool to address high frequency topics or concerns. This was done in a series of meetings where study team members discussed each question, including all topics or concerns, as well as potential solutions suggested by the participants to determine how to improve the SPARK Tool. Decisions made for each question were recorded. Topics or concerns were addressed by either modifying question wording, adding, editing or removing answer options, or replacing the question. Once a first draft was created, the Tool was presented to the larger study team, comprised of researchers and patient partners, to provide feedback on the changes. The Tool was also presented to an advisory committee that included subject matter experts on sociodemographic data. Feedback was received from other subject matter experts that were not part of the study team or advisory committee through emails and brief one-on-one meetings. Once all the feedback was incorporated, the SPARK Tool was finalized (Table 2).

General perspectives on the SPARK Tool

Throughout the SPARK Tool, many participants indicated they were unclear why some questions were asked in the context of receiving healthcare. Based on this feedback, we added descriptions to each question, to explain the purpose of the question and its potential use in care at an individual and organisational level. Many participants also indicated being unfamiliar with terms used, and descriptions also provide definitions.

For many questions about social needs, participants expressed confusion concerning the timeframe for and frequency of the social need they were being asked to indicate. For example, with the question, “Do you have difficulty making ends meet at the end of the month?”, participants indicated they could make ends meet some months (frequency) or could regularly make ends meet a year ago but not currently (timeframe). To address this feedback, we updated the questions to include a specific timeframe (e.g. ‘in the past 12 months’) and, in some cases, updated answer options to include frequency (e.g., ‘always’ or ‘sometimes’).

For questions that had ‘not applicable’ as an option, many participants were confused about the difference between ‘no’ and ‘not applicable’. Therefore, we qualified the ‘not applicable’ option with a short phrase. For example, for the question about paying one’s rent or mortgage on time, we replaced ‘not applicable’ with ‘not applicable, I do not have to pay rent or mortgage’.

For certain questions, such as those concerning whether the participant was born in Canada, sex at birth, gender identity, and sexual orientation, several participants noted they may be unwelcome or uncomfortable to complete. To address this concern, the research team included a ‘prefer not to answer’ option to all the questions in the final survey.

Additional and optional questions

Based on feedback from study participants, research staff, patient partners, advisory committee members, and other subject matter experts, additional domains were added to the questions on social needs, including questions on food security, and phone and internet access. Ethnicity and religion were added as optional questions. These optional questions should be used at the discretion of healthcare leaders if feel that they are necessary for use at their respective healthcare organizations.

Language preference for healthcare and need for translation

The wording of the question concerning language needs, “If it could be arranged, would translation into another language be helpful at your next appointment?”, was unclear for only seven (3.6%) participants. Five (2.6%) participants were unsure if the question was referring to a healthcare appointment or any appointments. To clarify, we added ‘healthcare’ to the updated question. In addition, to reduce data entry error, we added a list of optional drop-down answer choices for language preferred. The options are based on the top 20 languages and top five Indigenous languages spoken most often at home in Canada based on the 2016 Statistics Canada Census data.¹⁹

Born in Canada

Among 71 (36%) participants that were not born in Canada, three participants could not recall what year they arrived to Canada. Given that this question is aimed at determining newcomer status, we revised the answer options to include ‘less than 5 years’, ‘5 to 9 years’ and ‘10 years or more’ instead of an open-text field. This reduces data entry error and addresses recall issues. It also fits with Statistics Canada’s definition of a recent immigrant or newcomer, which refers to a person who obtained a landed immigrant or permanent resident status up to five years prior to a given census year.²² A limitation of this question is that it does not account for people who arrived more than once in Canada. Three (1.5%) participants expressed confusion, indicating that they moved to Canada more than once.

Indigenous identity

In this paper, we have not reported cognitive interview results from study participants on this question as we believe that the reporting of these results should be governed and led by Indigenous communities. We have included in the descriptor that this data must be collected with engagement with local First Nations, Métis, and Inuit governance bodies in accordance with the First Nations OCAP, Métis OCAS, and Inuit Qaujimajatuqangit data governance and sovereignty principles.^23–26

Race

The race question includes a preamble intended to explain why race is being asked about at a healthcare appointment. Twelve (6.1%) participants indicated that the preamble was too technical or overwhelming or requested simpler language. Eight (4.1%) participants noted that ‘not based in science’ is technically incorrect or unnecessary (e.g., sociology is considered a science, skin color differences are studied in certain scientific disciplines i.e., melanin), and some noted that ‘not based in biology’ is a more appropriate phrase. In addition to suggested edits to the preamble, eighteen (9.2%) participants requested examples of countries or regions for each race category in the answer options, which we have now included. Fifteen participants wanted the ‘White’ and ‘Black’ option to be separated into different “types” of White or Black people.

For example, participants suggested splitting the ‘Black’ option into African, Afro-Caribbean, African American etc. However, based on the definition of race and ethnicity, which we describe in the newly added descriptors, “types” of white or black people would correspond with ethnicity. The revised SPARK Tool provides an optional ethnicity question to address this topic. Eight (4.1%) participants indicated preference for an ethnicity question instead of a race question. While the participant feedback on this question was valuable and informed the creation of and changes to other questions, we did not edit this question in light of the cognitive interview. Rather, in keeping with national standards, we opted to align our final question including the preamble and answer options, with the standards for collecting race-based data released by the Canadian Institute for Health Information (CIHI) in March 2022.²⁷

Disability status

For 38 (19.5%) participants, the disability question, “In general, do you experience any of the following due to a physical, mental, or emotional condition?”, and its answer options were seen as too broad and subject to interpretation. For example, ‘difficulty communicating’ may refer to a speech impairment or language barriers. Some participants (34, 17.4%) were unsure if a difficulty they experienced should be included. More specifically, there were 12 participants (6.1%) who were unsure if they should say they have a difficulty if it could be corrected with assistance (e.g., glasses for ‘difficulty seeing’). Based on the feedback we received, the study team revised the question to include ‘due to a severe and persistent physical, mental or emotional condition’. Additionally, the study team revised the answer options based on participant feedback. For example, it was unclear what ‘self-care’ meant to 19 (9.7%) participants, so we changed that option to ‘personal hygiene’. Including examples in the answer options and adding a descriptor helped to clarify that this question is intended to identify conditions that are severe and disabling. Eleven (5.6%) participants felt the options provided were limited and did not account for difficulties associated with having a mental health condition. The addition of answer options to include ‘difficulty with activities for daily living’ captures difficulties mentioned in the cognitive interviews that were not present in the original answer option.

Sex at birth

For the question, “What was your sex assigned at birth?”, six (3.1%) participants noted that ‘intersex’ status is not necessarily always ‘assigned’ at birth, therefore ‘assigned’ was removed from the question.

Gender identity

For the question, “What is your current gender identity?”, based on feedback from subject matter experts, the term ‘current’ was removed from the question as it was deemed redundant or unnecessary for answering the question. Nine (4.6%) participants felt the options provided for gender identity were limited. Based on feedback from the participants as well as from subject matter experts, additional options, including ‘transgender woman’ and ‘transgender man’ were included in the question. For five (2.6%) participants, the difference between ‘sex at birth’ and ‘gender identity’ was unclear, and hence, the descriptor includes an explanation.

Sexual orientation

Two additional options, ‘demisexual’ and ‘asexual’, were added to the sexual orientation question (‘which best describes your sexual orientation?’) as 17 (8.7%) participants felt that the original options were too limited, and a few participants specifically mentioned those options should be included. This change was supported by the study team’s review of the literature and consultations with subject matter experts. We removed the definition in the ‘heterosexual’ option (‘male/female relationships or two different binary genders’) based on feedback from 16 (8.2%) participants who felt it was unnecessary, some of whom found it offensive. ‘Queer’ and ‘pansexual’ were separated into two different options as 9 (4.6%) participants highlighted that these terms are not synonymous. Some participants (2, 1.0%) indicated that they felt that gender could be represented by more than one answer option (e.g., queer and bisexual), therefore individuals can now select multiple options.

Education

For the education question, “What is the highest level of education you have completed?”, three (1.5%) participants and a patient partner on our study team felt that the phrase ‘highest level of education’ in the question was un-inclusive, hierarchical, elitist, or offensive, so the term ‘highest’ has been removed. Twenty-one (10.8%) participants mentioned that the education answer options were too limited and, in some cases, confusing. Therefore, several options were added or revised including replacing ‘trades certificate/diploma’ and ‘college/university degree’ with ‘college, CEGEP or other non-university certificate or diploma’, ‘completed registered apprenticeship or other trades certificate or diploma’ and ‘undergraduate degree’. Examples were added to clarify the term ‘postgraduate’ as 14 (7.2%) participants, many of which spoke English as a Second Language (ESL) or completed their education outside of Canada, were unfamiliar with the term. We clarified or removed the word ‘some’ in the options and replaced with ‘ongoing’ in most cases. One limitation to the updated question is that participants are not able to select more than one option for their level of education as requested by 9 (4.6%) participants. For example, in instances where a participant has attended both college and university and considers both education levels to be equivalent, they are forced to pick just one. Multiple choice selection was not added to this question as its purpose is to ascertain literacy level indicated by education level.

Finances

We changed the question from “Do you have difficulty making ends meet at the end of the month?” to “Do you currently have difficulty paying for basic needs?” as 26 (13.3%) participants, many of whom were ESL speakers, found the term ‘making ends meet’ difficult to understand. For 7 (3.6%) participants, it was unclear how to define a basic or essential need, so we included clarification in the descriptor. One limitation to the updated question is that it does not capture people who are able to pay for basic needs like food, shelter, clothing, but still have serious financial difficulties. This concern was identified by 7 (3.6%) participants. This brief question may not accurately capture every aspect of a person’s financial situation. However, it has been validated as a good predictor of poverty and identifying individuals living below the “poverty line” or low-income cut-off (LICO).²⁸

Access to prescription medication insurance

Three (1.5%) participants felt that the question about medication access (‘in the last 12 months, did you avoid filling a prescription or do anything to make a prescription last longer because of the cost?’) did not consider medical devices or supplies (e.g., for diabetes management), as well as over-the-counter (OTC) medications, which may not be prescribed. To account for patients’ diverse medical needs, ‘prescription’ was replaced with ‘medication or medical supplies’.

Housing

We changed the question from “What is your current housing?” to “What is your current housing situation?” as the previous wording of the question was cumbersome according to 6 (3.1%) participants. There were no adequate options for 35 (17.9%) participants who lived with family or friends by choice (and not because of a lack of alternatives). To address this, we replaced the ‘own home’ and ‘rent’ options with ‘a place you or your family owns’ and ‘a place you or your family rents’. Some participants found the terms ‘couch surfing’ (12 participants) and ‘on the street’ (9 participants) stigmatizing, inappropriate or offensive. We replaced ‘couch surfing’ with ‘staying in someone else’s place because you have no alternative’ and ‘On the street’ with ‘Experiencing homelessness’. Nineteen (9.7%) participants felt that the options were limited and did not capture other living situations such as transitional housing or a nomadic lifestyle. We added additional options including supportive housing or group home, long-term care facility and correctional facility. We added “social housing, subsidized housing or rent-geared-to-income’ as an option to this question and therefore removed the follow-up question “Is your current housing social housing, subsidized housing or rent-geared-to-income?” from the SPARK Tool. Seven (3.6%) participants indicated wanting to be asked about their living arrangements, specifically about who they live with. For example, if they rent, to specify that they live with roommates. One participant mentioned that the ‘rent’ option is a wide category that could include living with multiple roommates or alone in a “big fancy apartment”. We added a follow-up question asking “who do you live with? select all that apply”. For 18 (9.2%) participants, the options were inadequate and did not capture people who did not have to pay rent or people who were only able to make partial payment of the rent. ‘Not applicable, I do not have to pay rent or mortgage’ was added to address the first problem. The updated question does not yet capture participants who were only able to make partial rent payment, which is a limitation of the question.

Transportation

In the transportation question used in this study, “In the past 12 months, did you avoid attending an important appointment because of the cost of transportation?”, it was unclear to four (2.0%) participants what type of appointment the question was referring to. This has been clarified in the updated question. The updated question was adapted from the American Academy of Family Physicians (AAFP) Social Needs Screening Tool and the PREPARE Toolkit.^29,30

Utilities

Eleven (5.6%) people were unsure what constituted a utility bill, therefore, in the revised question and the created descriptor, we provided examples of utilities for clarity. The descriptor explains ‘the specific utilities you pay for may depend on where you live’, indicating that electric, gas/oil and water bills are only examples of utility bills. There were no adequate options for 8 (4.1%) participants whose utilities are included in the rent. If these participants experienced difficulty paying a bill it would be captured in part three of housing question which asks the participant, “in the past 12 months, was there a time when you were not able to pay the mortgage or rent on time?” Further, we have expanded the ‘not applicable’ answer option to state “not applicable, I did not have to pay utility bills in the past 12 months or utilities already included in rent”.

Social supports

Social support is captured by two questions, both of which were moved to follow the utilities question (initially placed after housing) as some participants found the transition from housing to social supports abrupt. For the first question, ‘do you feel you have family or close friends who you can open up to?’, 10 (5.1%) participants, many of whom were ESL speakers, the term ‘open up to’ was difficult to understand. We therefore revised the question adding ‘confide in’ for clarity (i.e., ‘do you feel you have people who you can open up to or confide in’). Six (3.1%) participants felt unable to provide a yes or no response to social support questions, explaining that they may not be able to open up all the time or may be able to open up about some things but not others. To address this we added ‘yes, always or sometimes’ to the ‘yes’ option. In addition, ‘family or close friends’ was replaced with ‘people’ in the question as three participants said they may be able to open up to people not in those categories, e.g., pastor, teacher, etc. For the follow-up question, ‘are you able to rely on them if you need help (e.g., transportation, emotional or financial assistance)?’, 32 (16.4%) participants, felt the categories or examples provided in the question were too specific. For example, participants may be able to rely on the people they referred to in the first question emotionally but not financially. We removed the examples and left the question open to any kind of help. Some of these categories, e.g., transportation, are addressed in other questions.

Employment

The SPARK Tool administered in this study included three questions aimed at identifying individuals experiencing precarious employment. However, 34 (17.4%) participants felt unable to answer the employment questions which were not inclusive of all employment situations, including full-time, permanent, self-employed, did not capture people in between jobs or working multiple jobs, and failed to capture participants that were retired. To address this, we added two questions: ‘are you currently employed?’ and ‘are you currently looking for work?’ before asking the follow up questions about precarious employment. These revisions will capture individuals unemployed and looking for employment and therefore may help connect people to employment resources. Among the precarious employment questions, the first of which asked ‘are you employed in a casual, short-term, or temporary position?’, 12 (6.1%) participants were not familiar with the terms ‘casual’ and ‘short-term’. The umbrella terms for precarious work ‘temporary’ or ‘part-time’ were used instead. Examples of types of temporary or part-time employment were also included. Based on feedback from 4 (2.0%) participants who had multiple jobs, this question was revised to include ‘main job’ in the question to clarify that this is referring to primary source of employment. For the second precarious employment question, ‘do you feel fearful that you could be fired if you raise employment concerns?’, the phrase ‘raise employment concerns’ was confusing for 4 (2.0%) participants. To clarify this, we added examples of types of employment concerns. The question has also been modified to simplify the language. Based on feedback from research staff, ‘fired’ was replaced with ‘negatively affected’ in the question to account for people who may not be fired but endure negative consequences if or when they raise concerns. For the question about income instability, ‘does your pay vary a lot from month to month?’, we replaced ‘vary’ with ‘change’ for 12 (6.1%) participants who wanted the question reworded. Several of these participants also mentioned that their pay may change a lot due to other factors unrelated to precarious employment such as self-employment or shift work (e.g., nurses). This is a limitation that the updates have not addressed.