Improving Clarity and Interpretability of Items in a Bilingual Index of Propensity to Integrate Research Evidence into Clinical Decision-Making in Rehabilitation
=================================================================================================================================================================

* Jacqueline Roberge-Dao
* Nancy Mayo
* Annie Rochette
* Keiko Shikako
* Aliki Thomas

## Abstract

**Purpose** To contribute evidence for the clarity and interpretability of a new five-item bilingual multidimensional index of a rehabilitation clinician’s propensity to integrate research evidence into clinical decision-making.

**Methods** This study was conducted in three sequential steps: (1) We conducted a focus group with occupational therapists, physical therapists, and researchers to review the items and response options for clarity, consistency, and interval properties and agree on equivalency in English and French. (2) We conducted cognitive interviews whereby clinicians elaborated on their interpretation of the item, comprehensibility of items, and appropriateness of response options. Accepted modifications were integrated and tested with subsequent participants. (3) We conducted an online survey to validate the English and French equivalency of response options on a 0 to 100 scale.

**Results** During the qualitative revision process (one focus group with 7 participants followed by 27 interviews), the index was revised 12 times with substantial modifications to the *use of research evidence* and *attitudes* items.

**Conclusion** This research increases the clinical relevance and reduces measurement error of this brief index which can inform on individual or organizational factors influencing a clinician’s propensity of integrating research evidence into decision-making and ultimately, improve rehabilitation outcomes.

Keywords
*   evidence-based practice
*   clinical decision-making
*   measurement
*   psychometrics
*   cognitive interviews
*   content validity
*   multidimensional index

## Introduction

To optimize quality of care, occupational (OTs) and physical therapists (PTs) as rehabilitation clinicians are expected to engage in evidence-based practice (EBP), that is, integrate best available research evidence, their clinical expertise, and patient values and preferences when making clinical decisions.1–4 Rehabilitation clinicians acknowledge that clinical experience is essential to integrating research evidence into practice13–15 and as such, they must make sense of the quality, pertinence and applicability of research evidence using their judgment and tacit knowledge.16,17 Importantly, patient-centered practice is considered a basic tenet of occupational therapy (OT) and physical therapy (PT) and accommodating patients’ goals, values and preferences are vital for positive patient outcomes and satisfaction.18,19

The tripartite conceptualization of EBP (i.e. research evidence, clinical expertise, and patient values) has traditionally been depicted by the “three circles” model.5–8 In more recent years, there have been refinements and additions to this conceptualization of EBP reflected in the inclusion of the context (e.g., organizational, economic, professional) to highlight the external influences to clinical decision-making (CDM).9–12 Despite the purported benefits of such a CDM approach and the implementation of EBP content into most entry level OT and PT curricula1,4 globally, clinicians report difficulties integrating research evidence into practice.20–25 Lack of allotted time for activities related to EBP, poor access to research evidence; low confidence in applying research to practice; lack of knowledge that evidence-based interventions exist; and inadequate equipment to implement new practices are among the reasons for the underutilization of research evidence in practice.15,20–22,25–27

Robust measurement practices are needed to identify the factors related to EBP that should be improved or the strong areas that must be maintained.28–30 Identifying which areas require improvement can inform targeted allocation of resources to support EBP and ultimately, improve patient outcomes. There exists a vast selection of questionnaires measuring the core factors influencing an OT or PT’s likelihood to integrate research evidence.28,31–33 However, there are shortcomings to current EBP measures such as failure to concurrently measure multiple domains, the inappropriate analysis of items derived from ordinal scales and the unknown relative weight of domains.33

In our previous work, Al Zoubi et al. identified the six most salient domains influencing a rehabilitation clinician’s likelihood of integrating research evidence into CDM.30 One best performing item was chosen per domain to form a brief, multidimensional index in English and French, as described by Roberge-Dao et al. (submitted). The five domains included in the index are: *use of research evidence, self-efficacy, resources, attitudes,* and *activities related to EBP.* However, as the selected items stem from five different questionnaires, there is inconsistency between items (and response options) in terms of the terminology and formulation which can increase respondent burden and introduce measurement bias. In addition, the English and French versions of these items may present cultural or linguistic discrepancies that can introduce systematic differences in scores.34

The aim of this study was to contribute evidence for the clarity and interpretability of items and response options for a new bilingual measure, the *Propensity to Integrate Research Evidence into Clinical Decision-Making Index* (PIRE-CDMI). Specifically, the primary objective was to review and revise the included items in the prototype index in English and French. The secondary objective was to estimate the equivalency of response option labels in both languages.

## Methods

This study involves a three-phased qualitative review process as illustrated in figure 1. Ethics approval was obtained from *The Faculty of Medicine and Health Sciences Institutional Review Board* at McGill University for all phases of this study before commencement and recruitment.

![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/08/08/2024.08.07.24310514/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/F1)

Figure 1. 
Overview of the item revision process for the Propensity to Integrate Research Evidence into Clinical Decision-Making Index (PIRE-CDMI)

### Phase 1: Focus group

#### Focus group participants

Practicing OTs and PTs and EBP researchers (defined as researchers having experience in EBP research and having published a minimum of one EBP-related publication) were recruited purposefully from the networks of the research team to participate in a 90-minute online focus group. The pool of participants was expected to be bilingual and have equal representation of both French and English native speakers.

#### Focus group process

The aim of this focus group was to review the items and response options of the prototype PIRE-CDMI for clarity, consistency, and interval properties and arrive at a consensus on modifications that would be needed to have equivalent versions in English and French.35,36 Participants were asked to establish equivalence in both languages such that the items, instructions, and response options were conceptually (i.e., do people in both groups see the concept in the same way) and semantically (i.e., the meaning attached to words in an item) comparable.36,37 Consenting participants were sent the items with a reminder of the study aim a week before the focus group.

The online focus group was conducted via Zoom and structured as follows: (1) welcoming and overview of the study; (2) objectives, instructions, and an example for item rewriting; (3) breakout room with two individuals per room for five minutes to allow attempts at reviewing one item; (4) attend to any questions that arose during breakout room; (5) item rewriting exercise altogether using the share screen function.

During the item rewriting exercise, the moderator (first author JRD) structured the discussion and a note taker recorded the suggested modifications on a shared online document. Participants were asked to rewrite items from question-item format into declarative statements from the perspective of a clinician (see figure 2 for an example of this process). Probing questions included: (1) How would you rewrite this item into a declarative statement? (2) Is the wording clear, and if not, how would you change it? (3) How difficult would it be for OTs and PTs to answer these items? For each item, the French translation was discussed simultaneously. Once every item was discussed, the moderator asked participants to verify that the overall index was coherent in terms of wording and length, that items read well together and that everyone agreed on the final set of items. After the focus group, the research team (consisting of bilingual EBP researchers in rehabilitation) reviewed the suggested final set of items and resolved any withstanding discrepancies.

![Figure 2.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/08/08/2024.08.07.24310514/F2.medium.gif)

[Figure 2.](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/F2)

Figure 2. 
Example of the item rewriting exercise in the focus group

### Phase 2: Cognitive interviews

#### Cognitive interview participants

New participants were recruited for cognitive interviews. Clinicians were eligible to participate in the cognitive interviews if they were (1) practicing OTs and PTs in Canada; (2) native French or English speaking; and (3) had been practicing for a minimum of one year. We used social media (Twitter and Facebook) and the University’s newsletter to advertise the project. Interested participants entered their contact information in an online form. A member of the research team contacted them to provide more information on the study. Participants gave informed consent to participate in the research.

#### Cognitive interview process

Cognitive interviews were conducted by the first author (JRD) by telephone or by Zoom conferencing with potential respondents of the PIRE-CDMI (i.e., OTs and PTs) to identify and rewrite any problematic items to increase the overall readability, functioning and interpretability of the measure.36,38–42 All interviews were audio recorded and expected to last between 15 to 30 minutes.

Participants were provided with a copy of the newly reviewed PIRE-CDMI at least one day before the scheduled interview. As presented in table 1, the interviewer used the verbal probing method to elicit participants’ comprehension of all five items by asking specific questions regarding meaning, clarity, and interpretation of items.40,43 These questions were adapted from the authors of a study using similar methods in developing a preference-based index for multiple sclerosis.44 Participants were encouraged to think out loud while going through the measure, allowing for insight into how a participant perceived and interpreted the items.43

View this table:
[Table 1:](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/T1)

Table 1: 
Cognitive interview probing questions

#### Analysis of cognitive interviews

English and French interviews were conducted in parallel so that no language was prioritized. After each day of interviews, the interviewer reviewed comments and revised the problematic items based on participants’ suggestions. The research team reviewed the feedback before implementing the change and proposed suggestions based on best practices of item development such as having simple items that express a single idea, using common vocabulary, and avoiding colloquialisms.36,41,45 Changes were implemented in both languages simultaneously, when applicable. The revised version of the PIRE-CDMI was then tested on the next round of participants. interviews were conducted until no further changes were necessary as suggested by three consecutive participants.44,46

### Phase 3: Survey

Given Canada’s linguistic diversity, methods to ensure the equivalence of questionnaire versions in the two official languages are warranted to decrease systematic differences between language groups. Validation of translations through quantitative response scaling can contribute to demonstrating that respondents are interpreting items in a similar fashion and consist of asking respondents to denote the position of response options on a visual analogue scale (VAS) (i.e., a line from 0-100) and to compare the ratings between languages. These methods have been used in previous studies.47,48

#### Survey respondents

This phase consisted of a cross-sectional online survey to generate additional evidence on the equivalency of PIRE-CDMI response option labels in English and French. The target population was healthcare professionals and students provided that (1) they were native English or French speakers and (2) worked or studied in a healthcare professional or graduate program in Canada. We used convenience sampling and did not exclude respondents based on profession or level of training, because the nature of the survey was such that respondents solely needed to have the abilities to interpret common words and rate response option labels on a numerical scale. We recruited through social media and interested respondents were invited to follow a link with study information, a consent statement, and an invitation to start the survey in the native language of their choice. No identifying nor sociodemographic information data were collected.

#### Survey procedure and analysis

The survey was piloted with seven graduate students, all of whom were also practicing clinicians in rehabilitation. Modifications were integrated to improve survey clarity and task comprehension. The survey was open from November to December 2021.

Each PIRE-CDMI item was associated with three response option labels. Respondents were asked to indicate the position of each of the three response option labels on a 0-100 VAS between two anchors. Response option labels belonging to the same set appeared on a single page sequentially. This method has been previously reported for health-related quality of life measures such as the SF-3647 and EuroQol-5d48. Specifically, participants were asked: “On the line, where would you position each of the three response option labels between [the bottom anchor] and [the top anchor]?” Appendix I presents the three response option labels for each item and associated response anchors. The full PIRE-CDMI item was also stated on the same page as the response set to provide context.

It was hypothesized that respondents would position the lowest response option label closest to the zero anchor, the middle response option label in the middle, and the highest response option label closest to the 100 anchor. Post hoc analyses were conducted to remove respondents who likely misunderstood the task reflected in having either: (1) more than two sets of disordinal response patterns; (2) the same rating for all three response options for one item or more; or (3) two or more extreme outlier ratings (0 or 100). Multiple linear regression was used to estimate the extent to which VAS ratings per item (0 to 100 scale) depended on language (English, French), response option (low, middle, high) and the interaction between language and response option. The normal probability plot of standardized residuals was visually examined.

## Results

### Focus group item rewriting

Four PTs and three OTs, all doctoral candidates with research experience in EBP participated in the 90-minute focus group. Each of the five question-item-response sets was transformed into five sets of three declarative statements which were then clarified and harmonized in English and French (version 2 of the PIRE-CDMI). Participants agreed that response options including the word “never” (for example, “I never integrate research evidence”) were perceived as being undesirable because choosing these response options would make them be perceived as incompetent. Given that clinicians would not opt for these options, the participants removed the word “never” from the *use of research evidence* and *activities* items. For the *activities* item, focus group participants communicated that it was important not to confine research evidence to scientific articles; they suggested replacing “read research” with “consult research evidence”. Further, participants agreed that omitting the verb “reading” was more inclusive to individuals who may have visual impairments. In French, multiple terms were proposed for “research evidence” but the agreed upon term was “*données probantes”* which was said to be most employed and recognized among clinicians.

Three issues remained unresolved after the focus group and were subsequently discussed within the research team. Modifications were made to the items before starting the cognitive interviews (version 3). First, it was unclear which term was preferred between “patient” and “client” as the terms are often used interchangeably in rehabilitation contexts depending on the setting and the population. The research team agreed to use “patient” consistently and added a footnote to explain the interchangeable nature (N.B. the final wording of items did not include the words “patient” nor “client”). Second, participants could not decide which term was best between “organization” and “clinical setting”. The research team modified the item to focus on the broad availability of resources and refrained from using either term. Finally, participants could not come to a consensus between the verbs “willing” and “inclined” in the *attitudes* item as they were compelled to answer high on the *attitudes* item; although clinicians may be willing or inclined to use evidence, they may not actually do so in practice. Participants reported that the adjectives “willing” and “inclined” did not have equivalent translations in common French that would be suitable for a self-report measure (“*enclin à”* or “*disposé à”* are not commonly used words). Thus, the *attitudes* item was reframed from “I am willing to use EBP” to the notion of “it is worth the effort to [use EBP]”. Appendix II reports the step-by-step changes at each step of the qualitative rewriting process.

### Item modifications from the cognitive interview process

Twenty-four individual cognitive interviews were conducted with 10 PTs and 14 OTs in Canada (13 native English speakers, 12 native French speakers; one bilingual participant provided feedback in both languages). Appendix III presents an overview of the item evolution process during cognitive interviews. An overview of the modifications made to the items are described below.

The *self-efficacy* item underwent three iterations. From the initial item, “I am (very confident/somewhat/not very confident) in my ability to integrate evidence into my intervention plan”, the word “integrate” was replaced with “apply” to be more action oriented. The words “intervention plan” were first replaced with “clinical cases” to avoid discriminating clinicians who solely perform assessments. The words “clinical cases” were then simplified to “practice” to avoid any confusion associated with the variability in clinical cases. Finally, the response option label “very confident” was changed to “confident” because participants stated it was difficult to endorse being very confident with one’s ability to apply research evidence to practice. In the final version, the wording of two response option labels (“somewhat” and “not very” confident) was not an exact translation in French (“*moyennement”* and “*peu confidant”)*.

The item on *use of research evidence* underwent three iterations. At the start of the cognitive interviews, this item consisted of asking respondents about the source of information, between research evidence, colleagues, or clinical experience, that they would seek when faced with a practice uncertainty. Participants found this item particularly difficult to answer because it was dependent on the case at hand (e.g., the availability of evidence for a clinical diagnosis or patient values) and the organizational context (e.g., whether colleagues were available and/or had experience related to the case). Clinicians reported that they often used a combination of all three sources and that it was difficult to select one to describe their typical behavior. The final wording focused on the frequency of using research evidence when faced with a practice uncertainty which avoids the conflicting response options of colleagues and clinical experience. Finally, an asterisk was added to define practice uncertainty as “a situation in which there is a gap in your knowledge relating to a clinical decision”.

With six versions, the *attitudes* item underwent the highest number of iterations. At the start of the cognitive interviews, respondents were asked the extent to which incorporating evidence into practice was worth the effort. Participants suggested that the item not contain the connotation of “worth the effort” because it was (1) prone to social desirability bias (e.g., participants felt pressured to respond the best and highest level) and (2) did not translate well in French (e.g., “*cela vaut l’effort”* or “*cela vaut la peine”*). The item was modified to focus on EBP requiring effort (e.g., “It requires little/some/a lot of effort to integrate research evidence into practice”). The response option label “some (effort)” was changed to “moderate (effort)” to clarify the middle level response, and the words “(requires… effort) for me” were added to clarify the intent of eliciting the individual’s perception of effort rather than a general belief. In French, the direct translation of “it requires little effort for me to…” is “*cela me requiert peu d’efforts pour*…” which was problematic for two reasons. First, starting a sentence in French with “*cela”* was too informal. Second, the verb “*requiert”* was too formal. The structure of the French sentence was changed to place the object (“*intégrer les données probantes dans ma pratique”*) before the verb and to replace “*me requiert”* with “*me demande”*.

The *resources* item underwent three iterations. The initial item was “I feel that I have the/only some of/do not have the necessary resources to integrate research evidence into my practice”. Participants suggested omitting the words “I feel that …” and questioned which resources the item was referring to. An asterisk was added to clarify meaning and enumerate examples of resources facilitating EBP. In French, participants prefered the verb “*je possède (les ressources nécessaires)”* to “*j’ai (les ressources nécessaires)”*. For one response option label (“some of”), the final French wording was not an exact translation (“*une partie des”*).

Finally, the *activities* item underwent the least number of modifications. The only modification consisted of changing the words “consult research evidence” to “keep up to date with research evidence”. Participants were interpreting the initial item as the frequency of using evidence in their practice which was already reflected in the *use of research evidence* item. The revised item reflects the concept of staying up to date with research evidence as an activity outside of routine CDM. Though three of the 24 participants suggested that we explicitly describe and quantify the three adverbs (regularly, occasionally, and rarely), the research team decided to avoid quantifying these adverbs as there is no agreed upon best practice for behavioral frequency of consulting the literature. By providing these three response options without specifying the exact range, the research team intended to capture clinician’s self-report relative to their temporal understanding of keeping up to date with research evidence in their field. In French, “keeping up to date with research evidence” did not exactly translate, so the following modification was retained for conceptual equivalence “*se tenir à jour quant aux données probantes”*.

The initial instructional prompt was “For each group of statements, select ONE statement which best applies to you. Please respond as honestly as possible”. The prompt was modified three times until the final version, “Please select ONE statement from each box which best reflects your current practice and context.”

The visual presentation of the measure was improved following participant suggestions. Specifically, the lettering of each response option was bolded to make discriminating between levels easier. It was also suggested to number the five items (1 to 5) and letter the three response options (a, b, c) to reduce cognitive burden involved in completing the index. The final version of the PIRE-CDMI in English can be found in Appendix IV. The final version of the PIRE-CDMI in French can be obtained from the corresponding author by request.

### Scaling of response option labels

Among the 129 individuals who started the online survey, 60 were Canadian French native (46%) and 69 were English native (54%). Of the 129, 42 Francophones (32%) and 38 Anglophones (30%) were included for analysis. The rest were excluded due to incomplete surveys (n=25, 19%) and task miscomprehension (n=24, 19%). Descriptive results for the rating of the five response option sets by the 80 respondents is presented in table 2. The ordinal nature of the ratings of response sets is illustrated in figure 3. Multiple linear regression results did not suggest any important main effects of language on score for the five items, nor any important interaction of language and response option. Sparring the presence of a few outliers in all items, the residuals were normally distributed.

View this table:
[Table 2:](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/T2)

Table 2: 
Descriptive statistics for the rating of response options on a 0-100 scale (n=80)

![Figure 3.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/08/08/2024.08.07.24310514/F3.medium.gif)

[Figure 3.](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/F3)

Figure 3. 
Histogram illustrating the frequency distribution of mean ratings of response option labels on a 0-100 scale for the five PIRE-CDMI items in English and French

## Discussion

This study describes a robust item revision and rewriting process of an index measuring a rehabilitation clinician’s propensity to integrate research evidence into CDM. This three-step approach to qualitative revision has not been previously reported in the EBP literature. Overall, the index (PIRE-CDMI) underwent 12 iterations to increase the clinical relevance and reduce measurement error with important changes made to the *use of research evidence* and *attitudes* items.

The *use of research evidence* item changed considerably with early iterations relating to (1) the frequency of integrating the three EBP pillars (research evidence, clinical expertise, and patient preferences) and (2) clinicians’ primary source of knowledge (research evidence, colleagues, and clinical experience). These two iterations failed to produce useful information because clinicians attested to integrating all three pillars of EBP into CDM and relying on all proposed sources of knowledge to various extents. In fact, the tripartite definition is foundational to how rehabilitation clinicians conceptualize EBP.50 However, asking clinicians to select their most relied upon pillar of EBP or to determine the frequency at which they integrate the three components is anathema to the reality of CDM whereby these elements are inextricably intertwined.23 Further, asking clinicians to select the most relied-upon source of knowledge in a measure relating to EBP appears to introduce high levels of social desirability bias.51,52 For example, a respondent may interpret the desirable answer to be “research evidence” and the undesirable answer to be “colleague” or “clinical experience”. This item formulation could inadvertently imply that consulting a colleague or relying on clinical experience is ill-advised when, in fact, these sources of knowledge are foundational to being a competent, reflexive, and evidence-based clinician.17,24,50,53 Though plurality of knowledge in CDM is increasingly invoked in the field of EBP,54 rehabilitation clinicians continue to voice challenges related to the integration of research evidence into CDM signaling a deep-rooted need for support with this component of EBP.21,24,26,27

The notion of a clinical uncertainty was introduced to the *use of research evidence* item to contextualize the behavior of seeking and using research evidence in CDM. The addition of contextual cues in items has been said to increase the validity of responses, notably when the behaviour has an element of automaticity.55 CDM often relies on automatic and intuitive reasoning rather than analytical reasoning, a phenomenon which is hypothesized to become stronger over time.56,57 This clinical uncertainty allows a clinician to tap into their analytical reasoning and can be compared to *the event* proposed in the reflective practice literature which is defined as “an event that occurs in everyday practice […] that leaves the occupational therapist with the urge to revisit it to make sense of it for the benefit of his or her future practice”.58(p345) The need to nuance this item with a clinical uncertainty is further reinforced by a possible mechanism whereby, over time, research-based knowledge becomes consolidated into tacit or experiential knowledge; in such cases, it may no longer be distinguished as *research evidence* but rather transformed into expert practice that is adapted to the practice context.59,60 Correspondingly, it may be difficult for clinicians to discern how frequently they use research evidence on a day to day basis. Lastly, consulting research evidence in everyday practice may not be realistic nor be a desirable behavior as it could conceal other professional difficulties such as low confidence in one’s clinical reasoning abilities.61 Thus, the more compelling question is not so much whether clinicians consult formal sources of evidence every day, but whether they do so when confronted with a gap in their knowledge.

Many modifications were made to the *attitudes* item to remedy the social desirability bias reported by participants. Despite changes to the item, participants, most of whom have been educated in Canada, continued to report that EBP was a desirable process and were compelled to select the highest response option for attitudes. The evidence demonstrating the relationship between attitudes towards EBP and EBP behavior is inconclusive. While some studies have suggested that holding positive attitudes towards EBP is an important precursor to EBP behavior,63–67 others have demonstrated that they do not translate into effective EBP behavior.20–22,68–71 We postulate that measuring attitudes towards EBP may not be useful in the context of this brief index given that (1) the relationship between attitudes and EBP behaviors is uncertain; (2) it is well-established that rehabilitation clinicians are generally convinced of the value of EBP and believe it to be a desirable and necessary process;20–22,69,71–73 and (3) value-laden items which can prejudice respondents should be omitted from measures36 and attitudes are inherently value-laden. Furthermore, when assessing attitudes for predicting behaviors, it is recommended to avoid measuring attitudes towards a general concept and to focus on specific behaviors.74 Using *effort* instead of *attitudes* circumvents asking clinicians whether they consider EBP to be valuable and highlights the perceived cost of integrating research evidence into practice.75 *Effort* is defined by the Cambridge Dictionary as “physical or mental activity needed to achieve something”. Social psychology and behavioral theorists have identified effort as largely contributing to behavioral motivation.76–78 People are less likely to engage in a behavior if it requires a large amount of effort79; several studies have shown that rehabilitation clinicians perceive the enactment of EBP as being effortful.23,24,69

One noteworthy modification to the *activities* item is the inclusion of various sources of research evidence outside of scientific articles. As such, one could associate keeping up to date with research evidence to leading or assisting a journal club, reading email subscription alerts, or gaining research-based knowledge from a colleague. This departure from formal sources of research-based knowledge is more aligned with the behaviors and preferences of rehabilitation clinicians who favor informal, quick methods of gaining research evidence, and tend to keep up with research through a variety of informal sources including consultation with trusted peers and email reminders.15,23,24,47,80–82 The process by which rehabilitation clinicians rely on colleagues for research-based knowledge is starting to gain importance in the EBP literature as a recognized and beneficial mechanism of EBP.24,82–84 Still, this is a promising avenue for future research.

In the third and last step of this study, the effect of language on VAS ratings was trivial, supporting the equivalency of the response option labels in English and French for all items. The distribution of ratings also demonstrates the ordinal consistency of response options and the quasi-interval nature of the scales (i.e., response options are equally spaced). The *keeping up to date* item had the largest variation in ratings between languages and within the same language, however these differences were less than five on 100 and are considered negligeable.98 A possible explanation for this variability in ratings may be because of the subjective nature of these behavioral frequency adverbs (potentially dependent on the area of practice) and lack of consensus on how often rehabilitation clinicians should keep up to date with the literature. As knowledge is produced at different rates for different areas of practice, and that this index is meant to be used in various settings, no explicit frequency denominator was attributed to this item. For instance, “regularly” could mean once every two months for a clinician in stroke rehabilitation or once a year for a clinician in palliative care. Our intent with this item is to capture the respondent’s self-rating relative to their understanding of what “regularly”, “occasionally” and “rarely” mean relative to their field of practice and relative to their perception of what is feasible given their clinical reality.

### Limitations

A strength of this study is the multi-phased rigorous qualitative review process which included target end-users and EBP researchers. In developing the response option scaling survey, we aimed to provide adequate guidance to maximize respondents’ comprehension of the task. Pilot testing enabled us to add examples and clarify the instructions. Still, given that 25 individuals did not complete the survey (19%) and that an additional 24 had to be excluded due to apparent miscomprehension of the task (19%), this exercise may have been perceived as difficult and burdensome, a finding also reported by others in the context of a valuation exercise for the EQ-5D.99 The task involved an unfamiliar method of placing response option labels on a 0 to 100 scale which required an ability for abstract reasoning. Given the lack of available demographic data, it is impossible to discern who misunderstood the task. Comprehension may have been improved with a quick instructional video.

Before deploying the PIRE-CDMI, there remains an important developmental step which consists of estimating the relative weights of each dimension-level. This work, which the research team has started, will allow for the generation of a more accurate total score that takes into consideration end-users’ perceived relative contribution of dimensions on the overall construct of propensity to integrate research evidence into CDM. We acknowledge that the initial mathematical properties of the prototype PIRE-CDMI established in previous research (Roberge-Dao et al., submitted) may have changed due to item rewriting. While this must be confirmed in future testing, our findings pertaining to the quasi-interval spacing of response option labels gives us reason to believe that the interval properties of the scale still hold. Due to important linguistic differences between different countries, we suggest undergoing a thorough cultural adaptation and reassessment before using the PIRE-CDMI with English and French-speaking individuals outside of Canada.

Finally, while some authors have stated that short scales are a limitation and can compromise the validity and reliability of inferences drawn from a measure,100,101 others have found value in the efficiency of short scales.102,103 There exists a delicate trade-off between scale comprehensiveness and feasibility. Given the resource-strained healthcare context, the aim was to create a short index capable of rapidly estimating elements of EBP requiring improvement. This measure can be used as an efficient global outcome measure of a clinician’s propensity to integrate research evidence into CDM for research purposes and professional self-reflection which may then be complemented with more comprehensive and lengthier measures. Intervention strategies can then be developed to target the specific areas requiring support.

## Conclusion

The three consecutive phases described in this paper illustrate a rigorous approach to developing a brief multidimensional index of propensity to integrate research evidence into CDM in rehabilitation that is coherent, clear, and relevant to Canadian OTs and PTs. A focus group and cognitive interviews gave rise to important item modifications in English and French to minimize ambiguity, measurement bias and cognitive burden on respondents. Finally, response option labels in English and French were found to be equivalent through a cross-sectional online survey wherein response option labels were compared on 0 to 100 scales in both languages. It is hoped that the use of this practical index will help identify research and practice needs, better support clinicians and improve the quality of rehabilitation care.

## Data Availability

All data produced or analyzed in the present study are available upon reasonable request to the authors

## Declaration of interest statement

The authors declare no conflict of interest.

## Acknowledgments

This study was funded by the Canadian Institutes of Health Research (grant number 148544). The funding body did not influence the study results and was not involved in the study in any way.

## Appendices

## Appendix I. Five PIRE-CDMI response sets

View this table:
[Table3](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/T3)

## Appendix II. Step-by-step changes at each step of the item rewriting process in both languages for each item English version of the index (please contact the author for French version of the index)

View this table:
[Table4](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/T4)

## Appendix III. Overview of the item evolution process during cognitive interviews

View this table:
[Table5](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/T5)

## Appendix IV. Final index in English (please contact the author for the Final index in French)

**Propensity to Integrate Research Evidence into Clinical Decision-Making Index (PIRE-CDMI)**

View this table:
[Table6](http://medrxiv.org/content/early/2024/08/08/2024.08.07.24310514/T6)

*   Received August 7, 2024.
*   Revision received August 7, 2024.
*   Accepted August 8, 2024.


*   © 2024, Posted by Cold Spring Harbor Laboratory

The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission.

## References

1.  1.CAOT. Profile of occupational therapist practice in Canada. Published online 2012. Accessed September 17, 2020. [https://www.caot.ca/document/3653/2012otprofile.pdf](https://www.caot.ca/document/3653/2012otprofile.pdf)
    
    

2.  2.Dawes M, Summerskill W, Glasziou P, et al. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5(1):1–7. doi:10.1186/1472-6920-5-1
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1472-6920-5-1&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15634359&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

3.  3.Emparanza JI, Cabello JB, Burls AJE. Does evidence-based practice improve patient outcomes? An analysis of a natural experiment in a Spanish hospital. J Eval Clin Pract. 2015;21(6):1059–1065. doi:10.1111/jep.12460
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/jep.12460&link_type=DOI) 

4.  4.Group NPA. Competency profile for physiotherapists in Canada. Natl Physiother Advis Group. Published online 2017. [https://www.peac-aepc.ca/pdfs/Resources/Competency%20Profiles/Competency%20Profile%20for%20PTs%202017%20EN.pdf](https://www.peac-aepc.ca/pdfs/Resources/Competency%20Profiles/Competency%20Profile%20for%20PTs%202017%20EN.pdf)
    
    

5.  5.Haynes RB, Sackett DL, Gray JMA, Cook DJ, Guyatt GH. Transferring evidence from research into practice: 1. The role of clinical care research evidence in clinical decisions. Evid Based Med. 1996;1(7):196.
    
    

6.  6.Haynes RB, Sackett DL, Richardson WS, Rosenberg W, Langley GR. Evidence-based medicine: How to practice & teach EBM. Can Med Assoc J. 1997;157(6):788.
    
    [FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6MzoiUERGIjtzOjExOiJqb3VybmFsQ29kZSI7czo0OiJjbWFqIjtzOjU6InJlc2lkIjtzOjk6IjE1Ny82Lzc4OCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDI0LzA4LzA4LzIwMjQuMDguMDcuMjQzMTA1MTQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

7.  7.Sackett DL, Rosenberg WM, Gray JM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. Br Med J. Published online 1996:71–72.
    
    

8.  8.Sackett DL, Wennberg JE. Choosing the Best Research Design for Each Question: It’s Time to Stop Squabbling over the “Best” Methods. British Medical Journal Publishing Group; 1997.
    
    

9.  9.DiCenso A, Ciliska D, Guyatt G. Introduction to evidence-based nursing. Evid-Based Nurs Guide Clin Pract. Published online 2005:3–19.
    
    

10. 10.Gutenbrunner C, Nugraha B. Decision-Making in Evidence-Based Practice in Rehabilitation Medicine: Proposing a Fourth Factor. Am J Phys Med Rehabil. 2020;99(5):436–440. doi:10.1097/PHM.0000000000001394
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/PHM.0000000000001394&link_type=DOI) 

11. 11.Kitson A, Harvey G, McCormack B. Enabling the implementation of evidence based practice: a conceptual framework. BMJ Qual Saf. 1998;7(3):149–158.
    
    [Abstract](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjcvMy8xNDkiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyNC8wOC8wOC8yMDI0LjA4LjA3LjI0MzEwNTE0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. 12.Satterfield JM, Spring B, Brownson RC, et al. Toward a Transdisciplinary Model of Evidence-Based Practice. Milbank Q. 2009;87(2):368–390. doi:10.1111/j.1468-0009.2009.00561.x
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1468-0009.2009.00561.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19523122&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000266711000003&link_type=ISI) 

13. 13.Craik J, Rappolt S. Theory of Research Utilization Enhancement: A Model for Occupational Therapy. Can J Occup Ther. 2003;70(5):266–275. doi:10.1177/000841740307000503
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/000841740307000503&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=14753069&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

14. 14.Craik J, Rappolt S. Enhancing research utilization capacity through multifaceted professional development. Am J Occup Ther. 2006;60(2):155–164.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.5014/ajot.60.2.155&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16596919&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000236083600005&link_type=ISI) 

15. 15.Robertson L, Graham F, Anderson J. What Actually Informs Practice: Occupational Therapists’ Views of Evidence. Br J Occup Ther. 2014;76:317–324. doi:10.4276/030802213X13729279114979
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4276/030802213X13729279114979&link_type=DOI) 

16. 16.Kothari AR, Bickford JJ, Edwards N, Dobbins MJ, Meyer M. Uncovering Tacit Knowledge: A Pilot Study to Broaden the Concept of Knowledge in Knowledge Translation. BMC Health Serv Res. 2011;11(1):1–10. doi:10.1186/1472-6963-11-198
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1472-6963-11-1&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21199575&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

17. 17.Thornton T. Tacit knowledge as the unifying factor in evidence based medicine and clinical judgement. Philos Ethics Humanit Med. 2006;1(1):2. doi:10.1186/1747-5341-1-2
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1747-5341-1-2&link_type=DOI) 

18. 18.Kashaf MS, McGill E. Does shared decision making in cancer treatment improve quality of life? A systematic literature review. Med Decis Making. 2015;35(8):1037–1048.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0272989X15598529&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26246515&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

19. 19.Shay LA, Lafata JE. Where Is the Evidence? A Systematic Review of Shared Decision Making and Patient Outcomes. Med Decis Making. 2015;35(1):114–131. doi:10.1177/0272989X14551638
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0272989X14551638&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25351843&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

20. 20.Bernhardsson S, Johansson K, Nilsen P, Öberg B, Larsson MEH. Determinants of Guideline Use in Primary Care Physical Therapy: A Cross-Sectional Survey of Attitudes, Knowledge, and Behavior. Phys Ther. 2014;94(3):343–354. doi:10.2522/ptj.20130147
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToicHRqb3VybmFsIjtzOjU6InJlc2lkIjtzOjg6Ijk0LzMvMzQzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDgvMDgvMjAyNC4wOC4wNy4yNDMxMDUxNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

21. 21.da Silva T, da Cunha Menezes Costa L, Garcia AN, Costa LOP. What do physical therapists think about evidence-based practice? A systematic review. Man Ther. 2015;20(3):388–401. doi:10.1016/j.math.2014.10.009
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.math.2014.10.009&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25458142&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

22. 22.Graham F, Robertson L, Anderson J. N ew Z ealand occupational therapists’ views on evidence-based practice: A replicated survey of attitudes, confidence and behaviours. Aust Occup Ther J. 2013;60(2):120–128.
    
    

23. 23.Jeffery H, Robertson L, Reay K. Sources of evidence for professional decision-making in novice occupational therapy practitioners: clinicians’ perspectives. Br J Occup Ther. 2020;84:030802262094139. doi:10.1177/0308022620941390
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0308022620941390&link_type=DOI) 

24. 24.Rochette A, Brousseau M, Vachon B, Engels C, Amari F, Thomas A. What occupational therapists’ say about their competencies’ enactment, maintenance and development in practice? A two-phase mixed methods study. BMC Med Educ. 2020;20. doi:10.1186/s12909-020-02087-4
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12909-020-02087-4&link_type=DOI) 

25. 25.Upton D, Stephens D, Williams B, Scurlock-Evans L. Occupational Therapists’ Attitudes, Knowledge, and Implementation of Evidence-Based Practice: A Systematic Review of Published Research. Br J Occup Ther. 2014;77:24. doi:10.4276/030802214X13887685335544
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4276/030802214X13887685335544&link_type=DOI) 

26. 26.Juckett LA, Wengerd LR, Faieta J, Griffin CE. Evidence-Based Practice Implementation in Stroke Rehabilitation: A Scoping Review of Barriers and Facilitators. Am J Occup Ther Off Publ Am Occup Ther Assoc. 2020;74(1):7401205050p1–7401205050p14. doi:10.5014/ajot.2020.035485
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.5014/ajot.2020.035485&link_type=DOI) 

27. 27.Paci M, Faedda G, Ugolini A, Pellicciari L. Barriers to Evidence-Based Practice Implementation in Physiotherapy: A Systematic Review and Meta-Analysis. Int J Qual Health Care. 2021;(mzab093). doi:10.1093/intqhc/mzab093
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/intqhc/mzab093&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34110410&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

28. 28.Fernández-Domínguez JC, Sesé-Abad A, Morales-Asencio JM, Oliva-Pascual-Vaca A, Salinas-Bueno I, de Pedro-Gómez JE. Validity and reliability of instruments aimed at measuring Evidence-Based Practice in Physical Therapy: a systematic review of the literature: Evidence-based physiotherapy instruments. J Eval Clin Pract. 2014;20(6):767–778. doi:10.1111/jep.12180
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/jep.12180&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24854712&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

29. 29.Martinez RG, Lewis CC, Weiner BJ. Instrumentation issues in implementation science. Implement Sci. 2014;9(1):118. doi:10.1186/s13012-014-0118-8
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13012-014-0118-8&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25185799&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

30. 30.Al Zoubi F, Mayo N, Rochette A, Thomas A. Applying modern measurement approaches to constructs relevant to evidence-based practice among Canadian physical and occupational therapists. Implement Sci. 2018;13(1):152. doi:10.1186/s13012-018-0844-4
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13012-018-0844-4&link_type=DOI) 

31. 31.Buchanan H, Siegfried N, Jelsma J. Survey Instruments for Knowledge, Skills, Attitudes and Behaviour Related to Evidence-based Practice in Occupational Therapy: A Systematic Review: Evidence-based Practice Survey Instruments. Occup Ther Int. 2016;23(2):59–90. doi:10.1002/oti.1398
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/oti.1398&link_type=DOI) 

32. 32.Glegg SMN, Holsti L. Measures of knowledge and skills for evidence-based practice: A systematic review. Can J Occup Ther Ott. 2010;77(4):219–232.
    
    

33. 33.Roberge-Dao J, Maggio LA, Zaccagnini M, et al. Quality, methods, and recommendations of systematic reviews on measures of evidence-based practice: an umbrella review. JBI Evid Synth. 2022;20(4):1004–1073.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.11124/JBIES-21-00118&link_type=DOI) 

34. 34.McGorry SY. Measurement in a cross-cultural environment: survey translation issues. Qual Mark Res Int J. Published online 2000.
    
    

35. 35.Haynes SN, Richard D, Kubany ES. Content validity in psychological assessment: A functional approach to concepts and methods. Psychol Assess. 1995;7(3):238.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037//1040-3590.7.3.238&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1995TP48600003&link_type=ISI) 

36. 36.Streiner DL, Norman GR, Cairney J. Health Measurement Scales: A Practical Guide to Their Development and Use. Oxford University Press, USA; 2015.
    
    

37. 37.Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417–1432.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/0895-4356(93)90142-N&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=8263569&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1993MP96000010&link_type=ISI) 

38. 38.American Educational Research Association., American Psychological Association., National Council on Measurement in Education., Joint Committee on Standards for Educational and Psychological Testing (U.S.). Standards for Educational and Psychological Testing.; 2014.
    
    

39. 39.Willis GB. Analysis of the Cognitive Interview in Questionnaire Design. Oxford University Press; 2015.
    
    

40. 40.Beatty PC, Willis GB. Research synthesis: The practice of cognitive interviewing. Public Opin Q. 2007;71(2):287–311.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/poq/nfm006&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000250350500006&link_type=ISI) 

41. 41.Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Front Public Health. 2018;6. doi:10.3389/fpubh.2018.00149
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3389/fpubh.2018.00149&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29942800&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

42. 42.Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003;12(3):229–238.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1023/A:1023254226592&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12769135&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000182069400003&link_type=ISI) 

43. 43.Drennan J. Cognitive interviewing: verbal data in the design and pretesting of questionnaires. J Adv Nurs. 2003;42(1):57–63.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1046/j.1365-2648.2003.02579.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12641812&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000181653100015&link_type=ISI) 

44. 44.Kuspinar A, Bouchard V, Moriello C, Mayo NE. Development of a Bilingual MS-Specific Health Classification System. Int J MS Care. 2016;18(2):63–70. doi:10.7224/1537-2073.2014-106
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7224/1537-2073.2014-106&link_type=DOI) 

45. 45.Clark LA, Watson D. Constructing validity: Basic issues in objective scale development. Published online 2016.
    
    

46. 46.Emmel N. Sampling and Choosing Cases in Qualitative Research: A Realist Approach. SAGE; 2013.
    
    

47. 47.Keller SD, Ware JE, Gandek B, et al. Testing the Equivalence of Translations of Widely Used Response Choice Labels: Results from the IQOLA Project. Published online 1998:12.
    
    

48. 48.Luo N, Li M, Chevalier J, Lloyd A, Herdman M. A comparison of the scaling properties of the English, Spanish, French, and Chinese EQ-5D descriptive systems. Qual Life Res. 2013;22(8):2237–2243. doi:10.1007/s11136-012-0342-0
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11136-012-0342-0&link_type=DOI) 

49. 49.Sheatsley PB. Questionnaire construction and item writing. Handb Surv Res. 1983;4(1):195–230.
    
    

50. 50.Hallé MC, Mylopoulos M, Rochette A, et al. Attributes of evidence-based occupational therapists in stroke rehabilitation. Can J Occup Ther. 2018;85(5):351–364. doi:10.1177/0008417418802600
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0008417418802600&link_type=DOI) 

51. 51.Fisher RJ. Social desirability bias and the validity of indirect questioning. J Consum Res. 1993;20(2):303–315.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1086/209351&link_type=DOI) 

52. 52.King MF, Bruner GC. Social desirability bias: A neglected aspect of validity testing. Psychol Mark. 2000;17(2):79–103.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/(SICI)1520-6793(200002)17:2<79::AID-MAR2>3.0.CO;2-0&link_type=DOI) 

53. 53.Buetow S, Kenealy T. Evidence-based medicine: the need for a new definition: Examining the meaning of EBM. J Eval Clin Pract. 2000;6(2):85–92. doi:10.1046/j.1365-2753.2000.00237.x
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1046/j.1365-2753.2000.00237.x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=10970002&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000088579100002&link_type=ISI) 

54. 54.Kinsella EA, Whiteford GE. Knowledge generation and utilisation in occupational therapy: Towards epistemic reflexivity. Aust Occup Ther J. 2009;56(4):249–258.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20854525&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

55. 55.Presseau J, McCleary N, Lorencatto F, Patey AM, Grimshaw JM, Francis JJ. Action, actor, context, target, time (AACTT): a framework for specifying behaviour. Implement Sci. 2019;14(1):102. doi:10.1186/s13012-019-0951-x
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13012-019-0951-x&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

56. 56.Potthoff S, Rasul O, Sniehotta FF, et al. The relationship between habit and healthcare professional behaviour in clinical practice: a systematic review and meta-analysis. Health Psychol Rev. 2019;13(1):73–90. doi:10.1080/17437199.2018.1547119
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/17437199.2018.1547119&link_type=DOI) 

57. 57.Reyna VF. A theory of medical decision making and health: fuzzy trace theory. Med Decis Making. 2008;28(6):850–865.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/0272989X08327066&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19015287&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000261037800008&link_type=ISI) 

58. 58.Bannigan K, Moores A. A model of professional thinking: Integrating reflective practice and evidence based practice. Can J Occup Ther. 2009;76(5):342–350.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/000841740907600505&link_type=DOI) 

59. 59.Paterson M, Higgs J, Wilcox S. Developing expertise in judgement artistry in occupational therapy practice. Br J Occup Ther. 2006;69(3):115–123.
    
    

60. 60.Paterson M, Wilcox S, Higgs J. Exploring dimensions of artistry in reflective practice. Reflective Pract. 2006;7(4):455–468.
    
    

61. 61.Thomas A, Chin-Yee B, Mercuri M. Thirty years of teaching evidence-based medicine: have we been getting it all wrong? Adv Health Sci Educ. 2022;27(1):263–276.
    
    

62. 62.Tilson JK, Kaplan SL, Harris JL, et al. Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ. 2011;11(1):1–10. doi:10.1186/1472-6920-11-78
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1472-6920-11-1&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21219639&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

63. 63.Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Ment Health Serv Res. 2004;6(2):61–74.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1023/B:MHSR.0000024351.12294.65&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15224451&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

64. 64.Ajzen I. From intentions to actions: A theory of planned behavior. In: Action Control. Springer; 1985:11–39.
    
    

65. 65.Heiwe S, Kajermo KN, Tyni-Lenné R, et al. Evidence-based practice: attitudes, knowledge and behaviour among allied health care professionals. Int J Qual Health Care. 2011;23(2):198–209. doi:10.1093/intqhc/mzq083
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/intqhc/mzq083&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=21242158&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000288250600013&link_type=ISI) 

66. 66.Melas CD, Zampetakis LA, Dimopoulou A, Moustakis V. Evaluating the properties of the Evidence-Based Practice Attitude Scale (EBPAS) in health care. Psychol Assess. 2012;24(4):867.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/a0027445&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22409449&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

67. 67.Rye M, Torres EM, Friborg O, Skre I, Aarons GA. The Evidence-based Practice Attitude Scale-36 (EBPAS-36): a brief and pragmatic measure of attitudes to evidence-based practice validated in US and Norwegian samples. Implement Sci. 2017;12(1):44.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

68. 68.Arumugam V, MacDermid JC, Walton D, Grewal R. Attitudes, knowledge and behaviors related to evidence-based practice in health professionals involved in pain management. JBI Evid Implement. 2018;16(2):107–118. doi:10.1097/XEB.0000000000000131
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/XEB.0000000000000131&link_type=DOI) 

69. 69.Hitch D, Nicola-Richmond K, Richards K, Stefaniak R. Student perspectives on factors that influence the implementation of evidence-based practice in occupational therapy. JBI Evid Implement. 2021;19(4):409–418. doi:10.1097/XEB.0000000000000285
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/XEB.0000000000000285&link_type=DOI) 

70. 70.Iles R, Davidson M. Evidence based practice: a survey of physiotherapists’ current practice. Physiother Res Int. 2006;11(2):93–103.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/pri.328&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16808090&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

71. 71.Scurlock-Evans L, Upton P, Upton D. Evidence-Based Practice in physiotherapy: a systematic review of barriers, enablers and interventions. Physiotherapy. 2014;100(3):208–219. doi:10.1016/j.physio.2014.03.001
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.physio.2014.03.001&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24780633&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

72. 72.Dijkers MP, Murphy SL, Krellman J. Evidence-Based Practice for Rehabilitation Professionals: Concepts and Controversies. Arch Phys Med Rehabil. 2012;93(8):S164–S176. doi:10.1016/j.apmr.2011.12.014
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.apmr.2011.12.014&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22683207&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

73. 73.Thomas A, Zoubi FA, Mayo NE, et al. Individual and organizational factors associated with evidence-based practice among physical and occupational therapy recent graduates: A cross-sectional national study. J Eval Clin Pract. 2020;n/a(n/a). doi:10.1111/jep.13518
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/jep.13518&link_type=DOI) 

74. 74.Fishman J, Yang C, Mandell D. Attitude theory and measurement in implementation science: a secondary review of empirical studies and opportunities for advancement. Implement Sci. 2021;16(1):87. doi:10.1186/s13012-021-01153-9
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s13012-021-01153-9&link_type=DOI) 

75. 75.Eccles J. Expectancies, values and academic behaviors. Achiev Achiev Motiv. Published online 1983.
    
    

76. 76.Contreras-Huerta LS, Pisauro MA, Apps MAJ. Effort shapes social cognition and behaviour: A neuro-cognitive framework. Neurosci Biobehav Rev. 2020;118:426–439. doi:10.1016/j.neubiorev.2020.08.003
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.neubiorev.2020.08.003&link_type=DOI) 

77. 77.Croxson PL, Walton ME, O’Reilly JX, Behrens TE, Rushworth MF. Effort-based cost– benefit valuation and the human brain. J Neurosci. 2009;29(14):4531–4541.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Njoiam5ldXJvIjtzOjU6InJlc2lkIjtzOjEwOiIyOS8xNC80NTMxIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDgvMDgvMjAyNC4wOC4wNy4yNDMxMDUxNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

78. 78.Kurzban R, Duckworth A, Kable JW, Myers J. An opportunity cost model of subjective effort and task performance. Behav Brain Sci. 2013;36(6):661–679.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1017/S0140525X12003196&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24304775&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

79. 79.Inzlicht M, Shenhav A, Olivola CY. The Effort Paradox: Effort Is Both Costly and Valued. Trends Cogn Sci. 2018;22(4):337–349. doi:10.1016/j.tics.2018.01.007
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.tics.2018.01.007&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29477776&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

80. 80.Condon C, McGrane N, Mockler D, Stokes E. Ability of physiotherapists to undertake evidence-based practice steps: a scoping review. Physiotherapy. 2016;102(1):10–19.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

81. 81.Myers CT, Lotz J. Practitioner Training for Use of Evidence-Based Practice in Occupational Therapy. Occup Ther Health Care. 2017;31(3):214–237. doi:10.1080/07380577.2017.1333183
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/07380577.2017.1333183&link_type=DOI) 

82. 82.Iqbal MZ, Rochette A, Mayo NE, et al. Exploring how evidence-based practice of occupational and physical therapists evolves: A longitudinal mixed methods national study. Submitted to Plos One on July 08, 2022 (PONE-D-22-19314).
    
    

83. 83.Gabbay J, May A le. Evidence based guidelines or collectively constructed “mindlines?” Ethnographic study of knowledge management in primary care. BMJ. 2004;329(7473):1013. doi:10.1136/bmj.329.7473.1013
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjkvNzQ3My8xMDEzIjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDgvMDgvMjAyNC4wOC4wNy4yNDMxMDUxNC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

84. 84.Salter KL, Kothari A. Knowledge ‘Translation’as social learning: negotiating the uptake of research-based knowledge in practice. BMC Med Educ. 2016;16(1):1–10.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12909-015-0506-z&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26727954&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

85. 98.Scott NW, Etta JA, Aaronson NK, et al. An evaluation of the response category translations of the EORTC QLQ-C30 questionnaire. Qual Life Res. 2013;22(6):1483–1490. doi:10.1007/s11136-012-0276-6
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s11136-012-0276-6&link_type=DOI) 

86. 99.Devlin NJ, Hansen P, Selai C. Understanding health state valuations: a qualitative analysis of respondents’ comments. Qual Life Res. 2004;13(7):1265–1277.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1023/B:QURE.0000037495.00959.9b&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=15473505&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000223205900010&link_type=ISI) 

87. 100.Morgado FFR, Meireles JFF, Neves CM, Amaral ACS, Ferreira MEC. Scale development: ten main limitations and recommendations to improve future research practices. Psicol Reflex E Crítica. 2017;30(1):3. doi:10.1186/s41155-016-0057-1
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s41155-016-0057-1&link_type=DOI) 

88. 101.Raykov T. Alpha if item deleted: A note on loss of criterion validity in scale development if maximizing coefficient alpha. Br J Math Stat Psychol. 2008;61(2):275–285.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1348/000711007X188520&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17535482&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F08%2F2024.08.07.24310514.atom) 

89. 102.Rammstedt B, Beierlein C. Can’t we make it any shorter? J Individ Differ. Published online 2014.
    
    

90. 103.Ziegler M, Kemper C, Kruyen P. Short Scales - Five Misunderstandings and Ways to Overcome Them. J Individ Differ. 2014;35:185–189. doi:10.1027/1614-0001/a000148
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1027/1614-0001/a000148&link_type=DOI)