Translation, adaptation, and validation of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) for use in Japan: A multicenter cross-sectional study ============================================================================================================================================================================== * Takuya Aoki * Yosuke Yamamoto * Tomoaki Nakata ## Abstract **Objectives** The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) is a well-established and internationally recognized scale for measuring patients’ experience with hospital inpatient care. This study aimed to develop a Japanese version of the HCAHPS and to examine its structural validity, criterion-related validity, and internal consistency reliability. **Design** Multicenter cross-sectional study. **Setting** A total of 48 hospitals in Japan. **Participants** Patients aged ≥ 16 years who were discharged from the participating hospitals. **Results** We translated the HCAHPS into Japanese according to the guidelines. Psychometric properties were examined using data from 6,522 patients. A confirmatory factor analysis showed excellent goodness of fit of the same factor structure as that of the original HCAHPS, with the following composites: communication with nurses, communication with doctors, responsiveness of hospital staff, hospital environment, communication about medicines, and discharge information. All hospital-level Pearson correlation coefficients between the Japanese HCAHPS composites and overall hospital rating exceeded the criteria. Results of inter-item correlations indicated adequate internal consistency reliability. **Conclusions** We developed the Japanese HCAHPS, and evaluated its structural validity, criterion-related validity, and internal consistency reliability. This scale could be used for quality improvement based on the assessment of patients’ experience with hospital care and for health services research in Japan. **Strengths and limitations of this study** * The Japanese HCAHPS is the first validated scale measuring patients’ experience with hospital inpatient care in Japan. * Our data were collected from a large number of hospitals that were distributed widely throughout Japan, and covered various hospital sizes and regions. * Although we examined the structural validity, criterion-related validity, and internal consistency reliability of the Japanese version developed in this study, other psychometric properties, including convergent and discriminant validity, test-retest reliability, and interpretability, have not been assessed. Keywords * HCAHPS * hospital quality * patient-centered care * patient experience * translation * validation ## INTRODUCTION In recent years, better patients’ perceptions of quality of health care have been deemed as one of the crucial goals of health care. Thus, patient experience has been globally considered as an important quality indicator in a wide range of settings.[1, 2] Patient experience is integrally tied to the principles and practices of patient- and family-centered care. Embedded within patient experience is a focus on individualized care, and tailoring services to meet patients’ needs and engage them as partners in their care.[3] Patient experience has recently replaced patient satisfaction because there are some limitations regarding the assessment of patient satisfaction, such as poor discriminability.[4] Several studies have shown that patient experience is consistently positively associated with clinical effectiveness, patient safety, and patient behaviors across a wide range of disease areas, settings, population groups, and outcome measures.[5–8]. The Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) is a well-established and internationally recognized scale for measuring patients’ experience with hospital care.[9] This scale was developed by the Centers for Medicare and Medicaid Services (CMS) in partnership with and funded by the Agency for Healthcare Research and Quality (AHRQ).[10] In the United States, HCAHPS results have been linked to financial reimbursement from Medicare and other insurers for promoting quality improvement in hospitals.[11] Additionally, these results are posted on the website for helping patients’ decision-making process by enabling comparisons across hospitals.[12] In Japan, activities for assessment of patient experience have just begun in limited settings, and systematic approaches for quality improvement based on patients’ perceptions of health care are still inadequate. In recent years, several scales have been developed and validated to assess outpatients’ experience, mainly in the primary care setting.[13–15] However, there are no validated scales for assessing patients’ experience with hospital inpatient care in Japan. Accordingly, the present study aimed to develop a Japanese version of the HCAHPS, and to examine its structural validity, criterion-related validity, and internal consistency reliability. ## METHODS ### Design, setting, and participants This multicenter cross-sectional study was conducted in 48 hospitals from September to December 2019, in cooperation with the Nihon Hospital Alliance (NHA), which is a group purchasing organization in Japan. Since 2014, the NHA has conducted an annual patient experience survey to evaluate and improve patient-centeredness in hospitals in Japan.[16] The participating hospitals voluntarily participated in the present study. Table 1 shows the characteristics of the participating hospitals. These hospitals were distributed widely throughout Japan, covering both urban and rural areas. Majority of the hospitals were large (≥ 400 beds), publicly owned, general hospitals, and they had an intensive care unit (ICU). A self-administered questionnaire was distributed to patients aged ≥ 16 years who were discharged from one of the participating hospitals during the survey period. Patients who were unable to respond to the questionnaire due to severe physical or mental disorders were excluded. We collected completed surveys by mail. View this table: [Table 1.](http://medrxiv.org/content/early/2020/05/18/2020.05.14.20102442/T1) Table 1. Hospital characteristics of 48 participating hospitals ### Measures #### The HCAHPS The original HCAHPS is a 19-item tool comprising six composites, two global ratings, and three screening items.[9] The composites are communication with nurses (Q1–Q3), communication with doctors (Q5–Q7), responsiveness of hospital staff (Q4 and 11), hospital environment (Q8 and 9), communication about medicines (Q13 and 14), and discharge information (Q16 and 17). The global ratings include overall hospital rating (Q18) and willingness to recommend the hospital to friends and family (recommended hospital) (Q19). We obtained permission for translating the HCAHPS into Japanese from the AHRQ and CMS. According to the guidelines for translating CAHPS® surveys provided by the AHRQ,[17] we translated the HCAHPS into Japanese through the following steps. First, two forward translations from English to Japanese were performed independently by two bilingual translators who had prior professional experience in translating survey instruments for health services. Subsequently, the two forward translations were reviewed by a translation reviewer, who was a native speaker of Japanese and had prior experience in translating survey instruments. After reviewing the translations, the reviewer produced a reconciled version of the translation. The final version of the translation was then produced through discussion in a committee composed of the two translators and the reviewer. The reconciled version from the original review was modified as needed, based on the committee’s decision for cross-cultural adaptation. The final wording of each survey item and response option was determined by consensus. The HCAHPS survey uses several different response scales: a dichotomous scale (1 = Yes, 2 = No), a global rating scale (0 = Worst to 10 = Best), and four-point Likert scales (1 = Never, 2 = Sometimes, 3 = Usually, and 4 = Always; and 1 = Definitely no, 2 = Probably no, 3 = Probably yes, and 4 = Definitely yes). To make the results easier to understand, we converted all scales to normalized scores ranging from 0 to 100 using the following formula: Normalized Score = 100 * (Respondent’s selected response value – Minimum response value on the scale) / (Maximum response value – Minimum response value) In the Japanese version, assuming the convergence in each composite as in the original version, the score for each of the six composites was computed as the mean value for all normalized scores in the scale that would fall in the range of 0–100 points, with higher scores indicating better performance. ### Statistical analysis To validate the Japanese HCAHPS, we first conducted a confirmatory factor analysis to evaluate the structural validity of the Japanese HCAHPS composites. In the factor analysis, we hypothesized the same factor structure (six-factor solution) as that of the original HCAHPS. The appropriateness of the resulting structure was determined by examining if factor loadings were 0.40 or greater.[18] The model fitness was assessed by the comparative fit index (CFI), Tucker-Lewis index (TLI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). Guidelines suggest that models with CFI and TLI close to 0.95 or higher, RMSEA close to 0.06 or lower, and SRMR close to 0.08 or lower are representative of models with a good fit.[19] Subsequently, the Japanese HCAHPS composite scores and the overall hospital rating were used to examine criterion-related validity. Validity was assessed using Pearson correlation coefficients with each Japanese HCAHPS composite, to predict the overall hospital rating at the hospital level. A correlation coefficient greater than 0.30 was considered meaningful.[20] Hospital-level correlations are a more important criterion for measurement than are individual-level correlations because the former are benchmarking tools to compare one hospital with another. To examine hospital-level correlations, we used each hospital’s mean score on HCAHPS composites and the overall hospital rating. The internal consistency reliability was examined by inter-item correlations and the Cronbach’s alpha. For a scale to be considered sufficiently reliable, an inter-item correlation of 0.30 and a Cronbach’s alpha value of 0.70 is recommended.[21] Finally, descriptive statistics were performed for the Japanese HCAHPS scores, including the mean, standard deviation, and observed range. To deal with missing data, in the confirmatory factor analysis, we used full information maximum likelihood estimation to enable use of information collected from participants with missing data. In the evaluation of criterion-related validity and internal consistency, we conducted complete case analyses. All statistical analyses were conducted using R version 3.6.3 (R Foundation for Statistical Computing, Vienna, Austria; [www.R-project.org](http://www.R-project.org)). ### Ethics approval This study was approved by the Ethics Committee of Kyoto University Graduate School of Medicine (approval number R2331) and was conducted in accordance with the Declaration of Helsinki. We obtained consent for participation from each participant. ## RESULTS Of the total 15,512 eligible participants, 6,522 (42.0%) responded to the survey. Table 2 shows the participants’ responses to each item of the Japanese HCAHPS. The Top Box score for each item, which is the percentage of participants who provided most positive responses on that item, ranged from 31.1% to 82.5%. Regarding the mean Top Box score for composites, the highest score was observed for discharge information (77.2%) while the lowest score was for hospital environment (49.1%). The bottom box score, which is the percentage of participants with least positive responses on the item, ranged from 0.5% to 23.2%. View this table: [Table 2.](http://medrxiv.org/content/early/2020/05/18/2020.05.14.20102442/T2) Table 2. Response to Japanese HCAHPS items: number (%) (N = 6,522) ### Structural validity Figure 1 shows the path diagrams of the confirmatory factor analysis to assess the structural validity of the Japanese HCAHPS composites. All factor loadings of each items onto each factor were above the 0.40 criteria, ranging from 0.41 to 0.91. The correlation coefficients among factors ranged from 0.30 to 0.85. The conceptual model showed excellent goodness of fit, with CFI = 0.987, TLI = 0.981, RMSEA = 0.031, and SRMR = 0.020. ![Figure 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2020/05/18/2020.05.14.20102442/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2020/05/18/2020.05.14.20102442/F1) Figure 1. Factor structure of Japanese HCAHPS (confirmatory factor analysis) ### Criterion-related validity Table 3 shows the Pearson correlation coefficients between the Japanese HCAHPS composites and the overall hospital rating at the hospital level. All correlations were statistically significant (*P* < 0.01), and they exceeded the 0.30 criterion. The composite “communication with doctors” (*r* = 0.63) had the highest correlation with the overall rating. View this table: [Table 3.](http://medrxiv.org/content/early/2020/05/18/2020.05.14.20102442/T3) Table 3. Pearson correlation coefficients between Japanese HCAHPS composites and overall hospital rating ### Internal consistency reliability and descriptive statistics Table 4 indicates the score distribution and internal consistency reliability for the Japanese HCAHPS. All inter-item correlations were above the 0.30 criteria, ranging from 0.31 to 0.73. For communication with nurses, communication with doctors, and communication about medicines, the Cronbach’s alpha was above 0.70. In contrast, for responsiveness of hospital staff, hospital environment, and discharge information, the Cronbach’s alpha did not exceed the 0.70 criterion. Descriptive statistics showed that the most highly-scored scale was communication with doctors (mean score = 87.9), and the most poorly-scored scale was recommended hospital (mean score = 75.5). The full range of possible scores was observed for all scales. View this table: [Table 4.](http://medrxiv.org/content/early/2020/05/18/2020.05.14.20102442/T4) Table 4. Descriptive features and internal consistency reliability of Japanese HCAHPS (N = 6,522) ## DISCUSSION Measurement of patient experience plays an important role in the improvement of a wide range of medical services, including inpatient care. We translated the HCAHPS, which is a validated international scale, into Japanese and examined its structural validity, criterion-related validity, and internal consistency reliability in 48 hospitals in Japan. This study was the first to develop a Japanese version of the HCAHPS and to examine its psychometric properties for assessing patients’ experience with hospital inpatient care. Standard psychometric evaluation methods were used to evaluate the Japanese HCAHPS. The confirmatory factor analysis supported the scale’s structural validity and the same six-factor solution as that of the original HCAHPS, with good statistical fitness. Correlation coefficients between all Japanese HCAHPS composites and the overall hospital rating for assessing criterion-related validity exceeded the meaningful value at the hospital level. In internal consistency analyses, the Cronbach’s alpha for responsiveness of hospital staff, hospital environment, and discharge information did not exceed the optimum criterion. The Cronbach’s alpha is quite sensitive to the number of items in the scale; therefore, it is common to find low Cronbach’s alpha for scales with few items (especially 2-item scales).[22] Likewise, a study conducted in the United States [23] found that the Cronbach’s alpha was low for some aspects of the original HCAHPS scale. In this case, it is more appropriate to report the inter-item correlation of items. In our study, all inter-item correlations were greater than the criterion, which indicated adequate internal consistency of the scales. To our knowledge, the Japanese HCAHPS is the first validated scale measuring patients’ experience with hospital inpatient care in Japan. The HCAHPS is one of the most widely studied and endorsed patient experience measure of hospital care worldwide. Our data were collected from a large number of hospitals that were distributed widely throughout Japan and they covered various hospital sizes and regions. Therefore, the study results have relatively high external validity. However, there are several potential limitations to our study. First, the response rate was a concern. A previous study of patient experience surveys showed that a low participation rate is less likely to introduce selective nonresponse bias;[24] however, it is possible that patients with worse experience were less likely to respond to our survey. Second, although we examined the structural validity, criterion-related validity, and internal consistency reliability of the Japanese HCAHPS in this study, other psychometric properties, including convergent and discriminant validity, test-retest reliability, and interpretability have not been assessed.[25] These measurement properties of the scale need to be evaluated in future studies. Third, this study was limited by the fact that the participating hospitals voluntarily participated in this study; thus, the preset sample may represent hospitals that have a higher interest in the quality of health care. Accordingly, the participating hospitals may not have sufficiently represented Japanese hospitals at national level. Therefore, the Japanese HCAHPS should be used for research in other settings. ## CONCLUSION We developed the Japanese HCAHPS and evaluated its structural validity, criterion-related validity, and internal consistency reliability. This scale could be used for quality improvement based on the assessment of patients’ experience with hospital care and for health services research in Japan. ## Data Availability Additional unpublished data is still being analyzed for another research and only available to the members of the study team. ## Contributors TA designed the study and participated in the implementation, data analysis and writing of the manuscript. YY contributed to the design of the study and critically reviewed the manuscript. TN contributed to the design of the study, data collection, and critically reviewed the manuscript. All authors gave the final approval of the manuscript before submission. ## Funding None. ## Competing interests None declared. ## Patient consent for publication Not required. ## Data sharing statement Additional unpublished data is still being analyzed for another research and only available to the members of the study team. ## Patient and Public Involvement statement We did not directly include PPI in this study, but the database used in the study was developed with PPI. ## Acknowledgments We would like to thank Toshio Goto and Yoko Endo (Nihon Hospital Alliance) for their assistance with the administration of this study. We are grateful to Prof. Kiyoshi Ando, Prof. Keiichi Saito, and Kaori Soga (Association for Patient eXperience Japan) for collaboration on the early stages of this work. * Received May 14, 2020. * Revision received May 14, 2020. * Accepted May 18, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/) ## Reference 1. 1.Institute of Medicine. Committee on Quality of Health Care in America. Crossing the quality chasm: A new health system for the 21st century. Washington, DC: National Academies Press; 2001. 2. 2.Berwick DM, Nolan TW, Whittington J. The Triple Aim: Care, Health, And Cost. Health Aff. 2008;27(3):759–769. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjg6IjI3LzMvNzU5IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjAvMDUvMTgvMjAyMC4wNS4xNC4yMDEwMjQ0Mi5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 3. 3.Wolf J, Niederhauser V, Marshburn D, et al. Defining patient experience. Patient Exp J. 2014;1(1):7–19. 4. 4.Salisbury C, Wallace M, Montgomery AA. Patients’ experience and satisfaction in primary care: secondary analysis using multilevel modelling. BMJ. 2010;341:c5004. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNDEvb2N0MTJfMS9jNTAwNCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzA1LzE4LzIwMjAuMDUuMTQuMjAxMDI0NDIuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 5. 5.Doyle C, Lennox L, Bell D. A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013;3:e001570. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qb3BlbiI7czo1OiJyZXNpZCI7czoxMToiMy8xL2UwMDE1NzAiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMC8wNS8xOC8yMDIwLjA1LjE0LjIwMTAyNDQyLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 6. 6.Anhang Price R, Elliott MN, Zaslavsky AM, et al. Examining the role of patient experience surveys in measuring health care quality. Med Care Res Rev. 2014;71(5):522–554. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/1077558714541480&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25027409&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) 7. 7.Aoki T, Miyashita J, Yamamoto Y, et al. Patient experience of primary care and advance care planning: A multicentre cross-sectional study in Japan. Fam Pract. 2017;34(2):206–212. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) 8. 8.Aoki T, Yamamoto Y, Ikenoue T, et al. Effect of Patient Experience on Bypassing a Primary Care Gatekeeper: a Multicenter Prospective Cohort Study in Japan. J Gen Intern Med 2018;33(5):722–728. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) 9. 9.Centers for Medicare and Medicaid Services. HCAHPS. [https://www.cms.gov/Research-Statistics-Data-and-Systems/Research/CAHPS/hcahps1.html](https://www.cms.gov/Research-Statistics-Data-and-Systems/Research/CAHPS/hcahps1.html) (accessed 1 April 2020). 10. 10.Goldstein E, Farquhar M, Crofton C, et al. Measuring hospital care from the patients’ perspective: An overview of the CAHPS® Hospital Survey development process. Health Serv Res. 2005;40(6II):1977–1995. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1475-6773.2005.00477.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16316434&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000233289500002&link_type=ISI) 11. 11.Mehta SJ. Patient satisfaction reporting and its implications for patient care. AMA J Ethics. 2015;17(7):616–621. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) 12. 12.Centers for Medicare and Medicaid Services. Official hospital compare data archive. [https://data.medicare.gov/data/archives/hospital-compare](https://data.medicare.gov/data/archives/hospital-compare) (accessed 1 April 2020). 13. 13.Aoki T, Inoue M, Nakayama T. Development and validation of the Japanese version of Primary Care Assessment Tool. Fam Pract. 2015;33(1):112–117. 14. 14.Aoki T, Fukuhara S, Yamamoto Y. Development and validation of a concise scale for assessing patient experience of primary care for adults in Japan. Fam Pract. 2020;37(1):137–142. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) 15. 15.Kijima T, Akai K, Matsushita A, et al. Development of the Japanese version of the general practice assessment questionnaire: Measurement of patient experience and testing of data quality. BMC Fam Pract. 2018;19:181. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) 16. 16.Nihon Hospital Alliance. Other solutions. [https://nha-gpo.or.jp/other-solutions/](https://nha-gpo.or.jp/other-solutions/) (in Japanese) (accessed 1 April 2020). 17. 17.Agency for Healthcare Research and Quality. Translating CAHPS Surveys. [https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/resources/cahpsGuidelines\_Translation.htm](https://www.ahrq.gov/cahps/surveys-guidance/helpful-resources/resources/cahpsGuidelines_Translation.htm) (accessed 1 April 2020). 18. 18.Peterson RA. A Meta-Analysis of Variance Accounted for and Factor Loadings in Exploratory Factor Analysis. Marketing Letters. 2000;11:261–275. 19. 19.Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equation Model. 1999;6(1):1–55. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1080/10705519909540118&link_type=DOI) 20. 20.Cohen, J. Statistical Power Analysis. Current Directions in Psychological Science 1992;1(3):98–101. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/1467-8721.ep10768783&link_type=DOI) 21. 21.Robinson J, Shaver P, Wrightsman L, eds. Measures of Personality and Social Psychological Attitudes. San Diego, Calif: Academic Press; 1991. 22. 22.Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use (5th ed.). Oxford, UK: Oxford University Press; 2015. 23. 23.Keller S, O’Malley AJ, Hays RD, et al. Methods used to streamline the CAHPS® Hospital Survey. Health Serv Res. 2005;40(6 II):2057–2077. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/j.1475-6773.2005.00478.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16316438&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000233289500006&link_type=ISI) 24. 24.Roland M, Elliott M, Lyratzopoulos G, et al. Reliability of patient responses in pay for performance schemes: analysis of national General Practitioner Patient Survey data in England. BMJ. 2009;339: b3851. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzkvc2VwMjlfMy9iMzg1MSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzA1LzE4LzIwMjAuMDUuMTQuMjAxMDI0NDIuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 25. 25.Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63(7):737–45. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2010.02.006&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=20494804&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F05%2F18%2F2020.05.14.20102442.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000278412800009&link_type=ISI)