Judgements about carer assessments for carers of people with dementia: case vignette study ========================================================================================== * Paul Clarkson * Lucie Mills * Asif Butt * Baber Malik * Ruth Eley * Cecilia Toole * Caroline Sanders * Ian Sheriff ## Abstract **Objectives** UK carer assessments, in primary and social care, intend to discover what carers need in their caring roles and more widely. Evidence points to these not being configured sufficiently around carers of people with dementia, with potentially their breadth of needs not being recognised. We evaluated the extent of agreement, between carers of people with dementia, primary care, and social care professionals, on their recommendations from assessing carers’ needs in a range of circumstances. It is intended for findings to be taken forward as recommendations for policy and practice. **Methods** Comparison of judgements, between carers, primary and social care professionals, on whether real-life circumstances in 9 anonymised case vignettes necessitated a range of 14 services to support carers appropriately. Participants were 6 carers of people with dementia, 7 primary care staff, and 2 social care staff. We presented participants with each vignette and asked them to make binary judgements of whether they would recommend a range of services in each case. Percentage agreement and Fleiss’ kappa coefficients measured the level of agreement amongst multiple carers, primary and social care staff and overall. These agreements were then compared. **Results** Carers agreed in their judgements more than primary or social care professionals. The overall level of agreement from judgements made by all participants, however, was ‘slight’ with variability between participant groups and overall. The need for First Language Support in some cases was recognised, an improvement from previous evidence. **Conclusions** Case vignettes are useful for investigating judgements concerning these carers’ needs, so raising issues for policy and practice. It is essential for carer assessments to be more reliable in recommending services based on need to ensure less variability, depending on assessor and carers circumstances. ## Introduction Over 670,000 people (UK) are primary, unpaid carers for people with dementia [1]. Carer assessments are a formal mechanism for identifying what carers need and require in their caring roles and more widely. In social care, these assessments are a legal duty, [2] and in primary care are tied to incentives and are more discretionary, often linked to health checks and signposting by general practices [3]. These assessments are not specific to carers of people with dementia, although their early effects for this group of carers, before current legislation under the 2014 Care Act, were evidenced [4]. Evidence points to these assessments not being configured sufficiently around carers of people with dementia, with potentially their breadth of needs not being recognised. More recent evidence points to these carers often not receiving an assessment, despite this being mandated, with often carers not seeing these assessments as valuable [5]. Although carer assessments are a requirement, their content and the types of needs judged important for those carers of people with dementia are variable and often not recognised. The experiences of two groups of carers of people with dementia are of note. Those carers from south Asian backgrounds report that the multiplicity of informal carer networks and the distinct needs of second-generation carers are not recognised or neglected [6]. Carers of people living with young onset dementia have distinct needs, which also tend not to be recognised, including impact on relationships, financial implications, and a sense of loss [7]. As part of a wider study, we aimed to examine whether the needs and preferences of carers of people with dementia were potentially covered sufficiently in these assessments. We aimed to evaluate the extent of agreement, between carers of people with dementia, primary care, and social care professionals, on their recommendations from assessing carers’ needs in a range of circumstances. We used case vignettes, relatively short accounts describing realistic carer’s situations, as a method to identify how different groups of raters assessed identical scenarios. ## Material and Methods ### Design Nine detailed case vignettes of anonymised real-life cases of carers’ circumstances were designed and used. These vignettes were created to express a range of situations that carers may be in as they interfaced with health and social care systems. The vignettes were given to participants belonging to three groups: carers of people with dementia, primary care staff, and social care staff. Participants were approached, respectively, through national and local carers’ charities, general practice and Primary Care Network, and local authority consortium. All participants gave informed consent to take part. Each participant was told that the vignettes were based on real cases and experiences but were anonymised as to person and place. Vignettes were completed online, through Zoom, email, or (for primary care staff) in person at a workshop meeting. Each participant was asked to read each vignette and make a binary judgement (yes/no) of whether they would recommend a range of services (n=14) in each case, that were warranted to help support the carers’ needs. Participants were also given the option of describing their reasons behind the judgements. The vignettes and responses were returned anonymously to the research team. ### Case vignettes This was a co-produced study with carers and carer representative organisations. The anonymised vignettes were written jointly by carers of people with dementia, through carers organisations and members of the research team, taken from carers’ experiences and circumstances. The vignettes were designed to express a range of different circumstances and possible service responses, but particularly (although not exclusively) focusing on the needs of carers of people with younger onset dementia and those from a south Asian background. The potential needs of carers from these groups, highlighted in vignettes, included financial, employment and relationship need, [8] multigenerational caring, language, and cultural needs [9]. Possible service responses to the vignettes, which participants were asked to indicate if warranted, were included after consultation with carers groups and charities supporting carers of people with dementia. Fig 1 shows an example of an anonymised case vignette with the same standardized format used for all vignettes. ![Fig 1.](http://medrxiv.org/https://www.medrxiv.org/content/medrxiv/early/2024/04/13/2024.04.11.24305691/F1.medium.gif) [Fig 1.](http://medrxiv.org/content/early/2024/04/13/2024.04.11.24305691/F1) Fig 1. Sample vignette. Description of carer’s circumstances and request to signal service responses that would be considered appropriate. Each vignette expressed circumstances, presenting insights from carers’ perspectives to generate participants’ judgements, but without leading too directly. The vignettes had an average length of 180 words (ranging from 91 to 318). In general, each vignette took participants about 5-10 minutes to read and complete. ### Participants All participants were approached in writing with Participant Information Sheets and invited to take part in the study and had to give their written consent to take part. Recruitment took place between 14/12/2022 and 31/12/2023. Carers of people with dementia were interested carers who had received carers assessments before and who wished to express their views about carers’ needs in the context of these assessments. These carers were not involved in the construction of the case vignettes. Primary care staff were professionals working in or with primary care practices who had an interest in dementia or older people’s healthcare. Social care staff were present or former statutory assessors within the context of carer assessments and were social workers of local authorities. It was left to participants to decide which of the nine vignettes they rated and the completion of reasons for their judgments was entirely optional. ### Ethics statement The study was approved by the University of Manchester Research Ethics Committee (Ref: 2022-14569-26301; 14/12/2022) and, additionally, for primary care staff, the Health Research Authority (IRAS Project ID: 326181; 23/HRA/1321). ### Statistical analysis Data analysis was undertaken using SPSS v28 for Windows. Descriptive statistics (percentage agreement) were used to compare judgements of each of the groups, about whether each of the possible 14 service responses was warranted in each case. Fleiss’ kappa coefficients, for multiple raters, [10] measured the level of agreement (inter-rater reliability) amongst carers, primary and social care staff and overall. the Kappa coefficient. Based on standards outlined by Landis and Koch [11], a Kappa coefficient <0 was considered to be poor agreement, 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60, moderate, 0.61–0.80 substantial, and 0.81–1.0 almost perfect agreement. Tests were two-tailed and P < 0.05 was considered statistically significant. ## Results Full data were available from 6 carers, 7 primary care staff and 2 social care staff. Two carers cared for people with younger onset dementia. Primary care staff included general practitioners, an assistant practice manager, nurses, a practice pharmacist, and a paramedic. Social care staff were social workers familiar with statutory carer’s assessments. ### Comparison of judgements by carers and professional staff Table 1 summarises the agreement of participants in each group and overall concerning whether they would recommend each of the 14 services in each case. Percentage agreement and Fleiss’ kappa coefficients are compared. Total agreement (100%) between all groups was seen in only two cases: case vignette 8 for Sitting Service and case vignette 1 for Financial Advice. There was relatively high agreement for case vignette 7, Sitting Service; case vignette 8 for Financial Advice; and case vignette 1, First Language Support, which was a case of a carer and person with dementia from a south Asian background. The remaining cases showed a range of differing judgements about whether each service was recommended. View this table: [Table 1.](http://medrxiv.org/content/early/2024/04/13/2024.04.11.24305691/T1) Table 1. Proportion (%) of each group judging case warrants service and overall level of agreement; Fleiss’ kappa (95% confidence interval); p value, between participants Kappa values for all participant groups showed only ‘slight’ agreement [11] across the case vignettes, but ‘fair’ agreement for First Language Support. Across the groups (Total column in Tables 1 and 2), agreement for Social Support Worker, Care Co-ordinator and Live in Carer was no better than chance. Carers had the highest level of agreement: ‘slight’ (9 services), ‘moderate’ (1 service) and ‘almost perfect’ (1 service; First Language Support). For carers, agreement on three services (Care Co-ordinator, Live in Care, Befriending Scheme) was no better than chance. Primary care staff had ‘slight’ agreement for 8 services, ‘fair’ agreement for 1 service, with agreement no better than chance for the remaining 5 services. Social care staff had the lowest level of agreement, ‘slight’ (2 services) fair’ (2 services), ‘moderate (1 service), and no better than chance for the remaining 8 services. However, social care staff had ‘almost perfect’ agreement on the need for First Language Support. ### Reasons behind the judgements Examination of the stated reasons behind the judgements made by participants in the vignettes helped to discern some of the wider factors influencing decision-making in these assessments. These tended to vary according to the groups of participants undertaking the exercise. For carers, judgements were made based on perceived risk, such that if there was a notion that needs were increasing to an unsustainable level then more and different services would be recommended. Carers often recommended services with the aim of building confidence in the carer’s ability to continue caring. Carers were also concerned about people’s entitlements and ensuring that these were supported through assessments but also that this depended on the local supply of services where the carer lived. Primary care staff, perhaps understandably, stressed the need for a formal dementia diagnosis for the carer’s relative, so that they could be keyed into the support services available. The thought of ‘carer burnout’ was also evident in the reasoning of primary care staff and that the carers needed to know all the options and support available to them to perhaps prevent that. Social care staff again stressed a consideration with carers’ entitlements and ensuring that they were eligible for certain services and financial help. Assessment was not seen as a one-off process by social care staff, and there was a notion of establishing a continuity to decisions and referring the carer on further for help from other professionals. ## Discussion The nature and process of carer assessments are no different than for assessments in general. Previous work on social care assessments has shown wide variability, across professionals, in the identification of older people’s needs [12,13]. For carer assessments of those caring for someone with dementia, this variability continues to be an issue, with many carers dissatisfied that their needs had not been identified or met [14,15]. In this case vignette study, we found wide variability in the reliability to which carers’ needs and suitable service responses were identified that concur with some of these judgements. Total agreement on needs occurred in only two vignettes for two service responses and there was mostly only slight agreement across the case vignettes. Carers tended to agree the most, with primary care staff second and social care staff agreeing the least. There was relatively high agreement in terms of the need for First Language Support, amongst carers and social care staff (showing ‘almost perfect’ agreement), particularly for case vignette 1 which was a case of a south Asian carer. This finding is encouraging considering previous evidence of this type of support being undervalued in carer assessments and support [16]. On the other hand, there was relatively low agreement by professionals on the need for Befriending Support, for the vignettes focusing on south Asian carers. This type of support has been highly valued by south Asian carers [16] and is seen as providing a different focus than more traditional services such as respite and day care. ### Strengths and Limitations of the Study Our study has limitations. We examined the judgements of a relatively small number of participants, [17]particularly social care staff. This limits the statistical power by which the indices of agreement were produced. Smaller samples also raise the potential of bias, in that it may be likely for raters who tended to choose particular categories (service responses) as present or absent to influence the results. This can produce situations where there is high percentage agreement amongst raters but a low kappa [18]. This was the case for our study, where, for example, for Financial Advice and Dementia Specialist Coordinator, rated by social care staff, there was high percentage agreement but with very low kappa scores, resultant agreement being no better than chance. However, this notwithstanding, our study has some strengths. We used real-life, detailed vignette descriptions of carers’ circumstances and these were able to prompt responses that mimic the decision-making processes of assessments. This makes it possible to draw conclusions that have ecological validity [19] and can be more realistically generalised to inform practice and policy. There is no ‘gold standard’ by which to judge the reliability of assessments, particularly for assessments that have a legislative basis, such as these carer assessments in the UK. The assessment process is complex and variable and is influenced by many factors, both internal and external to the assessment ‘encounter’. In addition, there is very little prior research investigation of how carer assessments operate in primary care in the UK, primarily because they do not have a legislative basis and are informed only by guidance, usually as part of UK health checks. Thus, this study adds to the evidence base in this area by considering the judgements of social care and primary care staff against those of carers, the subject of these assessments. Carers, arguably, are the best judge of their own needs for support and it is noteworthy that carers here agreed the most across the service responses and vignettes. Our study design recognised this in terms of co-producing the vignettes with carers to stimulate participant’s judgement decisions. ## Conclusion Case vignettes are a useful tool for investigating different professionals’ judgements concerning the needs of carers of people with dementia and so raising issues for policy and practice. This study showed these assessments can still be highly variable in terms of identifying the kinds of needs and service responses that might be required to support carers in their caring role. The need for First Language Support in some cases was recognised, an improvement from previous evidence. However, efforts to improve the reliability with which carer assessments are undertaken, so that prominent needs are identified and acted on in a consistent way, are still required. ## Data Availability The datasets generated and analysed in this study and the full anonymized case vignettes are publicly available via the Figshare data repository, at [https://doi.org/10.48420/25483270.v1](https://doi.org/10.48420/25483270.v1) Participants consented to make anonymized data publicly available. [https://doi.org/10.48420/25483270.v1](https://doi.org/10.48420/25483270.v1) ## Acknowledgements We thank the carers, primary care, and social care staff for their time in taking part in the study. Together in Dementia Everyday (TIDE), Sight and Mind CIC, Dementia United (Greater Manchester), Kirklees Council, Bury Primary Care Network, and the Greater Manchester Social Work Academy assisted in the recruitment of participants. ## Footnotes * [Dementia carer assessments case vignette study] * Received April 11, 2024. * Revision received April 11, 2024. * Accepted April 13, 2024. * © 2024, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. 1.Alzheimer’s Society. Alzheimer’s Society’s view on carer support. 2014; Available from: [https://www.alzheimers.org.uk/about-us/policy-and-influencing/what-we-think/carer-support](https://www.alzheimers.org.uk/about-us/policy-and-influencing/what-we-think/carer-support) 2. 2.Seddon D, Robinson C. Carer assessment: continuing tensions and dilemmas for social care practice. Health Soc Care Community. 2015; 23(1): 14–22. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/hsc.12115&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25243745&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F13%2F2024.04.11.24305691.atom) 3. 3.BMA/NHS England. Quality and Outcomes Framework guidance for 2021/22. 2021. Available from: [https://www.england.nhs.uk/wp-content/uploads/2021/03/B0456-update-on-quality-outcomes-framework-changes-for-21-22-.pdf](https://www.england.nhs.uk/wp-content/uploads/2021/03/B0456-update-on-quality-outcomes-framework-changes-for-21-22-.pdf) 4. 4.Seddon D, Robinson C. (2001) Carers of older people with dementia: assessment and the Carers Act. Health Soc Care Community. 2021; 9(3): 151–158. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1046/j.1365-2524.2001.00291.x&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11560731&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F13%2F2024.04.11.24305691.atom) 5. 5.Newbronner L, Chamberlain R, Borthwick R, Baxter M, Glendinning C. A Road Less Rocky – Supporting Carers of People with Dementia. London, Carer’s Trust. 2013 Available at: [https://carers.org/resources/all-resources/84-a-road-less-rocky-a-supporting-carers-of-people-with-dementia](https://carers.org/resources/all-resources/84-a-road-less-rocky-a-supporting-carers-of-people-with-dementia) 6. 6.Giebel CM, Zubair M, Jolley D, Bhui KS, Purandare N, Worden A, et al. South Asian older adults with memory impairment: improving assessment and access to dementia care. Int J Geriatr Psychiatry. 2015; 30(4): 345–56. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/gps.4242&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=25503751&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F13%2F2024.04.11.24305691.atom) 7. 7.Together In Dementia Everyday (TIDE). Young Onset Dementia Survey Findings. 2020. Available from: [https://www.tide.uk.net/young-onset-dementia/](https://www.tide.uk.net/young-onset-dementia/) 8. 8.Stamou V, La Fontaine J, Gage H, Jones B, Williams P, O’Malley M, et al. Services for people with young onset dementia: The ‘Angela’ project national UK survey of service use and satisfaction. Int J Geriatr Psychiatry. 2021; 36: 411–422. 9. 9.Baghirathan S, Cheston R, Hui R, Chacon A, Shears P, Currie K. A grounded theory analysis of the experiences of carers for people living with dementia from three BAME communities: Balancing the need for support against fears of being diminished. Dementia. 2020; 19(5); 1672–1691. 10. 10.Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971; 76: 378–82. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1037/h0031619&link_type=DOI) 11. 11.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33: 159–74. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/2529310&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=843571&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F04%2F13%2F2024.04.11.24305691.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1977CY39700012&link_type=ISI) 12. 12.Clarkson P. What research tells social workers about their work with older people. In: Davies M, editor. Social Work with Adults. Basingstoke, Hampshire: Palgrave Macmillan; 2012. pp. 300–313. 13. 13.Ellis K. Squaring the circle: user and carer participation in needs assessment. York: Joseph Rowntree Foundation; 1993. 14. 14.Carers UK. (2022) State of Caring 2022 report. London: carers UK. 2022. Available from: [https://www.carersuk.org/media/p4kblx5n/cukstateofcaring2022report.pdf](https://www.carersuk.org/media/p4kblx5n/cukstateofcaring2022report.pdf) 15. 15.Stock C, Lambert S. (2011) Who cares wins? Carers’ experiences of assessment since the introduction of the Carers (Equal Opportunities) Act 2004. Research Policy and Planning. 2011; 28: 173–184. 16. 16.Hepworth D. (2005) Asian carers’ perceptions of care assessment and support in the community. Br J Soc Work. 2005; 35: 337–353. 17. 17.Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005; 85: 257–68. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToicHRqb3VybmFsIjtzOjU6InJlc2lkIjtzOjg6Ijg1LzMvMjU3IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDQvMTMvMjAyNC4wNC4xMS4yNDMwNTY5MS5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 18. 18.Dettori JR, Norvell DC. Kappa and beyond: Is there agreement? Global Spine J. 2020; 10(4): 499–501. 19. 19.Kieffer S. ECOVAL: Ecological validity of cues and representative design in user experience evaluations. AIS Transactions on Human-Computer Interaction. 2017; 9 (2): 149–172.