Factor Structure, Reliability and Validity of the Institutional Culture Assessment Scale (ICAS)

Introduction

It is now a well-established axiom within the organizational literature that the institutional environment plays a key role in the efficiency, health and well-being of professionals. Institutional culture subtly shapes attitudes, biases, and behaviors, defining what is encouraged, discouraged, accepted or rejected within the workplace or profession. However, the essence of institutional culture, sometimes referred to as organizational culture, is a somewhat elusive and complex construct that defies simple definition. There are a myriad of formal definitions and a variety of models and methods for assessing it, some of which are likely to be found in the human resources field, where the focus is mainly on performance evaluation. Most of the scales assess the degree of satisfaction with the perceived climate at work in terms of self-realization, workload, co-workers, salary structure, leadership, nature of the work, promotions, and other markers in an effort to catalog the multiple factors that make up the structure. The end result is a more global assessment of the work environment, one that is less focused on factors related to employees’ well-being and psychological health.^1–4

The institutional culture may be of particular importance to those who are health care professionals. Several authors have observed that health professionals as a group have been intensely affected by changes in the administrative structure of hospitals and the decline of physician autonomy in the current culture of health institutions.^5–7 In fact, a recent meta-analysis revealed a significantly smaller effect of interventions aimed at the individual level compared to those geared to the institutional level.^8,9 Rates of exhaustion and mental health concerns are currently affecting 30-60% of all doctors and physicians in training in North America.¹⁰ In the face of the COVID-19 pandemic, these rates can be expected to rise exponentially.^11–13

Adequate tools to identify the specific aspects in the culture that contribute to staff well-being versus those that result in distress and psychiatric symptoms are badly needed. Although some instruments exist that measure the link between good relationships and work-related variables¹⁴, to our knowledge, there is not an instrument that evaluates the relationships with others that exist within the institution as well as the leadership style along with the relation of the professionals to the institution itself (e.g., the feeling of belonging, a collaborative atmosphere and an alignment between the values of doctors/ health professionals and the institution). Those factors seem to be highly relevant for the configuration of the culture, in that they may be instrumental in preventing an erosion of the professionals’ role, their psychological health, the quality of service provided to the clients, and in the case of health professionals the quality of patient care, besides the cost to society.¹⁵

With the intention of identifying these aspects Carneiro Monteiro et al.¹⁶ developed the Work Environment Evaluation Instrument (WEEI), to evaluate the work environment in medical training settings. The WEEI was developed and tested in a specific population, i.e., residents in training. The aim of the present study was to develop the Institutional Culture Assessment Scale (ICAS), based on an adaptation of the WEEI’s items, testing its factor structure, reliability and validity in a Brazilian sample of medical students and physicians in different settings and phases of the medical career.

Methods

In this online cross-sectional study, 2,537 individuals were evaluated in October 2019, using a snowball method over fifteen days. We chose an electronic questionnaire in order to reach a greater number of respondents. Online questionnaires also have the potential advantage of enhancing reliability by augmenting the perception of anonymity.^17,18 The advertisement explicitly stated the survey’s anonymity, and the subject only continued to the questionnaire if agreed with the online informed consent form. After completion, the questionnaire provided telephone and electronic contact information for suicide prevention and support centers located in Brazil. The study was approved by the ethics committee of the Hospital de Clínicas de Porto Alegre (Porto Alegre, Brazil) (protocol 70231617.6.0000.5327).

Results

Parallel analysis and Exploratory Factor Analysis

Parallel analysis revealed that 2 factors represents the best solution to explain variance in the WEEI adapted version items (Figure 1). The first factor explained 36% of the variance, whereas the second explained 15% of the variance. The first factor encompassed items involving the institution (Table 2; items 2, 4, 6) and supervisors (Table 2, items 1, 3, 7, 9). The second factor encompassed items involving peers (Table 2, items 5, 8, 10). The item 11 presented a factor loading which is markedly lower than the average of the factor loading from the other items of the first dimension (λ=0.23 vs. average λ=0.73). The correlation between the first and second factors was moderate 0.47.

View this table:

• View inline
• View popup
• Download powerpoint

Table 2

Exploratory Factor Analysis Results from the first half of the sample (n=1269)

• Download figure
• Open in new tab

Figure 1. Parallel Analysis Scree Plots for determining the number of factors in Exploratory Factor Analysis

Because peers formed a second factor, separated from institution and superiors/supervisors’ factors and presented with only 3 items, we decided to exclude the peer domain and performed the next analysis with a unidimensional constructed based solely on institution/supervisor items.

Item Response Theory

Graded Response Model revealed high discrimination for all items (all above 1.4) and difficulties ranging from −2.1 for the lower threshold to 1.5 for the higher threshold. Test information function (Figure 2, Panel A) reveals the test captures reliable information from −2 (alpha 0.8) until +2 area (alpha 0.72) of the latent trait as revealed by the irt.fa function and has a good potential in differentiate both healthy and toxic environments. Item Information Curves (Supplemental Figure 1) and Item Characteristic Curves are (Supplemental Figure 2) are presented in supplemental material. The ICAS showed approximately normal distributions in summed and IRT-based scores (Figure 2, Panel B). Conversion tables between summed and IRT-based scores are presented in Supplemental Table 1.

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 2. Test Information Function for the revised structure of the ICA and histograms of the summed and IRT-based factor scores

Discussion

This study presents a valid and reliable scale to assess institutional culture aspects connected with the relationships with superiors/supervisors and the relation to the institutions themselves. The ICAS revealed an excellent fit of the data (RMSEA=0.053, CFI=0.997, TLI=0.996) as well as high internal consistency (McDonald’s Omega = 0.89), proving to be a suitable tool for the evaluation of institutional culture dimensions that might be related to staff well-being versus those that result in distress and psychiatric symptoms. The correlation with burnout score in this sample provide additional evidence in this sense. The ICAS also encompasses many concepts of self-determinant theory research that focuses on the consequences of the extent to which individuals are able to satisfy the needs within social environments.²

To the best of our knowledge, there isn’t an instrument that evaluates the relationships with superiors/supervisors along with the relation to the institution itself as dimensions of the institutional culture that may impact employees’ health. There are some instruments like the Working Environment Scale (WES-10)²⁶ that assesses an overall degree of satisfaction with the perceived climate at work in terms of self-realization, workload, conflict and nervousness. Another is the Organizational Culture Assessment Instrument (OCAI)²⁷ that aims to evaluate institutional culture regarding six attributes: dominant characteristics, organizational leadership, management of employee, organization glue, strategic emphases, and criteria of success. The ICAS has the advantage of being a brief self-applied instrument that takes into account the perception of individuals in different stages of career and the holding of different positions within the institution. The items evaluated have been proven to be related to burnout symptoms in previous studies with medical residents as well in this national sample of medical studies and physicians.^16,19 As the aspects evaluated are related to leadership style (e.g. fostering collaboration, emphatic listening and attitudes towards employees), and to the perception of the institution (e.g., collaborative climate, shared values), the ICAS is expected to be associated with employee mental health in populations other than health professionals and should be tested in different settings. It is important to point out that the aspects evaluated by the ICAS are possible to change by interventions in the institutional level. This approach can be a more effective strategy than limiting the scope to interventions at the individual level that tend to foster the augmentation of the individual resilience, but, in many cases, ignore some of the causes.^7,8,28

It is well established that the work environment plays an important role in subjects’ well-being and/or distress. However, the aspects of the institutional culture that are more related to psychological distress in the work setting are still not defined. A variety of factors, e.g., individual vulnerability, institutional pressures, and socioeconomic factors, are involved in the phenomena that lead individuals to become severely distressed in the context of the workplace.^5,29,30 When individual factors interact with institutional characteristics that reinforce the unhealthy aspects of the individual (such as an excess of competitiveness, criticism, low self-esteem and so forth), there can be devastating health consequences.

This investigation has notable strengths. First, we were able to develop a fast and self-administered instrument with high internal consistency that evaluated institutional environment in a sample of medical students and physicians. Second, we were able to test the instrument in a high number of participants. Third, we could evaluate aspects that are poorly addressed in the literature concerning the work environment features. Finally, we found an association between the ICAS score and burnout dimensions, reinforcing its validity in measuring aspects related to work related distress vs. well-being. On the other hand, this study has limitations typical of preliminary studies that need replication in other samples and contexts. On the other hand, this study has limitations inherent to preliminary studies that need replication in other samples and contexts. Also, it used a convenience sample in an online study, so only participants with easy access to internet were likely and able to participate in the present study.

Several authors have emphasized that changing the culture within institutions may be the most efficient way to stem the rising rates of emotional distress among health professionals.⁷ In the case of health professionals, a recent paper by Hartzband and Groopman⁶ call the attention to the fact that the modifications implemented in the health systems aiming at rising productivity and efficiency have eroded the very sense of the identity of being a doctor by taking away the important triad of autonomy, competence and relatedness, described by Gagne and Deci² as essential to support the professional’s intrinsic motivation and psychological well-being. This might well be the case in other professions where the sense of effectiveness has followed an excessive materialistic view at the expense of essential human emotional needs. In this sense a validated and reliable instrument like the ICAS may help in calling the attention to those aspects that seem to be essential to foster well-being within any institution.