Monitoring sustainable development goal 5.2: Cross-country cross-time invariance of measures for intimate partner violence

Sample and data on IPV

The DHS is a U.S. Agency for International Development (USAID)-funded program operating across more than 90 LMICs that collects data on population and health, including IPV among ever-partnered women. Per DHS protocols, between 15% and 100% of sampled households are administered a DVM, for which one woman 15-49 years in the household is randomly selected and interviewed (Table 1). To ensure similarity in the number and wording of items across administrations of the DHS, we restricted our sample to DVM versions V through VII, administered between 2005 and 2019. Within this frame, final samples included ever-partnered women 15-49 years from 19 countries and 18-49 years from one country in which at least two DHS were administered at intervals of 1 to 12 years with the same seven physical-IPV items (44 surveys total). For analyses of the controlling-behaviors items, two Rwanda surveys in the above sample were excluded; one survey did not administer the controlling behaviors items, bringing the number of countries to 19 and the number of surveys to 42.

The total sample included 380,012 women who were selected and administered the DVM and were not skipped out of the IPV items due to never-partnered status. Of these, 2,512 were missing data on all physical-IPV items, bringing the final analytic sample to 377,500 ever-partnered women across 20 countries and 44 DHS in analyses of the physical IPV items. The final analytic sample for controlling-behaviors items was 374,628 women across 19 countries and 42 DHS. Removal of individuals with missing and “don’t know” responses for all controlling behaviors items brought the analytical sample for controlling behaviors to 371,846. All DHS samples were downloaded with written permission from the DHS program.

For items on physical, sexual, and psychological IPV, participants across all DHS were asked whether their husband or partner had ever done each act (see Supplemental Table S1 for item wordings). Participants who responded yes were asked whether their husband or partner had done the act often, sometimes, or never within the past 12 months. For controlling behaviors, participants were asked whether their husband or partner did or did not do each of the behaviors without reference to a time frame. We elected to use the lifetime rather than the past-12-month timeframe for responses to the physical-IPV items for greater comparability across the two item sets. In a subset of countries, the DHS also included a maximum of three sexual-IPV items and a maximum of three psychological-IPV items (Supplemental Table S1). We did not use these item sets in our final analyses due to their questionable content validity relative to uniform definitions of these constructs (14, 15) and the small number of included items (16).

Analytic strategy

In step 1 we tested the measurement invariance of the set of seven physical-IPV items, and separately, the five controlling-behaviors items, over repeated cross-sectional surveys within each country. We performed multiple group confirmatory factor analysis (MGCFA) using weighted least squares estimation, comparing the fit of configural models, in which all loadings and thresholds were estimated freely across repeated cross-sectional surveys, and scalar models, in which all loadings and thresholds were constrained to be equal across repeated cross-sectional surveys. For countries with three repeated cross-sectional surveys, we performed invariance testing across each combination of two surveys. We used DHS-generated probability weights and cluster variables in all models to account for selection probabilities and clustering. We used several indices to assess the fit of configural models: chi-square (χ²), Root Mean Square Error of Approximation (RMSEA, adequate fit ≤0.08, good fit ≤0.05), and Comparative Fit Index and Tucker-Lewis Index (CFI, TLI, ≥0.95) (17). We used the χ² difference test to assess invariance over repeated cross-sectional surveys (18, 19).

In step 2, we conducted a pooled analysis of all DHSs that showed configural invariance in each individual-country MGCFA. In step 3, we used MGCFA with maximum likelihood (ML) estimation to assess metric invariance across repeated cross-sectional surveys within each country and in a pooled analysis across all DHSs. When pooled analyses showed a lack of evidence for metric invariance, we used the alignment optimization (AO) approach in step 4 to perform an approximate invariance test in the pooled sample of all datasets. This approach relaxes some assumptions of MGCFA by allowing estimated country-specific model parameters to vary from the estimated model parameters in the pooled dataset following a normal distribution. The criterion for approximate invariance is evidence that ≤25% of model parameters (loadings and thresholds) are non-invariant. In step 5, where approximate invariance was not supported in the pooled sample of all datasets, we restricted the sample to countries that showed metric invariance over repeated cross-sectional surveys in individual-country analyses. We then reran alignment optimization using this subset of surveys and countries. We used STATA 17 (20) for data cleaning and management. All measurement invariance testing was performed in MPlus 8 (21).

Characteristics of included surveys

Survey characteristics, including logistics and design, are summarized across the full sample of 44 surveys (Table 1). The duration of enumerator training varied across surveys from between one to six weeks, with most surveys (43%) conducting training in four weeks. Across all surveys, data collection was conducted by an average of 42 field teams. The total number of survey field teams ranged from five in Senegal to 789 in India. Most surveys (91%) were translated into at least one local dialect. India had the most translations, into 17-18 dialects. All surveys included three sensitive modules on HIV, contraception, and sexual activity that preceded the DVM. The DVM typically was the last module in the women’s questionnaire, with at least nine modules preceding it. Nearly half of the surveys (21 of 44) reported a mean duration of the women’s interview of 45 to 60 minutes; however, the interview duration ranged from 31.9 minutes to 76.6 minutes.

Surveys were administered between 2005 and 2019; 16% were administered in 2018. Within countries, the average number of years between repeated survey administrations was five. To create the DVM sample, all surveys selected between 15% and 100% of households interviewed in the main DHS, and then sampled one woman per household for the DVM. A plurality of surveys (39%) sampled 50% of interviewed households to create the household sample for the DVM. On average, 10,520 women across surveys were selected and interviewed for the DVM; however, sample sizes for the DVM ranged from 1,865 women in the 2019 Senegal DHS to 83,703 women in the 2005-06 India DHS. In most surveys (91%), both ever-married and never-married women were eligible for the DVM. Two surveys in Jordan and two surveys in Pakistan interviewed only ever-married women for the DVM. Among the surveys that interviewed all women for the DVM, only ever-married women and women who ever lived with a man were eligible for the physical IPV and the controlling behaviors items. However, in the 2008 Philippines DHS, women who have (had) a boyfriend or dating partner previously or at the time of the survey were eligible for the physical-IPV and controlling-behaviors items. All surveys interviewed women ages 15 to 49 years for the DVM, except the 2015 Mozambique DHS, which included women 18 to 59 years.

Country-specific and pooled time invariance of physical IPV items and controlling behaviors items

Of the 20 countries included in the measurement-invariance testing of the seven physical-IPV items, all showed good fit for the individual-country configural model across at least two DHS administrations (Table 2). According to changes-in-fit-statistics criteria (ΔCFI, ΔTLI), individual-country models showed scalar invariance over time; however, according to the χ2 difference test between scalar and configural models, individual-country models for only five countries showed scalar invariance over time. In metric invariance testing using maximum likelihood estimation, 12 countries had a non-significant likelihood ratio test across repeated DHS administrations in individual-country analyses, suggesting metric invariance (Table 3).

In a pooled analysis of all 20 countries, while configural invariance was evident, neither metric (Table 3) nor scalar (Table 2) invariance was achieved according to difference testing. Changes in fit statistics, however, did not provide evidence of non-invariance in the scalar models. When the pooled sample was restricted to the 27 DHSs (from 12 countries) that showed evidence of metric invariance across repeated DHS administrations in within-country analyses, metric invariance still was not evident (Table 3).

In analyses of the five controlling-behaviors items, all 19 individual-country analyses showed good fit of the configural model (Table 2). Six countries showed evidence of scalar invariance over time according to χ2 difference testing, with 11 showing evidence of metric invariance according to the likelihood ratio test in maximum likelihood models (Table 3). For the pooled sample of 19 countries, neither metric nor scalar invariance was suggested by the likelihood ratio test or the χ2 difference test, respectively. Metric invariance was not evident in the 11-country pooled sample for which repeated cross-sectional DHSs showed metric invariance over time in individual-country analyses. Neither the individual-country nor pooled analyses showed evidence of non-invariance according to changes in fit statistics in weighted least squares models.

Tests of approximate invariance of physical-IPV items and controlling-behaviors items

Table 4 presents the AO-based results, in which we assessed approximate measurement invariance separately for the seven physical-IPV items (Panel 1) and the five controlling-behaviors items (Panel 2). For physical IPV, 118 (or 38% of) estimated thresholds, 44 (or 14% of) estimated loadings, and 26% of all parameter estimates were measurement non-invariant (Supplemental Table S2). For controlling behaviors, 132 (or 61% of) estimated thresholds, 78 (or 36% of) estimated loadings, and 49% of all parameter estimates were measurement non-invariant. A guideline of 25% or fewer total non-invariant parameter estimates is recommended for trustworthy latent mean estimates and their comparison across groups. The results suggested that neither item set exhibited approximate measurement invariance across the 20 countries and repeated DHS administrations. Among the seven physical-IPV items, the item ‘slap’ had a low degree of threshold and loading invariance, as shown by its low R² (Table 4).

Table 5 presents the AO-based results in which we assessed approximate measurement invariance separately for the physical-IPV items and the controlling-behaviors items for a subset of countries that displayed metric invariance across at least two administrations of the DHS (Table 5). For physical IPV, 61 (or 36% of) estimated thresholds, 15 (or 9% of) estimated loadings, and 20% of all parameter estimates were measurement non-invariant. For controlling behaviors, 47 (or 39% of) estimated thresholds, 21 (or 17.5% of) estimated loadings, and 28% of all parameter estimates were measurement non-invariant. Thus, the results suggested that DHS physical-IPV items but not the controlling-behaviors items exhibited approximate measurement invariance across countries and repeated administrations showing within-country metric invariance and allowed acceptable alignment performance. Additionally, the R² values showed that all seven physical-IPV items had a reasonable degree of threshold and loading invariance (Table 5).

Summary of findings

Testing of within-country cross-time measurement invariance, relevant for national efforts to monitor IPV trends using the DHS DVM, revealed that the seven physical-IPV items and the five controlling-behaviors items functioned equivalently in repeated survey administrations within a subset of LMIC countries. In the second stage, we examined cross-country and cross-time invariance in pooled samples including multiple countries with two or more survey administrations each. While these two item sets were not strictly equivalent in these samples, the physical-IPV item set exhibited approximate invariance over time and across countries in a restricted sample of countries exhibiting within-country, cross-time metric invariance of the item set. The five controlling-behaviors items did not meet the recommended threshold for non-invariant parameters to infer approximate invariance across time and across countries. A prior analysis found evidence of approximate invariance for physical-IPV and controlling-behaviors item sets across 36 DHS administered in 36 countries during 2012-2018 (13, 22). The present analysis corroborates that the physical-IPV items function comparably across time and across very diverse national contexts and highlights the cross-time invariance of the controlling-behaviors items within selected countries. Evidence of greater threshold than loading non-invariance, especially for controlling-behaviors items, suggests greater comparability in item interpretation across contexts and time, but less comparability in the likelihood of endorsing items (responding yes to acts of IPV) across contexts and time.

Limitations and strengths

Findings should be interpreted considering the study’s limitations and strengths. The study assessed measurement properties of item sets in the DHS; therefore, findings cannot be extended to item sets that measure other forms of IPV nor to item sets that are used in other, non-DHS IPV survey modules. However, the DHS are widely administered across LMICs and represent approximately half the data being reported to monitor progress toward SDG5.2.1. It is the single largest contributor to SDG5.2.1 monitoring and has IPV items like those used in WHO surveys, making it possibly the most important source for rigorous psychometric testing. Findings reported here may represent a best-case scenario. The DHS program provides technical support for survey administration, which, while not entirely uniform across countries or time periods (Table 1), does provide a level of consistency in administration that does not exist across the wide variety of survey formats and forms of administration that represent the data pool available for SDG5.2.1 monitoring. This level of consistency bolsters its use for research, but potentially limits study findings to the item sets tested using similarly consistent methods of administration. Finally, in pooled analyses, we were unable to account for possible auto-correlation across national surveys within countries. Despite their limitations, the findings are based on 44 DHS conducted in 20 diverse countries spanning four regions (Africa, Asia, Latin America, and the Middle East) and 15 years (2005-2019).

Implications for research and policy

These findings suggest the seven DHS physical-IPV items are promising for comparing and monitoring national trends in IPV toward achieving SDG5.2.1, to eliminate IPV against women. The low R² for the item ‘slap’ in AO analysis suggests the potential benefit of focused cognitive testing of this item across diverse contexts to improve its measurement properties, and thereby, the item set as a whole. The five DHS controlling-behaviors items, in their present formulation, show promise in some countries for monitoring within-country trends in this form of IPV; however, their lack of approximate invariance in full and restricted pooled samples of countries with repeated DHS administrations caution against their use to compare and to monitor national trends in this form of IPV toward achieving SDG5.2.1. Cognitive testing of these items, and psychometric testing of a revised controlling-behaviors item set in diverse, multi-country samples of women, may improve their measurement properties and their utility for monitoring SDG5.2.1 cross-nationally and over time.

Improved global measures of controlling behaviors also will improve our estimates of the impacts of these forms of IPV on the health of victims and their children worldwide, providing insights into strategies for prevention and response. These advances are critical, given that controlling behaviors in an intimate partnership often indicate more severe forms of IPV. Improved measurement of controlling behaviors is motivated further by changes in some criminal codes to include ‘controlling or coercive behaviors’ as prosecutable offenses (23). Thus, promoting standard, contextually informed, definitions of controlling behaviors and enhancing the measurement properties of controlling-behaviors items will strengthen the capacity for cross-national monitoring of trends, and may stimulate changes to other national criminal codes to include controlling behaviors as a prosecutable offense. Such changes would provide new legal norms about the nature and scope of IPV and new mechanisms to deter controlling behaviors (24). Such changes also offer the potential to move away from a narrow focus on physical injury towards emotional and psychological trauma in criminal cases of IPV (23), expanding response options for victims of these forms of IPV.

This analysis did not assess the cross-country and cross-time measurement properties of DHS item sets measuring psychological IPV (typically 3 items) and sexual IPV (typically 2-3 items). Presently, these DHS item sets align only narrowly with uniform definitions of these forms of IPV (15, 25), suggesting a notable lack of content validity. Still, there may be practical benefit in future analyses to assess the psychometric properties of these limited item sets to establish an evidence-base regarding the extent of their cross-country and cross-time measurement invariance. The current lack of content validity of these item sets, however, has important practical implications for interpreting trends in these forms of violence. Namely, the current content of the item sets implies that only certain underlying ranges of these forms of IPV are observable. As a result, estimated trends in these forms of IPV—even if they are shown to be measurement invariant—may inaccurately capture true underlying trends. For example, if reductions in sexual IPV using measured physical tactics occurs alongside increases in sexual IPV using unmeasured non-physical tactics, observed rates of sexual IPV will appear to decline, when, trends in the totality of sexual IPV are stable or increasing. Indeed, focused studies using more comprehensive measures confirm the high levels of forms of sexual IPV (26) and psychological IPV (27) that the DHS does not measure. Hence, expanding these item sets is needed to capture the full range of relevant behaviors for accurate monitoring of SDG5.2.1. Such an effort need not result in large item sets, because the process of psychometric assessment can identify a precise subset that is reasonably content valid. Therefore, we recommend desk reviews of validated instruments and qualitative research in diverse settings to generate expanded item pools for sexual and psychological IPV, cognitive testing of these expanded item pools, repeated cross-cultural pilot surveys, and rigorous psychometric assessment to identify item sets that are content valid and measurement invariant across-context and across-time. Such an effort would round out the much-needed evidence to identify a common, validated item pool for inclusion in national surveys of violence against women. Agencies like the United Nations (UN), national governments, and global donors would have the evidence needed to make maximally informed decisions about the allocation of resources to prevent and to respond to IPV, based on trends in all domains of IPV that are optimally measured.