First assessment of a cost questionnaire for its agreement with administrative databases in females with stress urinary incontinence in Switzerland: a validation study

Questionnaire development

The questionnaire was developed for a previous COI analysis in patients with stress urinary incontinence. However, validity of the data gathered by this tool was never evaluated. The “Health status, comorbidities and cost-of-illness in females with stress urinary incontinence living in the Canton of Bern” (12) was embedded in a randomized controlled trial investigating the effects of two pelvic floor muscle training protocols (13). The first version of the instrument was developed based on a questionnaire that has already been used in a study from the Universities of Ghent and Antwerp (Belgium), which evaluated health status and costs of obesity in adult females in Flanders (14). Face validity of the adapted cost questionnaire was operationalized by comprehensibility and completeness of its items and assessed by six experts in the field of pelvic floor rehabilitation. Additionally, the questionnaire was confirmed for comprehensibility by two pilot-test subjects. To review criterion validity in a first point, these two test subjects filled out the cost questionnaire retrospectively for the last four months and provided the health insurance bills for the same period of observation. The comparison with the survey data did not raise any major concerns, or the need for further modifications (12).

The present cost questionnaire under investigation assessed participants’ characteristics, health status and health related costs retrospectively over four months. This included all direct medical costs such as medical and therapeutic treatment costs, raises for dental services, complementary medicine, and medications. Consumed service units and price units were asked separately. In addition, direct non-medical costs (e.g., incontinence aids) and indirect costs (e.g., productivity loss such as work absenteeism and presenteeism) were assessed. “Filter questions” were used as guiding assistance for costs reporting (6). The questionnaire in full version can be found under “additional files”.

Setting

The previous COI study of Koenig et al. (12) used a prevalence based, bottom-up approach. According to a prevalence based approach guidelines, questions of the survey targeted not only expenditures due to SUI but the entirety of healthcare expenditure (15). Out of 96 participants in the original randomized controlled trial (RCT), 37 adult female participants volunteered in the COI study and completed the cost questionnaire in an individual four-month time horizon between 2016 and 2017. Each participant returned her questionnaire during a post-intervention measurement for the RCT. The answers were then checked by the responsible examiner for completeness and missing data were completed immediately in the presence of the participants. Based on unit prices, Koenig et al. (12) estimated the total healthcare costs of the investigated four months.

The Swiss healthcare system is highly decentralized. Compulsory health insurance is offered by competing insurers with minimal annual deductibles of 300 Swiss Francs (CHF) for insured adults. Insured persons may increase deductibles up to 2’500 CHF. Insured persons also pay 10% coinsurance of these services with an annual limit of 700 CHF for adults (16). Residents are free to take out a supplementary insurance. In contrast to the compulsory health insurance, coverage of services may differ between insurance providers.

Supplementary insurances might include, for example, complementary medicine, dental treatment, or health prevention services. There are different methods of charging the fee: service providers can either invoice the insurances directly or they can bill patients, which settle up the account and claim reimbursement from the insurer (16).

Study participants and procedure

Each participant received written information about the study design and procedures and signed the informed consent form between October 2019 and February 2020. With volunteer’s power of attorney, one of the authors (CM) was allowed insight into personal health insurances billing data of the investigated four-month period between 2016 and 2017.

The inclusion criteria for the previous randomized controlled trial were: age range 18 to 60 years, body mass index (BMI) between 18 to 30 kg/m², nulliparous or at least 12 months postpartum with a negative pregnancy test, ability to read and understand the German language and to run on a treadmill. Exclusion criteria were: acute urinary tract or vaginal infections, urogenital surgery, predominant overactive bladder and urogenital prolapse over grade one pelvic organ prolapse quantification system (POP-Q). These criteria met the International Consultation on Incontinence’s recommendation for diagnosis of SUI (17).

Statistical analysis

Characteristics of the sample were compared to the original cohort of the RCT with a Student’s t-test for independent samples. As a reference for the degree of incontinence, the Oxford Grading Scale was used. This scale is a classification system of the pelvic floor muscle strength where zero represents no palpable contraction and the maximum score of five signifies strong contraction (18). Median and interquartile range (IQR) of observed muscle strength values were calculated for the original and the current samples.

For comparability with other studies (19), descriptive parametric statistics were used to analyze the cost data, despite the probability of a skewed distribution. The items “medical care” and “physiotherapy” fell into the scope of compulsory health insurance. The items “dental care”, “complementary treatment” and “medication” were categorized as services within supplementary insurance. To measure criterion validity, the differences between reported costs and health insurance billings were calculated and compared with “0” difference using a one-sample t-test. For a first visual impression of the relationship between the datasets, a scatterplot was performed. Cost data is usually skewed due to, for example, medical complications. Therefore, data was log transformed for approximate normality to conduct calculations of correlation (20,21).

As a preliminary test, consistency of variables was indicated by Pearson’s correlation coefficient (r). Agreement was measured using intraclass correlation coefficient (ICC). Since the variables were continuous and reported in the same units, ICC was appropriate (22). ICC two-way, mixed model formula for agreement was applied. As a reference value for small sample sizes, Lin’s concordance correlation coefficient (LINCCC) was added (23). Pearson’s r, ICC and LINCCC are measurements of correlation ranging from 1 (perfect agreement) to -1 (perfect inverse agreement) (23,24). The following levels for interpreting accuracy were applied: “poor” = < 0.40, “fair to good” = 0.40 to 0.74, “excellent” = 0.75 to 1.00 (25). Standard error of measurement (SEM) was calculated (26). Systematic bias was assessed using limits of agreement (LOA) and were presented in a Bland Altman plot (6). The visual result was supported by the calculation of linear regression of the difference between the costs over mean value of the costs. The procedure was doublechecked by two of the authors independently.

All data analyses were performed using SPSS (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp.) and with a macro written in Excel for the calculation of LINCCC. Significance was set at 5% level of error.

Assessment of validity

This study aimed to validate a questionnaire assessing costs in females with stress urinary incontinence living in the larger area around Bern in Switzerland. Although analysis of correlation is known as a reliability process, it is also used as a method to determine criterion validity between the measurement instrument and the gold standard (22). As a gold standard, health insurance billing was used in this study.

Students’ t-test for independent samples showed no significant difference in demographic data of the 14 participants enrolled in the current study to the original cohort of 96 females. Since the participants of the RCT were diagnosed according to the criteria of the International Consultation on Incontinence, the current sample was representative for females with stress urinary incontinence.

Healthcare expenditures analyzed by the items “medical care”, “physiotherapy”, “dental care”, “complementary treatment” and “medications” showed large standard deviations in both reported and administrative data. Since each of those items represented different medical services, this variability seemed to have occurred naturally. For example, a respondent could accrue a ticket for a simple laboratory service of about CHF 20 in the category of medical care as well as a surgical procedure of CHF 1’500. Despite the observed variability, when comparing mean values of reported costs and billing data, subgroups appeared similar. This was particularly relevant for the items “medical care” and “physiotherapy”. Participants seemed to estimate costs for these services accurately and consistently with insurance billings. The differences of datasets get larger, when it comes to the costs of dental care and complementary treatments. Reported dentist costs were three times higher than insurances recorded billings. Since Swiss residents are free to take out a supplementary insurance of their needs, different services might be covered or not. If participants had a supplementary dental health insurance, dentists’ costs would be covered by the insurance. If not, residents would most likely settle the bill out of pocket without submitting the invoice to the insurance. Consequently, companies might be incompletely informed and reflect the true costs insufficiently. Thus, the patient reported questionnaire could be more accurate than the so-called gold standard itself. Concerning complementary treatments, billings were higher than reported costs. Since overestimation of billing data is unlikely (5), underestimation of patient reported cost must be assumed. Pinto et al. (5) observed this behavior of underreporting regarding doctor visits. Recall bias side lined, difference of reported costs may also be a result of the “reverse telescoping” effect.

Telescoping occurs, when participants unconsciously lengthen their period of investigation and report events erroneously into the recall time. In contrast, treatments within the time frame may be “reverse telescoped” out of the recall period (27). True costs might even be higher than insurance’ billings revealed. Depending on the coverage of services, participants may or may not have submitted the invoices to the companies. Therefore, unsubmitted billings must be assumed, which would lead to a simultaneous underestimation of insurances expenditures for complementary treatments. In the case of medications, insurance companies’ information is generally restricted to prescribed drugs. Whereas the patient reported cost questionnaire captured all types of medication, including out of pocket utilizations. Overall sums of reported costs and billings were similar but with large standard deviations. Subsequently, the data range had a similar impact on each arm of the distribution and averaged itself out. Correspondingly similar values of reported costs were found after dividing the costs into categories of compulsory and supplementary services. Differences between the data sets were nonsignificant in all the three calculations. Therefore, Pearson’s r correlation coefficient was calculated. In contrast to ICC agreement, Pearson’s r does not take systematic error into account (22). Whereas ICC measures the degree to which one variable can be equated to another variable, Pearson’s r assesses the linearity of the relationship between the two variables. Lin’s concordance correlation coefficient is known for its robustness even with small sample sizes (23,24). It evaluates the degree to which pairs fall on the 45° line when plotted (28). In this present study, variability between the degree of correlation between the three methods occurred. Given the small number of samples, Pearson’s r and ICC may have yielded unstable calculations. Hence, ICC reflected a higher correlation than Pearson’s r despite the more restrictive formula (22). Point estimations of correlation for overall data were “fair/ good” to “excellent” according to the levels of accuracy with a Pearson’s r of 0.54 [95% CI 0.16 to 1.00], ICC of 0.79 [95% CI 0.33 to 0.93] and LINCCC of 0.75 [95% CI 0.19 to 0.86]. Nevertheless, the large confidence intervals indicated an ambiguous result. As expected, due to the descriptive values, correlations of supplementary services were lower than compulsory services with point estimations rated “fair to good” for supplementary services respectively, “fair to good” and “excellent” for compulsory services.

The Bland Altman plot (figure 1) depicts the mean value of the pairs against differences between the pairs. Each dot on the plot represents data from one participant. There are six dots each above and below the mean visible. The limits of agreement are wide and thus consistent with the large confidence intervals of the correlations. Out of 14 dots, 13 are located within the limits of agreement and one is situated exteriorly. There is a trend noticeable, where the more the mean increases, the larger the differences become. This finding suggests that participants who have less health service consumption were able to estimate their costs better than those with much healthcare consumption. The calculation of the linear regression confirmed the first impression of no evident systematic bias. Beta coefficient was -0.17 (p = 0.564), indicating that there was no proportional bias involved.

Indeed, the plot shows that the deviations of the mean were evenly balanced between under- and overreporting of study participants. In the systematic review of Leggett et al. (2), a tendency of underreporting was noticed. The 15 included self-reported questionnaires seemed to capture the number of hospitalizations and emergency room visits accurately, whereas participants tended to underestimate the number of visits to a general practitioner (GP).

Pinto et al. (5) developed a questionnaire to capture healthcare use and costs in patients with osteoarthritis in New Zealand. This study included 50 participants. The reported costs of the three-month investigation period were compared to administrative data. GP services, hospital services and medication use were collated. According to the authors, correlations were acceptable concerning GP services and hospitalizations (LINCCC >0.4) but exhibited poor agreement for medication (LINCCC < 0.4). These results were in line with findings of the current included females with SUI. Depending on the cost categories, concordance between survey and administrative data corresponded well and the questionnaire seemed to give an appropriate reflection of direct medical costs. Nonetheless, administrative databases were limited as gold standard, since they captured a more restricted perspective of costs than the patient reported questionnaires.

This study attempted an important step for standardized economic research in the field of pelvic floor medicine and rehabilitation. Further research is needed to evaluate and strengthen the results of this present study on a greater sample size. Since the cost questionnaire is only available on paper, a digital version is required. For this future version, a feasibility study will be needed.

Limitations

In their systematic review, Leggett et al. (2) showed that few studies have aimed to measure the accuracy of healthcare service use and cost data questionnaires. Thus, no unambiguous method is known (29) and limitations need to be declared. The first obstacle for assessing accuracy of such questionnaire is the lack of an appropriate “gold standard” (29). Insurance companies do not cover all types of healthcare costs and are therefore not informed about the totality of expenses. Due to the decentralized Swiss healthcare system, there is no single omniscient administrative database. Survey data were thus compared to a deficient criterion. Secondly, since there is no appropriate comparator for a broad perspective of costs, the cost questionnaire was assessed for a subset of questions only. Hence, declaration of validity was only possible for the part of direct medical costs. Direct non-medical and indirect costs were neglected. Thirdly, sample size of the present study was low (n = 14). Due to the fact, that the study population was originally participating in an RCT and was committed to the following COI analysis, only few people agreed with another observation. Consequently, the low number of participants may have generated unstable calculations. An unambiguous result would have been achieved by a greater cohort.

Conclusion

The developed questionnaire for measuring costs in females with stress urinary incontinence in Switzerland seemed to capture direct medical costs sufficiently. Overall correlation of patient reported data and health insurance billings yielded in a Pearson’s r of 0.64 [95% CI 0.16 to 1], ICC of 0.79 [95% CI 0.33 to 0.93] and LINCCC of 0.75 [95% CI 0.19 to 0.86].

Compulsory services such as medical care and physiotherapy corresponded better with administrative database than supplementary services such as expenses for dentist, complementary treatment, and medication. No proportional bias of under- or overreporting occurred.