Volume 22 Issue 3 September 2013
Comparison between an online self-administered and an interviewer-administered version of the Asthma Control Questionnaire: a cross-sectional validation study
*Persijn J Honkoop1,2, Rik JB Loijmans3, Evelien H Termeer4, Jiska B Snoeck-Stroband1, Gerben ter Riet3, Tjard RJ Schermer4, Jacob K Sont1, for the ACCURATE Study Group
1 Department of Medical Decision Making, Leiden University Medical Center, The Netherlands
2 Department of Public Health and Primary Care, Leiden University Medical Center, the Netherlands
3 Department of General Practice, Academic Medical Center, Amsterdam, The Netherlands
4 Department of Primary Care, University Medical Center St Radboud, Nijmegen, The Netherlands
Received 29 August 2012 • Accepted 13 February 2013 • Online 25 April 2013
Background: Online self-management programmes for asthma have recently become available. International guidelines suggest that the Asthma Control Questionnaire (ACQ) can be used in these programmes. In order to assess the current level of control and guide therapy, the same cut-off values are being used as in conventional asthma management. However, results might differ between different types of administration of the ACQ.
Aims: To assess the agreement between an online self-administered version of the ACQ and an interviewer-administered version at a routine visit.
Methods: Cross-sectional data from primary care asthma patients in the Asthma Control Cost Utility Randomized Trial Evaluation (ACCURATE) trial aged 18–50 years and prescribed inhaled steroids were analysed. We selected patients who self-administered an ACQ online and subsequently had an ACQ completed by a nurse practitioner within 7 days at a trial-related control visit. ACQ scores were calculated and agreement assessed by paired t-tests, Pearson's correlation coefficient and a Bland-Altman plot.
Results: A total of 351 patients were eligible (68% female, mean age 40 years). The time interval between the two versions was 3.2 days. There was a significant difference of 0.14 (95% CI 0.09 to 0.20; p<0.001) between the results of the online self-administered ACQ (mean 1.04±0.04) and the interviewer-administered ACQ results (0.90±0.04). The Pearson correlation coefficient was 0.79. The limits of agreement (–0.86, 1.14) exceeded the predefined minimal clinically important difference between results (±0.5). The Bland-Altman plot therefore showed insufficient agreement.
Conclusions: Assessment of asthma control by the ACQ is influenced by the type of administration. Our results suggest that better control of asthma is perceived when interacting with a caregiver than by online self-assessment.
Cite as: Honkoop PJ, Loijmans RJB, Termeer EH, Snoeck-Stroband JB, ter Riet G, Schermer TRJ, Sont JK, for the ACCURATE Study Group. Comparison between an online self-administered and an interviewer-administered version of the Asthma Control Questionnaire: a cross-sectional validation study. Prim Care Respir J 2013;22(3):284-289. DOI: http://dx.doi.org/10.4104/pcrj.2013.00041
asthma management, monitoring, asthma control, interviewer-administered, self-administered, online
* Corresponding author. Persijn J Honkoop Tel: +31 (0)71 5264904 Fax: +31(0)71 5266838 Email: email@example.com
© 2013 Primary Care Respiratory Society UK. All rights reserved.
Online monitoring of asthma symptoms provides an opportunity of optimising patient-centred daily control. Online self-management programmes have therefore been developed which offer similar questionnaires to those being used in daily practice for assessment of asthma control. This policy is advocated in current guidelines which state that composite symptom scores such as the Asthma Control Questionnaire (ACQ) can be used in different settings by both patients and care providers to assess current asthma control.1-4 As a consequence, even though the ACQ was originally developed and validated for guided self-administration, it is now administered in a variety of ways – for example, in self-management plans using pen and paper or internet applications,5,6 self-administered but under guidance of a professional during regular control visits, or interviewer-administered by a practice nurse (PN), general practitioner or chest physician based on patients’ responses.1–3 Recent studies did not show statistically significant differences between paper and electronic versions of the ACQ7 or between a postal-administered version and a version under guidance of a healthcare professional.8 However, little is known about the agreement between online self-administered and interviewer-administered versions of the ACQ.
The aim of this study was to assess the agreement between an online self-administered version of the ACQ and an interviewer-administered ACQ by a PN during a routine control visit in adults with asthma.
The ACQ results of patients collected in the Asthma Control Cost Utility Randomized Trial Evaluation (ACCURATE) trial were used.9 The trial was approved by the Medical Ethics Committee of the Leiden University Medical Center, Leiden, the Netherlands, and compares three different treatment strategies for asthma in primary care. Patients were included in the study if they were aged 18–50 years, had a doctor’s diagnosis of asthma, and had received a prescription for inhaled corticosteroids in the previous year. Patients were excluded if they suffered from significant co-morbidity, if they were unable to understand written or oral Dutch instructions, or if they had been prescribed oral corticosteroids in the previous month.
In the ACCURATE trial, patients visited the general practice where ACQs were administered by a PN (interviewer-administered ACQ). Appointments were scheduled at approximately three-monthly intervals at a convenient time for both practices and participants. In addition, patients completed online questionnaires monthly at home, including the ACQ (online self-administered ACQ). Since the time between completion of the two versions of the ACQ was variable, we only selected observations from patients who self-administered an ACQ online and subsequently visited the PN for a trial-related control visit within 7 days. This time window was selected because it lies well within the advised range of 2–14 days for assessing reproducibility of instruments.10 We excluded measurements where online self-administered ACQs were completed after the visit to the PN because trial-related medication changes might influence asthma control. We used a cross-sectional design to ensure that each participant was included only once.
Asthma Control Questionnaire (ACQ)
The ACQ consists of six questions on symptoms and one on pre-bronchodilator percentage predicted forced expiratory volume in one second (FEV1% predicted). In the present study, FEV1% predicted was assessed with a hand-held spirometer (PIKO-1) in the online self-administered ACQ and by routine spirometry in the interviewer-administered ACQ. In the online self-administered version, patients completed the questionnaire online at home without supervision, whereas in the interviewer-administered version the PN completed all questions based on patient responses. An overview of all methods of ACQ assessment is shown in Table 1. All questions were scored on a 7-point scale from 0 to 6 (where 0=good control, 6=poor control), and the composite ACQ7 score is the mean of the seven responses. The minimal important difference (MID) between results is 0.5.11 Current control of asthma was divided into three levels: controlled (ACQ <0.75); partly controlled (0.75>ACQ<1.5); uncontrolled (ACQ >1.5).1,3
Paired t-tests were used to assess whether there were systematic differences between online self-administered and interviewer-administered ACQ results. The strength of the relationship was assessed using Pearson correlation coefficients. A Bland-Altman plot was used to assess the agreement between the two versions. In order to interpret a Bland-Altman plot, a clinically significant difference had to be predefined and results agreed sufficiently if they remained within this limit.12 In the present study the MID of 0.5 was used for this purpose.
Linear regression and correlation analysis were performed to asses whether the time between self-administered and interviewer-administered questionnaires, age of participants, smoking behaviour, years of asthma history, or differences in sex between patients and PNs influenced ACQ results. All analyses were performed with STATA Version 11 (StataCorp LP).
Of 611 patients participating in the ACCURATE trial, 351 visited the PN within one week after completion of the online questionnaire at home and were therefore eligible for the current analysis. Their mean age was 40.0 years, 68% were female, the mean inhaled corticosteroid dose was 861 µg beclomethasone equivalent, and 50% used a long-acting bronchodilator (Table 2). There was a statistically significant difference between the online self-administered (mean±SE 1.04±0.04, range 0–4.1) and interviewer-administered ACQ results (mean±SE 0.90±0.04, range 0–4.3) (difference 0.14; 95% CI 0.09 to 0.20; p<0.001). The Pearson correlation coefficient was 0.79.
When comparing the level of control, in 23% of patients the online self-administered version indicated a worse current level of control than the interviewer-administered version. This would lead to a step-up in maintenance treatment if the online self-administered ACQ were to be used to guide treatment decisions, which would not occur with the interviewer-administered ACQ. In 6% of patients the level of control was better, potentially triggering a step-down in medication. Overall kappa was 0.54, indicating moderate agreement (see Table 3 and Figure 1).
The Bland-Altman plot showed a mean difference of 0.14 between the two versions and the 95% limits of agreement (at ±1.96SD from the mean) were at –0.86 and 1.14 points. Both limits of agreement were beyond the predefined acceptable difference of (–)0.5 and therefore there was insufficient agreement between the two versions (Figure 2). Since the dispersion of the difference increased as the mean increased, we repeated the Bland-Altman analysis after log transformation and obtained similar results: mean log-transformed difference 0.16 (95% CI 0.10 to 0.22, p<0.001), 95% limits of agreement –0.98 and 1.31.
The difference between the two versions of the ACQ exceeded the MID of 0.5 in 25.1% of patients, of which 79.5% had a higher result in the online self-administered version and 20.5% in the interviewer-administered version. The differences were especially prominent in uncontrolled asthma. Furthermore, when questions in the ACQ were assessed separately, each question was significantly higher in the online self-administered version (p<0.03), except the question on FEV1 % predicted which showed no significant difference (p=0.39, Table 4).
The mean time between completion of the two versions of the ACQ was 3.2 days (95% CI 2.9 to 3.5). Linear regression and correlation analyses showed no significant association between the difference in ACQ scores and the time between visits (Pearson correlation coefficient (r)=0.03, p=0.54), age of participants (r=0.02, p=0.73), smoking behaviour (yes or no) (r=0.06, p=0.23), history of allergy (yes or no) (r=0.03, p=0.54), years of asthma history (r=0.04, p=0.52), and sex differences between patient and healthcare professional (r=0.01, p=0.81).
This study shows that the level of current asthma control is influenced by the type of administration of the instrument to assess control status. More severe symptoms were reported with the online self-administered questionnaire than with the interviewer-administered questionnaire by a PN based on patients’ responses. This difference is of particular relevance due to the increasing use of questionnaires in clinical practice13 and the appearance of online self-management programmes for asthma.5,6 These programmes adjust therapy based on the same cut-off points of asthma control as those used in conventional asthma management, which may lead to different treatment advice in 29% (95% CI 24.6% to 34.1%) of cases.
Strengths and limitations of the study
Since we included patients from both rural and urban areas with the full range of uncontrolled to well-controlled asthma and less than 1% of all eligible patients for the ACCURATE trial were excluded due to exclusion criteria other than age, the strength of our study is that the 351 patients included comprise a representative sample of adult asthma patients aged 18–50 years. Also, the interviewer-administered ACQs were assessed by PNs in 119 different general practices so selection bias seems unlikely. A limitation of our study is that the results may perhaps not be applied straightforwardly to patients aged >50 years who were excluded because of the increased prevalence of chronic obstructive pulmonary disease and fixed airways obstruction in this group. Furthermore, online self-administered versions were always completed before the interviewer-administered version, which might have introduced an order bias. However, an analysis of 68 patients who self-administered an online ACQ within one week after a trial-related visit and in whom no trial-related change in asthma medication was prescribed confirmed the higher ACQ results in the online self-administered version (difference 0.17, 95% CI 0.04 to 0.29, p=0.01). The time difference between administrations of the two versions probably did not influence the results since the correlation analysis of difference in ACQ results and time between both administrations showed no significant association (r=0.03, p=0.54) (data not shown).
The online self-administered ACQ was used in a previous study6 and validated in a preceding pilot study (which was not formally published). We showed that the repeatability was very good (intraclass correlation coefficient (ICC) 0.86) and the agreement was also very good (ICC 0.88). The obtained validation sample of subjects was a relatively homogeneous population with low mean scores for the ACQ (0.60) and no high results. Even though the ICC of 0.88 is slightly below the required 0.9 for validation,3 we considered the online version validated since the ICC is highly dependent on between-subject variability. In our population sample the between-subject variation was relatively low and, therefore, the ICC of 0.88 underestimates the actual agreement. On the other hand, assessment by a PN based on patient responses is a non-validated method of assessment, albeit widely used. Therefore, in the present study we compared a validated method of assessment with a non-validated method of assessment which could (partly) explain the difference in results.
Interpretation of findings in relation to previously published work
Two previous studies compared a postal-administered version of the ACQ with a version under guidance of a healthcare professional,8 and an electronic version with a paper self-administered version.7 In both studies the ICC and Pearson’s correlation were high and paired t-tests showed no statistically significant differences. Juniper et al. selected a more homogeneous group of only patients with uncontrolled asthma, whereas we included patients with the full range of asthma control. Furthermore, in the same study, Juniper et al. did find a significant difference between the electronic and paper versions of two other questionnaires with a more favourable outcome (i.e. fewer symptoms) with the paper version.7 Also, in these studies, all assessments of the ACQ were completely self-administered while in our study it was completed by the PN based on patients’ responses. Therefore, the actual person administering the questionnaire might explain the difference in ACQ results. In a review of different methods of administration of questionnaires, Bowling showed that patients reported more symptoms in self-administered than in interviewer-administered questionnaires in a number of fields of medicine, although not consistently.14 For certain asthma symptoms, McDonald et al. recently showed that patients and healthcare professionals rate the importance differently.15 Hence, the differences may be explained by a different interpretation of the question between patients and healthcare professionals, or PNs may weigh patients’ answers and score their own interpretation. Finally, patients may express their symptoms more freely online or under-report symptoms when answering a PN due to social desirability, a known phenomenon when assessing highly sensitive personal behaviour.16
Several previous papers have shown that patients reported more symptoms in an online self-administered questionnaire than in a paper self-administered version,14,17,18 suggesting that the format of an online questionnaire might have an effect. However, this was not a consistent finding,14,18,19 and we also showed very good agreement between the online self-administered version and a paper self-administered version in our pilot study.
Implications for future research, policy and practice
Our results suggest that, during face-to-face contact with a healthcare professional, on average, symptoms of asthma are reported as less severe than with self-assessment of symptoms when the same questionnaire (i.e. the ACQ) is used. It remains to be seen if asthma is better controlled if guided by online self-administered (risking overtreatment) or by healthcare professional-administered measurement scales (risking undertreatment). In order to guide treatment by the level of asthma control, the type of assessment – even when using standardised tools – should therefore be taken into account.
Our study also underscores the relevance of validating new methods of assessment of questionnaires before using them in clinical practice.
The level of asthma control measured by the ACQ depends on the type of administration. Symptoms of asthma are reported as less severe in the interaction with a healthcare professional compared with online self-assessment at home.
Handling editor Hilary Pinnock
Statistical review Gopal Netuveli
Acknowledgements The authors thank Professor EF Juniper for permission to use a web-based version of the Asthma Control Questionnaire.
Conflicts of interest The authors declare that they have no conflicts of interest in relation to this article.
Contributorship PH, ET and RL contributed to the data collection and analysis. JBS, GR, TS and JS contributed to the study design and planning. PH wrote the first draft and all authors contributed to the interpretation of the data and the writing of the manuscript.
Funding This study was funded by the Netherlands Organisation for Health Research and Development (ZON-MW, Health Care Efficiency Research, sub-programme Effects and Costs 80-82310-98-8627), and by the Netherlands Asthma Foundation (NAF 3.4.07.044).
1. National Heart Lung and Blood Institute. National Asthma Education and Prevention Program (NAEPP) Expert Panel Report 3. Guidelines for the diagnosis and management of asthma. 2007.
2. Global Initiative for Asthma (GINA). Global strategy for asthma management and prevention. 2006.
3. Juniper EF, O'Byrne PM, Guyatt GH, Ferrie PJ, King DR. Development and validation of a questionnaire to measure asthma control. Eur Respir J 1999;14:902-07. http://dx.doi.org/10.1034/j.1399-3003.1999.14d29.x
4. The Dutch General Practice Society (NHG) guideline. Asthma in adults. 2007.
5. Araújo L, Jacinto T, Moreira A, et al. Clinical efficacy of web-based versus standard asthma self-management. J Investig Allergol Clin Immunol 2012;22:28-34.
6. Van der Meer V, Bakker MJ, Van Den Hout WB, et al. Internet-based self-management plus education compared with usual care in asthma: a randomized trial. Ann Intern Med 2009;151:110-20. http://dx.doi.org/10.7326/0003-4819-151-2-200907210-00008
7. Juniper EF, Langlands JM, Juniper BA. Patients may respond differently to paper and electronic versions of the same questionnaires. Respir Med 2009;103:932-4. http://dx.doi.org/10.1016/j.rmed.2008.10.019
8. Pinnock H, Juniper EF. Concordance between supervised and postal administration of the Mini Asthma Quality of Life Questionnaire (MiniAQLQ) and Asthma Control Questionnaire (ACQ) was very high. J Clin Epidemiol 2005;58:809-14. http://dx.doi.org/10.1016/j.jclinepi.2005.01.010
9. Honkoop PJ, Loymans RJ, Termeer EH, et al. Asthma control cost-utility randomized trial evaluation (ACCURATE): the goals of asthma treatment. BMC Pulm Med 2011;53. http://dx.doi.org/10.1186/1471-2466-11-53
10. Fitzpatrick R, Davey C, Buxton MJ, Jones DR. Evaluating patient-based outcome measures for use in clinical trials. Health Technol Assess 1998;2:1-74.
11. Juniper EF, Svensson K, Mörk AC, Stahl E. Measurement properties and interpretation of three shortened versions of the Asthma Control Questionnaire. Respir Med 2005;99:553-8. http://dx.doi.org/10.1016/j.rmed.2004.10.008
12. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1(8476):307-10. http://dx.doi.org/10.1016/j.ijnurstu.2009.10.001
13. Pinnock H, Burton C, Campbell S, et al. Clinical implications of the Royal College of Physicians three questions in routine asthma care: a real-life validation study. Prim Care Respir J 2012;21:288-94. http://dx.doi.org/10.4104/pcrj.2012.00052
14. Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health 2005;27:281-91. http://dx.doi.org/10.1093/pubmed/fdi031
15. McDonald VM, Higgins I, Simpson JL, Gibson PG. The importance of clinical management problems in older people with COPD and asthma: do patients and physicians agree? Prim Care Respir J 2011;20:389-95. http://dx.doi.org/10.4104/pcrj.2011.00025
16. Richman W, Weisband S, Kiesler S, Drasgow F. A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires and interviews. J Appl Psychol 1999;84:754-75. http://dx.doi.org/10.1037/0021-9010.84.5.754
17. McCue P, Buchanan T, Martin CR. Screening for psychological distress using internet administration of the Hospital Anxiety and Depression Scale (HADS) in individuals with chronic fatigue syndrome. Br J Clin Psychol 2006;45:483-98. http://dx.doi.org/10.1348/014466505X82379
18. Whitehead L. Methodological issues in internet-mediated research: a randomized comparison of internet versus mailed questionnaires. J Med Internet Res 2011;13:e109. http://dx.doi.org/10.2196/jmir.1593
19. Holländare F, Andersson G, Engström I. A comparison of psychometric properties between internet and paper versions of two depression instruments (BDI-II and MADRS-S) administered to clinic patients. J Med Internet Res 2010;12:e49. http://dx.doi.org/10.2196/jmir.1392