Summary
Background
In epidemiologic research, incidence is often estimated from data arising from an imperfect diagnostic test performed at unequally spaced intervals over time.
Methods
We developed a likelihood-based method to estimate incidence when disease status is measured imperfectly and assays are performed at multiple unequally spaced visits. We assumed conditional independence, no remission, known constant levels of sensitivity and specificity, and constant incidence rates over time. The method performance was evaluated by examining its bias, accuracy (i.e., mean squared error (MSE)), and coverage probability in a simulation study of 4000 datasets, and then we applied the proposed method to a study of hepatitis C virus (HCV) infection in a cohort of pregnant women in the period 1997–2006.
Results
The simulation revealed that our method has minimal bias and low MSE, as well as good coverage probability of the resulting confidence intervals. In the application to HCV study, the standard incidence rate estimate which ignores the imperfections of the diagnostic test (number of events/person-years), was 13.7 new HCV cases per 1000 person-years (95% confidence interval 10.1, 17.4). The adjusted incidence estimates (obtained using our proposed method) ranged from 0.4 cases per 1000 person-years (when sensitivity and specificity were assumed to both be 95%) to 13.7 cases per 1000 person-years (when sensitivity and specificity were both 100%). The magnitude of difference between standard and adjusted estimates varied depending on specificity and sensitivity assumptions. Specificity had the greatest impact on the magnitude of bias.
Conclusions
Scientists should be aware of the impact of misclassification on incidence estimates. Appropriate study design, proper selection of the diagnostic test, and adjustment for misclassification probabilities in the analysis is necessary to obtain the most accurate incidence estimates.
Keywords
1. Introduction
Determining disease incidence is important for measuring disease burden, identifying risk factors, assessing prevention efforts, and informing policy. The standard approach to estimating disease incidence by following a cohort, calculating the number of events and dividing it by the person-time at risk may be complicated by two problems: (1) misclassification of disease status if measured using an imperfect diagnostic test, and (2) uncertainty regarding the exact timing of disease acquisition if disease status is measured only periodically. A number of methods have been proposed to handle one or both of these complications,
1
, 2
, 3
, 4
, 5
including an algebraic approach for adjusting cumulative incidence5
and more sophisticated models such as hidden Markov models (HMM).1
In the HMM approach adapted by Bureau et al.1
and by Smith and Vounatsou,2
disease status was modeled as hidden states of a continuous-time process, and the diagnostic test provided imperfect measurements of disease state. However, performance and interpretation of HMM is highly dependent on the degree to which the assumptions of the model are fulfilled.1
The aforementioned problem of incidence estimation in the presence of an imperfect diagnostic assay for disease status is exemplified by epidemiologic studies of hepatitis C virus (HCV). HCV infection is a chronic and silent killer estimated to affect about 200 million people worldwide
6
and approximately 4 million people in the USA alone, who have a 2.37 times higher all-cause mortality rate ratio compared to the non-HCV-infected.7
, 8
Egypt is estimated to have the highest prevalence of HCV in the world with approximately 15% of the population infected at some time in the past and 8 million people with chronic infection.9
High incidence rates were found among children born to HCV-infected mothers,10
spouses of the HCV-infected,11
and household contacts of infected persons living in rural villages.12
, 13
These previous estimates of disease incidence might be inaccurate due to misclassification. The reliability (sensitivity, specificity, negative and positive predictive values) of HCV diagnostic assays has been found to vary greatly depending on the population studied, HCV prevalence, the laboratory that performs the assay, and the type of assay used.14
, 15
, 16
Due to differences in the laboratory methods used and the varied reliabilities of the HCV assays, comparing inferences from different study results are somewhat difficult.14
, 15
, 16
In this paper, we describe a likelihood-based method to estimate disease incidence using an imperfect diagnostic test performed at multiple unequally spaced visits. We apply our proposed method to a community-based study of HCV seroincidence in a cohort of pregnant women. We include comparisons to standard estimates of the incidence rate.
2. Methods
2.1 Misclassification-adjusted incidence rate estimation
We developed a likelihood-based method to estimate incidence when disease status is measured imperfectly and assays are performed at multiple unequally spaced visits. We assumed: (1) conditional independence: i.e., conditional on the true disease status, test results obtained from the same person at different time points are independent; (2) no remission: i.e., no probability of spontaneously reverting from established chronic disease status to non-disease status; (3) levels of sensitivity and specificity are assumed to be known constants (rather than parameters to estimate); and (4) incidence rates are constant over time. Our proposed method for incidence estimation is an adaptation of the HMM method described by Bureau et al.
1
Our method can adjust for differential misclassification via the incorporation of different misclassification probabilities for different subjects, or at different time points. Allowing for differential misclassification is relevant to HCV studies because of change in the testing method over time and the subsequent change in the sensitivity and specificity of the assay. We developed R code, which is available upon request, to implement the method.The likelihood function can be constructed based on these assumptions (Supplementary Material). We treated disease status as missing data, and used an expectation-maximization (EM) algorithm to maximize the likelihood. Variances of the estimates can be estimated in a standard way based on the observed information matrix.
17
EM involves using the standard formula for incidence rate (incident cases/person-time) but replaces it with E (incident cases)/E (person-time) where E(.) denotes expected value, and the expected values are estimated based on the probability that each person is an incident case (or not). The expected number of incident cases can be calculated based on the probability that each person is an incident case given the person's test results, and given the current estimates of incidence and assumed values of sensitivity and specificity. For example, if a person was measured at two time points, given the assumption of no remission, there are three underlying ‘hidden’ possibilities: (1) the person was uninfected at both time points, (2) the person was uninfected at the first time point and was infected at the second time point (an incident case), or (3) the person was infected at both time points (a chronically infected prevalent case). Given the person's test results at each time point we can calculate the probability that the person is truly in each of those hidden states, and then use those probabilities to calculate the value of E (incident cases). In addition, we can estimate the expected time of follow-up for each person (including the expected time of seroconversion given the possibility that the person was an incident case) and use this to calculate E (person-time). These calculations depend on the current estimate of the incidence rate, and result in a revised estimate of incidence rate. Therefore, this process needs to be iterated until it converges to a stable estimate of incidence. The detailed calculation is provided in the Supplementary Material. In order to obtain the 95% confidence interval (CI) we computed the approximated variance/covariance of the adjusted incidence and prevalence estimates using a standard likelihood approach based on the inverse of the information matrix (Supplementary Material).2.2 Application of the proposed method to simulated datasets
To examine the characteristics of the calculated adjusted estimate in finite samples, given various incidence, sensitivity and specificity assumptions, we simulated 4000 data sets (500 for each of eight scenarios).The scenarios differed by sample size (1000 vs. 5000), true incident rate (0.01 vs. 0.10), and misclassification rates (95% sensitivity and specificity vs. 80% sensitivity and specificity). Misclassified disease status was simulated from Bernoulli distribution, conditioned on true disease status and sensitivity and specificity assumptions. We assessed the performance of our method by evaluating the proposed method bias, accuracy, and coverage.
18
For each scenario, the average adjusted estimate was calculated. The bias was defined as the difference between the average adjusted estimate and the simulated estimate IRsim. The amount of bias that exceeded 2SE() was considered problematic.18
Accuracy was assessed by the mean squared error (MSE), a measure of both the bias and the variability. MSE was defined as the average squared deviation between estimate and truth, i.e., ()2/number of simulations. The lower the MSE the better the performance of the method.18
To further evaluate our method we calculated coverage probability of the resulting confidence intervals, defined as the probability that the confidence interval contains the underlying true incidence parameter IRsim. A probability coverage would be deemed reasonable if the coverage falls within approximately two SEs of the nominal coverage probability (p), where SE(p) = √(p(1 − p)/number of simulations).19
Therefore, for a 95% confidence interval based on 500 independent simulations, a coverage probability between 93.05% and 96.95% was acceptable.2.3 Application of the proposed method to the HCV incidence study
We applied our method to a longitudinal prospective cohort study of HCV incidence in three rural villages in Menoufia Governorate in the Nile Delta in Egypt. The study enrolled 3410 pregnant women attending community health centers for prenatal care who consented to participate. All women were first seen during their second or third trimester of pregnancy and asked to return at 2 months postpartum and yearly thereafter until their child was 5 years old. Information was gathered regarding socio-demographic characteristics, as well as their risk factors for acquiring HCV infection, and HCV antibodies (anti-HCV) status was assessed at each visit.
The standard (naïve) incidence estimate, , of HCV among pregnant women was calculated by dividing the number of new apparent HCV cases (determined by the imperfect assay) over the total person-time at risk. Thus, is the maximum likelihood estimator when the number of disease cases follows a Poisson distribution. For women seroconverters, we used the midpoint of the interval between laboratory assays as time of seroconversion for the person-time calculation. The adjusted incidence estimate of HCV among pregnant women, , was calculated as described above, using a range of different values of sensitivity and specificity.
3. Results
3.1 Simulation study
Table 1 presents the results of the simulation study. Bias estimates were within acceptable range, such that none of the bias estimates exceeded two times the SE (). For a simulated rate of 0.01, the bias ranged from 6.94E−05 to 1.06E−03 (from approximately 1% to 10% of the true incidence). The least bias was observed in simulations of larger sample size, and with best sensitivity and specificity assumption. The highest bias was for the simulation with the smaller sample size and the worst sensitivity and specificity assumption. For a higher simulated rate of 0.1, the bias was less than 1.2% of the true incidence for all scenarios, and was the least for bigger sample size and the best sensitivity and specificity assumption. MSE ranged from 2.30E−06 to 7.19E−05 for the simulation scenarios of sample size 1000 and 2.27E−06 to 1.70E−05 for the simulation scenarios of sample size 5000. In particular, lower MSE was observed for simulations with larger sample sizes and with better sensitivity and specificity. Coverage probabilities ranged from 95.0% to 96.4% for the simulation scenarios of sample size 1000 and 93.2% to 96.6% for the simulation scenarios of sample size 5000. In general, coverage probabilities increased with larger sample size and with better sensitivity and specificity. However, when there was less variability in the calculated estimate, the coverage probability was lower for larger sample size.
Table 1Results and performance of the method in a simulation study of 4000 studies with different rates, sensitivity, specificity and sample size scenarios
| Simulated incidence rate | Sensitivity and specificity | Sample size – 1000 | Sample size – 5000 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Mean estimated incidence rate | Bias | MSE | CI coverage | Mean estimated incidence rate | Bias | MSE | CI coverage | ||
| 0.01 | 95% and 95% | 0.0101 | 9.64E−05 | 2.30E−06 | 95.8 | 0.0101 | 6.94E−05 | 2.27E−06 | 96.6 |
| 80% and 80% | 0.0111 | 1.06E−03 | 1.12E−05 | 96.4 | 0.0109 | 9.37E−04 | 2.75E−06 | 93.2 | |
| 0.1 | 95% and 95% | 0.1005 | 5.43E−04 | 2.71E−05 | 95.0 | 0.1001 | 1.25E−04 | 5.21E−06 | 96.6 |
| 80% and 80% | 0.1010 | 9.79E−04 | 7.19E−05 | 96.0 | 0.1012 | 1.23E−03 | 1.70E−05 | 93.2 | |
MSE, mean squared error; CI, confidence interval.
3.2 Standard (naïve) HCV seroincidence rate estimation
Among 2801 mothers at risk, 55 became anti-HCV positive after 4004.3 person-years of follow-up. Thus the standard incidence rate estimate; , was 13.7 new HCV cases per 1000 person-years (95% CI 10.1, 17.4)
3.3 Misclassification-adjusted incidence analysis: a sensitivity analysis approach
To perform a sensitivity analysis, we calculated incidence assuming that both assay sensitivity and specificity ranged from 0.95 to 1.0.
20
We ran the computer algorithm using different sets of initial values and the estimates remained the same.When sensitivity and specificity were both assumed to be 100%, then the proposed estimator is almost identical to , because the assumption is tantamount to assuming that there is no misclassification. The only difference is the fact that the adjusted estimate does not assume that seroconversion occurs at the midpoint of an interval, but is based on an estimate of the expected time of seroconversion. This difference has little effect when incidence is low.
However, as illustrated in Table 2, minimal declines in specificity and sensitivity caused sizeable change in the adjusted incidence estimate; for example, when sensitivity and specificity were both assumed to equal 99% at all time points, was 5.0 per 1000 person-years (95% CI 2.2, 7.8), which was 36% of , with a ratio () of 2.7.
Table 2Sensitivity analysis of the impact of various misclassification assumptions on adjusted incidence rate estimates
| Sensitivity | Specificity | Adjusted incidence | Difference | Ratio |
|---|---|---|---|---|
| 100% | 100% | 13.7 | 0 | 1 |
| 100% | 99.8% | 10.2 | 3.5 | 1.35 |
| 100% | 99% | 6.2 | 7.5 | 2.20 |
| 100% | 98% | 4.6 | 9.1 | 2.97 |
| 100% | 97% | 3.8 | 9.9 | 3.58 |
| 100% | 96% | 3.2 | 10.5 | 4.27 |
| 100% | 95% | 2.7 | 11.0 | 5.09 |
| 99% | 100% | 12.0 | 1.7 | 1.14 |
| 99% | 99.8% | 8.7 | 5.0 | 1.57 |
| 99% | 99% | 5.0 | 8.7 | 2.74 |
| 99% | 98% | 3.4 | 10.3 | 4.02 |
| 99% | 97% | 2.6 | 11.1 | 5.32 |
| 99% | 96% | 1.9 | 11.8 | 7.07 |
| 99% | 95% | 1.3 | 12.4 | 10.24 |
| 98% | 100% | 11.2 | 2.5 | 1.23 |
| 98% | 99.8% | 8.0 | 5.7 | 1.70 |
| 98% | 99% | 4.5 | 9.2 | 3.08 |
| 98% | 98% | 3.0 | 10.7 | 4.53 |
| 98% | 97% | 2.3 | 11.4 | 6.02 |
| 98% | 96% | 1.6 | 12.1 | 8.46 |
| 98% | 95% | 1.0 | 12.7 | 13.40 |
| 97% | 100% | 10.6 | 3.1 | 1.30 |
| 97% | 99.8% | 7.7 | 6.0 | 1.78 |
| 97% | 99% | 4.2 | 9.5 | 3.26 |
| 97% | 98% | 2.8 | 10.9 | 4.84 |
| 97% | 97% | 2.1 | 11.6 | 6.56 |
| 97% | 96% | 1.4 | 12.3 | 9.61 |
| 97% | 95% | 0.8 | 12.9 | 16.94 |
| 96% | 100% | 10.1 | 3.6 | 1.35 |
| 96% | 99.8% | 7.4 | 6.3 | 1.85 |
| 96% | 99% | 4.0 | 9.7 | 3.41 |
| 96% | 98% | 2.6 | 11.1 | 5.22 |
| 96% | 97% | 1.9 | 11.8 | 7.22 |
| 96% | 96% | 1.2 | 12.5 | 11.11 |
| 96% | 95% | 0.6 | 13.1 | 22.98 |
| 95% | 100% | 9.6 | 4.1 | 1.42 |
| 95% | 99.8% | 7.2 | 6.5 | 1.91 |
| 95% | 99% | 3.8 | 9.9 | 3.57 |
| 95% | 98% | 2.5 | 11.2 | 5.57 |
| 95% | 97% | 1.7 | 12.0 | 7.90 |
| 95% | 96% | 1.0 | 12.7 | 13.15 |
| 95% | 95% | 0.4 | 13.3 | 33.03 |
a Number of cases per 1000 person-years.
b Difference = naïve unadjusted incidence (i.e., 13.7 cases per 1000 person-years) − adjusted incidence.
c Ratio = naïve unadjusted incidence/adjusted incidence.
Similarly, given reasonably good sensitivity (95%) and specificity (95%), was 0.4/1000 person-years (95% CI 0.0, 2.9), which was 2.9% of , with a ratio () of 33.0. In general, as presumed sensitivity or specificity declined, decreased (Table 2) and the difference increased (Table 2). It was also noted that in our study of HCV incidence, where the incidence was fairly low, the magnitude of the effect of imperfect specificity on was much higher than the magnitude of imperfect sensitivity. For example, in the situation of perfect sensitivity and 99% specificity, was 6.2/1000 person-years (95% CI 3.3, 9.1; ratio = 2.2), while in the situation of perfect specificity and 99% sensitivity, was 12.0/1000 person-years (95% CI 8.4, 15.6; ratio = 1.1).
Of note, did not meaningfully differ when the algorithm was run using different starting values. For instance for a sensitivity and specificity of 95%, ranged from 0.415 to 0.406 per 1000 person-years when the initial starting points ranged from 0.0137 to 0.5.
4. Discussion
Reports of disease incidence can have important implications for policy, costs, and health decisions. Thus determining incidence with the greatest possible accuracy is crucial. Unfortunately, incidence estimation is not only a function of case definition but also of the reliability of the assay used.
Our results suggest that small imperfections in the assay can have large effects on estimates of incidence. This was exemplified by our findings of a minimum HCV incidence of 0.4 cases per 1000 person-years (when sensitivity and specificity were both assumed to be 95%) and a maximum incidence of 13.7 cases per 1000 person-years (when sensitivity and specificity were both 100%). This big difference in estimate of disease burden might erroneously lead to inappropriate policies and allocation of funds.
One strategy to reduce this problem is to attempt to increase the accuracy of identification using more expensive or frequent assessments, or exclude some observations that were based on less-accurate diagnostic tests. However, this may be costly, time-consuming, and might lead to attrition, loss of some available information, or categorizing some cases as indeterminate.
21
, 22
, 23
, 24
Our proposed method adjusts for misclassification and has the advantage of allowing inclusion of all available information, thus reducing the need for using strict case definitions or extra confirmatory laboratory testing, and thus can be less costly.As our results revealed, accounting for possible misclassification can sometimes result in dramatically different estimates. We detected a higher impact of specificity compared with sensitivity on the magnitude of bias in the HCV study. Generally, previous reports of rare diseases have also shown that imperfect specificity produces results with a greater magnitude of bias than does imperfect sensitivity.
3
, 4
, 5
, 25
We used the EM algorithm to estimate incidence, prevalence, and seroconversion time parameters, an approach that was used previously when outcomes were measured with uncertainty, or with incomplete data.
1
, 4
, 26
, 27
Although one of the drawbacks of the EM algorithm is that it may reach a local maximum, not a global maximum, this is unlikely in our study where we had only two parameters (incidence and prevalence); thus the likelihood is believed to be uni-modal. Furthermore, running the EM algorithm with different initial values resulted in the same estimates. In addition, our simulation results revealed that our method has minimal bias and low MSE, as well as good coverage probability of the resulting confidence interval.Although our method is useful in obtaining a misclassification-adjusted incidence estimate, it has some limitations. First, we assumed the absence of spontaneous remission, i.e., once a person has disease he/she continues to have disease or some marker of disease. This assumption is applicable to many diseases, and is thought to apply to HCV seroconversion. However, if remission is to be expected, our method can be extended following the methods in Bureau et al.
1
Another limitation that should be noted is the assumption of conditional independence, which may not be true in all situations. We also assumed that misclassification probabilities (i.e., sensitivity and specificity) are known constants, and we did not provide a way of estimating them. While these probabilities are often not known, the method we described can still be useful to assess the degree to which estimates might be affected by different levels of misclassification. The method can be extended to simultaneously estimate the misclassification probabilities, although non-identifiable problems and instability of estimates may arise.In light of our study, it is important for the researchers to realize the impact of misclassification on their results, and to address this issue in the early stages of study design and during the analysis in an attempt to minimize the resultant bias.
In conclusion, we have developed a relatively simple method to address a frequently overlooked question: what is the best estimate of an incidence rate given the use of diagnostic tests with imperfect sensitivity and specificity? As our study illustrates, even in the presence of near perfect assays, results of incidence studies can be severely biased. To obtain the desired accurate estimates, it is of paramount importance to choose an appropriate diagnostic assay during the study design, and to use proper statistical methods to adjust for misclassification during the analysis.
Conflict of interest: The authors declare no conflict of interests.
Appendix A. Supplementary data
References
- Applications of continuous time hidden Markov models to the study of misclassified disease outcomes.Stat Med. 2003; 22: 441-462
- Estimation of infection and recovery rates for highly polymorphic parasites when detectability is imperfect, using hidden Markov models.Stat Med. 2003; 22: 1709-1724
- Discrete proportional hazards models for mismeasured outcomes.Biometrics. 2003; 59: 947-954
- Product limit estimation for infectious disease data when the diagnostic test for the outcome is measured with uncertainty.Biostatistics. 2000; 1: 341-354
- Estimating disease rates from a diagnostic test.Am J Epidemiol. 1984; 119: 1015-1023
- Global burden of disease (GBD) for hepatitis C.J Clin Pharmacol. 2004; 44: 20-29
- All-cause, liver-related, and non-liver-related mortality among HCV-infected individuals in the general US population.Clin Infect Dis. 2011; 53: 150-157
- The burden of hepatitis C in the United States.Hepatology. 2002; 36: S30-S34
Fatma E, Way A. Egypt demographic and health survey 2008. Cairo, Egypt: Ministry of Health, El-Zanaty and Associates, and Macro International; 2009.
- Prospective cohort study of mother-to-infant infection and clearance of hepatitis C in rural Egyptian villages.J Med Virol. 2009; 81: 1024-1031
- Estimation of the risk of transmission of hepatitis C between spouses in Egypt based on seroprevalence data.Int J Epidemiol. 2005; 34: 160-165
- Intrafamilial transmission of hepatitis C in Egypt.Hepatology. 2005; 42: 683-687
- Transmission of hepatitis C virus between parents and children.Am J Trop Med Hyg. 2006; 75: 16-20
- Analysis of sera indeterminate by Ortho-HCV RIBA-2 by using three confirmatory assays for anti-hepatitis C virus antibody.J Clin Microbiol. 1994; 32: 2071-2075
- Study on reliability of commercially available hepatitis C virus antibody tests.J Clin Microbiol. 1995; 33: 620-624
- Anti-HCV commercial panel data (update 1). Comparative performance data for anti-HCV assays using commercial seroconversion and other panels. MDA 02029.Medical Devices Agency, London2002
- Finding the observed information matrix when using the EM algorithm.J R Stat Soc Series B Stat Methodol. 1982; 44: 226-233
- The design of simulation studies in medical statistics.Stat Med. 2006; 25: 4279-4292
- A comparison of imputation methods in a longitudinal randomized clinical trial.Stat Med. 2005; 24: 2111-2128
- Comparison of second- and third-generation enzyme immunoassays for detecting antibodies to hepatitis C virus.J Clin Microbiol. 2002; 40: 1656-1659
- Increase in HCV incidence in HIV-1 infected patients followed since primary infection.Sex Transm Infect. 2006; 82: 458-460
- [Incidence and risk factors of HCV infection in a cohort of intravenous drug users in the North and East of France].Rev Epidemiol Sante Publique. 2006; 54 (1S15–1S22)
- Incidence of HCV infection in French hemodialysis units: a prospective study.J Med Virol. 2005; 77: 70-76
- Risk of hepatitis C virus (HCV) transmission by anti-HCV-negative blood components in Austria and Germany.Ann Hematol. 1996; 72: 35-39
- Bias due to misclassification in the estimation of relative risk.Am J Epidemiol. 1977; 105: 488-495
- Logistic regression when the outcome is measured with uncertainty.Am J Epidemiol. 1997; 146: 195-203
- Monitoring epidemiologic surveillance data using hidden Markov models.Stat Med. 1999; 18: 3463-3478
Article info
Publication history
Published online: April 30, 2012
Accepted:
February 28,
2012
Received in revised form:
February 20,
2012
Received:
June 28,
2011
Corresponding Editor: Hubert Wong, Vancouver, CanadaIdentification
Copyright
© 2012 International Society for Infectious Diseases. Published by Elsevier Inc.
User license
Elsevier user license | How you can reuse 
Elsevier's open access license policy
Elsevier user license
Permitted
For non-commercial purposes:
- Read, print & download
- Text & data mine
- Translate the article
Not Permitted
- Reuse portions or extracts from the article in other works
- Redistribute or republish the final article
- Sell or re-use for commercial purposes
Elsevier's open access license policy
