If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
SARS-CoV-2 rapid antigen tests provide fast identification of infectious patients
•
Prediction models for identification of false negative test results were developed
•
One investigated clinical model reached an area under the curve of 0.971
•
Prediction models can be routinely applied in the healthcare sector
•
The prevention of nosocomial infections positively influences the pandemic's course
Abstract
Objectives
SARS-CoV-2 rapid antigen tests (RAT) provide fast identification of infectious patients when RT-PCR results are not immediately available. We aimed to develop a prediction model for identification of false negative (FN) RAT results.
Methods
In this multicenter trial, patients with documented paired results of RAT and RT-PCR between October 1st 2020 and January 31st 2021 were retrospectively analyzed regarding clinical findings. Variables included demographics, laboratory values and specific symptoms. Three different models were evaluated using Bayesian logistic regression.
Results
The initial dataset contained 4,076 patients. Overall sensitivity and specificity of RAT was 62.3% and 97.6%. 2,997 cases with negative RAT results (FN: 120; true negative: 2,877; reference: RT-PCR) underwent further evaluation after removal of cases with missing data. The best-performing model for predicting FN RAT results containing 10 variables yielded an area under the curve of 0.971. Sensitivity, specificity, PPV and NPV for 0.09 as cut-off value (probability for FN RAT) were 0.85, 0.99, 0.7 and 0.99.
Conclusion
FN RAT results can be accurately identified through ten routinely available variables. Implementation of a prediction model in addition to RAT testing in clinical care can provide decision guidance for initiating appropriate hygiene measures and therefore helps avoiding nosocomial infections.
), the coronavirus disease 2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), emerged to a major burden for the population in general but especially brought great challenges for the healthcare sector (
). Recent statistics presented by the German federal government agency Robert-Koch-Institute (RKI) show that cough, fever, nasal congestion, sore throat and loss of smell or taste are the most common symptoms caused by COVID-19 (
. Other symptoms like dyspnea, fatigue, myalgia and gastrointestinal symptoms (e.g., diarrhea, nausea) were frequently reported among studies in the early phase of the pandemic (
The gold standard for the detection of SARS-CoV-2 is the nucleic acid amplification in a nasopharyngeal sample by real-time reverse transcription polymerase chain reaction (RT-PCR) according to German (
AWMF. Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 - Living Guideline; 2021. Available from: https://www.awmf.org/leitlinien/detail/ll/113-001LG.html. [Accessed May 20, 2021].
) guidelines. However, because RT-PCR is an expensive and time-consuming technique, numerous rapid antigen tests (RAT) for point-of-care (PoC)-testing were developed during the past months and approved by federal agencies (
EC-DG. European Commission Directorate-General For Health And Food Safety (EC-DG SANTE): A common list of COVID-19 rapid antigen tests, including those of which their test results are mutually recognised, and a common standardised set of data to be included in COVID-19 test result certificates; 2021. Available from: https://ec.europa.eu/health/sites/default/files/preparedness_response/docs/covid-19_rat_common-list_en.pdf. [Accessed May 21, 2021].
). Those tests are easily applicable, cost-efficient, provide fast test results and do not require specifically trained personnel when compared to RT-PCR (
). Authors of a recent meta-analysis report pooled sensitivity and specificity estimates of 73.1% and 99.7% respectively for different widely used RATs including FIA-based tests (
Brümmer LE, Katzenschlager S, Gaeddert M, Erdmann C, Schmitz S, Bota M, et al. The accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis. medRxiv 2021:2021.02.26.21252546.
. In the healthcare sector, PoC-testing can provide a huge benefit for the fast identification of infectious patients and initiation of appropriate measures like consecutive isolation. Confirmation of positive RAT test results with RT-PCR is required since false positive tests are possible (
). Every inpatient should be tested by RT-PCR at admission in addition to PoC-testing to prevent nosocomial SARS-CoV-2 infections according to German guidelines (
AWMF. Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 - Living Guideline; 2021. Available from: https://www.awmf.org/leitlinien/detail/ll/113-001LG.html. [Accessed May 20, 2021].
). To date, no standardized approach exists to identify SARS-CoV-2 patients using clinical parameters, especially in the context of a negative PoC-test. However, false negative (FN) RAT results can have a considerably great impact on the healthcare sector because isolation and appropriate hygiene measures would be possibly not initiated when the patient is initially tested negative. Consecutively, this may also influence the course of the pandemic. The aim of the present study was therefore to identify different clinical predictors for FN results of RATs, determined by positive RT-PCR as a reference standard, in a real-world multicenter patient cohort and to develop a corresponding prediction model.
Methods
We conducted a multicenter trial in three centers of the German Elblandkliniken GmbH group. The local ethics committee has not raised any objections to the collection and further analyses of data as part of the study (EK-BR-30/21-1). Informed consent has not been obtained due to the retrospective study design.
Characteristics of patients treated in the participating hospitals between October 1st 2020 and January 31st 2021 with documented paired results of RAT and RT-PCR conducted within the first 24 hours from admission were retrospectively collected and analyzed. Specimen collection for both tests was performed with a combined oropharyngeal and bilateral deep nasal swab. For the RAT, Standard F COVID-19 Ag FIA (SD Biosensor Inc., Gyeonggi-do, Republic of Korea) was used for all patients (hereafter called Standard F). According to the manufacturer's instructions (SD
), a Cut-Off-Index (COI) of 1.0 was used for test evaluation with COI ≥ 1 considered a positive result and COI < 1 considered a negative result.
RT-PCR was carried out with the Allplex™ SARS-CoV-2/FluA/FluB/RSV Assay (Seegene, Seoul, Republic of Korea) targeting the N, RdRP and S genes of SARS-CoV-2 (
). Cycle threshold (Ct) values for each gene (N, RdRP, S) were collected and transferred to the study database.
The patients’ symptoms, known COVID-19 contact (14 days prior to hospital admission) and status post COVID-19 were assessed by a physician during the initial medical interview for both regular and emergency room admissions. The following medical history and physical examination variables were considered as relevant for study purposes and consecutively extracted from the hospital information system (HIS):
•
known COVID-19 contact 14 days prior to hospital admission
•
history of fever (defined as body temperature ≥ 38.0 degrees Celsius),
Collected laboratory values included: Standard C-reactive protein (CRP), leucocyte count, platelet count, lactate dehydrogenase (LDH), aspartate aminotransferase (AST). Furthermore, imaging data (X-ray and chest CT), if present, were reviewed for COVID-specific findings.
First, a confusion matrix was created for the whole dataset displaying the RAT and RT-PCR results (RT-PCR results considered as the ground truth). Discrimination metrics (sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV)) were computed. As our primary goal was to create a model predicting the probability of an RAT negative patient being SARS-CoV-2 positive (therefore predicting a FN RAT, as determined by positive RT-PCR), we focused on the proportion of patients with negative RAT results (true negative, TN and false negative, FN) for further analyses.
In a first step, we identified missing values in the dataset of TN and FN patients. Due to the high number of missing values, the medical history variables “known COVID-19 contact” and “loss of smell and taste” were not taken into account for model development as were LDH and AST for laboratory values and all imaging data. The dataset was split in 75%/25% portions for model testing and model training. The sampling was stratified for RT-PCR results (RT-PCR positive/RT-PCR negative ratio was identical in each subset). Remaining laboratory values (CRP, leucocyte and platelet count) were transformed into categorical variables, using the median as a cut-off value (CRP = 7.4 mg/l, leucocytes = 9.4/nl, thrombocytes = 237/nl). We evaluated three different models for prediction of FN RAT results as determined by RT-PCR consisting of two demographic variables, three laboratory values and five symptoms (Figure 1).
With regard to statistical methods, we utilized a Bayesian implementation of logistic regression, using weak priors (i.e. normally distributed prior, with a mean of 0 and a large variance) (
). Models were fitted using four Markov chains (Monte Carlo-algorithm), each with 10,000 warm-up iterations followed by 10,000 sampling iterations. Comparisons between models and for individual parameters were carried out using the Bayes Factor (BF) with appropriate interpretation (Supplementary Table S2) (
). Model performance was further evaluated by computing receiver operating characteristic (ROC) area under the curve (AUC). Pearson's Chi-squared Test and Mann-Whitney-U-Test were used for intergroup comparisons where appropriate. All statistical analyses were carried out using the rstanarm package in the R environment for statistical computing (version 4.0.2).
Results
Overall sensitivity and specificity for the Standard F RAT in the whole dataset containing 4,076 patients was 62.3% and 97.6% respectively (PPV = 91.5%; NPV = 86.0%). The dataset which was used for model development (TN and FN patients with regard to RAT results) contained 2,997 patients (49% male; mean age [SD] = 63.7 [22.9]) after all cases with missing variables were removed (Figure 2). Patient characteristics for the TN and FN group are displayed in Table 1. Variable expressions for the train (n = 2,248) and test (n = 749) dataset were well balanced (Supplementary Table S1). Ct-values for all SARS-CoV-2 genes in the FN group were high (median [IQR] for N, RdRP, S gene: 28.0 [23.9;31.3]; 28.5 [24.7;32.4]; 28.1 [24.3;31.8]).
Table 2 summarizes the estimates for specific variables for all three models. Superior performance could be demonstrated for model 3, which includes all variables, over model 1 and 2 both regarding the direct comparison using the BF (Supplementary Table S3) and by comparing the ROC AUCs (Figure 3). Model 3 displayed an excellent discriminatory performance with regard to AUC (0.971). Calibration plots for the test dataset are provided in the Supplementary Material (Figure S1). Most important variables for model performance with respect to BF were (Table 2): leucocyte count, fever, breathing frequency, dyspnea and musculoskeletal symptoms. The variables “high leucocyte count” and “high platelet count” show a negative estimate which means that COVID-19 in our study was rather associated with leukopenia and thrombocytopenia.
Table 2Estimates of the parameter values with 95% confidence intervals
Variable
Model 1
Model 2
Model 3
Estimates (95%CI)
BF
Estimates (95%CI)
BF
Estimates (95%CI)
BF
Age
0.016 (0.003; 0.029)
1.3
0.013 (-0.005; 0.032)
0.2
0.011 (-0.008; 0.032)
0.2
Female
-0.477 (-0.926; -0.038)
0.4
-0.167 (-0.879; 0.536)
0.1
-0.076 (-0.828; 0.672)
0.1
High CRP
1.337 (0.853; 1.849)
>100
0.672 (-0.08; 1.438)
0.4
High leucocyte count
-1.495 (-2.031; -0.993)
>100
-1.935 (-2.904; -1.054)
>100
High platelet count
-0.27 (-0.733; 0.182)
0.1
-0.37 (-1.141; 0.381)
0.1
Fever (Temperature ≥ 38°C)
2.88 (1.626; 4.261)
>100
2.662 (1.288; 4.156)
67.8
Breathing frequency > 15/min
3.047 (2.155; 3.935)
>100
3.383 (2.471; 4.313)
>100
Dyspnea
3.758 (2.809; 4.729)
>100
3.635 (2.672; 4.651)
>100
Abdominal symptoms
2.162 (0.815; 3.528)
3.6
2.615 (1.253; 3.97)
17.7
Musculoskeletal symptoms
3.278 (1.408; 5.133)
12.2
3.835 (1.913; 5.782)
39.9
95%CI = 95%-confidence interval; BF = Bayes factor; CRP = C-reactive protein; High CRP is defined by values above 7.4 mg/l; High leucocyte count is defined by values above 9.4/nl; High platelet count is defined by values above 237/nl
The probability for a RT-PCR positive result at the time point of the patient's hospital admission can be calculated by using a specific formula derived from the models. As a first step, all the variables used in the model have to be collected and each coefficient (estimates, see Table 2) is then multiplied by the value of the variable. For “Age”, this is the age of the patient given in years and for categorical variables the binary numbers 0 and 1 (no/yes) have to be used. The cut-off for laboratory values has to be applied like defined previously: “High CRP” is defined by values above 7.4 mg/l, “High leucocyte count” is defined by values above 9.4/nl and “High platelet count” is defined by values above 237/nl.
These multiplications are summed, and -5.08 is added, corresponding to the estimate of the intercept. Finally, this sum can be transformed into a probability using the following equation:
Y is the sum defined above, P is the probability, and e the Euler's number.
In order to compute classical discrimination metrics (sensitivity, specificity, PPV, NPV), a cut-off threshold was defined (Youden's index) (
). For model 3, we defined the cut-off threshold as 0.09 (9% probability for positive RT-PCR result) because this index yields a high sensitivity of 85% together with a PPV of 70% (specificity = 0.99; NPV = 0.99; accuracy = 0.98; see Figure 4). Transferred to the clinical perspective and performance of RATs, this threshold, applied on the test dataset, would result in 85% of the patients being classified correctly as FN with subsequent isolation while 30% (1-PPV) are unnecessarily isolated.
Figure 4Discrimination metrics for model 3 (cut-off 0.09)
In a multicenter retrospective study of 4,076 patients, we evaluated clinical predictors for probable SARS-CoV-2 infection and compared different prediction models for positive RT-PCR results in the setting of an initially negative RAT. As a result, we propose a model (model 3) utilizing ten easily collectable and fast available variables, consisting of demographics, symptoms and laboratory parameters, with high accuracy and an AUC of 0.971. As the pandemic is ongoing, application of our model in clinical routine could provide a great benefit for the fast assessment of both inpatients and outpatients regarding their COVID-19 status in addition to PoC-testing by RATs as it is crucial to differentiate between FN and TN results in the healthcare sector in order to initiate appropriate hygiene measures and avoid nosocomial infections. For practical use, the development of, for example, a computerized tool which uses the above mentioned formula to calculate individual probabilities is conceivable. In a hospital setting, the patient-specific variables can be obtained directly at the patient's admission (after performance of RAT and RT-PCR respectively) and entered manually into the calculator. The resulting probability value for present COVID-19 (in patients with negative RAT results) can guide hospital staff on whether an initial isolation of the patient while awaiting RT-PCR results is necessary despite negative RAT. This process could easily be carried out and implemented as a routine measure. Automatic calculators utilizing the HIS as a data source represent another option for real-world application of our prediction models.
Sensitivity of RATs for PoC-testing decreases with increasing Ct-values of RT-PCR (
). Pooled sensitivity for Standard F was reported as 70.9% and specificity as 98.5% in a recent meta-analysis (preprint format) (Brümmer et al., 2021). These numbers can be confirmed by other studies (
) and compare well with our findings. The manufacturer of Standard F indicates a higher sensitivity and specificity of 93.70% and 99.63% respectively (SD
). These differences between trial and real-world data can be explained with bias due to test application and its evaluation under study conditions. Furthermore, RATs are mostly validated for virus detection in symptomatic patients only (
Validation of a rapid antigen test as a screening tool for SARS-CoV-2 infection in asymptomatic populations. Sensitivity, specificity and predictive values.
). Additionally, the sample in the latter trial included patients with high viral loads and only 64 samples were analyzed which limits scalability of the presented results. RT-PCR Ct-values of FN patients in our study were high which is a possible explanation for the observed lower sensitivity.
Lower leukocyte counts as well as lower platelet counts were associated with increased probability of present SARS-CoV-2 infection among our prediction models. These results are concordant with recently published findings. A heterogeneous distribution of leukocytosis as well as leukopenia among COVID-19 patients has been shown in several studies (
Different working groups identified predictors for COVID-19 during the course of the pandemic and developed respective prediction models. Bayat et al. trained and tested a machine learning (ML) model on a large dataset of SARS-CoV-2 positive and negative patients using laboratory values only which reached an accuracy of 86.4% (
). In our opinion, limitations to this study arise regarding a missing reference standard (both RT-PCR and RAT were used for SARS-CoV-2 detection) and the possible unavailability of certain laboratory values (20 parameters were required for the model) which hinders application in clinical routine. Analogously, the use of laboratory parameters paired with ML application for prediction of RT-PCR results was proposed by another working group but the model displayed an AUC of 0.74 only which is considerably lower in comparison to our model integrative of demographics, symptoms and laboratory values (
Regarding utilization of imaging data for predicting RT-PCR results, a Chinese study reported a high sensitivity (97%) of initially performed chest CTs (
). As the CT represents a crucial tool for fast assessment of critically ill patients, it should not be used for screening purposes due to radiation exposure. One integrative prediction model for COVID-19 was presented by Sun et al. in the early phase of the pandemic comprising of demographics, symptoms, physical examination results, imaging data and laboratory values (
). Different models displayed an AUC in range of 0.65-0.91 whereas the best performing model utilized 16 in part highly COVID-specific variables like CT/X-ray findings suspicious for viral pneumonia. Similar model performance (AUC = 0.91) was reported by Ng et al., but again, imaging data besides other variables was required (
). Albeit these studies show that imaging findings are often included as variables for COVID-19 prediction models, there are concerns about the clinical applicability in our opinion because only few patients would receive a CT/X-ray examination directly at hospital admission. This, high costs and the time factor therefore prevents routine application of those models as a screening tool.
The use of ML algorithms for prediction of SARS-CoV-2 infections has been proposed by several authors. Besides the above mentioned work (
), Zoabi et al. presented a model using eight demographic and medical history variables for prediction of positive RT-PCR which yielded a high accuracy (
). Similar results were reported in another study albeit the presented model required a large number of variables and the evaluated case number was low (
Development of machine learning models to predict RT-PCR results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in patients with influenza-like symptoms using only basic clinical data.
Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine.2020; 28: 113
). However, ML-derived prediction of COVID-19 probabilities is a valuable and promising approach which demands further evaluation in future studies using large datasets.
In contrast to the above mentioned studies, we evaluated the specific constellation of RAT negative/RT-PCR positive patients. In a recently published study by Ford et al., a similar approach was pursued but in a real-life and not a hospital setting (
Epidemiologic characteristics associated with SARS-CoV-2 antigen-based test results, rRT-PCR cycle threshold values, subgenomic RNA, and viral culture results from university testing.
). Paired swabs for RAT and RT-PCR were collected on a university campus. A major advantage of this trial was the verification of positive RT-PCR or RAT results by subsequent viral culture. FN results occurred among symptomatic and asymptomatic patients whereas odds were higher for asymptomatic patients and for higher Ct-values. This trial highlights symptom status, also including light symptoms like nasal congestion, as an important predictor of COVID-19 infections which should be taken into account when interpreting RAT results. Another working group identified factors associated with false negative first RT-PCR results in patients with a final diagnosis of COVID-19 (
). Interestingly, higher platelet and leucocyte counts yielded a higher risk and symptoms (e.g. fatigue, fever) a lower risk for a false negative first RT-PCR result which is in accordance with our observations where patient-reported symptoms and lower platelet/leucocyte counts were associated with present SARS-CoV-2 infections.
Limitations
We acknowledge several limitations in connection with this study. First, data collection was performed retrospectively which could have influenced data quality, especially regarding symptom assessment and also resulted in a significant amount of missing values. Furthermore, application to an external dataset did not take place. Prospective evaluation and validation of our prediction model in future studies is required. Second, a large number of RT-PCR positive (RAT false negative) patients were excluded from the analysis prior to model development due to missing values which could result in weaker model performance when the model is applied to other datasets. Third, we removed several variables (LDH, AST, loss of smell and taste, COVID-19 contact) which could have improved the performance. Nevertheless, accuracy and AUC of model 3 were excellent. Forth, we used RT-PCR as a reference standard while sensitivity and specificity of this method is not 100% (
), especially in patients with very low viral loads, but those have a low risk of transmission. The Allplex™ assay used in our study proved to be reliable in previous studies (
) and 3 viral genes were targeted which enhances diagnostic accuracy. Fifth, patients with false negative RAT results are not necessarily infectious as has been stated previously (
) and isolation may not be required but this risk should not be taken in view of a hospital setting since at the time of testing it is unknown in which direction viral load will develop.
Conclusion
We evaluated clinical predictors of FN results of RATs and propose a prediction model consisting of widely available and easily collectable variables. The model showed an excellent discriminatory performance. Its routine implementation in healthcare both in an inpatient and outpatient setting could help identifying patients at high risk for COVID-19 despite negative RAT result and could therefore influence the course of the pandemic with respect to avoiding nosocomial infections.
Author contributions
JL and VP contributed equally to this manuscript in view of study design, data analysis and interpretation and the writing of the manuscript (joint first-authorship). AN, SK, SH, IN, GH, CK, BR, JB, AP, JS, JP, AB, MW contributed substantially to the study design, data analysis and interpretation and revision of the manuscript and gave final approval for publishing.
Conflicts of interest
We declare no conflicts of interest associated with this publication. Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Acknowledgements
Guarantor statement: Mr. Johannes Leiner (JL) takes full responsibility for the content of the manuscript, including the data and analysis.
AWMF. Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 - Living Guideline; 2021. Available from: https://www.awmf.org/leitlinien/detail/ll/113-001LG.html. [Accessed May 20, 2021].
Brümmer LE, Katzenschlager S, Gaeddert M, Erdmann C, Schmitz S, Bota M, et al. The accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis. medRxiv 2021:2021.02.26.21252546.
EC-DG. European Commission Directorate-General For Health And Food Safety (EC-DG SANTE): A common list of COVID-19 rapid antigen tests, including those of which their test results are mutually recognised, and a common standardised set of data to be included in COVID-19 test result certificates; 2021. Available from: https://ec.europa.eu/health/sites/default/files/preparedness_response/docs/covid-19_rat_common-list_en.pdf. [Accessed May 21, 2021].
Validation of a rapid antigen test as a screening tool for SARS-CoV-2 infection in asymptomatic populations. Sensitivity, specificity and predictive values.
Epidemiologic characteristics associated with SARS-CoV-2 antigen-based test results, rRT-PCR cycle threshold values, subgenomic RNA, and viral culture results from university testing.
Development of machine learning models to predict RT-PCR results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in patients with influenza-like symptoms using only basic clinical data.
Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine.2020; 28: 113
00717-7&id=ga1.jpg){kind=link}
00717-7&id=gr1.jpg){kind=link}
00717-7&id=gr2.jpg){kind=link}
00717-7&id=gr3.jpg){kind=link}
00717-7&id=gr4.jpg){kind=link}