Advertisement

SARS-CoV-2 rapid antigen testing in the healthcare sector: A clinical prediction model for identifying false negative results

Open AccessPublished:September 10, 2021DOI:https://doi.org/10.1016/j.ijid.2021.09.008

      Highlights

      • SARS-CoV-2 rapid antigen tests provide fast identification of infectious patients
      • Prediction models for identification of false negative test results were developed
      • One investigated clinical model reached an area under the curve of 0.971
      • Prediction models can be routinely applied in the healthcare sector
      • The prevention of nosocomial infections positively influences the pandemic's course

      Abstract

      Objectives

      SARS-CoV-2 rapid antigen tests (RAT) provide fast identification of infectious patients when RT-PCR results are not immediately available. We aimed to develop a prediction model for identification of false negative (FN) RAT results.

      Methods

      In this multicenter trial, patients with documented paired results of RAT and RT-PCR between October 1st 2020 and January 31st 2021 were retrospectively analyzed regarding clinical findings. Variables included demographics, laboratory values and specific symptoms. Three different models were evaluated using Bayesian logistic regression.

      Results

      The initial dataset contained 4,076 patients. Overall sensitivity and specificity of RAT was 62.3% and 97.6%. 2,997 cases with negative RAT results (FN: 120; true negative: 2,877; reference: RT-PCR) underwent further evaluation after removal of cases with missing data. The best-performing model for predicting FN RAT results containing 10 variables yielded an area under the curve of 0.971. Sensitivity, specificity, PPV and NPV for 0.09 as cut-off value (probability for FN RAT) were 0.85, 0.99, 0.7 and 0.99.

      Conclusion

      FN RAT results can be accurately identified through ten routinely available variables. Implementation of a prediction model in addition to RAT testing in clinical care can provide decision guidance for initiating appropriate hygiene measures and therefore helps avoiding nosocomial infections.

      Graphical Abstract

      Keywords

      Abbreviations:

      AST (Aspartate aminotransferase), AUC (Receiver operating characteristic area under the curve), BF (Bayes Factor), COI (Cut-Off-Index), COVID-19 (Coronavirus disease 2019), CRP (C-reactive protein), Ct (Cycle threshold), CT (Computed tomography), FIA (Fluorescence-immunoassays), FN (False negative), HIS (Hospital information system), ICU (Intensive care unit), LDH (Lactate dehydrogenase), ML (Machine learning), NPV (Negative predictive value), PCR (Polymerase chain reaction), PoC (Point-of-care), PPV (Positive predictive value), RAT (Rapid antigen test), RT-PCR (Reverse transcription polymerase chain reaction), RKI (Robert-Koch-Institute), ROC (Receiver operating characteristic), SARS-CoV-2 (Severe acute respiratory syndrome coronavirus 2), Standard F (Standard F COVID-19 Ag FIA (SD Biosensor Inc.)), TN (True negative), WHO (World Health Organization)

      Introduction

      Since its onset (
      • Zhu N
      • Zhang D
      • Wang W
      • Li X
      • Yang B
      • Song J
      • et al.
      A novel Coronavirus from patients with Pneumonia in China, 2019.
      ), the coronavirus disease 2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), emerged to a major burden for the population in general but especially brought great challenges for the healthcare sector (
      • Miller IF
      • Becker AD
      • Grenfell BT
      • Metcalf CJE.
      Disease and healthcare burden of COVID-19 in the United States.
      ). Recent statistics presented by the German federal government agency Robert-Koch-Institute (RKI) show that cough, fever, nasal congestion, sore throat and loss of smell or taste are the most common symptoms caused by COVID-19 (

      RKI. Robert-Koch-Institut (RKI): Most frequent symptoms caused by the coronavirus (COVID-19) in Germany in 2021. Statista. Statista Inc.; 2021a. Available from: https://www.statista.com/statistics/1105523/coronavirus-covid-19-symptoms-most-frequent-germany/ [Accessed June 15, 2021].

      ). High rates (approximately 50%) of loss of smell or taste among COVID patients were reported in several trials (
      • Tong JY
      • Wong A
      • Zhu D
      • Fastenberg JH
      • Tham T.
      The prevalence of olfactory and gustatory dysfunction in COVID-19 patients: a systematic review and meta-analysis.
      ) but those numbers cannot be confirmed in a real-world setting

      RKI. Robert-Koch-Institut (RKI): Most frequent symptoms caused by the coronavirus (COVID-19) in Germany in 2021. Statista. Statista Inc.; 2021a. Available from: https://www.statista.com/statistics/1105523/coronavirus-covid-19-symptoms-most-frequent-germany/ [Accessed June 15, 2021].

      . Other symptoms like dyspnea, fatigue, myalgia and gastrointestinal symptoms (e.g., diarrhea, nausea) were frequently reported among studies in the early phase of the pandemic (
      • Manoharan L
      • Cattrall JWS
      • Harris C
      • Newell K
      • Thomson B
      • Pritchard MG
      • et al.
      Evaluating clinical characteristics studies produced early in the Covid-19 pandemic: A systematic review.
      ).
      The gold standard for the detection of SARS-CoV-2 is the nucleic acid amplification in a nasopharyngeal sample by real-time reverse transcription polymerase chain reaction (RT-PCR) according to German (

      AWMF. Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 - Living Guideline; 2021. Available from: https://www.awmf.org/leitlinien/detail/ll/113-001LG.html. [Accessed May 20, 2021].

      ) and US (

      NIH. National Institutes of Health (NIH): COVID-19 Treatment Guidelines; 2021. Available from: https://www.covid19treatmentguidelines.nih.gov/overview/. [Accessed May 20, 2021].

      ) guidelines. However, because RT-PCR is an expensive and time-consuming technique, numerous rapid antigen tests (RAT) for point-of-care (PoC)-testing were developed during the past months and approved by federal agencies (

      BfArM. Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM): Antigen-Tests zum direkten Erregernachweis des Coronavirus SARS-CoV-2; 2021. Available from: https://antigentest.bfarm.de/ords/f?p=110:100:16064132278745:::::&tz=2:00. [Accessed May 21, 2021].

      ,

      EC-DG. European Commission Directorate-General For Health And Food Safety (EC-DG SANTE): A common list of COVID-19 rapid antigen tests, including those of which their test results are mutually recognised, and a common standardised set of data to be included in COVID-19 test result certificates; 2021. Available from: https://ec.europa.eu/health/sites/default/files/preparedness_response/docs/covid-19_rat_common-list_en.pdf. [Accessed May 21, 2021].

      ). RATs for qualitative detection of SARS-CoV-2 are mostly based on lateral flow immunochromatography or fluorescence-immunoassays (FIA) (
      • Porte L
      • Legarraga P
      • Iruretagoyena M
      • Vollrath V
      • Pizarro G
      • Munita J
      • et al.
      Evaluation of two fluorescence immunoassays for the rapid detection of SARS-CoV-2 antigen-new tool to detect infective COVID-19 patients.
      ). Those tests are easily applicable, cost-efficient, provide fast test results and do not require specifically trained personnel when compared to RT-PCR (
      • Hirotsu Y
      • Maejima M
      • Shibusawa M
      • Amemiya K
      • Nagakubo Y
      • Hosaka K
      • et al.
      Prospective study of 1308 nasopharyngeal swabs from 1033 patients using the LUMIPULSE SARS-CoV-2 antigen test: Comparison with RT-qPCR.
      ,
      • Paul G PT
      • Sethi S
      • Schilling T
      • Wienand O
      • Jürgensen J S
      • Menzel C U
      Klinische Performance eines neuen SARS-CoV-2-AntigenTests in der Notaufnahme eines Maximalversorgers.
      ). Sensitivity and specificity of available RATs vary between manufacturers (
      • Dinnes J
      • Deeks JJ
      • Berhane S
      • Taylor M
      • Adriano A
      • Davenport C
      • et al.
      Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.
      ) but overall accuracy is commonly acknowledged as inferior in comparison to RT-PCR (
      • Corman VM
      • Haage VC
      • Bleicker T
      • Schmidt ML
      • Mühlemann B
      • Zuchowski M
      • et al.
      Comparison of seven commercial SARS-CoV-2 rapid point-of-care antigen tests: a single-centre laboratory evaluation study.
      ,
      • Mak GC
      • Cheng PK
      • Lau SS
      • Wong KK
      • Lau C
      • Lam ET
      • et al.
      Evaluation of rapid antigen test for detection of SARS-CoV-2 virus.
      ). Authors of a recent meta-analysis report pooled sensitivity and specificity estimates of 73.1% and 99.7% respectively for different widely used RATs including FIA-based tests (

      Brümmer LE, Katzenschlager S, Gaeddert M, Erdmann C, Schmitz S, Bota M, et al. The accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis. medRxiv 2021:2021.02.26.21252546.

      ). Sensitivity decreases markedly when asymptomatic persons are tested (
      • Dinnes J
      • Deeks JJ
      • Berhane S
      • Taylor M
      • Adriano A
      • Davenport C
      • et al.
      Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.
      ). PoC-testing is an important component of the national strategy for pandemic response in Germany and therefore widely implemented

      RKI. Robert-Koch-Institut (RKI): Nationale Teststrategie – wer wird in Deutschland auf das Vorliegen einer SARS-CoV-2 Infektion getestet?; 2021b. Available from: https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Teststrategie/Nat-Teststrat.html;jsessionid=82E51EEE7744B4D6DFE4B229ACB2EE65.internet082?nn=13490888#doc14279904bodyText5. [Accessed May 25, 2021].

      . In the healthcare sector, PoC-testing can provide a huge benefit for the fast identification of infectious patients and initiation of appropriate measures like consecutive isolation. Confirmation of positive RAT test results with RT-PCR is required since false positive tests are possible (
      • Seifried J BS
      • von Kleist M
      • Jenny MA
      • Antão E OD
      • Jung-Sendzik T
      • Broich K
      • Denkinger C
      • Bartenschlager T SL
      • Hamouda O
      • Mielke M
      Antigentests als ergänzendes Instrument in der Pandemiebekämpfung.
      ,

      WHO. World Health Organization (WHO): Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays - interim guidance 2021. Available from: https://www.who.int/publications/i/item/antigen-detection-in-the-diagnosis-of-sars-cov-2infection-using-rapid-immunoassays. [Accessed May 25, 2021].

      ). Every inpatient should be tested by RT-PCR at admission in addition to PoC-testing to prevent nosocomial SARS-CoV-2 infections according to German guidelines (

      AWMF. Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 - Living Guideline; 2021. Available from: https://www.awmf.org/leitlinien/detail/ll/113-001LG.html. [Accessed May 20, 2021].

      ).
      Clinical predictors of probable SARS-CoV-2 infections have been identified in several studies. Proposed prediction models include laboratory values (
      • Bayat V
      • Phelps S
      • Ryono R
      • Lee C
      • Parekh H
      • Mewton J
      • et al.
      A severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) prediction model from standard laboratory tests.
      ,
      • Tschoellitsch T
      • Dünser M
      • Böck C
      • Schwarzbauer K
      • Meier J.
      Machine learning prediction of SARS-CoV-2 polymerase chain reaction results with routine blood tests.
      ), symptoms and demographic data (
      • Zoabi Y
      • Deri-Rozov S
      • Shomron N.
      Machine learning-based prediction of COVID-19 diagnosis based on symptoms.
      ), imaging data (
      • Ai T
      • Yang Z
      • Hou H
      • Zhan C
      • Chen C
      • Lv W
      • et al.
      Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases.
      ) and all of the last mentioned variables (
      • Ng MY
      • Wan EYF
      • Wong HYF
      • Leung ST
      • Lee JCY
      • Chin TW
      • et al.
      Development and validation of risk prediction models for COVID-19 positivity in a hospital setting.
      ,
      • Sun Y
      • Koh V
      • Marimuthu K
      • Ng OT
      • Young B
      • Vasoo S
      • et al.
      Epidemiological and clinical predictors of COVID-19.
      ). To date, no standardized approach exists to identify SARS-CoV-2 patients using clinical parameters, especially in the context of a negative PoC-test. However, false negative (FN) RAT results can have a considerably great impact on the healthcare sector because isolation and appropriate hygiene measures would be possibly not initiated when the patient is initially tested negative. Consecutively, this may also influence the course of the pandemic. The aim of the present study was therefore to identify different clinical predictors for FN results of RATs, determined by positive RT-PCR as a reference standard, in a real-world multicenter patient cohort and to develop a corresponding prediction model.

      Methods

      We conducted a multicenter trial in three centers of the German Elblandkliniken GmbH group. The local ethics committee has not raised any objections to the collection and further analyses of data as part of the study (EK-BR-30/21-1). Informed consent has not been obtained due to the retrospective study design.
      Characteristics of patients treated in the participating hospitals between October 1st 2020 and January 31st 2021 with documented paired results of RAT and RT-PCR conducted within the first 24 hours from admission were retrospectively collected and analyzed. Specimen collection for both tests was performed with a combined oropharyngeal and bilateral deep nasal swab. For the RAT, Standard F COVID-19 Ag FIA (SD Biosensor Inc., Gyeonggi-do, Republic of Korea) was used for all patients (hereafter called Standard F). According to the manufacturer's instructions (SD

      SD Biosensor. SD Biosensor/Bestbion dx: Standard F COVID-19 Ag FIA Instruction Manual; 2021b. Available from: https://bestbion.com/wp-content/uploads/2021/03/bestbiondx_GA_COVID19Ag_FIA_20210304.pdf. [Accessed June 3, 2021].

      ), a Cut-Off-Index (COI) of 1.0 was used for test evaluation with COI ≥ 1 considered a positive result and COI < 1 considered a negative result.
      RT-PCR was carried out with the Allplex™ SARS-CoV-2/FluA/FluB/RSV Assay (Seegene, Seoul, Republic of Korea) targeting the N, RdRP and S genes of SARS-CoV-2 (

      Seegene. Seegene: Allplex™ SARS-CoV-2/FluA/FluB/RSV Assay; 2021. Available from: https://www.seegene.com/assays/allplex_sars_cov_2_flua_flub_rsv_assay. [Accessed June 3, 2021].

      ) in connection with the CFX96™ Dx Real-time PCR System cycler (

      Bio-Rad/Seegene. Bio-Rad/Seegene: CFX96™ Dx Real-time PCR detection system; 2021. Available from: https://www.seegene.com/instruments/cfx96_dx. [Accessed June 3, 2021].

      ). Cycle threshold (Ct) values for each gene (N, RdRP, S) were collected and transferred to the study database.
      The patients’ symptoms, known COVID-19 contact (14 days prior to hospital admission) and status post COVID-19 were assessed by a physician during the initial medical interview for both regular and emergency room admissions. The following medical history and physical examination variables were considered as relevant for study purposes and consecutively extracted from the hospital information system (HIS):
      • known COVID-19 contact 14 days prior to hospital admission
      • history of fever (defined as body temperature ≥ 38.0 degrees Celsius),
      • breathing frequency > 15/min.
      • dyspnea
      • loss of smell and taste
      • abdominal symptoms (abdominal pain, diarrhea, emesis)
      • musculoskeletal symptoms (myalgia, arthralgia, headache, fatigue).
      Collected laboratory values included: Standard C-reactive protein (CRP), leucocyte count, platelet count, lactate dehydrogenase (LDH), aspartate aminotransferase (AST). Furthermore, imaging data (X-ray and chest CT), if present, were reviewed for COVID-specific findings.
      First, a confusion matrix was created for the whole dataset displaying the RAT and RT-PCR results (RT-PCR results considered as the ground truth). Discrimination metrics (sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV)) were computed. As our primary goal was to create a model predicting the probability of an RAT negative patient being SARS-CoV-2 positive (therefore predicting a FN RAT, as determined by positive RT-PCR), we focused on the proportion of patients with negative RAT results (true negative, TN and false negative, FN) for further analyses.
      In a first step, we identified missing values in the dataset of TN and FN patients. Due to the high number of missing values, the medical history variables “known COVID-19 contact” and “loss of smell and taste” were not taken into account for model development as were LDH and AST for laboratory values and all imaging data. The dataset was split in 75%/25% portions for model testing and model training. The sampling was stratified for RT-PCR results (RT-PCR positive/RT-PCR negative ratio was identical in each subset). Remaining laboratory values (CRP, leucocyte and platelet count) were transformed into categorical variables, using the median as a cut-off value (CRP = 7.4 mg/l, leucocytes = 9.4/nl, thrombocytes = 237/nl). We evaluated three different models for prediction of FN RAT results as determined by RT-PCR consisting of two demographic variables, three laboratory values and five symptoms (Figure 1).
      With regard to statistical methods, we utilized a Bayesian implementation of logistic regression, using weak priors (i.e. normally distributed prior, with a mean of 0 and a large variance) (
      • Gelman A
      • Jakulin A
      • Pittau MG
      • Su Y-S.
      A weakly informative default prior distribution for logistic and other regression models.
      ). Models were fitted using four Markov chains (Monte Carlo-algorithm), each with 10,000 warm-up iterations followed by 10,000 sampling iterations. Comparisons between models and for individual parameters were carried out using the Bayes Factor (BF) with appropriate interpretation (Supplementary Table S2) (
      • Jeffreys H.
      Theory of probability.
      ). Model performance was further evaluated by computing receiver operating characteristic (ROC) area under the curve (AUC). Pearson's Chi-squared Test and Mann-Whitney-U-Test were used for intergroup comparisons where appropriate. All statistical analyses were carried out using the rstanarm package in the R environment for statistical computing (version 4.0.2).

      Results

      Overall sensitivity and specificity for the Standard F RAT in the whole dataset containing 4,076 patients was 62.3% and 97.6% respectively (PPV = 91.5%; NPV = 86.0%). The dataset which was used for model development (TN and FN patients with regard to RAT results) contained 2,997 patients (49% male; mean age [SD] = 63.7 [22.9]) after all cases with missing variables were removed (Figure 2). Patient characteristics for the TN and FN group are displayed in Table 1. Variable expressions for the train (n = 2,248) and test (n = 749) dataset were well balanced (Supplementary Table S1). Ct-values for all SARS-CoV-2 genes in the FN group were high (median [IQR] for N, RdRP, S gene: 28.0 [23.9;31.3]; 28.5 [24.7;32.4]; 28.1 [24.3;31.8]).
      Table 1Baseline characteristics.
      VariableTN, n = 2,877FN, n = 120P-Value
      Demographic
      Age (years), mean (SD)63.34 (23.09)72.00 (15.56)<0.001
      Gender: female, n (%)1,480 (51%)50 (42%)0.036
      Gender: male, n (%)1,397 (49%)70 (58%)0.036
      Laboratory values
      CRP (mg/l), mean (SD)34.33 (62.79)63.85 (62.91)<0.001
      Leucocyte count (/nl), mean (SD)10.59 (6.03)7.87 (3.41)<0.001
      Platelet count mean (/nl), mean (SD)250.00 (98.98)222.09 (104.24)<0.001
      Symptoms
      Fever (Temperature ≥ 38°C), n (%)7 (0.2%)38 (32%)<0.001
      Breathing frequency > 15/min, n (%)28 (1.0%)72 (60%)<0.001
      Dyspnea, n (%)16 (0.6%)65 (54%)<0.001
      Abdominal symptoms, n (%)10 (0.3%)18 (15%)<0.001
      Musculoskeletal symptoms, n (%)7 (0.2%)20 (17%)<0.001
      CRP = C-reactive protein; FN = false negative RAT result; SD = standard deviation; TN = true negative RAT result
      Table 2 summarizes the estimates for specific variables for all three models. Superior performance could be demonstrated for model 3, which includes all variables, over model 1 and 2 both regarding the direct comparison using the BF (Supplementary Table S3) and by comparing the ROC AUCs (Figure 3). Model 3 displayed an excellent discriminatory performance with regard to AUC (0.971). Calibration plots for the test dataset are provided in the Supplementary Material (Figure S1). Most important variables for model performance with respect to BF were (Table 2): leucocyte count, fever, breathing frequency, dyspnea and musculoskeletal symptoms. The variables “high leucocyte count” and “high platelet count” show a negative estimate which means that COVID-19 in our study was rather associated with leukopenia and thrombocytopenia.
      Table 2Estimates of the parameter values with 95% confidence intervals
      VariableModel 1Model 2Model 3
      Estimates (95%CI)BFEstimates (95%CI)BFEstimates (95%CI)BF
      Age0.016 (0.003; 0.029)1.30.013 (-0.005; 0.032)0.20.011 (-0.008; 0.032)0.2
      Female-0.477 (-0.926; -0.038)0.4-0.167 (-0.879; 0.536)0.1-0.076 (-0.828; 0.672)0.1
      High CRP1.337 (0.853; 1.849)>1000.672 (-0.08; 1.438)0.4
      High leucocyte count-1.495 (-2.031; -0.993)>100-1.935 (-2.904; -1.054)>100
      High platelet count-0.27 (-0.733; 0.182)0.1-0.37 (-1.141; 0.381)0.1
      Fever (Temperature ≥ 38°C)2.88 (1.626; 4.261)>1002.662 (1.288; 4.156)67.8
      Breathing frequency > 15/min3.047 (2.155; 3.935)>1003.383 (2.471; 4.313)>100
      Dyspnea3.758 (2.809; 4.729)>1003.635 (2.672; 4.651)>100
      Abdominal symptoms2.162 (0.815; 3.528)3.62.615 (1.253; 3.97)17.7
      Musculoskeletal symptoms3.278 (1.408; 5.133)12.23.835 (1.913; 5.782)39.9
      95%CI = 95%-confidence interval; BF = Bayes factor; CRP = C-reactive protein; High CRP is defined by values above 7.4 mg/l; High leucocyte count is defined by values above 9.4/nl; High platelet count is defined by values above 237/nl
      Figure 3:
      Figure 3Receiver operating characteristic (ROC) curves for different models
      The probability for a RT-PCR positive result at the time point of the patient's hospital admission can be calculated by using a specific formula derived from the models. As a first step, all the variables used in the model have to be collected and each coefficient (estimates, see Table 2) is then multiplied by the value of the variable. For “Age”, this is the age of the patient given in years and for categorical variables the binary numbers 0 and 1 (no/yes) have to be used. The cut-off for laboratory values has to be applied like defined previously: “High CRP” is defined by values above 7.4 mg/l, “High leucocyte count” is defined by values above 9.4/nl and “High platelet count” is defined by values above 237/nl.
      These multiplications are summed, and -5.08 is added, corresponding to the estimate of the intercept. Finally, this sum can be transformed into a probability using the following equation:
      P=ey1+ey


      Y is the sum defined above, P is the probability, and e the Euler's number.
      In order to compute classical discrimination metrics (sensitivity, specificity, PPV, NPV), a cut-off threshold was defined (Youden's index) (
      • Youden WJ.
      Index for rating diagnostic tests.
      ). For model 3, we defined the cut-off threshold as 0.09 (9% probability for positive RT-PCR result) because this index yields a high sensitivity of 85% together with a PPV of 70% (specificity = 0.99; NPV = 0.99; accuracy = 0.98; see Figure 4). Transferred to the clinical perspective and performance of RATs, this threshold, applied on the test dataset, would result in 85% of the patients being classified correctly as FN with subsequent isolation while 30% (1-PPV) are unnecessarily isolated.
      Figure 4:
      Figure 4Discrimination metrics for model 3 (cut-off 0.09)

      Discussion

      In a multicenter retrospective study of 4,076 patients, we evaluated clinical predictors for probable SARS-CoV-2 infection and compared different prediction models for positive RT-PCR results in the setting of an initially negative RAT. As a result, we propose a model (model 3) utilizing ten easily collectable and fast available variables, consisting of demographics, symptoms and laboratory parameters, with high accuracy and an AUC of 0.971. As the pandemic is ongoing, application of our model in clinical routine could provide a great benefit for the fast assessment of both inpatients and outpatients regarding their COVID-19 status in addition to PoC-testing by RATs as it is crucial to differentiate between FN and TN results in the healthcare sector in order to initiate appropriate hygiene measures and avoid nosocomial infections. For practical use, the development of, for example, a computerized tool which uses the above mentioned formula to calculate individual probabilities is conceivable. In a hospital setting, the patient-specific variables can be obtained directly at the patient's admission (after performance of RAT and RT-PCR respectively) and entered manually into the calculator. The resulting probability value for present COVID-19 (in patients with negative RAT results) can guide hospital staff on whether an initial isolation of the patient while awaiting RT-PCR results is necessary despite negative RAT. This process could easily be carried out and implemented as a routine measure. Automatic calculators utilizing the HIS as a data source represent another option for real-world application of our prediction models.
      Sensitivity of RATs for PoC-testing decreases with increasing Ct-values of RT-PCR (
      • Bruzzone B
      • De Pace V
      • Caligiuri P
      • Ricucci V
      • Guarona G
      • Pennati BM
      • et al.
      Comparative diagnostic performance of rapid antigen detection tests for COVID-19 in a hospital setting.
      ,
      • Lanser L
      • Bellmann-Weiler R
      • Öttl K-W
      • Huber L
      • Griesmacher A
      • Theurl I
      • et al.
      Evaluating the clinical utility and sensitivity of SARS-CoV-2 antigen testing in relation to RT-PCR Ct values.
      ,
      • Liotti FM
      • Menchinelli G
      • Lalle E
      • Palucci I
      • Marchetti S
      • Colavita F
      • et al.
      Performance of a novel diagnostic assay for rapid SARS-CoV-2 antigen detection in nasopharynx samples.
      ) and increases when symptomatic patients are tested (
      • Brihn A
      • Chang J
      • Oyong K
      • Balter S
      • Terashita D
      • Rubin Z
      • et al.
      Diagnostic performance of an antigen test with RT-PCR for the detection of SARS-CoV-2 in a hospital setting - Los Angeles County.
      ). Pooled sensitivity for Standard F was reported as 70.9% and specificity as 98.5% in a recent meta-analysis (preprint format) (Brümmer et al., 2021). These numbers can be confirmed by other studies (
      • Dinnes J
      • Deeks JJ
      • Berhane S
      • Taylor M
      • Adriano A
      • Davenport C
      • et al.
      Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.
      ) and compare well with our findings. The manufacturer of Standard F indicates a higher sensitivity and specificity of 93.70% and 99.63% respectively (SD

      SD Biosensor. SD Biosensor/Bestbion dx: Standard F COVID-19 Ag FIA; 2021a. Available from: https://bestbion.com/wp-content/uploads/2021/03/bestbiondx-standardf-covid19-ag-nasal.pdf. [Accessed May 25, 2021].

      ) which is supported by a study by Porte et al. (90.6% and 96.9%) (
      • Porte L
      • Legarraga P
      • Iruretagoyena M
      • Vollrath V
      • Pizarro G
      • Munita J
      • et al.
      Evaluation of two fluorescence immunoassays for the rapid detection of SARS-CoV-2 antigen-new tool to detect infective COVID-19 patients.
      ). These differences between trial and real-world data can be explained with bias due to test application and its evaluation under study conditions. Furthermore, RATs are mostly validated for virus detection in symptomatic patients only (
      • Fernandez-Montero A
      • Argemi J
      • Rodríguez JA
      • Ariño AH
      • Moreno-Galarraga L.
      Validation of a rapid antigen test as a screening tool for SARS-CoV-2 infection in asymptomatic populations. Sensitivity, specificity and predictive values.
      ). Additionally, the sample in the latter trial included patients with high viral loads and only 64 samples were analyzed which limits scalability of the presented results. RT-PCR Ct-values of FN patients in our study were high which is a possible explanation for the observed lower sensitivity.
      Lower leukocyte counts as well as lower platelet counts were associated with increased probability of present SARS-CoV-2 infection among our prediction models. These results are concordant with recently published findings. A heterogeneous distribution of leukocytosis as well as leukopenia among COVID-19 patients has been shown in several studies (
      • Goyal P
      • Choi JJ
      • Pinheiro LC
      • Schenck EJ
      • Chen R
      • Jabri A
      • et al.
      Clinical characteristics of Covid-19 in New York City.
      ,
      • Huang C
      • Wang Y
      • Li X
      • Ren L
      • Zhao J
      • Hu Y
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan.
      ). Patients treated in an intensive care unit (ICU) and invasively ventilated patients more frequently show leukocytosis (
      • Goyal P
      • Choi JJ
      • Pinheiro LC
      • Schenck EJ
      • Chen R
      • Jabri A
      • et al.
      Clinical characteristics of Covid-19 in New York City.
      ,
      • Huang C
      • Wang Y
      • Li X
      • Ren L
      • Zhao J
      • Hu Y
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan.
      ) which is also associated with worse outcomes (
      • Loomba RS
      • Villarreal EG
      • Farias JS
      • Aggarwal G
      • Aggarwal S
      • Flores S.
      Serum biomarkers for prediction of mortality in patients with COVID-19.
      ,
      • Zhou F
      • Yu T
      • Du R
      • Fan G
      • Liu Y
      • Liu Z
      • et al.
      Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
      ). Lower platelet counts are frequently observed in COVID-19 patients and are associated with increased mortality (
      • Goyal P
      • Choi JJ
      • Pinheiro LC
      • Schenck EJ
      • Chen R
      • Jabri A
      • et al.
      Clinical characteristics of Covid-19 in New York City.
      ,
      • Zong X
      • Gu Y
      • Yu H
      • Li Z
      • Wang Y.
      Thrombocytopenia is associated with COVID-19 severity and outcome: an updated meta-analysis of 5637 patients with multiple outcomes.
      ). Elevation of inflammatory markers such as CRP is also a common finding (
      • Guan W-J
      • Ni Z-Y
      • Hu Y
      • Liang W-H
      • Ou C-Q
      • He J-X
      • et al.
      Clinical characteristics of Coronavirus disease 2019 in China.
      ) in addition to lymphocytopenia and elevation of LDH and AST (
      • Goyal P
      • Choi JJ
      • Pinheiro LC
      • Schenck EJ
      • Chen R
      • Jabri A
      • et al.
      Clinical characteristics of Covid-19 in New York City.
      ,
      • Guan W-J
      • Ni Z-Y
      • Hu Y
      • Liang W-H
      • Ou C-Q
      • He J-X
      • et al.
      Clinical characteristics of Coronavirus disease 2019 in China.
      ).
      Different working groups identified predictors for COVID-19 during the course of the pandemic and developed respective prediction models. Bayat et al. trained and tested a machine learning (ML) model on a large dataset of SARS-CoV-2 positive and negative patients using laboratory values only which reached an accuracy of 86.4% (
      • Bayat V
      • Phelps S
      • Ryono R
      • Lee C
      • Parekh H
      • Mewton J
      • et al.
      A severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) prediction model from standard laboratory tests.
      ). In our opinion, limitations to this study arise regarding a missing reference standard (both RT-PCR and RAT were used for SARS-CoV-2 detection) and the possible unavailability of certain laboratory values (20 parameters were required for the model) which hinders application in clinical routine. Analogously, the use of laboratory parameters paired with ML application for prediction of RT-PCR results was proposed by another working group but the model displayed an AUC of 0.74 only which is considerably lower in comparison to our model integrative of demographics, symptoms and laboratory values (
      • Tschoellitsch T
      • Dünser M
      • Böck C
      • Schwarzbauer K
      • Meier J.
      Machine learning prediction of SARS-CoV-2 polymerase chain reaction results with routine blood tests.
      ).
      Regarding utilization of imaging data for predicting RT-PCR results, a Chinese study reported a high sensitivity (97%) of initially performed chest CTs (
      • Ai T
      • Yang Z
      • Hou H
      • Zhan C
      • Chen C
      • Lv W
      • et al.
      Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases.
      ). As the CT represents a crucial tool for fast assessment of critically ill patients, it should not be used for screening purposes due to radiation exposure. One integrative prediction model for COVID-19 was presented by Sun et al. in the early phase of the pandemic comprising of demographics, symptoms, physical examination results, imaging data and laboratory values (
      • Sun Y
      • Koh V
      • Marimuthu K
      • Ng OT
      • Young B
      • Vasoo S
      • et al.
      Epidemiological and clinical predictors of COVID-19.
      ). Different models displayed an AUC in range of 0.65-0.91 whereas the best performing model utilized 16 in part highly COVID-specific variables like CT/X-ray findings suspicious for viral pneumonia. Similar model performance (AUC = 0.91) was reported by Ng et al., but again, imaging data besides other variables was required (
      • Ng MY
      • Wan EYF
      • Wong HYF
      • Leung ST
      • Lee JCY
      • Chin TW
      • et al.
      Development and validation of risk prediction models for COVID-19 positivity in a hospital setting.
      ). Albeit these studies show that imaging findings are often included as variables for COVID-19 prediction models, there are concerns about the clinical applicability in our opinion because only few patients would receive a CT/X-ray examination directly at hospital admission. This, high costs and the time factor therefore prevents routine application of those models as a screening tool.
      The use of ML algorithms for prediction of SARS-CoV-2 infections has been proposed by several authors. Besides the above mentioned work (
      • Bayat V
      • Phelps S
      • Ryono R
      • Lee C
      • Parekh H
      • Mewton J
      • et al.
      A severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) prediction model from standard laboratory tests.
      ,
      • Tschoellitsch T
      • Dünser M
      • Böck C
      • Schwarzbauer K
      • Meier J.
      Machine learning prediction of SARS-CoV-2 polymerase chain reaction results with routine blood tests.
      ), Zoabi et al. presented a model using eight demographic and medical history variables for prediction of positive RT-PCR which yielded a high accuracy (
      • Zoabi Y
      • Deri-Rozov S
      • Shomron N.
      Machine learning-based prediction of COVID-19 diagnosis based on symptoms.
      ). Similar results were reported in another study albeit the presented model required a large number of variables and the evaluated case number was low (
      • Langer T
      • Favarato M
      • Giudici R
      • Bassi G
      • Garberi R
      • Villa F
      • et al.
      Development of machine learning models to predict RT-PCR results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in patients with influenza-like symptoms using only basic clinical data.
      ). However, ML-derived prediction of COVID-19 probabilities is a valuable and promising approach which demands further evaluation in future studies using large datasets.
      In contrast to the above mentioned studies, we evaluated the specific constellation of RAT negative/RT-PCR positive patients. In a recently published study by Ford et al., a similar approach was pursued but in a real-life and not a hospital setting (
      • Ford L
      • Lee C
      • Pray IW
      • Cole D
      • Bigouette JP
      • Abedi GR
      • et al.
      Epidemiologic characteristics associated with SARS-CoV-2 antigen-based test results, rRT-PCR cycle threshold values, subgenomic RNA, and viral culture results from university testing.
      ). Paired swabs for RAT and RT-PCR were collected on a university campus. A major advantage of this trial was the verification of positive RT-PCR or RAT results by subsequent viral culture. FN results occurred among symptomatic and asymptomatic patients whereas odds were higher for asymptomatic patients and for higher Ct-values. This trial highlights symptom status, also including light symptoms like nasal congestion, as an important predictor of COVID-19 infections which should be taken into account when interpreting RAT results. Another working group identified factors associated with false negative first RT-PCR results in patients with a final diagnosis of COVID-19 (
      • Lascarrou J-B
      • Colin G
      • Le Thuaut A
      • Serck N
      • Ohana M
      • Sauneuf B
      • et al.
      Predictors of negative first SARS-CoV-2 RT-PCR despite final diagnosis of COVID-19 and association with outcome.
      ). Interestingly, higher platelet and leucocyte counts yielded a higher risk and symptoms (e.g. fatigue, fever) a lower risk for a false negative first RT-PCR result which is in accordance with our observations where patient-reported symptoms and lower platelet/leucocyte counts were associated with present SARS-CoV-2 infections.

      Limitations

      We acknowledge several limitations in connection with this study. First, data collection was performed retrospectively which could have influenced data quality, especially regarding symptom assessment and also resulted in a significant amount of missing values. Furthermore, application to an external dataset did not take place. Prospective evaluation and validation of our prediction model in future studies is required. Second, a large number of RT-PCR positive (RAT false negative) patients were excluded from the analysis prior to model development due to missing values which could result in weaker model performance when the model is applied to other datasets. Third, we removed several variables (LDH, AST, loss of smell and taste, COVID-19 contact) which could have improved the performance. Nevertheless, accuracy and AUC of model 3 were excellent. Forth, we used RT-PCR as a reference standard while sensitivity and specificity of this method is not 100% (
      • Dinnes J
      • Deeks JJ
      • Berhane S
      • Taylor M
      • Adriano A
      • Davenport C
      • et al.
      Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.
      ,
      • van Kasteren PB
      • van der Veer B
      • van den Brink S
      • Wijsman L
      • de Jonge J
      • van den Brandt A
      • et al.
      Comparison of seven commercial RT-PCR diagnostic kits for COVID-19.
      ), especially in patients with very low viral loads, but those have a low risk of transmission. The Allplex™ assay used in our study proved to be reliable in previous studies (
      • Farfour E
      • Lesprit P
      • Visseaux B
      • Pascreau T
      • Jolly E
      • Houhou N
      • et al.
      The Allplex 2019-nCoV (Seegene) assay: which performances are for SARS-CoV-2 infection diagnosis?.
      ) and 3 viral genes were targeted which enhances diagnostic accuracy. Fifth, patients with false negative RAT results are not necessarily infectious as has been stated previously (
      • Homza M
      • Zelena H
      • Janosek J
      • Tomaskova H
      • Jezo E
      • Kloudova A
      • et al.
      Five antigen tests for SARS-CoV-2: virus viability matters.
      ,
      • Möckel M
      • Corman VM
      • Stegemann MS
      • Hofmann J
      • Stein A
      • Jones TC
      • et al.
      SARS-CoV-2 antigen rapid immunoassay for diagnosis of COVID-19 in the emergency department.
      ) and isolation may not be required but this risk should not be taken in view of a hospital setting since at the time of testing it is unknown in which direction viral load will develop.

      Conclusion

      We evaluated clinical predictors of FN results of RATs and propose a prediction model consisting of widely available and easily collectable variables. The model showed an excellent discriminatory performance. Its routine implementation in healthcare both in an inpatient and outpatient setting could help identifying patients at high risk for COVID-19 despite negative RAT result and could therefore influence the course of the pandemic with respect to avoiding nosocomial infections.

      Author contributions

      JL and VP contributed equally to this manuscript in view of study design, data analysis and interpretation and the writing of the manuscript (joint first-authorship). AN, SK, SH, IN, GH, CK, BR, JB, AP, JS, JP, AB, MW contributed substantially to the study design, data analysis and interpretation and revision of the manuscript and gave final approval for publishing.

      Conflicts of interest

      We declare no conflicts of interest associated with this publication. Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

      Acknowledgements

      Guarantor statement: Mr. Johannes Leiner (JL) takes full responsibility for the content of the manuscript, including the data and analysis.

      Appendix. Supplementary materials

      References

        • Ai T
        • Yang Z
        • Hou H
        • Zhan C
        • Chen C
        • Lv W
        • et al.
        Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases.
        Radiology. 2020; 296: E32-E40
      1. AWMF. Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF): Empfehlungen zur stationären Therapie von Patienten mit COVID-19 - Living Guideline; 2021. Available from: https://www.awmf.org/leitlinien/detail/ll/113-001LG.html. [Accessed May 20, 2021].

        • Bayat V
        • Phelps S
        • Ryono R
        • Lee C
        • Parekh H
        • Mewton J
        • et al.
        A severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) prediction model from standard laboratory tests.
        Clinical Infectious Diseases. 2020;
      2. BfArM. Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM): Antigen-Tests zum direkten Erregernachweis des Coronavirus SARS-CoV-2; 2021. Available from: https://antigentest.bfarm.de/ords/f?p=110:100:16064132278745:::::&tz=2:00. [Accessed May 21, 2021].

      3. Bio-Rad/Seegene. Bio-Rad/Seegene: CFX96™ Dx Real-time PCR detection system; 2021. Available from: https://www.seegene.com/instruments/cfx96_dx. [Accessed June 3, 2021].

        • Brihn A
        • Chang J
        • Oyong K
        • Balter S
        • Terashita D
        • Rubin Z
        • et al.
        Diagnostic performance of an antigen test with RT-PCR for the detection of SARS-CoV-2 in a hospital setting - Los Angeles County.
        MMWR Morbidity and mortality weekly report. 2021; 70 (California, June-August 2020): 702-706
      4. Brümmer LE, Katzenschlager S, Gaeddert M, Erdmann C, Schmitz S, Bota M, et al. The accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis. medRxiv 2021:2021.02.26.21252546.

        • Bruzzone B
        • De Pace V
        • Caligiuri P
        • Ricucci V
        • Guarona G
        • Pennati BM
        • et al.
        Comparative diagnostic performance of rapid antigen detection tests for COVID-19 in a hospital setting.
        International Journal of Infectious Diseases. 2021; 107: 215-218
        • Corman VM
        • Haage VC
        • Bleicker T
        • Schmidt ML
        • Mühlemann B
        • Zuchowski M
        • et al.
        Comparison of seven commercial SARS-CoV-2 rapid point-of-care antigen tests: a single-centre laboratory evaluation study.
        The Lancet Microbe. 2021;
        • Dinnes J
        • Deeks JJ
        • Berhane S
        • Taylor M
        • Adriano A
        • Davenport C
        • et al.
        Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection.
        Cochrane Database of Systematic Reviews. 2021;
      5. EC-DG. European Commission Directorate-General For Health And Food Safety (EC-DG SANTE): A common list of COVID-19 rapid antigen tests, including those of which their test results are mutually recognised, and a common standardised set of data to be included in COVID-19 test result certificates; 2021. Available from: https://ec.europa.eu/health/sites/default/files/preparedness_response/docs/covid-19_rat_common-list_en.pdf. [Accessed May 21, 2021].

        • Farfour E
        • Lesprit P
        • Visseaux B
        • Pascreau T
        • Jolly E
        • Houhou N
        • et al.
        The Allplex 2019-nCoV (Seegene) assay: which performances are for SARS-CoV-2 infection diagnosis?.
        European Journal of Clinical Microbiology & Infectious Diseases. 2020; 39: 1997-2000
        • Fernandez-Montero A
        • Argemi J
        • Rodríguez JA
        • Ariño AH
        • Moreno-Galarraga L.
        Validation of a rapid antigen test as a screening tool for SARS-CoV-2 infection in asymptomatic populations. Sensitivity, specificity and predictive values.
        EClinicalMedicine. 2021; : 37
        • Ford L
        • Lee C
        • Pray IW
        • Cole D
        • Bigouette JP
        • Abedi GR
        • et al.
        Epidemiologic characteristics associated with SARS-CoV-2 antigen-based test results, rRT-PCR cycle threshold values, subgenomic RNA, and viral culture results from university testing.
        Clinical Infectious Diseases. 2021;
        • Gelman A
        • Jakulin A
        • Pittau MG
        • Su Y-S.
        A weakly informative default prior distribution for logistic and other regression models.
        The Annals of Applied Statistics. 2008; 2 (24): 1360-1383
        • Goyal P
        • Choi JJ
        • Pinheiro LC
        • Schenck EJ
        • Chen R
        • Jabri A
        • et al.
        Clinical characteristics of Covid-19 in New York City.
        New England Journal of Medicine. 2020; 382: 2372-2374
        • Guan W-J
        • Ni Z-Y
        • Hu Y
        • Liang W-H
        • Ou C-Q
        • He J-X
        • et al.
        Clinical characteristics of Coronavirus disease 2019 in China.
        New England Journal of Medicine. 2020; 382: 1708-1720
        • Hirotsu Y
        • Maejima M
        • Shibusawa M
        • Amemiya K
        • Nagakubo Y
        • Hosaka K
        • et al.
        Prospective study of 1308 nasopharyngeal swabs from 1033 patients using the LUMIPULSE SARS-CoV-2 antigen test: Comparison with RT-qPCR.
        International Journal of Infectious Diseases. 2021; 105: 7-14
        • Homza M
        • Zelena H
        • Janosek J
        • Tomaskova H
        • Jezo E
        • Kloudova A
        • et al.
        Five antigen tests for SARS-CoV-2: virus viability matters.
        Viruses. 2021; 13: 684
        • Huang C
        • Wang Y
        • Li X
        • Ren L
        • Zhao J
        • Hu Y
        • et al.
        Clinical features of patients infected with 2019 novel coronavirus in Wuhan.
        China. The Lancet. 2020; 395: 497-506
        • Jeffreys H.
        Theory of probability.
        3rd ed. Clarendon Press, Oxford1961
        • Langer T
        • Favarato M
        • Giudici R
        • Bassi G
        • Garberi R
        • Villa F
        • et al.
        Development of machine learning models to predict RT-PCR results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in patients with influenza-like symptoms using only basic clinical data.
        Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine. 2020; 28: 113
        • Lanser L
        • Bellmann-Weiler R
        • Öttl K-W
        • Huber L
        • Griesmacher A
        • Theurl I
        • et al.
        Evaluating the clinical utility and sensitivity of SARS-CoV-2 antigen testing in relation to RT-PCR Ct values.
        Infection. 2021; 49: 555-557
        • Lascarrou J-B
        • Colin G
        • Le Thuaut A
        • Serck N
        • Ohana M
        • Sauneuf B
        • et al.
        Predictors of negative first SARS-CoV-2 RT-PCR despite final diagnosis of COVID-19 and association with outcome.
        Scientific Reports. 2021; 11
        • Liotti FM
        • Menchinelli G
        • Lalle E
        • Palucci I
        • Marchetti S
        • Colavita F
        • et al.
        Performance of a novel diagnostic assay for rapid SARS-CoV-2 antigen detection in nasopharynx samples.
        Clin Microbiol Infect. 2021; 27: 487-488
        • Loomba RS
        • Villarreal EG
        • Farias JS
        • Aggarwal G
        • Aggarwal S
        • Flores S.
        Serum biomarkers for prediction of mortality in patients with COVID-19.
        Annals of clinical biochemistry. 2021; 45632211014244
        • Mak GC
        • Cheng PK
        • Lau SS
        • Wong KK
        • Lau C
        • Lam ET
        • et al.
        Evaluation of rapid antigen test for detection of SARS-CoV-2 virus.
        Journal of Clinical Virology. 2020; 129104500
        • Manoharan L
        • Cattrall JWS
        • Harris C
        • Newell K
        • Thomson B
        • Pritchard MG
        • et al.
        Evaluating clinical characteristics studies produced early in the Covid-19 pandemic: A systematic review.
        PLOS ONE. 2021; 16e0251250
        • Miller IF
        • Becker AD
        • Grenfell BT
        • Metcalf CJE.
        Disease and healthcare burden of COVID-19 in the United States.
        Nature Medicine. 2020; 26: 1212-1217
        • Möckel M
        • Corman VM
        • Stegemann MS
        • Hofmann J
        • Stein A
        • Jones TC
        • et al.
        SARS-CoV-2 antigen rapid immunoassay for diagnosis of COVID-19 in the emergency department.
        Biomarkers. 2021; 26: 213-220
        • Ng MY
        • Wan EYF
        • Wong HYF
        • Leung ST
        • Lee JCY
        • Chin TW
        • et al.
        Development and validation of risk prediction models for COVID-19 positivity in a hospital setting.
        International journal of infectious diseases: IJID: official publication of the International Society for Infectious Diseases. 2020; 101: 74-82
      6. NIH. National Institutes of Health (NIH): COVID-19 Treatment Guidelines; 2021. Available from: https://www.covid19treatmentguidelines.nih.gov/overview/. [Accessed May 20, 2021].

        • Paul G PT
        • Sethi S
        • Schilling T
        • Wienand O
        • Jürgensen J S
        • Menzel C U
        Klinische Performance eines neuen SARS-CoV-2-AntigenTests in der Notaufnahme eines Maximalversorgers.
        Epid Bull. 2021; 3 (2021): 10-15
        • Porte L
        • Legarraga P
        • Iruretagoyena M
        • Vollrath V
        • Pizarro G
        • Munita J
        • et al.
        Evaluation of two fluorescence immunoassays for the rapid detection of SARS-CoV-2 antigen-new tool to detect infective COVID-19 patients.
        PeerJ. 2021; 9 (e10801-e)
      7. RKI. Robert-Koch-Institut (RKI): Most frequent symptoms caused by the coronavirus (COVID-19) in Germany in 2021. Statista. Statista Inc.; 2021a. Available from: https://www.statista.com/statistics/1105523/coronavirus-covid-19-symptoms-most-frequent-germany/ [Accessed June 15, 2021].

      8. RKI. Robert-Koch-Institut (RKI): Nationale Teststrategie – wer wird in Deutschland auf das Vorliegen einer SARS-CoV-2 Infektion getestet?; 2021b. Available from: https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Teststrategie/Nat-Teststrat.html;jsessionid=82E51EEE7744B4D6DFE4B229ACB2EE65.internet082?nn=13490888#doc14279904bodyText5. [Accessed May 25, 2021].

      9. SD Biosensor. SD Biosensor/Bestbion dx: Standard F COVID-19 Ag FIA; 2021a. Available from: https://bestbion.com/wp-content/uploads/2021/03/bestbiondx-standardf-covid19-ag-nasal.pdf. [Accessed May 25, 2021].

      10. SD Biosensor. SD Biosensor/Bestbion dx: Standard F COVID-19 Ag FIA Instruction Manual; 2021b. Available from: https://bestbion.com/wp-content/uploads/2021/03/bestbiondx_GA_COVID19Ag_FIA_20210304.pdf. [Accessed June 3, 2021].

      11. Seegene. Seegene: Allplex™ SARS-CoV-2/FluA/FluB/RSV Assay; 2021. Available from: https://www.seegene.com/assays/allplex_sars_cov_2_flua_flub_rsv_assay. [Accessed June 3, 2021].

        • Seifried J BS
        • von Kleist M
        • Jenny MA
        • Antão E OD
        • Jung-Sendzik T
        • Broich K
        • Denkinger C
        • Bartenschlager T SL
        • Hamouda O
        • Mielke M
        Antigentests als ergänzendes Instrument in der Pandemiebekämpfung.
        Epid Bull. 2021; 17 (2021): 14-25
        • Sun Y
        • Koh V
        • Marimuthu K
        • Ng OT
        • Young B
        • Vasoo S
        • et al.
        Epidemiological and clinical predictors of COVID-19.
        Clinical Infectious Diseases. 2020; 71: 786-792
        • Tong JY
        • Wong A
        • Zhu D
        • Fastenberg JH
        • Tham T.
        The prevalence of olfactory and gustatory dysfunction in COVID-19 patients: a systematic review and meta-analysis.
        Otolaryngology–Head and Neck Surgery. 2020; 163: 3-11
        • Tschoellitsch T
        • Dünser M
        • Böck C
        • Schwarzbauer K
        • Meier J.
        Machine learning prediction of SARS-CoV-2 polymerase chain reaction results with routine blood tests.
        Laboratory Medicine. 2021; 52: 146-149
        • van Kasteren PB
        • van der Veer B
        • van den Brink S
        • Wijsman L
        • de Jonge J
        • van den Brandt A
        • et al.
        Comparison of seven commercial RT-PCR diagnostic kits for COVID-19.
        Journal of Clinical Virology. 2020; 128104412
      12. WHO. World Health Organization (WHO): Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays - interim guidance 2021. Available from: https://www.who.int/publications/i/item/antigen-detection-in-the-diagnosis-of-sars-cov-2infection-using-rapid-immunoassays. [Accessed May 25, 2021].

        • Youden WJ.
        Index for rating diagnostic tests.
        Cancer. 1950; 3: 32-35
        • Zhou F
        • Yu T
        • Du R
        • Fan G
        • Liu Y
        • Liu Z
        • et al.
        Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
        The Lancet. 2020; 395: 1054-1062
        • Zhu N
        • Zhang D
        • Wang W
        • Li X
        • Yang B
        • Song J
        • et al.
        A novel Coronavirus from patients with Pneumonia in China, 2019.
        New England Journal of Medicine. 2020; 382: 727-733
        • Zoabi Y
        • Deri-Rozov S
        • Shomron N.
        Machine learning-based prediction of COVID-19 diagnosis based on symptoms.
        npj Digital Medicine. 2021; 4
        • Zong X
        • Gu Y
        • Yu H
        • Li Z
        • Wang Y.
        Thrombocytopenia is associated with COVID-19 severity and outcome: an updated meta-analysis of 5637 patients with multiple outcomes.
        Laboratory Medicine. 2021; 52: 10-15