Comparison of analytical sensitivity of SARS-CoV-2 molecular detection kits

  • Jing Yang
    Affiliations
    National Center for Clinical Laboratories, Beijing Hospital, National Center of Gerontology, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing, PR China

    Graduate School of Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, PR China

    Beijing Engineering Research Center of Laboratory Medicine, Beijing Hospital, Beijing, PR China
    Search for articles by this author
  • Yanxi Han
    Affiliations
    National Center for Clinical Laboratories, Beijing Hospital, National Center of Gerontology, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing, PR China

    Beijing Engineering Research Center of Laboratory Medicine, Beijing Hospital, Beijing, PR China
    Search for articles by this author
  • Runling Zhang
    Affiliations
    National Center for Clinical Laboratories, Beijing Hospital, National Center of Gerontology, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing, PR China

    Graduate School of Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, PR China

    Beijing Engineering Research Center of Laboratory Medicine, Beijing Hospital, Beijing, PR China
    Search for articles by this author
  • Rui Zhang
    Correspondence
    Corresponding authors: Jinming Li and Rui Zhang, National Center for Clinical Laboratories, No. 1 Dahua Road, Dongdan, Beijing 100730, People's Republic of China. Tel: +86 10 58115053. Fax: +86 10 65212064
    Affiliations
    National Center for Clinical Laboratories, Beijing Hospital, National Center of Gerontology, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing, PR China

    Beijing Engineering Research Center of Laboratory Medicine, Beijing Hospital, Beijing, PR China
    Search for articles by this author
  • Jinming Li
    Correspondence
    Corresponding authors: Jinming Li and Rui Zhang, National Center for Clinical Laboratories, No. 1 Dahua Road, Dongdan, Beijing 100730, People's Republic of China. Tel: +86 10 58115053. Fax: +86 10 65212064
    Affiliations
    National Center for Clinical Laboratories, Beijing Hospital, National Center of Gerontology, Institute of Geriatric Medicine, Chinese Academy of Medical Sciences, Beijing, PR China

    Graduate School of Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing, PR China

    Beijing Engineering Research Center of Laboratory Medicine, Beijing Hospital, Beijing, PR China
    Search for articles by this author
Open AccessPublished:August 21, 2021DOI:https://doi.org/10.1016/j.ijid.2021.08.043

      Highlights

      • This study is the first to evaluate other molecular detection assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), as well as rRT-PCR detection kits.
      • This is the most comprehensive evaluation of currently approved National Medical Products Administration detection kits.
      • Factors that may affect the sensitivity of the detection process were analyzed comprehensively.

      Abstract

      Objectives

      Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection has had a significant impact on global public health systems, making nucleic acid detection an important tool in epidemic prevention and control. Detection kits based on real-time reverse transcriptase PCR (rRT-PCR) have been used widely in clinics, but their analytical sensitivity (limit of detection, LOD) remains controversial. Moreover, there is limited research evaluating the analytical sensitivity of other molecular detection kits.

      Methods

      In this study, armored ribonucleic acid reference materials developed in-house were used to evaluate the analytical sensitivity of SARS-CoV-2 detection kits approved by the National Medical Products Administration. These were based on rRT-PCR and other molecular detection assays.

      Results

      The percentage retesting required with rRT-PCR kits was as follows: 0%, 7.69%, 15.38%, and 23.08% for samples with concentrations ranging from 50 000 to 781 copies/ml. In total, 93% of rRT-PCR kits had a LOD <1000 copies/ml. Only one kit had an LOD >1000 copies/ml. The LOD of other molecular detection kits ranged from 68 to 2264 copies/ml.

      Conclusions

      The study findings can help pharmaceutical companies optimize and improve detection kits, guide laboratories in selecting kits, and assist medical workers in their daily work.

      Graphical Abstract

      KEYWORDS

      Introduction

      The outbreak of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was first reported in December 2019. To date, more than 207 million people have been infected with SARS-CoV-2 and over 4 million people have died from COVID-19 globally (
      WHO
      COVID-19 Explorer.
      ). Asymptomatic infection in individuals has accelerated the transmission of the virus, making containment and mitigation difficult (
      • Han D
      • Li R
      • Han Y
      • Zhang R
      • Li J.
      COVID-19: Insight into the asymptomatic SARS-COV-2 infection and transmission.
      ). To prevent the situation from worsening, governments need to implement stronger prevention and control strategies to test and track patients, suspected cases, and asymptomatic cases.
      Currently, the diagnostic methods for COVID-19 include not only traditional molecular tests, serology tests, and computed tomography, but also field-effect transistor-based sensing and plasmonic sensing, as well as the use of high-throughput sensors (
      • Taleghani N
      • Taghipour F.
      Diagnosis of COVID-19 for controlling the pandemic: a review of the state-of-the-art.
      ;
      • Seo G
      • Lee G
      • Kim MJ
      • Baek SH
      • Choi M
      • Ku KB
      • et al.
      Rapid detection of COVID-19 causative virus (SARS-CoV-2) in Human Nasopharyngeal swab specimens using field-effect transistor-based biosensor.
      ;
      • Ahmadivand A
      • Gerislioglu B
      • Ramezani Z
      • Kaushik A
      • Manickam P
      • Ghoreishi SA.
      Functionalized terahertz plasmonic metasensors: femtomolar-level detection of SARS-CoV-2 spike proteins.
      ). Among these, molecular detection is recommended as the gold standard for SARS-CoV-2 detection (
      • Green DA
      • Zucker J
      • Westblade LF
      • Whittier S
      • Rennert H
      • Velu P
      • et al.
      Clinical performance of SARS-CoV-2 molecular tests.
      ). Real-time reverse transcriptase PCR (rRT-PCR) is a robust technology with a high specificity and sensitivity (
      • Yuce M
      • Filiztekin E
      • Ozkaya KG.
      COVID-19 diagnosis -a review of current methods.
      ). However, false-negative results from rRT-PCR are associated with substantial risks and public health implications. Many factors can lead to false-negative results, especially low viral loads (
      • Yu F
      • Yan L
      • Wang N
      • Yang S
      • Wang L
      • Tang Y
      • et al.
      Quantitative detection and viral load analysis of SARS-CoV-2 in infected patients.
      ;
      • Kucirka LM
      • Lauer SA
      • Laeyendecker O
      • Boon D
      • Lessler J.
      Variation in false-negative rate of reverse transcriptase polymerase chain reaction-based SARS-CoV-2 tests by Time since exposure.
      ;
      • Wikramaratna PS
      • Paton RS
      • Ghafari M
      • Lourenco J.
      Estimating the false-negative test probability of SARS-CoV-2 by RT-PCR.
      ). Accurate detection can help effectively identify infected individuals and limit the spread of the virus. Therefore, it is necessary to improve the analytical sensitivity to ensure the accuracy and reliability of the test results. The limit of detection (LOD) is the lowest concentration of SARS-CoV-2 RNA at which the positivity rate of the detection kit is ≥95%, known as the analytical sensitivity. It is an important performance parameter for evaluating a detection kit. At the beginning of the pandemic, only a few kits were evaluated for detection performance (
      • Alcoba-Florez J
      • Gil-Campesino H
      • Artola DG
      • Gonzalez-Montelongo R
      • Valenzuela-Fernandez A
      • Ciuffreda L
      • et al.
      Sensitivity of different RT-qPCR solutions for SARS-CoV-2 detection.
      ;
      • Wang B
      • Hu M
      • Ren Y
      • Xu X
      • Wang Z
      • Lyu X
      • et al.
      Evaluation of seven commercial SARS-CoV-2 RNA detection kits based on real-time polymerase chain reaction (PCR) in China.
      ;
      • Wang X
      • Yao H
      • Xu X
      • Zhang P
      • Zhang M
      • Shao J
      • et al.
      Limits of detection of 6 approved RT-PCR kits for the novel SARS-Coronavirus-2 (SARS-CoV-2).
      ;
      • Yan Y
      • Chang L
      • Luo W
      • Liu J
      • Guo F
      • Wang L.
      Comparison of seven commercial severe acute respiratory syndrome coronavirus 2 nucleic acid detection reagents with pseudovirus as quality control material.
      ). Currently, the number of kits approved by the National Medical Products Administration (NMPA) has increased significantly, and previously approved kits have been optimized. Notably, some of these kits are CE-certified and US Food and Drug Administration-approved for use in various countries. In addition, the evaluation of detection kits has mainly focused on the rRT-PCR method, with little focus on other molecular detection assays.
      Therefore, the aim of this study was to evaluate the analytical sensitivity of kits currently approved by the NMPA, including those using rRT-PCR and those using other molecular detection techniques, to assess whether each SARS-CoV-2 detection kit meets the LOD claimed by their manufacturer and to provide a theoretical basis for laboratories in selecting kits.

      Methods

       Preparation of the evaluation sample

      As shown in Figure 1a, SARS-CoV-2 ORF1a, RdRP, ORF1b/S, and N/E virus-like particles (VLPs) were prepared in the laboratory and used as evaluation samples for this study. Four gene fragment sequences were synthesized (Geneary, Shanghai, China). These fragments were cloned into the pACYC-MS2 vector, respectively. The recombinant plasmid was transformed into BL21 (DE3) cells, and protein expression was induced using isopropyl-β-d-thiogalactopyranoside. The VLPs were purified and digested with DNase and RNase to remove free nucleic acids from the surface. Specific primers and probes were used to identify the four target sequences contained in the VLPs by quantitative PCR (qPCR) and were quantified by droplet digital PCR (ddPCR) (Supplementary Material Table S1). According to the results, four VLPs were mixed in equal proportions at a concentration of 1  ×  108 copies/ml. The four-fold serially diluted mixture was used as the evaluation sample at concentrations of 50 000, 12 500, 3125, 781, 195, 49, and 12.25 copies/ml using Dulbecco's modified Eagle medium (DMEM). The samples were sub-packaged into small tubes and stored in a refrigerator at −80°C until testing.
      Figure 1
      Figure 1Flow chart of the analytical sensitivity evaluation of the SARS-CoV-2 detection kits. (a) Preparation of the evaluation sample. (b) Evaluation of the LOD of the detection kits. (c) Statistical analysis.
      SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; LOD, limit of detection; PEG, polyethylene glycol; VLP, virus-like particles; ddPCR, droplet digital PCR; qPCR, quantitative PCR.

       Preparation of the detection kits

      The LODs of the SARS-CoV-2 detection kits approved by the NMPA were evaluated in this study. The 13 commercial detection kits for rRT-PCR were as follows: Sansure, Da an (Da an 1 and Da an 2), BioGerm, Liferiver, Maccura, EasyDiagnosis, Bioperfect, Applied Biological, Fosun Long March, Kinghawk, GeneoDx, and BGI. The following five kits based on other molecular detection assays were also evaluated in this study: CapitalBio, Rendu, Zhongzhi 1, Zhongzhi 2, and Anbio. Evaluation of the 13 kits using the rRT-PCR method was performed in our laboratory. Biopharmaceutical companies were entrusted to complete the evaluation of the other five detection kits.
      Da an has two rRT-PCR kits approved by the NMPA, namely Da an 1 and Da an 2. Although the detection principles of these two kits are the same, the RNA preparation method, thermal cycling conditions, and cycling number differ. The method used by Anbio is a hybrid capture immunofluorescence assay. Zhongzhi 1 uses a dual amplification assay method, whereas Zhongzhi 2 uses a ribonucleic acid (RNA) isothermal amplification gold probe chromatography assay. The method used by Rendu is an RNA capture probe assay. CapitalBio uses an integrated isothermal amplification chip assay.

       Evaluation of the LOD of the rRT-PCR-based commercial kits

      As shown in Figure 1b, the rRT-PCR evaluation samples were grouped according to seven concentrations: 50 000, 12 500, 3125, 781, 195, 49, and 12.25 copies/ml. Each concentration group included 21 samples. For each detection kit, there were two negative and 147 positive samples. The procedure was performed according to the manufacturer's instructions. RNA was obtained by extraction and purification of nucleic acid, or lysis release. The viral RNA extraction kit (Qiagen, Shanghai, China) was used for RNA extraction and purification in the rRT-PCR-based kits, except for Da an 2, for which the lysis method for RNA preparation was used. Results were interpreted according to the manufacturer's instructions (Supplementary Material Table S2). In cases in which false positivity was suspected, the sample was retested. The results of the retest were interpreted according to the interpretation criteria of the retest result.

       Evaluation of the LOD of other commercial kits

      As shown in Figure 1b, five detection kits were tested by biopharmaceutical companies. All companies were provided with evaluation samples with concentrations of 50 000, 12 500, 3125, 781, 195, and 49 copies/ml. Each concentration group included 21 samples. There were three negative and 126 positive samples. The order of all samples was disrupted before handing them to the biopharmaceutical company, and the samples were transported through cold-chain transportation to maintain a low temperature environment. The evaluation samples were tested according to the manufacturer's instructions. All companies were required to report results within a week.

       Statistical analysis

      As shown in Figure 1c, probit regression analysis was used to evaluate the LOD of the test results in MedCalc Statistical Software version 19.6.1 (MedCalc Software Ltd, Ostend, Belgium). The Pearson Chi-square test was used to evaluate the positivity rate of target genes in IBM SPSS Statistics version 19.0 (IBM Corp., Armonk, NY, USA).

      Results

       Characteristics of SARS-CoV-2 VLPs

      Four VLPs were prepared, and specific probe primers for ORF1a, RdRP, ORF1b/S, and N/E were used to detect VLPs containing the target sequence (Supplementary Material Table S1). The four VLPs were quantified by droplet digital PCR using specific probe primers. The results showed ORF1a, RdRP, ORF1b/S, and N/E VLPs at concentrations of 5.44 × 1011, 1.273 × 1012, 4.48 × 1011, and 1.9 × 1012 copies/ml, respectively (Supplementary Material Table S3). The four VLPs were diluted and mixed to form a high-concentration sample with a concentration of 1 × 108 copies/ml. Three commercial kits that could detect ORF1a, RdRP, ORF1b/S, and N/E fragments were used to confirm that all target sequences were present in the mixture (Supplementary Material Tables S4–S6).

       Rate of retest calculation

      Serially diluted samples were detected using commercial detection kits, and the results were presented as positive or negative according to the manufacturer's instructions (Table 1). As described in some of the instructions, samples that were suspected to be positive in the first test were retested. The percentage of rRT-PCR kits that required retesting was 0% (0/13), 7.69% (1/13), 15.38% (2/13), and 23.08% (3/13) for samples with concentrations ranging from 50 000 to 781 copies/ml, respectively. With the decrease in sample concentration, the number of kits that required retesting increased. Therefore, the percentage of rRT-PCR kits retested was 69.23% (9/13), 92.31% (12/13), and 61.54% (8/13) for sample with concentrations from 195 to 12.25 copies/ml, respectively (Table 2, Figure 2a).
      Figure 2
      Figure 2Summary of the testing status of the SARS-CoV-2 rRT-PCR detection kits. (a) Retest rates of samples with different concentrations according to the rRT-PCR detection kits. (b) Measured LOD of each target gene and claimed LOD of each rRT-PCR kit. (c) Measured and claimed LODs of each rRT-PCR kit.
      SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; LOD, limit of detection.
      Figure 3
      Figure 3Estimated limit of detection (LOD) of the molecular detection assay kits using probit regression analysis. (a)–(m) Analysis graphs of probit regression for each of the 13 commercial detection kits investigated.
      The retest rate was calculated as follows: number of retest/number of replicates (retest rate, %) (Table 2). The retest rate of the Da an 2 detection kit was the highest. The retest rates were 14.29% (3/21), 57.14% (12/21), 71.43% (15/21), 57.14% (12/21), 23.81% (5/21), and 9.52% (2/21) for the samples of concentrations 12 500, 3125, 781, 195, 49, and 12.25 copies/ml, respectively. The Maccura kit had the second highest retest rate, which were as follows: 23.81% (5/21), 14.29% (3/21), and 71.43% (15/21) for concentrations of 3125, 781, and 195 copies/ml, respectively. For samples with low concentrations, the retest rates were zero because all test results were considered negative. For Sansure and kits based on other molecular detection assays, retesting of samples of each concentration was not required; therefore, the retest rates of these kits were not calculated (Figure 2a).

       Evaluation of the LOD of target genes

      All kits were designed to simultaneously detect different target genes. Hence, the positivity rate for each gene was calculated. The positivity rate was calculated as follows: number of positive/number of replicates (positivity rate, %) (Table 3). Except for BGI, the manufacturer's instructions did not indicate the target gene (Figure 2b). The positivity rate of the ORF gene in the Da an 2 kit was much higher than that of the N gene (10/21 vs 19/21, Pearson Chi-square test, P = 0.003), which affected the positivity rate of the 781 copies/ml sample. In contrast, the difference in the positivity rates of the 195 copies/ml samples showed that the N and E genes were more sensitive than the ORF gene (17/21 vs 21/21 vs 8/21, P < 0.001) in the Maccura kit. This phenomenon was more obvious in the low-concentration samples. However, Da an 1 showed similar positivity rates for the N and ORF genes (20/21 vs 20/21).

       Evaluation of the LOD of rRT-PCR kits

      Probit regression analysis of the positivity rate of 147 samples was performed to obtain the LOD of the rRT-PCR kit (Table 3). The LODs of Sansure, Liferiver, Fosun Long March, and Da an 1 were 48 copies/ml (95% confidence interval (CI) 32–131 copies/ml), 113 copies/ml (95% CI 69–321 copies/ml), 176 copies/ml (95% CI 115–432 copies/ml), and 185 copies/ml (95% CI 115–469 copies/ml), respectively. The 95% hit rate of the Applied Biological, BioGerm, Kinghawk, BGI, Bioperfect, and EasyDiagnosis kits were 311 copies/ml (95% CI 195–775 copies/ml), 331 copies/ml (95% CI 229–725 copies/ml), 392 copies/ml (95% CI 239–999 copies/ml), 439 copies/ml (95% CI 235–1385 copies/ml), 480 copies/ml (95% CI 275–1320 copies/ml), and 711 copies/ml (95% CI 420–1853 copies/ml), respectively. The LODs of Maccura, GeneoDx, and Da an 2 were 724 copies/ml (95% CI 478–1776 copies/ml), 827 copies/ml (95% CI 432–28 953 copies/ml), and 1217 copies/ml (95% CI 756–3039 copies/ml), respectively. The results of the probit regression analysis are shown in Figure 2c and Figure 3.

       Evaluation of the LOD of kits using other detection methods

      The LODs of kits using other detection methods were obtained to analyze the feedback from the biopharmaceutical companies. Similarly, probit regression analysis was performed on the positivity rate of 126 samples to obtain the LODs of the other detection assay kits (Table 3). As shown in Figure 4, the LOD of Zhongzhi 2 was 761 copies/ml (95% CI 457–2056 copies/ml); that of Anbio was 803 copies/ml (95% CI 365–5930 copies/ml); those of Zhongzhi 1 and CapitalBio were 820 copies/ml (95% CI 477–2365 copies/ml) and 2264 copies/ml (95% CI 1204–7041 copies/ml), respectively; and that of Rendu was 68 copies/ml. Notably, the Rendu kit reported 12 samples as positive, with a concentration of 49 copies/ml. Other positive samples with other concentrations were also completely detected. Therefore, the CI was not obtained after the probit regression analysis (Figure 4).
      Figure 4
      Figure 4Estimated limit of detection (LOD) of the other detection assays using probit regression analysis. (a) Measured and claimed LODs of five other detection assay kits. (b)–(e) LOD obtained via probit regression analysis using 21 replicates of four-fold serially diluted samples.

      Discussion

      During the process of RNA preparation, the volumes of the sample and elution buffer were different (Table 1). They are summarized as follows: (1) 140 μl of sample for extraction and 60 μl of buffer for elution; (2) the volume of the sample is 200 μl; (3) the elution volume is 80 μl. This difference could affect the RNA concentration. For instance, RNA obtained from (2) was two-fold more concentrated than that from (3). Similarly, the volumes of the reaction system and RNA template were also different. The discrepancy in the amount of template may affect the detection performance of a kit (
      • Wang B
      • Hu M
      • Ren Y
      • Xu X
      • Wang Z
      • Lyu X
      • et al.
      Evaluation of seven commercial SARS-CoV-2 RNA detection kits based on real-time polymerase chain reaction (PCR) in China.
      ). The RNA templates of the kits tested were 2/5, 1/5, 1/2, and 1/3 of the total reaction system. The largest difference between them was up to 2.5-fold. Remarkably, the LOD of Da an 2 was much worse than that of Da an 1, and the method of RNA preparation may be the reason for this huge disparity. However, considering the differences in the reaction solutions of the two kits, further studies are needed to determine the effect of RNA extraction on kit detection performance.
      Table 1Characteristics of National Medical Products Administration approved SARS-CoV-2 rRT-PCR detection kits.
      Study kitsSpecimensFluorescence channelTarget gene(s)LOD (copies/ml)Input Vol. (μl)Elution Vol. (μl)RNA Vol. (μl)Total reaction Vol. (μl)Cycling number
      SansureOP, BALFFAM, ROXORF1ab, N20014060205045
      Da an 1OP, sputumFAM, VICORF1ab, N5002006052545
      BioGermNP, OP, sputumFAM, HEX/VICORF1ab, N10001406052540
      LiferiverNP, sputum, BALFFAM, HEX/VIC, TEXAS REDRdRP, N, E2001406052545
      MaccuraOP, sputumFAM, ROX, Cy5ORF1ab, N, E100014080204040
      EasyDiagnosisNP, OP, sputumHEX, FAMORF1ab, N5001406052540
      BioperfectNP, OP, sputumFAM, VICORF1ab, N3501406052545
      Applied BiologicalOP, sputumFAM, VICORF1ab, N2002006052545
      FAME525
      Fosun Long MarchOP, sputumFAM, JOE, ROXORF1ab, N, E30020060103040
      KinghawkOP, sputumFAM, VICORF1ab, N5001406052540
      GeneoDxNP, OP, sputumTEXAS RED, FAMORF1ab, N5002006052545
      BGIOP, BALFFAM/10014060103040
      Da an 2OP, sputumFAM, VICORF1ab, N500//52532
      BALF, bronchoalveolar lavage fluid; LOD, limit of detection; NP, nasal pharyngeal; OP, oral pharyngeal; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; Vol., volume.
      Table 2Summary of the retest rate of SARS-CoV-2 rRT-PCR detection kits.
      Detection kitsNumber of retest/number of replicates (retest rate, %) at diluted concentrations (copies/ml)
      12 500 copies/ml3125 copies/ml781 copies/ml195 copies/ml49 copies/ml12.25 copies/ml
      Da an 10/21 (0)0/21 (0)0/21 (0)3/21 (14)7/21 (33)4/21(19)
      BioGerm0/21 (0)0/21 (0)0/21 (0)7/21 (33)7/21 (33)4/21(19)
      Liferiver0/21 (0)0/21 (0)0/21 (0)0/21 (0)2/21 (10)3/21 (14)
      Maccura0/21 (0)5/21 (24)3/21(14)15/21 (71)0/21 (0)0/21 (0)
      EasyDiagnosis0/21 (0)0/21 (0)0/21 (0)7/21 (33)8/21(38)4/21(19)
      Bioperfect0/21 (0)0/21 (0)0/21 (0)13/21(62)7/21 (33)0/21 (0)
      Applied Biological0/21 (0)0/21 (0)0/21 (0)2/21(10)3/21 (14)0/21 (0)
      Fosun Long March0/21 (0)0/21 (0)0/21 (0)0/21 (0)1/21(5)0/21 (0)
      KINGHAWK0/21 (0)0/21 (0)0/21 (0)10/21 (48)9/21 (43)2/21(10)
      GENEODX0/21 (0)0/21 (0)5/21 (24)7/21 (33)2/21(10)0/21 (0)
      Da an 23/21 (14)12/21 (57)15/21 (71)12/21 (57)5/21 (24)2/21(10)
      BGI0/21 (0)0/21 (0)0/21 (0)0/21 (0)2/21(10)2/21(10)
      SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.
      Table 3Summary of results for LOD evaluation.
      Detection kitsNumber of positive/number of replicates (positivity rate, %) at diluted concentrations (copies/ml)
      50 000 copies/ml12 500 copies/ml3125 copies/ml781 copies/ml195 copies/ml49 copies/ml12.25 copies/ml
      Sansure21/21 (100)21/21 (100)21/21 (100)21/21 (100)21/21 (100)20/21 (95)6/21 (29)
      Da an 121/21 (100)21/21 (100)21/21 (100)21/21 (100)20/21 (95)11/21 (52)1/21 (5)
      BioGerm21/21 (100)21/21 (100)21/21 (100)21/21 (100)16/21 (76)1/21 (5)0/21 (0)
      Liferiver21/21 (100)21/21 (100)21/21 (100)21/21 (100)21/21 (100)15/21 (71)4/21 (19)
      Maccura21/21 (100)21/21 (100)21/21 (100)20/21 (95)8/21 (38)0/21 (0)0/21 (0)
      EasyDiagnosis21/21 (100)21/21 (100)21/21 (100)19/21 (91)16/21 (76)2/21 (10)0/21 (0)
      Bioperfect21/21 (100)21/21 (100)21/21 (100)21/21 (100)17/21 (81)4/21 (19)2/21 (10)
      Applied Biological21/21 (100)21/21 (100)21/21 (100)21/21 (100)19/21(91)3/21 (14)1/21 (5)
      Fosun Long March21/21 (100)21/21 (100)21/21 (100)21/21 (100)20/21 (95)9/21 (43)0/21 (0)
      Kinghawk21/21 (100)21/21 (100)21/21 (100)21/21 (100)16/21 (76)6/21 (29)0/21 (0)
      GeneoDx21/21 (100)21/21 (100)21/21 (100)21/21 (100)3/21 (14)1/21 (5)0/21 (0)
      Da an 221/21 (100)21/21 (100)21/21 (100)19/21 (91)4/21 (19)1/21 (5)0/21 (0)
      BGI21/21 (100)21/21 (100)21/21 (100)21/21 (100)15/21 (71)14/21 (67)1/21 (5)
      CapitalBio21/21 (100)21/21 (100)21/21 (100)16/21 (76)7/21 (33)3/21 (14)/
      Rendu21/21 (100)21/21 (100)21/21 (100)21/21 (100)21/21 (100)11/21 (52)/
      Zhongzhi 121/21 (100)21/21 (100)21/21 (100)21/21 (100)9/21 (43)4/21 (19)/
      Zhongzhi 221/21 (100)21/21 (100)21/21 (100)21/21 (100)9/21 (43)3/21 (14)/
      Anbio21/21 (100)21/21 (100)20/21 (95)21/21 (100)18/21 (86)10/21 (48)/
      LOD, limit of detection.
      Moreover, the interpretation criteria for these detection kits and the sensitivity of each target gene affected retesting. The interpretation criteria consist of two parts, the first test result and the retest result. In terms of the interpretation criteria for the first test result, the criteria for diagnosis of positivity in Da an 1 were more stringent than those in Sansure and Liferiver. A single positive result for the target gene needed to be retested in the case of the Da an 1 kit. This undoubtedly increased the retest rate, especially when the target gene had poor sensitivity. The sensitivities of each target gene in the multiple rRT-PCR detection kits were not identical. Three studies reported that the primer and probe sets of the US Centers for Disease Control and Prevention and World Health Organization have different LODs (
      • Afzal A.
      Molecular diagnostic technologies for COVID-19: Limitations and challenges.
      ;
      • Nalla AK
      • Casto AM
      • Huang MW
      • Perchetti GA
      • Sampoleo R
      • Shrestha L
      • et al.
      Comparative performance of SARS-CoV-2 detection assays using seven different primer-probe sets and one assay kit.
      ;
      • Vogels CBF
      • Brito AF
      • Wyllie AL
      • Fauver JR
      • Ott IM
      • Kalinich CC
      • et al.
      Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT-qPCR primer-probe sets.
      ). In the present study, the positivity rates of the different target genes (ORF1ab, N, and E) of these kits were calculated (Figure 2b). The performance of the N gene in the Da an 2 kit and the ORF gene in the Maccura and Bioperfect kits was poor, which may have been caused by technical deficiencies, including primer design, reagent instability, or inappropriate reagent ratio (
      • Wang X
      • Yao H
      • Xu X
      • Zhang P
      • Zhang M
      • Shao J
      • et al.
      Limits of detection of 6 approved RT-PCR kits for the novel SARS-Coronavirus-2 (SARS-CoV-2).
      ). Therefore, Da an 2 and Maccura kits have a high retesting rate for high-concentration samples. A high retesting rate delays the test results, increases the workload of medical workers, and leads to the requirement for more detection reagents. However, retesting reminds medical workers to pay more attention to suspicious positive samples and avoid false-negative results.
      Furthermore, the interpretation criteria for these detection kits and the sensitivity of each target gene also affect SARS-CoV-2 detection. Taking the interpretation criteria of retest results as an example, the possibility of successful diagnoses of weakly positive samples is higher with the Da an 1 kit than with the EasyDiagnosis kit. The sensitivity of each target gene determines the sensitivity of the kit (
      • Vogels CBF
      • Brito AF
      • Wyllie AL
      • Fauver JR
      • Ott IM
      • Kalinich CC
      • et al.
      Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT-qPCR primer-probe sets.
      ;
      • Wan DY
      • Luo XY
      • Dong W
      • Zhang ZW.
      Current practice and potential strategy in diagnosing COVID-19.
      ). Additionally, primers are key to determining the sensitivity of target genes. The positivity rate of SARS-CoV-2 detection is related to the selection of target regions and amplification efficiency of the primers. These results suggest that target gene selection, kit optimization, and primer validation are important for improving detection performance. The amplification of primers is greatly affected by characteristics such as internal stability, melting temperature, secondary structure, or mutual interference. Better-performing primers preferentially amplify the target fragments, leading to unbalanced amplification and even differences in LOD between different targets in multiple PCR reactions (
      • Sint D
      • Raso L
      • Traugott M.
      Advances in multiplex PCR: balancing primer efficiencies and improving detection success.
      ). Hence, reducing the competition between primers and improving the efficiency of primer amplification is necessary for multiple rRT-PCR.
      In contrast to the rRT-PCR method, other molecular detection assays use the isothermal amplification method, except the Anbio kit. The Anbio kit adopts a hybrid capture immunofluorescence method, which is rapid, simple to perform, and has a low requirement for personnel and equipment. This kit is suitable for communities and for other scenarios outside hospitals. The reverse transcription recombinase-aided amplification and loop-mediated isothermal amplification used by CapitalBio have higher requirements for instruments than the other kits. Both Zhongzhi 1 and 2 use the transcription-mediated amplification method, but the detection methods for the amplified products are different. When compared to the operation process of Zhongzhi 2, that of Zhongzhi 1 is very complicated. The principle of Rendu is the same as that of Zhongzhi, but with a high analytical sensitivity. Among other molecular detection assays, only Rendu and Zhongzhi 2 reached their claimed detection limits.
      In general, it is necessary to ensure that the amplification efficiency of primers for different target genes is similar and is >90% to reduce the discrepancy in the sensitivity of different target genes and reduce the retest rate. The detection region of the target gene should be carefully selected to avoid competition between primers during the test. The kit instructions should clearly indicate the RNA extraction method and the corresponding kit. If the lysis method is used for RNA preparation, it should be indicated by the manufacturer of the sample preservation solution. All recommended kits, reaction reagents, and procedures in the instruction manual should be strictly verified. The kit should be fully verified with accurate quantitative international standard reference materials to determine the LOD of the kit. In the application process, weak positive quality control materials should be used to regularly evaluate the detection performance of the kit.
      Although we tried to perfect our research, there are limitations to this study. Compared with VLPs, real samples may contain more interference or inhibitory substances, and clinical samples should be used for sensitivity and specificity evaluations.
      In conclusion, although the measured LODs of several kits were inferior to those claimed, the analytical sensitivities of the kits approved by the NMPA are able to meet the needs of COVID-19 diagnosis in clinics. Next, biopharmaceutical companies should focus on the pitfalls of their detection kits and improve the detection performance of the kits. Laboratories need to emphasize the importance of quality control in daily work. All of these measures are essential for ensuring reliable test results.

      Author contributions

      JML and RZ conceived and designed the study. JY and RLZ completed the experiment. JY prepared the first version of the manuscript. JML, RZ, and YXH provided valuable comments on the manuscript. JY revised and finalized the manuscript. All authors read and approved the final version of the manuscript.

      Funding

      This work was supported by the “AIDS and Hepatitis, and Other Major Infectious Disease Control and Prevention” Program of China under Grant No. 2018ZX10102001.

      Ethical approval and consent to participate

      Not applicable.

      Conflict of interest

      None declared.

      Acknowledgements

      The authors would like to thank the agent manufacturers that provided the real-time RT-PCR kits for SARS-CoV-2 detection in this study.

      Appendix. Supplementary materials

      References

        • Afzal A.
        Molecular diagnostic technologies for COVID-19: Limitations and challenges.
        J Adv Res. 2020; 26: 149-159
        • Ahmadivand A
        • Gerislioglu B
        • Ramezani Z
        • Kaushik A
        • Manickam P
        • Ghoreishi SA.
        Functionalized terahertz plasmonic metasensors: femtomolar-level detection of SARS-CoV-2 spike proteins.
        Biosens Bioelectron. 2021; 177112971
        • Alcoba-Florez J
        • Gil-Campesino H
        • Artola DG
        • Gonzalez-Montelongo R
        • Valenzuela-Fernandez A
        • Ciuffreda L
        • et al.
        Sensitivity of different RT-qPCR solutions for SARS-CoV-2 detection.
        Int J Infect Dis. 2020; 99: 190-192
        • Green DA
        • Zucker J
        • Westblade LF
        • Whittier S
        • Rennert H
        • Velu P
        • et al.
        Clinical performance of SARS-CoV-2 molecular tests.
        J Clin Microbiol. 2020; 58: e00920-e00995
        • Han D
        • Li R
        • Han Y
        • Zhang R
        • Li J.
        COVID-19: Insight into the asymptomatic SARS-COV-2 infection and transmission.
        Int J Biol Sci. 2020; 16: 2803-2811
        • Kucirka LM
        • Lauer SA
        • Laeyendecker O
        • Boon D
        • Lessler J.
        Variation in false-negative rate of reverse transcriptase polymerase chain reaction-based SARS-CoV-2 tests by Time since exposure.
        Ann Intern Med. 2020; 173: 262-267
        • Nalla AK
        • Casto AM
        • Huang MW
        • Perchetti GA
        • Sampoleo R
        • Shrestha L
        • et al.
        Comparative performance of SARS-CoV-2 detection assays using seven different primer-probe sets and one assay kit.
        J Clin Microbiol. 2020; 58: e00520-e00557
        • Seo G
        • Lee G
        • Kim MJ
        • Baek SH
        • Choi M
        • Ku KB
        • et al.
        Rapid detection of COVID-19 causative virus (SARS-CoV-2) in Human Nasopharyngeal swab specimens using field-effect transistor-based biosensor.
        ACS Nano. 2020; 14: 5135-5142
        • Sint D
        • Raso L
        • Traugott M.
        Advances in multiplex PCR: balancing primer efficiencies and improving detection success.
        Methods Ecol Evol. 2012; 3: 898-905
        • Taleghani N
        • Taghipour F.
        Diagnosis of COVID-19 for controlling the pandemic: a review of the state-of-the-art.
        Biosens Bioelectron. 2021; 174112830
        • Vogels CBF
        • Brito AF
        • Wyllie AL
        • Fauver JR
        • Ott IM
        • Kalinich CC
        • et al.
        Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT-qPCR primer-probe sets.
        Nat Microbiol. 2020; 5: 1299-1305
        • Wan DY
        • Luo XY
        • Dong W
        • Zhang ZW.
        Current practice and potential strategy in diagnosing COVID-19.
        Eur Rev Med Pharmacol Sci. 2020; 24: 4548-4553
        • Wang B
        • Hu M
        • Ren Y
        • Xu X
        • Wang Z
        • Lyu X
        • et al.
        Evaluation of seven commercial SARS-CoV-2 RNA detection kits based on real-time polymerase chain reaction (PCR) in China.
        Clin Chem Lab Med. 2020; 58: e149-ee53
        • Wang X
        • Yao H
        • Xu X
        • Zhang P
        • Zhang M
        • Shao J
        • et al.
        Limits of detection of 6 approved RT-PCR kits for the novel SARS-Coronavirus-2 (SARS-CoV-2).
        Clin Chem. 2020; 66: 977-979
        • WHO
        COVID-19 Explorer.
        World Health Organization, Geneva2020 (accessed 17 Aug 2021)
        • Wikramaratna PS
        • Paton RS
        • Ghafari M
        • Lourenco J.
        Estimating the false-negative test probability of SARS-CoV-2 by RT-PCR.
        Euro Surveill. 2020; 252000568
        • Yan Y
        • Chang L
        • Luo W
        • Liu J
        • Guo F
        • Wang L.
        Comparison of seven commercial severe acute respiratory syndrome coronavirus 2 nucleic acid detection reagents with pseudovirus as quality control material.
        J Mol Diagn. 2021; 23: 300-309
        • Yu F
        • Yan L
        • Wang N
        • Yang S
        • Wang L
        • Tang Y
        • et al.
        Quantitative detection and viral load analysis of SARS-CoV-2 in infected patients.
        Clin Infect Dis. 2020; 71: 793-798
        • Yuce M
        • Filiztekin E
        • Ozkaya KG.
        COVID-19 diagnosis -a review of current methods.
        Biosens Bioelectron. 2021; 172112752