Advertisement

Comment to the article by Pedro Brotons: Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples, IJID 110 (2021) 363–370

Open AccessPublished:August 16, 2022DOI:https://doi.org/10.1016/j.ijid.2022.08.005
      Dear Editor,
      We read with interest the article entitled “Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples” (
      • Brotons P
      • Perez-Argüello A
      • Launes C
      • Torrents F
      • Subirats MP
      • Saucedo J
      • et al.
      Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples.
      ). This study validates and implements an optimized screening method for the detection of SARS-CoV-2 ribonucleic acid, integrating the use of self-collected raw saliva samples, single-step heat-treated virus inactivation and ribonucleic acid extraction, and direct reverse transcription-quantitative polymerase chain reaction (RT-qPCR). Although this article provides valuable information, we believe that when the authors evaluated the diagnostic accuracy of saliva-based direct reverse transcription-polymerase chain reaction (RT-PCR) versus standard RT-PCR on nasopharyngeal swabs, some results are worth discussing. We noticed that the agreement of the two assays was not assessed.
      Generally, the question of agreement, or consistency among samples collecting data immediately arises due to the variability in different diagnosis methods. Thus, well-designed research studies must consequently incorporate procedures that measure agreement among the various data collectors (
      • McHugh ML.
      Interrater reliability: the kappa statistic.
      ). There are a number of statistics that have been used to measure inter-rater reliability. A partial list includes overall accuracy (
      • Asai S
      • Seki A
      • Akai Y
      • Tazawa H
      • Kakizoe H
      • Ravzanaaadii MA
      • et al.
      Nationwide external quality assessment of SARS-CoV-2 nucleic acid amplification tests in Japan.
      ), Cohen's kappa (
      • McHugh ML.
      Interrater reliability: the kappa statistic.
      ), Pearson's R (
      • Salvagno GL
      • Henry BM
      • Lippi G.
      The strength of association between pre-and post-booster BNT162b2 anti-SARS-CoV-2 antibodies levels depends on the immunoassay.
      ), Spearman Rho (
      • Geisler S
      • Lytton SD
      • Toan NL
      • Nghia TH
      • Nam NM
      • Hung HV
      • et al.
      Neopterin levels and Kyn/Trp ratios were significantly increased in dengue virus patients and subsequently decreased after recovery.
      ), intraclass correlation coefficient (
      • Rezaeipandari H
      • Mohammadpoorasl A
      • Morowatisharifabad MA
      • Shaghaghi A.
      Psychometric properties of the Persian version of abridged Connor-Davidson Resilience Scale 10 (CD-RISC-10) among older adults.
      ), concordance correlation coefficient (
      • Campana C
      • van Koetsveld PM
      • Feelders RA
      • de Herder WW
      • Iyer AM
      • van Velthuysen MF
      • et al.
      Digital quantification of somatostatin receptor subtype 2a immunostaining: a validation study.
      ), Krippen-dorff's alpha (
      • Dupuis H
      • Ghesquière L
      • Pierache A
      • Subtil D
      • Houfflin-Debarge V
      • Garabedian C.
      Evaluation and impact of fetal physiology training on fetal heart rate analysis.
      ), and Matthews correlation coefficient (
      • Qorri E
      • Takács B
      • Gráf A
      • Enyedi MZ
      • Pintér L
      • Kiss E
      • et al.
      A comprehensive evaluation of the performance of prediction algorithms on clinically relevant missense variants.
      ). Here, we will only consider the most common measures, Cohen's kappa and overall accuracy (Table 1).
      Table 1Weighted kappa value and overall accuracy for calculating agreement between saliva-based direct RT-qPCR and standard RT-PCR on nasopharyngeal swab
      Standard RT-qPCRSaliva-based direct RT-qPCROverall accuracy
      k = 0.802

      (strong agreement)
      PositiveNegativeInconclusiveTotal97.36%, (22+273+0)/303
      Positive220123
      Negative02730273
      Inconclusive1607
      Total232791303
      Note: The data has been cited from the article published by
      • Brotons P
      • Perez-Argüello A
      • Launes C
      • Torrents F
      • Subirats MP
      • Saucedo J
      • et al.
      Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples.
      and undergone modification. k is the weighted kappa value calculated by us.
      RT-qPCR, reverse transcription-quantitative polymerase chain reaction.
      Generally, Cohen's kappa statistic is suitable for evaluating two raters (
      • McHugh ML.
      Interrater reliability: the kappa statistic.
      ). In Cohen's kappa statistic, weighted kappa statistic should be used to calculate the inter-rater reliability in the presence of more than two categories (
      • Li X
      • Zhao Y
      • Zhang Z
      • Zheng T
      • Li S
      • Yang G
      • et al.
      Correlations of magnetic resonance imaging classifications with preoperative functions among patients with refractory lateral epicondylitis.
      ).
      Weighted kappa is calculated as follows:
      kw=1i=1nj=1nwijpiji=1nj=1nwijpiqj
      (1)


      The value of ujj(ii) is the proportion of objects put in the same category j by both raters i and i. The value of pij is the proportion of objects that rater i assigned to category j. According to
      • McHugh ML.
      Interrater reliability: the kappa statistic.
      , the kappa result should be interpreted as follows: 0-0.20 indicating no agreement, 0.21-0.39 as minimal agreement, 0.40-0.59 as weak agreement, 0.60-0.79 as moderate agreement, 0.80-0.90 as strong agreement, and 0.91-1.00 as almost perfect agreement.
      Therefore, according to the authors’ data, the weighted kappa value between saliva-based direct RT-qPCR and standard RT-PCR on nasopharyngeal swab evaluated by us was 0.802 (95% confidence interval = 0.669-0.935), indicating a strong agreement. The overall accuracy between the two assays was 97.36%.

      CRediT authorship contribution statement

      Tianfei Yu: Writing – original draft. Fangfang Liu: Data curation. Haichang Yin: Data curation. Nana Yi: Data curation. Ming Li: Writing – review & editing.

      Declarations of competing interests

      The authors have no competing interests to declare.

      Funding source

      This research was supported by a grant (LH2020C110) from the Joint Guidance Project of Natural Science Foundation of Heilongjiang Province of China, a grant (HLJ2019017) Chinese Ministry of Education “Chunhui Plan” International Scientific Research Cooperation Project, a grant (145109136) from the Fundamental Research Funds in Heilongjiang Provincial Universities and a grant from Heilongjiang Province Leading Talent Echelon Reserve Leader Funding Project.

      Ethical approval

      Not applicable.

      References

        • Asai S
        • Seki A
        • Akai Y
        • Tazawa H
        • Kakizoe H
        • Ravzanaaadii MA
        • et al.
        Nationwide external quality assessment of SARS-CoV-2 nucleic acid amplification tests in Japan.
        Int J Infect Dis. 2022; 115: 86-92
        • Brotons P
        • Perez-Argüello A
        • Launes C
        • Torrents F
        • Subirats MP
        • Saucedo J
        • et al.
        Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples.
        Int J Infect Dis. 2021; 110: 363-370
        • Campana C
        • van Koetsveld PM
        • Feelders RA
        • de Herder WW
        • Iyer AM
        • van Velthuysen MF
        • et al.
        Digital quantification of somatostatin receptor subtype 2a immunostaining: a validation study.
        Eur J Endocrinol. 2022; 187: 399-411
        • Dupuis H
        • Ghesquière L
        • Pierache A
        • Subtil D
        • Houfflin-Debarge V
        • Garabedian C.
        Evaluation and impact of fetal physiology training on fetal heart rate analysis.
        J Gynecol Obstet Hum Reprod. 2021; 50102185
        • Geisler S
        • Lytton SD
        • Toan NL
        • Nghia TH
        • Nam NM
        • Hung HV
        • et al.
        Neopterin levels and Kyn/Trp ratios were significantly increased in dengue virus patients and subsequently decreased after recovery.
        Int J Infect Dis. 2020; 91: 162-168
        • Li X
        • Zhao Y
        • Zhang Z
        • Zheng T
        • Li S
        • Yang G
        • et al.
        Correlations of magnetic resonance imaging classifications with preoperative functions among patients with refractory lateral epicondylitis.
        BMC Musculoskelet Disord. 2022; 23: 690
        • McHugh ML.
        Interrater reliability: the kappa statistic.
        Biochem Med (Zagreb). 2012; 22: 276-282
        • Qorri E
        • Takács B
        • Gráf A
        • Enyedi MZ
        • Pintér L
        • Kiss E
        • et al.
        A comprehensive evaluation of the performance of prediction algorithms on clinically relevant missense variants.
        Int J Mol Sci. 2022; 23: 7946
        • Rezaeipandari H
        • Mohammadpoorasl A
        • Morowatisharifabad MA
        • Shaghaghi A.
        Psychometric properties of the Persian version of abridged Connor-Davidson Resilience Scale 10 (CD-RISC-10) among older adults.
        BMC Psychiatry. 2022; 22: 493
        • Salvagno GL
        • Henry BM
        • Lippi G.
        The strength of association between pre-and post-booster BNT162b2 anti-SARS-CoV-2 antibodies levels depends on the immunoassay.
        Int J Infect Dis. 2021; 111: 65-67