Dear Editor,
We read with interest the article entitled “Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples” (
Brotons et al., 2021
). This study validates and implements an optimized screening method for the detection of SARS-CoV-2 ribonucleic acid, integrating the use of self-collected raw saliva samples, single-step heat-treated virus inactivation and ribonucleic acid extraction, and direct reverse transcription-quantitative polymerase chain reaction (RT-qPCR). Although this article provides valuable information, we believe that when the authors evaluated the diagnostic accuracy of saliva-based direct reverse transcription-polymerase chain reaction (RT-PCR) versus standard RT-PCR on nasopharyngeal swabs, some results are worth discussing. We noticed that the agreement of the two assays was not assessed.Generally, the question of agreement, or consistency among samples collecting data immediately arises due to the variability in different diagnosis methods. Thus, well-designed research studies must consequently incorporate procedures that measure agreement among the various data collectors (
McHugh, 2012
). There are a number of statistics that have been used to measure inter-rater reliability. A partial list includes overall accuracy (Asai et al., 2022
), Cohen's kappa (McHugh, 2012
), Pearson's R (Salvagno et al., 2021
), Spearman Rho (Geisler et al., 2020
), intraclass correlation coefficient (Rezaeipandari et al., 2022
), concordance correlation coefficient (Campana et al., 2022
), Krippen-dorff's alpha (Dupuis et al., 2021
), and Matthews correlation coefficient (Qorri et al., 2022
). Here, we will only consider the most common measures, Cohen's kappa and overall accuracy (Table 1).Table 1Weighted kappa value and overall accuracy for calculating agreement between saliva-based direct RT-qPCR and standard RT-PCR on nasopharyngeal swab
| Standard RT-qPCR | Saliva-based direct RT-qPCR | Overall accuracy | ||||
|---|---|---|---|---|---|---|
| k = 0.802 (strong agreement) | Positive | Negative | Inconclusive | Total | 97.36%, (22+273+0)/303 | |
| Positive | 22 | 0 | 1 | 23 | ||
| Negative | 0 | 273 | 0 | 273 | ||
| Inconclusive | 1 | 6 | 0 | 7 | ||
| Total | 23 | 279 | 1 | 303 | ||
Note: The data has been cited from the article published by
Brotons et al., 2021
and undergone modification. k is the weighted kappa value calculated by us.RT-qPCR, reverse transcription-quantitative polymerase chain reaction.
Generally, Cohen's kappa statistic is suitable for evaluating two raters (
McHugh, 2012
). In Cohen's kappa statistic, weighted kappa statistic should be used to calculate the inter-rater reliability in the presence of more than two categories (Li et al., 2022
).Weighted kappa is calculated as follows:
(1)
The value of is the proportion of objects put in the same category j by both raters and . The value of is the proportion of objects that rater assigned to category . According to
McHugh, 2012
, the kappa result should be interpreted as follows: 0-0.20 indicating no agreement, 0.21-0.39 as minimal agreement, 0.40-0.59 as weak agreement, 0.60-0.79 as moderate agreement, 0.80-0.90 as strong agreement, and 0.91-1.00 as almost perfect agreement.Therefore, according to the authors’ data, the weighted kappa value between saliva-based direct RT-qPCR and standard RT-PCR on nasopharyngeal swab evaluated by us was 0.802 (95% confidence interval = 0.669-0.935), indicating a strong agreement. The overall accuracy between the two assays was 97.36%.
CRediT authorship contribution statement
Tianfei Yu: Writing – original draft. Fangfang Liu: Data curation. Haichang Yin: Data curation. Nana Yi: Data curation. Ming Li: Writing – review & editing.
Declarations of competing interests
The authors have no competing interests to declare.
Funding source
This research was supported by a grant (LH2020C110) from the Joint Guidance Project of Natural Science Foundation of Heilongjiang Province of China, a grant (HLJ2019017) Chinese Ministry of Education “Chunhui Plan” International Scientific Research Cooperation Project, a grant (145109136) from the Fundamental Research Funds in Heilongjiang Provincial Universities and a grant from Heilongjiang Province Leading Talent Echelon Reserve Leader Funding Project.
Ethical approval
Not applicable.
References
- Nationwide external quality assessment of SARS-CoV-2 nucleic acid amplification tests in Japan.Int J Infect Dis. 2022; 115: 86-92
- Validation and implementation of a direct RT-qPCR method for rapid screening of SARS-CoV-2 infection by using non-invasive saliva samples.Int J Infect Dis. 2021; 110: 363-370
- Digital quantification of somatostatin receptor subtype 2a immunostaining: a validation study.Eur J Endocrinol. 2022; 187: 399-411
- Evaluation and impact of fetal physiology training on fetal heart rate analysis.J Gynecol Obstet Hum Reprod. 2021; 50102185
- Neopterin levels and Kyn/Trp ratios were significantly increased in dengue virus patients and subsequently decreased after recovery.Int J Infect Dis. 2020; 91: 162-168
- Correlations of magnetic resonance imaging classifications with preoperative functions among patients with refractory lateral epicondylitis.BMC Musculoskelet Disord. 2022; 23: 690
- Interrater reliability: the kappa statistic.Biochem Med (Zagreb). 2012; 22: 276-282
- A comprehensive evaluation of the performance of prediction algorithms on clinically relevant missense variants.Int J Mol Sci. 2022; 23: 7946
- Psychometric properties of the Persian version of abridged Connor-Davidson Resilience Scale 10 (CD-RISC-10) among older adults.BMC Psychiatry. 2022; 22: 493
- The strength of association between pre-and post-booster BNT162b2 anti-SARS-CoV-2 antibodies levels depends on the immunoassay.Int J Infect Dis. 2021; 111: 65-67
Article info
Publication history
Published online: August 16, 2022
Accepted:
August 7,
2022
Received:
July 28,
2022
Identification
Copyright
© 2022 The Author(s). Published by Elsevier Ltd on behalf of International Society for Infectious Diseases.
User license
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0) | How you can reuse 
Elsevier's open access license policy
Creative Commons Attribution – NonCommercial – NoDerivs (CC BY-NC-ND 4.0)
Permitted
For non-commercial purposes:
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article (private use only, not for distribution)
- Reuse portions or extracts from the article in other works
Not Permitted
- Sell or re-use for commercial purposes
- Distribute translations or adaptations of the article
Elsevier's open access license policy
