Evaluation of the performance of 25 SARS-CoV-2 serological rapid diagnostic tests using a reference panel of plasma specimens at the Uganda Virus Research Institute

Introduction Serological testing is needed to better understand the epidemiology of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus. Rapid diagnostic tests (RDTs) have been developed to detect specific antibodies, IgM and IgG, to the virus. The performance of 25 of these RDTs was evaluated. Methods A serological reference panel of 50 positive and 100 negative plasma specimens was developed from SARS-CoV-2 PCR and antibody positive patients and pre-pandemic SARS-CoV-2-negative specimens collected in 2016. Test performance of the 25 RDTs was evaluated against this panel. Results A total of 10 RDTs had a sensitivity ≥98%, while 13 RDTs had a specificity ≥98% to anti-SARS-CoV-2 IgG antibodies. Four RDTs (Boson, MultiG, Standard Q, and VivaDiag) had both sensitivity and specificity ≥98% to anti-SARS-CoV-2 IgG antibodies. Only three RDTs had a sensitivity ≥98%, while 10 RDTs had a specificity ≥98% to anti-SARS-CoV-2 IgM antibodies. Three RDTs (Autobio, MultiG, and Standard Q) had sensitivity and specificity ≥98% to combined IgG/IgM. The RDTs that performed well also had perfect or almost perfect inter-reader agreement. Conclusions This evaluation identified three RDTs with a sensitivity and specificity to IgM/IgG antibodies of ≥98% with the potential for widespread antibody testing in Uganda.


Introduction
Coronavirus disease 2019  was declared a pandemic by the World Health Organization (WHO) on March 11, 2020. Globally as of September 1, 2021, there have been nearly 218 million cases reported to the WHO with over 4.5 million deaths, while in Uganda 99 408 cases with over 30 0 0 deaths have been recorded ( WHO, 2021 ). Since this evaluation, the number of new cases and deaths has continued to rise.
Standard laboratory confirmation of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is based on the detection of unique viral sequences in nasopharyngeal samples by nucleic acid amplification test (NAAT) ( WHO, 2020b ). Although the priority intervention from a public health perspective has been to identify those with acute infection and to quarantine them and their immediate contacts in order to control the spread of infection, it has become apparent that it is also important to identify convalescent cases through antibody testing in order to better understand the https://doi.org/10.1016/j.ijid.2021.09.020 1201-9712/Crown Copyright © 2021 Published by Elsevier Ltd on behalf of International Society for Infectious Diseases. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ) epidemiology of the virus and thereby to introduce effective control measures. Antibody testing has traditionally been conducted using enzyme-linked immunosorbent assays (ELISAs) or more recently with rapid diagnostic tests (RDTs). ELISAs facilitate the testing of large numbers of specimens per run, while RDTs are lateral flow devices for individual specimens. RDTs typically give results in less than 30 minutes and are therefore ideal for use at the point of care (POC). During the early days of the pandemic, commercial ELISAs and RDTs were hard to come by and none were approved for use in Uganda.
Most people infected with SARS-CoV-2 have an incubation period of 3-7 days before the appearance of symptoms. IgM seroconversion occurs within 10-14 days and IgG seroconversion within 12-14 days after symptom onset ( Long et al., 2020 ;Lou et al., 2020 ;To et al., 2020 ;Zhao et al., 2020 ) and can be detected in less than 40% of infected people within 1 week of symptom onset and in 100% by day 15 ( Batra et al., 2020 ;Zhao et al., 2020 ). Antibodies can take much longer to develop in those with a subclinical or mild infection ( WHO, 2020b ). The strength of the antibody response depends on a number of factors including age, nutritional status, and disease severity amongst others ( WHO, 2020a ). IgM antibodies start to disappear by week 5, and by week 7 they are no longer detectable; IgG antibodies persist beyond week 7 ( Xiao et al., 2020 ). It is not clear whether antibodies confer immunity to re-infection, although the recurrence of COVID-19 illness appears to be very uncommon ( CDC, 2020a ).
As a result of global shortages of reagents for molecular testing, a number of groups ( Lassaunière et al., 2020 ;Zhao et al., 2020 ) have investigated the potential use of antibody tests, particularly those for IgM, either singly or in combination, to diagnose acute SARS-CoV-2 infection. However, a reliable diagnosis of infection by antibody testing is only possible in the recovery phase when the possibility of intervening has passed and consequently serological diagnosis is not recommended for informing clinical management or contact-tracing ( WHO, 2020b ).
In Uganda, as in many other countries, there is a sense of urgency to understand the epidemiology of the virus in order to implement effective control measures. This requires mass screening of the population for anti-SARS-CoV-2 antibodies to determine among other things: how many people have been infected with the virus and how this changes over time; the risk factors for infection such as age, ethnicity, domicile, or underlying health issues; the proportion of infected people with mild or asymptomatic infection and how long antibodies can be detected in individuals who have been infected ( CDC, 2021 ). Many countries are now testing for SARS-CoV-2 antibodies at the population level or in specific groups, such as health workers, close contacts of known cases, or within households ( WHO, 2020c ).
The tools to conduct mass serological screening, including ELISA and RDT kits with emergency use approval, slowly became available in mid-2020 and plans were made to evaluate their performance.
Laboratory tests including RDTs that detect antibodies to SARS-CoV-2 in people need validation to determine their accuracy and reliability. Inaccurate RDT results would have serious consequences and would affect pandemic control effort s. In Uganda, all new diagnostic assays that are introduced into the market must undergo incountry laboratory validation at the Uganda Virus Research Institute (UVRI), which is a designated WHO and Africa Centres for Disease Control and Prevention (CDC) SARS-CoV-2 reference laboratory, before being recommended to the Ministry of Health for use in the country. Both the US Centers for Disease Control and Prevention, and WHO also advise that diagnostic and antibody tests should be validated in appropriate populations and settings before they are recommended ( CDC, 2020b ;WHO, 2017 ).
For this evaluation, ELISAs and chemiluminescent microparticle immunoassays (CMIA) with one or more approvals under the US Food and Drug Administration (FDA), Emergency Use Authorization (EUA) or the WHO Emergency Use Listing (EUL), or with the European Conformité Européenne (CE) mark were procured (supplementary files, Appendix A). For the rapid tests, local distributors provided 25 serological RDTs for a cost-free evaluation at the UVRI (supplementary files, Appendix B). Nine distributors provided enough test kits for the evaluation of 150 samples, while 16 distributors provided test kits for less than 150 samples (73-125).

Serological reference panel
At UVRI, the normal practice to validate new antibody test kits is to evaluate their performance on a reference panel of well-characterized plasma specimens. In mid-2020, no commercial or WHO reference panels were available and consequently UVRI developed its own reference panel. Presumptive anti-SARS-CoV-2 antibody-positive specimens were selected from available qRT-PCRconfirmed, symptomatic and asymptomatic SARS-CoV-2 cases; the number of days post symptom onset was reported for some of the symptomatic cases. Presumptive SARS-CoV-2 antibody-negative specimens were selected from the UVRI repository of specimens collected during an HIV national serosurvey conducted in 2016, long before the COVID-19 pandemic.
Fifty samples with the above profile were included in the SARS-CoV-2-positive reference panel ( UVRI, 2021 ).
One hundred samples with the above profile were included in the SARS-CoV-2-negative reference panel ( UVRI, 2021 ).

Rapid diagnostic tests
A total of 25 serological RDTs (Appendix B) were evaluated against the characterized serological reference panel. The antigen(s), spike and/or nucleocapsid protein, targeted by the RDTs was not disclosed in the IFUs for most of the RDTs. The majority of RDTs had a single reading window with a control line, an IgM test line, and an IgG test line, while one (Biocredit) had two separate cassettes. Two RDTs had a reading window for IgM/IgG (Sino Care and Wondfo), while one had reading windows for IgG and IgA/IgM and not IgM (Antai). The manufacturers' IFUs were followed and the results were read by two technicians blinded to each other's results.

Statistical analysis
The statistician conducting the data analysis was blinded to the RDT identity. Only concordant results between technicians were used to evaluate the performance of the RDTs; inter-reader variability was also documented.

Sensitivity
The sensitivity was calculated as the number of specimens determined as positive by the two technicians for each RDT under evaluation, divided by the number of specimens tested from the positive panel; this was expressed as a percentage.

Specificity
The specificity was calculated as the number of specimens determined as negative by the two technicians for each RDT under evaluation, divided by the number of specimens tested from the negative panel; this was expressed as a percentage.

Accuracy
The accuracy was calculated as the proportion of RDT test results that agreed with the panel source (positive and negative panels) and was expressed as a percentage. The sensitivity, specificity, and accuracy calculations were performed using the proportion command in STATA 15 and confidence intervals (CI) were produced with the Wilson score method ( Newcombe, 1998 ).
Sensitivity, specificity, and accuracy were also determined for combined IgM/IgG (either or both IgM and IgG).

Ethical considerations
The evaluation protocol was reviewed and approved by the Research Ethics Committee of the UVRI and the Uganda National Council for Science and Technology (UNCST). The panels were unlinked to personal identifiers and the results could not be traced to individual patients. Consent to participate and to store samples for future use was also sought.

Performance
Results by individual RDT performance are summarized (in alphabetical order) in Table 1 . Most RDTs showed poor performance, with none showing both anti-SARS-CoV-2 IgM and IgG antibody sensitivity and specificity ≥98%. Many RDTs that performed well in at least one reading window showed good reactivity to anti-SARS-CoV-2 IgG antibodies, with 10 having a sensitivity ≥98%, while 13 had a specificity ≥98% ( Table 1 ). Only three RDTs had an anti-SARS-CoV-2 IgM antibody sensitivity ≥98%, while 10 RDTs had a specificity ≥98% for anti-SARS-CoV-2 IgM antibodies. Three RDTs (Autobio, MultiG, and Standard Q) had a sensitivity and a specificity ≥98% to combined IgM/IgG. There were seven RDTs that had an accuracy ≥98% for anti-SARS-CoV-2 IgG antibodies, with three (Boson, Standard Q, and VivaDiag) having an accuracy of 100%. There were four RDTs (Boson, MultiG, Standard Q, and VivaDiag) where both anti-SARS-CoV-2 IgG antibody sensitivities and specificities were ≥98%.

Inter-reader agreement
There was almost perfect agreement between the two technicians for the determination of IgG in 20/23 RDTs (87.0%) (where there was a reading window for IgG). Four of the RDTs had perfect agreement for IgG, with a kappa statistic of 100% (Biocredit, BTNX, MultiG, Standard Q, and VivaDiag). Agreement in determining IgM was much lower, with only 11/23 tests (47.8%) where there was almost perfect agreement ( Table 2 ).

Discussion
The WHO continues to review the evidence on antibody responses to SARS-CoV-2 infection and has published guidance on adjusting public health and social measures for the next phase of the COVID-19 response ( WHO, 2019 ). The development of accurate RDTs for the diagnosis of anti-SARS-CoV-2 IgM and IgG antibodies will benefit epidemiological and surveillance studies in identifying past COVID-19 symptomatic and asymptomatic infections including those in 'hot-spots'. This will serve as an aid in determining the extent of herd immunity, although for how long immunity will last, especially with the appearance of SARS-CoV-2 variants, is not yet known ( Aschwanden, 2021 ). Accurate RDTs detecting the relevant antibodies will benefit vaccine studies in identifying SARS-CoV-2 vaccine responders and for how long one remains immune to the virus.
This evaluation of 25 RDTs showed significant variation in performance, emphasizing the need for more input in research and development in order to come up with more accurate tests. There were only four RDTs that had a sensitivity and specificity ≥98% for anti-SARS-CoV-2 IgG, i.e., Boson, MultiG, Standard Q, and VivaDiag. Of these four RDTs, only Boson had a sensitivity ≥98% for anti-SARS-CoV-2 IgM (sensitivity was 100% and corresponding specificity was 87.0% for anti-SARS-CoV-2 IgM; Table 1 ). When combining IgM and IgG, this evaluation identified three RDTs (Autobio, MultiG, and Standard Q) with a sensitivity and specificity ≥98% for both IgG and/or IgM.
There was better agreement between the two technicians for result determination of IgG compared to result determination of IgM, with 87.0% having almost perfect agreement for IgG compared to only 47.8% for IgM. Five of the RDTs had perfect agreement for IgG, with a kappa statistic of 100% (Biocredit, BTNX, MultiG, Standard Q, VivaDiag), while there was no such finding for IgM.
Combination IgG/IgM RDTs can provide unclear results given the po tential for cross-reactivity of antibodies with other coronaviruses and the often poor specificity of IgM serological assays ( IDSA, 2020 ). The use of some of these combination IgG/IgM RDTs may not be simple to interpret in the field if only the IgG test component of the RDT gives accurate results. Some RDTs had one cassette with separate IgM and IgG bands, while others had separate cassettes for IgG and IgM. If one used an RDT with a cassette having both IgM and IgG and with IgM having poor specificity, then the final result describing past exposure may be confusing and inaccurate, unless the reader disregards the IgM results altogether.
For the above reasons, it may be more practical to recommend the use of RDTs with both a sensitivity and specificity of ≥98% for IgM and IgG combined; Autobio, MultiG, and Standard Q fit this profile.   All of the test kits recommended above for IgG (Boston, Multi G, Standard Q, and VivaDiag) had almost perfect agreement; similarly for either IgG or IgM (IgG/IgM), MultiG and Standard Q had almost perfect agreement, while Autobio had substantial agreement.
To evaluate the performance of the 25 RDTs, a wellcharacterized reference panel of plasma specimens from SARS-CoV-2 qRT-PCR-positive individuals that had been screened by means of six SARS-CoV-2 ELISA/CMIA IgM and IgG assays was used. This approach is different from many other studies that have used any samples from qRT-PCR-positive individuals. The reason for this approach is that not all qRT-PCR-positive individuals have antibodies; the presence of antibodies depends on the time since infection, the severity of infection, and a number of other factors. A further step to increase the relevance of the reference panel was to include ELISA/CMIA kits that targeted either the nucleocapsid or the spike antigen where possible.
There are other published evaluation reports that have similarly shown poor serological RDT performance compared to that re-ported by the manufacturer ( Deeks et al., 2020 ;Jacobs et al., 2020 ;Vauloup-Fellous et al., 2021 ). Since some of these other studies used samples from any qRT-PCR-positive individuals, they showed lower sensitivity in the first week post symptom onset, with improved performance at later time points. Few studies have evaluated RDTs using samples taken beyond 1 month post symptom onset. Furthermore, there is limited information on the performance of these RDTs in asymptomatic participants ( Deeks et al., 2020 ).
While a serial or orthogonal ( Xu et al., 2020 ) testing approach has been recommended for surveillance especially when using ELISAs ( CDC, 2020b ), here we propose that parallel testing with RDTs could also be a viable approach. With this approach, two RDTs with ≥98% sensitivity and specificity could be used together with an equally accurate RDT as a tie-breaker for discrepant results.
This study evaluated only 25 RDTs, but we are aware that many SARS-CoV-2 RDT kits are on market. With the well characterized panel of samples we now have and with WHO serological reference standards now available from the National Institute for Biological Standards and Control (NIBSC), the validation of additional RDTs will be quicker.
The study had some limitations. The selection of the RDTs to evaluate was dictated by what was provided to us by the local distributors in the country. Some distributors provided test kits for evaluation, less than the desired 150 samples. Another limitation was that it was not possible to procure an IgM ELISA based on the nucleocapsid protein. Some of the ELISA kits procured did not perform well and hence the results with these kits were excluded from the development of the serological reference panel. Subsequent to the completion of this evaluation, it was discovered that authorization for the use of some of the RDTs had been revoked; they were removed from the FDA EUA notification list as of February 23, 2021.
In conclusion, using a reference panel of well characterized plasma samples and considering the sensitivity and specificity of combined IgG and IgM results, and inter-reader agreement, this evaluation found that three RDTs performed well: Autobio, MultiG, and Standard Q. Where only the IgG result is of interest, there were four RDTs that performed well: Boson, MultiG, Standard Q, and Vi-vaDiag.