Evaluation of In-House ELISA for Antirabies Antibodies Detection in Domestic Canine

Indonesia is known to be endemic for rabies in several areas, especially in Sumatra, Kalimantan, Sulawesi, and Flores Islands. Currently, vaccinating dogs has been shown to be the most cost-effective strategy for preventing rabies in humans. Postvaccination monitoring should be carried out to evaluate the success of vaccination by measuring antibody titers in serum of vaccinated dogs. Serological methods for monitoring rabies-specific antibody titers can be carried out using enzyme-linked immunosorbent assay (ELISA) methods as recommended by the World Organization for Animal Health (WOAH). Therefore, the development of the in-house ELISA (BukTi-Vet) that we have carried out in order to support postvaccination monitoring in dogs needs to be evaluated for its diagnostic performance compared to commercial ELISA kits. The diagnostic performance of each ELISA kit was evaluated using 111 known positive and 47 negative serums. Each known positive and negative serum will be tested using the three rabies ELISA kits used in this study. BukTi-Vet is an in-house ELISA for the detection of rabies-specific IgG antibodies that have been developed with sensitivity, specificity, and accuracy of 98.19%, 97.87%, and 98.1%, respectively. Based on the value of its positive and negative clinical utility index, BukTi-Vet is excellent for use in immunoassays directed for confirmatory (0.97) as well as screening (0.94) tests. BukTi-Vet shows a very good agreement with both Platelia II and RFFIT, so it is convincing to be further refined into a diagnostic kit. Tests of field sera from dogs vaccinated with various vaccines should be performed to provide more complete information on diagnostic performance. BukTi-Vet showed a very good agreement with RFFIT, while Pusvetma and Platelia II only showed good agreement. The average value of BukTi-Vet compatibility with RFFIT can reach 94%.


Introduction
Rabies is one of the oldest zoonotic diseases in the world and causes high fatality in humans. Tis disease is caused by the Rabies virus (RABV) and other Lyssavirus species from the Rhabdoviridae family [1,2]. Despite the lack of accurate data on the global burden of neglected tropical diseases, estimates of direct deaths from rabies are most commonly transmitted by bites from rabid animals, especially canines [1]. Most human deaths from rabies occur in Asia and Africa. Estimates of human deaths due to endemic rabies-carrying dogs in Asia and Africa annually exceed 30,000 and 23,000, respectively [1].
Indonesia is known to be endemic for rabies in several areas, especially in Sumatra, Kalimantan, Sulawesi, and Flores Islands [3]. However, outbreaks of rabies in animals and humans have been reported on two islands previously known as rabies-free zones, namely, Bali and Sumbawa Islands. Te rabies outbreak in Bali began in 2008-2011 [4][5][6] and cases continued until 2016 [7]. Recently, a rabies outbreak on the island of Sumbawa occurred in 2018-2019 [8].
Currently, rabies-infected dogs are the major reasons for the high incidence of human rabies, and therefore, vaccinating dogs has shown to be the most cost-efective strategy for preventing rabies in humans [2]. Postvaccination monitoring should be carried out to evaluate the success of vaccination by measuring antibody titers in serum of vaccinated dogs. Serological methods for monitoring rabiesspecifc antibody titers can be carried out using virus neutralization and enzyme-linked immunosorbent assay (ELISA) methods as recommended by the World Organization for Animal Health (WOAH) previously known as Ofce International des Epizooties (OIE) [9]. Two wellknown virus neutralization methods are the fuorescent antibody virus neutralization (FAVN) and the rapid fuorescent focus inhibition test (RFFIT) [9].
Even though the RFFIT is known to be the most reliable test for the evaluation of successful vaccination, unfortunately, this is time-consuming, expensive, and requires a live rabies virus; sometimes, the results cannot be read of due to the cytotoxic efect on the cells [10,11]. Virus neutralization technically requires highly skilled technicians, hard to standardize, sufers from some interlaboratory variations, difcult to perform at weekly intervals, and requires containment facilities, which is only conveniently carried out in reference laboratories [10,[12][13][14]. In order to overcome this, ELISA deserves to be developed as an alternative to RFFIT [11]. ELISA is considered technically simple, afordable, higher in throughput, safer, and rapid compared to RFFIT [1,15]. ELISA does not require live virus and high-containment facilities, easy to be validated, and has the greater assurance of consistent results [1,16]. Virus neutralization (RFFIT) and ELISA have also been found to have good compatibility [10,17]. Terefore, ELISA is very suitable for routine serological testing with a large number of samples, so it is very suitable for monitoring antibody titers after rabies vaccination. Te ELISA-based technique was recently adopted by the WOAH or the screening of sera for international pet movement schemes [18]. Te commercial ELISA kit commonly used in Indonesia requires quite expensive costs for the imported kit, while domestic products used have low accuracy. Terefore, the development of the in-house ELISA that we have carried out in order to support postvaccination monitoring in dogs needs to be evaluated for its diagnostic performance compared to commercial ELISA kits.

Rapid Fluorescent Focus Inhibition Test (RFFIT).
Te RFFIT technique was performed by following the procedure described by the WOAH [9]. Briefy, the positive control (0.5 IU/ml) and various dilutions of serum known positive and negative were incubated for 24 h in the presence of the challenge virus standard (CVS) 11 strain suspension infecting baby hamster kidney (BHK)-21 cells. Tey were grown in Dulbecco's Modifed Eagle's Medium (DMEM) (Termo Scientifc, USA) supplemented with 10% fetal bovine serum (FBS) (Termo Scientifc, USA), antibiotic and antifungal using Gibco Antibiotic-Antimycotic (Termo Scientifc, USA), and incubated in an incubator at 37°C with 50% CO 2 atmosphere. After 24 hours of incubation, the cells were washed and fxed and then incubated with fuorescein isothiocyanate (FITC)-conjugated antirabies monoclonal antibody (Fujirebio, Japan). Te serum titers were expressed in IU/ml (international units per milliliter) by determining the last dilution of serum which inhibited 50% of the initial fuorescent foci.

Enzyme-Linked Immunosorbent Assay (ELISA).
Te supernatant from cell cultures containing the Pasteur strain rabies virus was centrifuged at 5000 g, 4°C for 15 minutes. Te supernatant was then fltered and concentrated using a 10 K protein concentrator (Termo Scientifc, USA). Te fltrate was separated, and then, lysis bufer (Qiagen, Germany) was added to it (2 : 1). Protein concentration was determined by the Bradford method using the BioRad Protein Assay (BioRad, France).
Te fat bottomed MaxiSorp microplate (Nunc, Denmark) was coated with antigen at a concentration of 2 μg/ml in carbonate-bicarbonate bufer (Sigma Chem., USA) and incubated at 4°C (in a refrigerator) overnight. After three times washing using phosphate-bufered saline containing 0.05% Tween-20 (PBST), it was blocked using 0.5% bovine serum albumin (BSA) (Sigma Chem., USA) in PBST and incubated overnight in the refrigerator. Following three times washing with PBST, the microplate was ready to be used for immunoassay. Tis kit (including the procedure) hereinafter is referred to as an in-house ELISA rabies kit to be evaluated in this study and coded as BukTi-Vet.
Te ELISA procedure using BukTi-Vet will be briefy described in this section. Te positive control was diluted to obtain the control serum with titers of 4 IU, 2 IU, 1 IU, 0.5 IU, 0.25 IU, and 0.125 IU. Te known positive and known negative serums were diluted at 1 : 200 in PBST, respectively. All serum was put into each well as much as 100 μl. Incubation was carried out at room temperature (23-25°C) for one hour and then followed by three times washing using PBST. A diluted (1 : 20,000) anticanine IgG-HRP conjugate (SouthernBiotech, USA) was added to each well of the microplate. Subsequent incubation and washing were carried out in the same way as the previous step. Finally, the substrate 3,3′, 5,5-tetramethylbenzidine (TMB) solution (Invitrogen, USA) was added and incubated for about 5 minutes before being stopped using 2N H 2 SO 4 . Microplates were read at a wavelength of 450 nm using a BioTek ELx800 microplate reader (BioTek, USA). Te absorbance value was then converted into an antibody titer in EU/ml.
Te immunoassay procedure using a commercial ELISA kit was carried out according to the protocol described by the manufacturer. Te commercial ELISA kit used in this study is a commercial kit that is widely used in Indonesia, namely, the ELISA Rabies ® kit (Veterinary Pharma Center, Indonesia) hereinafter referred to as the Pusvetma. Another commercial kit is the Platelia ® Rabies II ad usum veterinarium ELISA kit (BioRad, France) and hereinafter referred to as the Platelia II.

Evaluation of ELISA Kit
Performance. Te diagnostic performance of each ELISA kit was evaluated using 111 known positive and 47 negative serums. Each known positive and negative serum will be tested using the three rabies ELISA kits used in this study. Seropositive and seronegative of the tested serum were determined using a threshold titer value of 0.5 EU/ml. Evaluation of the diagnostic performance of each rabies ELISA kit was analyzed using MedCalc V.20 and the Clinical Utility Index Calculator [19]. Furthermore, an evaluation was also carried out to determine the diagnostic performance of the three rabies ELISA kits against serum derived from the use of diferent vaccines. Te second evaluation was conducted to determine the performance of each rabies ELISA kit against 55 serum known positives from dogs vaccinated with Rabivet ® and 47 serum known negatives. Te third evaluation was conducted to determine the performance of the three rabies ELISA kits against 56 serum known positives from dogs vaccinated using Rabisin ® and 47 serum known negatives.

Interassay Agreement and Simulation of Serial and Parallel Testing.
Interassay agreement among the three rabies ELISA kits was carried out on all of 158 known positive and negative serums or also partially on known positive serum from dogs vaccinated with Rabivet ® or Rabisin ® . Te analysis of the results of interassay agreement among the three tests of the rabies ELISA kit was carried out using Agreestat360 [20] and the interpretation of the level of agreement was based on the benchmark from Altman [21]. Serial and parallel test simulations among the three ELISA kits were evaluated to determine the improvement in diagnostic test performance that could be achieved with the combination of the two ELISA kits. Simulation analysis of serial and parallel tests was carried out using Epitools [22].

Interassay Agreement of Tree ELISA Kits with RFFIT.
Te three ELISA kits were also evaluated qualitatively for their interassay agreement compared to the virus neutralization test using the RFFIT method. A total of 48 known positive and negative serums were selected randomly, and the results of their agreement were compared between the three ELISA methods with RFFIT. Te test results were analyzed by Agreestat360 [20], and the interpretation of the level of concordance was based on the benchmark from Altman [21].

Evaluation of In-House ELISA Performance by Diferent
Operators. Another series of immunoassays using the inhouse ELISA kit (BukTi-Vet) was carried out to evaluate the reliability of the kit by diferent operators. Operators consist of 2 people who have experience working with ELISA (Operators 1 and 2) and 1 person (Operator 3) who never worked with ELISA at all. Each operator worked on 80 serum known positive and negative which had previously been randomly selected. Te results of each operator were analyzed with MedCalc V.20, AgreeStat 360, and Clinical Utility Index Calculator.

Comparative Performance of In-House ELISA (BukTi-Vet).
Overall, the in-house ELISA kit that has been developed (BukTi-Vet) has excellent accuracy (98.1%) with a sensitivity and specifcity of 98.19% and 97.87%, respectively (Table 1). Tis result was not signifcantly diferent (p > 0.05) compared to Platelia II which also had an accuracy of 94.94% with a sensitivity and specifcity of 94.59% and 95.75%, respectively. On the other hand, the diagnostic performance of BukTi-Vet had a very signifcant diference (p < 0.01) compared to other commercial ELISA kits, namely, Pusvetma which only had an accuracy of 87.34% with a sensitivity and specifcity of 82.88% and 97.87%, respectively. However, between the two commercial ELISA kits, namely, Pusvetma and Platelia II, there was no signifcant diference in diagnostic performance (p > 0.05).
Te clinical utility index (CUI) value consists of a confrmatory test (CUI +ve) and a screening test (CUI −ve). Both clinical utility index values for BukTi-Vet are categorized as an excellent utility. Meanwhile, Pusvetma has a good clinical utility index value for the confrmatory test (CUI +ve) and screening test (CUI −ve). Platelia II has an excellent clinical utility index value for confrmatory tests (CUI +ve) and a good utility index for screening tests (CUI −ve). However, the diagnostic performance of Platelia II is better than Pusvetma because it has a higher clinical utility index score. Tis result is supported by evidence of a high false-negative rate in Pusvetma ( Figure 1).
Evaluation of the diagnostic performance of each known positive serum from two diferent vaccines showed surprising results. Te diagnostic performance of the three ELISA kits against dog serum vaccinated using Rabivet ® showed excellent accuracy with values ranging from 92.38% to 97.14% ( Table 2). All three showed a diagnostic performance that was not signifcantly diferent (p > 0.05) based on the area under the curve (AUC) value from the receiver operating characteristic (ROC) curve analysis. However, in the assay using known positive serum from dogs vaccinated with Rabisin ® , only Platelia II and BukTi-Vet had excellent accuracy of 96.08% and 97.09% (Table 3). Based on AUC value from the ROC curve analysis, the diagnostic Veterinary Medicine International

Interassay Agreement among Tree ELISA Kits and RFFIT.
Overall, the results of the analysis of the interassay agreement between BukTi-Vet and Platelia II showed a very good agreement of all coefcient values in accordance with the benchmarking of Altman [21]. Meanwhile, Pusvetma only has a good agreement with Platelia II and BukTi-Vet on all coefcient values (Table 4). Te interassay agreement between the three ELISA kits for known negative and positive serum from dogs vaccinated with Rabivet ® showed a very good agreement (Table 5). On the other hand, a very good agreement when evaluating known negative and positive serum from dogs vaccinated with Rabisin ® only occurred between Platelia II and BukTi-Vet (Table 6).
Te analysis of the interassay agreement between two commercial ELISA kits and one in-house ELISA kit showed satisfactory results. BukTi-Vet showed a very good agreement with RFFIT, while Pusvetma and Platelia II only showed good agreement (Table 7). Te average value of BukTi-Vet compatibility with RFFIT can reach 94%.

Sensitivity and Specifcity Improvement by Serial and
Parallel Testing Strategy. Applications in combination with two diferent ELISA kits sometimes improve diagnostic performance. Te combination of two of the three ELISA kits in this study, either serially or in parallel, has been shown to improve the sensitivity and specifcity (Table 8). Te serial combination of two of the three kits will increase the specifcity up to 99%. Te combination of two of the three kits in parallel will also increase the sensitivity up to 99%.

Diagnostic Performance of In-House ELISA under Different Operators.
In general, BukTi-Vet maintained its diagnostic performance even though it was performed by an operator who had never previously worked with ELISA. Comparing the test results from the three operators, there was no signifcant diference (p > 0.05) in the performance of the BukTi-Vet kit, even though the inexperienced operator (operator 3) caused a decrease in the specifcity of 13.04% (Table 9). However, when compared with the original performance of BukTi-Vet, the test results of the third operator caused a very signifcant reduction in performance (p < 0.01), while the test results of the frst and second operators did not cause a signifcant change in their diagnostic performance (p > 0.05).

Discussion
Te results of this study proved that Pusvetma has a diagnostic accuracy similar to the previous report of Dartini et al. [23]. Tey reported that the sensitivity and specifcity were 96.8% and 73.5%, respectively, with an estimated diagnostic accuracy of 86.6%. In our study, the sensitivity, specifcity, and diagnostic accuracy of Pusvetma were 82.88%, 97.87%, and 87.34% (Table 1). Tese results also indicate a possible inconsistency of diagnostic performance Terefore, the clinical utility of Pusvetma for case fnding (confrmation) and screening (ruling out) is categorized as a good utility. However, due to indications of inconsistent diagnostic performance, the ELISA kit (Pusvetma) has the potential to cause doubt and uncertainty to estimate the actual condition, especially in serum samples from animals   6 Veterinary Medicine International whose vaccination status and postvaccination immune response are unknown. Several existing reports on the diagnostic performance of Platelia II are relatively consistent which refers to the sensitivity value being always lower than the specifcity. In general, the sensitivity of Platelia II is often below 90%. It has been reported that the sensitivity, specifcity, and accuracy of Platelia II were 88.8%, 98.9%, and 90.5%, respectively, when    Veterinary Medicine International using positive control serum from the WOAH [17]. Tis diagnostic performance is then used as the fnal performance in the leafet that comes with this ELISA kit, namely, Platelia Rabies II ad usum veterinarium (or simply called Platelia II). However, in the same report, the authors also informed that the sensitivity, specifcity, and accuracy of Platelia II were 91.1%, 97.8%, and 92.2%, respectively, when using R4b as a positive control serum derived from the kit itself [17]. Another report documented that the sensitivity and specifcity of Platelia II were 83.9% and 100% with an estimated accuracy of about 85.3% when using R4b as the positive control serum included in the kit [24]. A similar report also stated that the sensitivity and specifcity of the Platelia II ELISA kit were 78.2% and 100% with an estimated accuracy of about 89.1% using the R4b-positive control serum included in the kit [14,25]. Based on the data reported by Servat et al. [17], the CUI +ve and CUI −ve values were calculated as 0.88 (excellent utility) and 0.62 (fair utility).
Otherwise, based on the data reported by Knoop et al. [24], it can be calculated that the value of CUI +ve and CUI −ve is 0.84 (excellent utility) and 0.38 (poor utility). Terefore, the clinical utility of Platelia II for case fnding (confrmation) is an excellent utility, while for screening (ruling out), it is a poor or fair utility. Referring to the data from the report above, Platelia II is not suitable for testing directed for screening purposes. All of the research results that have been reported by several authors above are still below the diagnostic performance of Platelia II as evaluated by this study. Our study showed that the sensitivity, specifcity, and diagnostic accuracy of Platelia II were 94.59%, 95.75%, and 94.4%, respectively. Te CUI +ve and CUI −ve values in our study were 0.93 (excellent utility) and 0.85 (excellent utility). Terefore, based on our study, the clinical utility of the Platelia II ELISA kit for case fnding (confrmation) and screening (ruling out) is categorized as excellent. Te clinical utility performance that is close to our study is the report that came from Servat et al. [17] when using R4b as a positive control serum that obtained CUI +ve and CUI −ve values of 0.91 (excellent utility) and 0.66 (good utility). Unfortunately, these data are not referred to as the fnal diagnostic performance in Platelia II.
BukTi-Vet has provided satisfactory diagnostic performance compared to the other two commercial ELISA kits. Te results of the evaluation of diagnostic performance are also consistent, which refers to the sensitivity value being always higher than the specifcity, both of which always have a value above 95%. Tese advantages can be observed from four points. First, the diagnostic accuracy is always above 90%, while the comparison commercial kit is below 90%. Second, the false-negative values were lower than the two comparison ELISA kits ( Figure 1). Tird, it has very good compatibility with RFFIT compared to the other two ELISA kits (Table 7). Fourth, based on the analysis of the clinical utility index (Table 1), BukTi-Vet is categorized as an excellent utility for confrmation tests (case fnding) and screening tests (ruling out). Tis condition is in accordance with the use of ELISA for the screening of sera for international pet movement schemes as adopted by the WOAH [18]. On the other hand, from this study, it is notable that Platelia II can be categorized as excellent for the confrmation test and quite good for the screening test. Although Pusvetma is categorized as a good utility for confrmation and screening tests, with its inconsistent diagnostic performance, it is necessary to reconsider its use for both purposes.
Te correlation of the test between ELISA and RFFIT is often used as the basis for assessing serological diagnostic performance for the detection of rabies-specifc antibodies. However, it should be understood that the two methods are actually not always comparable, especially the diferences in immunoglobulin classes detected by the two methods [26]. Te RFFIT is also known as a functional assay in which the ability of antibodies to inhibit rabies virus infection is determined (known as virus-neutralizing antibodies), while the ELISA is a nonfunctional assay that detects certain classes of binding antibodies to viral antigens [1,18,27]. Tis is one of the main factors that sometimes can lead to incompatibility between the two methods. Under experimental or laboratory conditions, both ELISA and RFFIT techniques have good compatibility, but in feld conditions, they are very likely to give diferent results, so their compatibility is low [18]. Testing between ELISA and RFFIT using controlled laboratory serum (known as positive and negative) can result in very good agreement. Tis condition is enough to give an expectation that when the ELISA kit is applied to feld samples, the diference with RFFIT is not excessive.
Considering this evidence, the diferences between Pusvetma and BukTi-Vet in the level of concordance against RFFIT were not associated with diferences in the immunoglobulin classes being detected. Te basic concept for the development of BukTi-Vet is very similar to that of Pusvetma. Both use native protein from whole virus lysate from the Pasteur strain. Tese diferences should be related to the formulas and techniques we have used for our in-house ELISA development. In simple terms, BukTi-Vet has the advantage of distinguishing seropositive and seronegative compared to Pusvetma giving it a better compatibility with RFFIT.
Evaluation of BukTi-Vet showed fairly good stability of diagnostic performance. Tis evidence comes from the results of tests carried out by three operators with diferent skills who were still able to maintain diagnostic performance without signifcant diferences (p > 0.05) between the three of them. Despite this, some values of the diagnostic performance indicators were seen to decrease, but the decrease was still within the tolerance threshold so that the overall diagnostic accuracy could still be maintained at a value of 95% (Table 9). Te signifcant decrease compared to the original performance (p < 0.05) occurred in the tests carried out by inexperienced operators who had never even worked with ELISA. Tis also gives an indication that the skill and experience of the operator will afect the performance of the ELISA diagnostic kit used in the real environment.
Te interassay agreement between the three ELISA kits showed that BukTi-Vet and Platelia II consistently had a very good agreement ( ® was categorized as good agreement. Tis indicates that Pusvetma has a tendency of different diagnostic performance in seropositive sera from vaccinated animals using diferent vaccines. Te ability to measure rabies-specifc antibodies from Pusvetma has a better preference for serum from dogs vaccinated with Rabivet ® . Tis is understandable because both the Rabivet ® vaccine and the rabies ELISA kit from Pusvetma are produced by the same company. On the other hand, Platelia II and BukTi-Vet have similar detection capabilities against seropositive serum from dogs vaccinated with Rabivet ® and Rabisin ® . Tis condition is consistent with the similar distribution of antibody titers from the two vaccine types used in known positive serum, both tested with Platelia II and BukTi-Vet ( Figure 2). Te distribution of antibody titers tested using Pusvetma on both known positive sera from diferent vaccine types was in contrast to the results of the other two ELISA kits. Tese data raise concerns that the diagnostic performance of Pusvetma is not good enough and inconsistent when used to monitor postvaccination antibody levels in serum from dogs using diferent vaccines. Based on the AUC score of ROC, the diagnostic performance of Pusvetma was not signifcantly diferent (p > 0.05) from Platelia II and BukTi-Vet , when used for known positive serum from dogs, vaccinated with Rabivet ® (Table 2). However, when used for known positive serum from dogs vaccinated with Rabisin ® (Table 3), the diagnostic performance of Pusvetma showed a signifcant diference (p < 0.05) with PlateliaII and BukTi-Vet.
Te strategy that can be applied to increase the diagnostic sensitivity and specifcity of Pusvetma is to perform a combination of serial and parallel testing. Te combination of serial testing with Platelia II and BukTi-Vet is calculated to increase the diagnostic specifcity up to 99.91% and 99.95% (Table 8). Likewise, parallel testing between Pusvetma and Platelia II and BukTi-Vet resulted in a diagnostic sensitivity of up to 99.07% and 99.69%, respectively. However, the decision to carry out serial and parallel testing is practically not easy and must be done with caution because it remains open to the incompatibility of the test results. Te most appropriate action is to use an ELISA kit that has good diagnostic performance and has good test compatibility with other commercial kits.

Conclusions
BukTi-Vet is an in-house ELISA for the detection of rabiesspecifc IgG antibodies that have been developed with sensitivity, specifcity, and accuracy of 98.19%, 97.87%, and 98.1%, respectively. Based on the value of its clinical utility index, BukTi-Vet is excellent for use in immunoassays directed for confrmatory as well as screening tests. BukTi-Vet shows a very good agreement with both Platelia II and RFFIT, so it is convincing to be further refned into a diagnostic kit. Tests of feld sera from dogs vaccinated with various vaccines should be performed to provide more complete information on diagnostic performance.

Data Availability
Te data used to support the fndings of this study are available from the corresponding author upon request.

Conflicts of Interest
Te authors declare that they have no conficts of interest.