Receiver Operating Characteristic (ROC) to Determine Cut-Off Points of Biomarkers in Lung Cancer Patients

The role of biomarkers in disease prognosis continues to be an important investigation in many cancer studies. In order for these biomarkers to have practical application in clinical decision making regarding patient treatment and follow-up, it is common to dichotomize patients into those with low vs. high expression levels. In this study, receiver operating characteristic (ROC) curves, area under the curve (AUC) of the ROC, sensitivity, specificity, as well as likelihood ratios were calculated to determine levels of growth factor biomarkers that best differentiate lung cancer cases versus control subjects. Selected cut-off points for p185erbB-2 and EGFR membrane appear to have good discriminating power to differentiate control tissues versus uninvolved tissues from patients with lung cancer (AUC = 89% and 90%, respectively); while AUC increased to at least 90% for selected cut-off points for p185erbB-2 membrane, EGFR membrane, and FASE when comparing between control versus carcinoma tissues from lung cancer cases. Using data from control subjects compared to patients with lung cancer, we presented a simple and intuitive approach to determine dichotomized levels of biomarkers and validated the value of these biomarkers as surrogate endpoints for cancer outcome.


Introduction
The role of biomarkers in disease prognosis continues to be an important investigation in many cancer studies. More specifically, the central role of growth factors and their receptors in the neoplastic process makes evaluation of these biomarkers pivotal in the early detection, diagnosis and development of targeted therapies [1]. Biomarkers have also been used as surrogate endpoints for cancer outcome. For example, many Phase II chemoprevention studies in patients at highrisk for cancer utilize biomarkers as surrogate endpoints of efficacy [2]. This allows for smaller, less expensive, and faster studies as compared to studies that utilize cancer as the endpoint [3].
Many studies have identified EGF-r 3 and two of its ligands, EGF and TGF-α, and a closely related receptor, p185 erbB−2 , as some of the major growth factors and growth factor receptors involved in neoplastic formations [4]. The role of these biomarkers in lung cancer has been studied by many investigators [5][6][7][8][9][10][11][12]. However, a great heterogeneity in the incidence of overexpression and in the prognostic significance of these growth factors and receptors has been observed both within and between various tumor types, including cancers of the lung.
In order for these biomarkers to have practical application in clinical decision making regarding patient treatment and follow-up, it is common to dichotomize patients into those with low vs. high expression levels [13]. Receiver operating characteristic (ROC) analysis is a widely-used method for comparing diagnostic accuracy of laboratory tests. The ROC curve is constructed by plotting the true positive rate (sensitivity) on the y-axis as a function of false positive rate (1specificity) on the x-axis. An ideal ROC curve climbs rapidly towards the upper left corner of the graph which indicates good discrimination property. Pina et al. [13] showed the use of ROC in obtaining optimum cutoff points for serum tumor markers in lung cancer patients. They also calculated the area under the ROC curve as a quantitative measure of the discrimination power of tumor markers between lung cancer cases vs. healthy controls or subjects with non-neoplastictype pulmonary pathologies. Furthermore, Choi has discussed how the slope of the ROC can be used to calculate different types of likelihood ratios [14].
This paper utilizes a case-control study of 60 lung cancer patients and 60 control patients (without lung cancer) to determine levels of biomarkers of growth factors (specifically EGFR, TGF-α, p185 erbB−2 ) and fatty acid synthase (FASE), a major component of the de novo fatty acid biosynthetic pathway, that will best differentiate between the two groups. These biomarkers were measured on a semi-quantitative scale using immunohistochemistry techniques [15]. Receiver operating characteristic (ROC) curves, area under the curve (AUC) of the ROC as well as likelihood ratios were calculated to determine levels of biomarkers that best differentiate lung cancer cases and control subjects.

Study population
Study subjects were selected from patient populations of the Veterans Administration Hospital and University Hospital in Birmingham, AL. Sixty noncancer specimens of the lung (controls) were selected at random from a list of lung surgeries performed between 1988-1998 for lung diseases or conditions other than lung cancer. From a patient list of SCC of the lung who had undergone surgery between 1989-1996, sixty cases were selected at random. Formalin fixed, paraffin embedded sections of the lung were obtained from the archival collection. One tissue block from each non-cancer patient was selected to provide a section that contained adequate normal specimen. Tissue blocks from each SCC patient were se-lected to provide sections that contained uninvolved and carcinoma specimens. Immunohistochemistry scores were evaluated for 4 biomarkers (EGFR, p185 erbB−2 , TGF-alpha, FASE). Methods of immunohistochemical analysis have been reported previously [15]. Briefly, biomarkers were graded on the intensity of immunostaining (0-4) as well as the percentage of cells at each intensity. The sum of the product of the intensity of immunostaining and percent cells at each intensity were calculated to arrive at a semi-quantitative immunohistochemistry score.

Analysis methods
Descriptive statistics were calculated to summarize immunohistochemistry scores from normal tissues of control subjects and from uninvolved and carcinoma tissues in lung cancer cases. Univariate comparison of scores between lung cancer cases versus controls were performed using Wilcoxon rank-sum tests while within patient comparison of scores were performed using Wilcoxon signed-ranks test.
Continuous levels of immunohistochemistry scores for each biomarker were grouped into multiple categories based on the distribution of biomarker scores. For each biomarker, a logistic regression model was fitted for cases versus controls using categorized levels of the biomarker score as independent variable. ROC curves based on the logistic regression model were constructed. Sensitivity, specificity, likelihood ratios (LR), and AUC of the ROC based on the trapezoidal rule were calculated. Values of AUC closer to 100% indicate better discrimination power of a biomarker between cases versus controls. The slope of the ROC curve represents a likelihood ratio for a diagnostic test [14,16]. The multiple LR [14] was calculated and presented in this paper. Optimal cut-off points for biomarkers were determined based on the best balance of sensitivity and specificity along with larger increases in LR's. Logistic regression models and the corresponding ROC curves were generated for the comparison of normal specimens in controls versus uninvolved specimens from lung cancer cases and for the comparison of normal specimens in controls versus carcinoma specimens from lung cancer cases. Table 1 shows mean ± standard deviation, median, and range of immunohistochemistry scores in lung can-cer cases and controls by cellular localization (i.e. basal, luminal, cytoplasm, membrane). Across all biomarkers, lung cancer cases exhibited higher scores compared to control subjects. For example, the mean level of basal p185 erbB−2 score was 0.51 for controls compared to 0.86 from uninvolved tissues of lung cancer cases. Furthermore, an increasing trend of scores from normal tissues of controls to carcinoma tissues of lung cancer cases was evident. For example, mean EGFR cytoplasm scores were equal to 0.38, 0.74, and 1.32 for controls, uninvolved tissues, and carcinoma tissues from lung cancer cases, respectively. A similar trend of increasing mean EGFR scores was observed in the membrane layer. The average of scores from the cytoplasm and membrane layers are also presented. Statistically significant differences in these scores between controls vs. uninvolved tissues from lung cancer cases were observed for all biomarkers in all locations (p < 0.0001). These differences also were observed for all biomarkers in all locations when comparing controls vs. carcinoma tissues from lung cancer cases (p < 0.005). Table 2 shows biomarker cut-off points when comparing control versus uninvolved tissues from lung cancer cases. The optimal cut-off points were selected based on levels of each biomarker that provide the best balance of sensitivity, specificity as well as larger increases in LR's. Levels of p185 erbB−2 scores that provide the best discrimination point between lung cancer cases versus controls were equal to 0.70, 1.30, and 0.40 for basal, cytoplasm, and membrane, respectively. These cut-off points provided moderate levels of sensitivity for basal and cytoplasm (60% and 65%, respectively) and a higher sensitivity for p185 erbB−2 in membrane (80%). Correspondingly, the highest AUC was achieved at 89% for p185 erbB−2 membrane. Thus, it appears that p185 erbB−2 membrane levels provide better discrimination power between lung cancer cases and controls. Specificity levels for these cut-off points were at least 85%; while the LR's indicate that p185 erbB−2 levels greater than or equal to the cut-off points were at least 2.0 times as likely to be found in lung cancer cases than in controls. EGFR cut-off points were equal to 1.00, 0.40 and 0.10 for basal, cytoplasm, and membrane, respectively. Moderate levels of sensitivity for these cut-off points were observed for both basal and cytoplasm while sensitivity was equal to 91% for EGFR membrane with a corresponding AUC equal to 90%. Similar to p185 erbB−2 , the best discrimination power was observed for levels of EGFR measured from the membrane layer. Cut-off points for TGF-α and FASE indicate only moderate but slightly better lev-els of discrimination from measurements in the basal layer (AUC=86% and 78%, respectively) as opposed to carcinoma layer. Table 3 shows biomarker cut-off points based on comparison between controls versus carcinoma tissues of lung cancer cases. The cut-off point equal to 0.40 for p185 erbB−2 membrane provided 89% sensitivity and 92% specificity with an AUC equal to 93%. A similar cut-off point of 0.40 was obtained for EGFR membrane providing high levels of sensitivity and specificity with an AUC equal to 97%. Interestingly, a cut-off point for FASE equal to 0.70 provided strong discrimination power (AUC = 99.6%) between controls vs. lung cancer cases.

Results
Finally, attributable proportions (AP) were calculated which provide a measure of cases that is attributable to a biomarker [17] (Table 4). Values of AP closer to 1.0 indicate a more valid biomarker as a surrogate endpoint for cancer outcome. Using the optimal cut-off points from Tables 2 and 3, the corresponding AP for each biomarker were calculated ( Table 4). As shown in Table 4, all biomarkers exhibited only small to moderate levels of AP. EGFR levels from the membrane exhibited the highest AP equal to 79% while only 63% of lung cancer cases is attributable to high levels of p185 erbB−2 from the membrane. As indicated by Schatzkin et al. [17], individual biomarkers are at best marginal surrogates for cancer outcome. Higher levels of AP were obtained from cut-off points arising from the comparison between controls vs. carcinoma tissues from lung cancer cases. Attributable proportions were equal to 83% and 78% for EGFR membrane and p185 erbB−2 membrane, respectively. In addition to providing strong discrimination power between controls versus tumor tissues from lung cancer patients (Table 3), FASE appears to be a good surrogate for lung cancer outcome providing an AP equal to 96%. A study by Piyathilake et al. [18] suggests that expression of FASE is an early event in the development and progression of SCC of the lung. Combinations of biomarkers also were evaluated in a multiple logistic regression model (data not shown). The combination of p185 erbB−2 and EGFR membrane levels improved AUC to 96% from an AUC of about 90% with either biomarker alone (Table 2) when comparing controls versus uninvolved tissues from lung cancer cases. The combination of FASE and EGFR membrane levels did not improve the AUC of 99.6% from the model with FASE alone when comparing controls vs. carcinoma tissues from lung cancer cases.

Discussion
Selected cut-off points for p185 erbB−2 and EGFR membrane levels appear to have good discriminating power to differentiate controls versus patients with lung cancer. This was true for both comparisons between controls versus uninvolved tissues from lung cancer cases and between controls versus carcinoma tissues from lung cancer cases. In fact, better discrimination based on higher AUC, was observed for the latter comparison, indicative of the trend of increasing levels of biomarkers from controls to uninvolved tissues to carci-  [18] reported a statistically significant step-wise increase in FASE expression from uninvolved bronchial mucosa to epithelial hyperplasia to SCC suggesting its possible involvement in the development and progression of SCC of the lung.
In this study, we utilized two independent patient groups, i.e. controls and patients with lung cancer to arrive at biomarker cut-off points that differentiate between the two samples. Methods for determining cut points of a prognostic variable in instances wherein only the sample of cancer patients are available include the use of a biological criteria, medians and upper limits of normality (mean ± 2x standard deviations), and the minimum p-value approach [19]. The latter approach which involves a systematic cut point search leads to inflation of the type I error arising from multiple testing. As such, the use of this method requires adjustment of the inflated p-values; however, methods for adjustment are not frequently used in many prognostic studies. The availability of data from control subjects allowed us to use a simple and intuitive approach that will best dichotomize biomarker levels into high expression/risk vs. low expression/risk. This provides an alternative approach to the use of arbitrarily defined cut points at the median value or the use of multiple testing to search for cut points. Since the cut-points were selected independent of patient outcome (e.g. response rate, overall survival, disease-free survival), they can then be used as dichotomized prognostic variables without adjustment on significance levels of hypothesis testing.
Our criteria for obtaining cut-off points was based on the best balance between sensitivity and specificity as recommended by Pina et al. [13]. However, our reference population consists of subjects with other types of lung diseases besides lung cancer. As such, we were not able to determine the potential improvement in sensitivity levels from the use of healthy controls [13] in this study. A cohort of healthy controls and additional lung cancer cases is currently being evaluated by authors of this study and will provide for this additional evaluation as well as a serve as an independent patient group to validate the biomarker cut-off points identified in this study.
The use of control subjects also allowed us to validate the value of biomarkers as surrogate endpoints for cancer outcome. With the exception of FASE, this study found only small to moderate proportions of lung cancer cases can be attributed to high levels of p185 erbB−2 , EGFR, and TGF-alpha. This stresses the importance of validating biomarkers in different study populations and in other cancer sites in order to arrive at a more definitive conclusion regarding the value of a biomarker as a surrogate for cancer outcome.