Automated Image Interpretation Computer-Assisted Diagnostics

Much of the difficulty in reaching consistent evaluations of radiology and pathology imaging studies arises from subjective impressions of individual observers. Developing strategies that can reliably transform complex visual observations into well-defined algorithmic procedures is an active area of exploration which can advance clinical practice, investigative research and outcome studies. The literature shows that when characterizations are based upon computer-aided analysis, objectivity, reproducibility and sensitivity improve considerably. Advanced imaging and computational tools could potentially enable investigators to detect and track subtle changes in measurable parameters leading to the discovery of novel diagnostic and prognostic clues which are not apparent by human visual inspection alone. The overarching objective of this book chapter is to provide readers with a summary of the origin, evolution and future directions for the fields of automated image interpretation and computer-assisted diagnostics. The chapter begins with a high-level overview of the fields of image processing, pattern recognition, and computer vision followed by a description of how these disciplines relate to the more comprehensive fields of computer-assisted diagnostics and image guided decision support. Throughout the remainder of the chapter we have supplied multiple illustrative examples demonstrating how recent advances and innovations in each of these areas have impacted clinical and research activities throughout pathology and radiology including high-throughput tissue microarray analysis, multi-spectral imaging, and image co-registration.


Introduction
The field of computer-assisted diagnostics is a discipline which has emerged as a result of sustained efforts throughout the clinical and scientific communities to devise systematic procedures and protocols to support physicians in rendering more informed diagnostic decisions. One of the most active and exciting branches of this research focuses on the automated interpretation of medical images. While most of the emphasis for these activities has been directed towards radiological imaging modalities including X-ray, MRI, and ultrasound, new advances in digital microscopy now make it possible for the fields of diagnostic and investigative pathology to follow a similar experimental and developmental path.

Overview of computer-assisted diagnostics (CAD) in radiology and pathology
The incentive for undertaking many of the earliest CAD studies grew out of efforts to identify a means for reducing the level of fatigue experienced by physicians while increasing the reliability and objectivity with which medical images are evaluated. Prior to the 1990's the primary emphasis was directed towards developing stand-alone "expert systems" that * Corresponding author: E-mail: foran@umdnj.edu. could review a given set of medical findings, including images, in order to deduce a reliable differential diagnosis using simple statistical methods. Although several of these early projects helped advance the field, most of these efforts presented so many contributing factors that standard modeling strategies were rendered grossly inadequate [1][2][3]. Experience gained by investigators during the course of these pioneering studies led to a shift away from attempting to automatically provide a definitive diagnosis towards developing methods that could provide objective, reliable clinical decision support for physicians. This new emphasis led to the term, computer -assisted diagnostics (CAD), which is commonly used to describe these activities today. To the extent that CAD is currently used in medical imaging applications, the majority of studies are directed towards: 1) pre-screening specimens to flag suspicious findings; 2) detecting and segmenting image regions that are likely to be diseased; 3) accurately measuring and quantifying image features that are known to be relevant to a given clinical decision; and 4) testing new image features and algorithms which could potentially provide more accurate clinical decisions.
Radiologists were among the first physicians to utilize CAD in conjunction with rigorous computational methods to support diagnostic decisions with much of their incentive driven by the need to discriminate among subtle changes in the size, shape, and texture of imaged organs and lesions. These tasks demanded accurate mathematical descriptions which could be incorporated into standardized techniques and protocols. In the 1960's Lusted published work that formally introduced the use of receiver operator curves (ROC) to measure performance and he promoted the use of modern statistical methods, including Bayesian statistics and likelihood ratios to support more accurate diagnoses and prognoses [4,5]. CAD techniques have been successfully utilized in number of applications ranging from the characterization of lung nodules to the classification of dermatologic conditions [6] and the detection of colonic polyps from CT colonoscopy [7,8]. One of the most publicized advances that have been achieved using CAD, however, has been in the detection of breast cancer [9][10][11][12]. Since that time, several commercial CAD systems have been developed and approved by the FDA for clinical use. Today, these technologies are used routinely as part of modern clinical practice [13]. Although the idea of using CAD in diagnostic pathology has taken a much longer time for acceptance, the advent of new digital imaging technologies including high-throughput whole slide scanners (virtual microscopes) are making a very compelling case as part of the clinical workflow. For example, the PAPNET system [14][15][16] along with specialized thinprep processing, were designed to optimally prepare and image cervical smear specimens prior to neural network-based screening. PAPNET is yet another FDA approved CAD system that has been introduced into the clinical workflow as a reliable means for detecting and flagging suspicious cases which require follow-up before a definitive classification can be rendered.
The next sections of this chapter provide a highlevel overview of the fields of image processing, pattern recognition and computer vision followed by a description of how these disciplines relate to the more comprehensive field of computer-assisted diagnostics. Throughout the remainder of the chapter we have supplied multiple illustrative examples demonstrating how recent advances and innovations in each of these areas have impacted clinical and research activities throughout pathology and radiology.

Image processing
The term image processing refers to the approaches, methods and technologies that are used to manipulate images from one form to another with the intent of rendering them more useful. During the course of these operations images may be enhanced, noise suppressed and blurring eliminated, but the results of these processes still require further assessment and/or interpretation. Figure 1 shows a representative imaged breast tissue histologic section along with the resultant edge detection map and corresponding intensity histogram.

Pattern recognition
Pattern recognition refers to the methods and processes that make it possible to perceive structure in data. The range of domains that benefit from the use of pattern recognition spans a wide number of applications including the interpretation of speech, analysis of seismic data and prediction of economic trends. For the purposes of this chapter, however, the term will be used to refer to the algorithms, methods and procedures that enable computer-based systems to arrive at conclusions which are consistent with what an expert pathologist or radiologist might observe if they were to evaluate the images using traditional approaches and practices. Figure 2 shows a multivariate clustering algorithm in the process of systematically interrogating and detecting the salient peaks and basins of attraction within a representative data set.

Computer vision
Several of the fundamental principles of the field of computer vision have grown out of attempts to mimic the human visual system. One of the most striking architectural features of the human visual system is the underlying structure which enables it to operate on the principle of convergence wherein there is an increased capacity for abstraction at ascending levels of processing and analysis. Figure 3 shows the histology of retina which initiates the hierarchical structure of the human visual system as an input signal ascends from the simple to the complex and hyper-complex cell. It is this aspect of the visual system that machine vision scientists have tried to exploit and incorporate into modern computer-based designs.
Over the course of more than 15 years our research team has designed, developed and implemented a host of new and innovative computer vision algorithms in projects addressing challenging clinical problems in diagnostic and investigative pathology and radiology. While it would exceed the scope of a single book chap- ter to provide a detailed description of any one of those efforts, we will provide a summary of several key efforts that our team has undertaken over the past 5-10 years along with a set of citations which refer to the most relevant efforts of other investigators throughout the field whose work demonstrates the tremendous impact that new and emerging technologies in imaging and high-performance computing are having throughout the clinical and research communities.

Image guided decision support
A significant driving force for advances in the field of computer-assisted diagnostics has been the growing emphasis throughout the clinical and research communities on personalized healthcare. As such, it has become increasingly important to improve the reliability and accuracy with which subclasses of disease are distinguished in order to facilitate drug discovery and identify subpopulations of patients who would most likely benefit most from a specific drug regimen or therapy.
Developing strategies that can reliably transform such complex concepts into well-defined algorithmic procedures is an active area of research with several major projects focusing on diagnostic pathology. These include the Pathex framework and the Pathex/Red system [17], developed at Ohio State University to assist pathologists in the assessment of laboratory data; ECLIPS [18] which was developed at the University of Illinois Urbana; and the PathFinder project from University of Southern California and Stanford which was directed towards the development of an expert system that provides a differential diagnosis based on the histo-282

D.J. Foran et al. / Automated image interpretation and computer-assisted diagnostics
logical feature(s) observed by pathologists. PathFinder suggests any additional features that are most likely to narrow the range of possible diagnoses, thus helping to screen for observations which are inconsistent with a given disease [19].

Content-based image retrieval
At approximately the same time that the concept of personalized medicine was becoming popular, there was a growing, concurrent excitement at the opposite side of research spectrum among computer scientists and engineers who were investigating the feasibility of developing data mining approaches which could reliably locate and retrieve images and graphical information based upon their underlying visual content rather than based upon the alphanumeric labels that were traditionally used to name them. As the field evolved, the idea of querying such data, based on image-based characteristics became known as Content-Based Image Retrieval (CBIR). While the mechanisms for accessing alphanumeric data have been studied extensively, content-based access of medical images, especially imaged pathology specimens, still remains largely unexplored.
The technologies that capture, describe, and index the visual essence of multimedia information rely on the methods and principles of image analysis, pattern recognition, and database theory. The individual strategies and approaches used to perform these analyses differ according to the degree of generality (general purpose versus domain specific), the level of feature abstraction (primitive features versus logical features), overall dissimilarity measure used in retrieval ranking, database indexing procedure, level of user intervention (with or without relevance feedback), and by the methods used to evaluate their performance.
There have been several general purpose contentbased image retrieval (CBIR) systems that have been reported over the years such as the IBM QBIC System [20], the Photobook System [21], the WBIIS System [22], the Blobworld System [23] and the SIMPLIcity System [24].
More recently, however, there has been increased interest and efforts applied to utilizing CBIR in medical applications. Wang from Pennsylvania State University emphasizes the use of wavelet technology and Integrated Region Matching (IRM) distances for char-acterizing pathology images [25]. The system indexed segments of images at different scales by partitioning the original image into smaller overlapping blocks. A CBIR engine was then interfaced with a server which allowed users to browse portions of the original matched image at different scales. During the same period of time, the Pittsburgh Supercomputing Center developed a system which utilized global characteristics of images to provide a measure of Gleason grade of prostate tumors [26]. The same team later reported a prototype system which enabled physicians to utilize standard desktops to access supercomputers resulting in decisions which exhibited a strong correspondence between the similarity distances generated by the computer algorithm and the pathological significance as judged by certified anatomic pathologists [27]. Based on advances made in this early work there have been a range of successful CBIR applications that have been reported [28][29][30][31][32][33][34][35].

CBIR for discriminating among malignant lymphomas and leukemia
In 1998, our team first began to undertake the challenge of developing an image-guided decision support system to discriminate among lymphoproliferative disorders that can sometimes by confused with one another when assessed during routine microscopic evaluation because of similarities in their morphologic profiles and visual appearance. The motivation for undertaking this project grew out of a pilot studying involving a relatively new entity, called Mantle Cell Lymphoma (MCL). One of the interesting aspects of this disorder is the fact that it exhibits a phenotype which closely resembles the morphologic profile exhibited by several other disorders including a wide spectrum of benign cells. Unfortunately, Mantle Cell Lymphoma runs a significantly more aggressive clinical course [36][37][38][39][40][41] than the other entities with which it is often confused. This problem presented a significant clinical problem with a large number of technical challenges. Another factor that made this project attractive was the fact that the diagnosis of each of the entities could be evaluated using immunophenotyping techniques which made it possible to build a "goldstandard" image archive and database of cases for which there was independent confirmation of the differential diagnosis of each specimen.   Figure 4 shows a prototype platform that was developed to enable us to conduct iterative development and optimization experiments. The proposed framework was developed using modular design to maximize its flexibility with each of the major modules designed to operate independently or in conjunction with one another.

Overview of system framework
Using this platform, once a query image entered the system, the client processor automatically performed segmentation and feature extraction. After which, statistical pattern recognition techniques allowed quick, reliable content-based image retrieval and comparison of the unclassified case with a gold-standard database of cases for which independent confirmation of the diagnoses had already been conducted using immunophenotyping [42,43].

Automated segmentation module
An important component of the client software which is central to it functionality is the segmenta-tion module which has been shown to automatically, reliably and reproducibly delineate the nuclear and cytoplasmic components of cells of interest. Xu and Prince [44] advanced the strategies used to segment images significantly when they reported the development of an algorithm based on active contours and a gradient vector flow (GVF) model which could support large capture regions while reliably navigate in and out of boundary concavities. Unfortunately, the GVF snake that they developed was not appropriate for color images. Several early experiments showed that simply transforming chromatic images into gray-level images and then applying the GVF segmentation algorithm was not a viable solution for many applications. This fact became even more problematic when used in diagnostic and investigative pathology applications where even subtle differences in color often provided essential cues for performing reliable analysis.
Several investigators attempted to address this challenge by calculating the color gradients through a simple summation of the response across the RGB channels of the image. This strategy was, in general, heuristic and had no theoretical basis. In our work, we developed a definition of gradients based upon classical Riemannian results. This approach was based on a solid theoretical background and constituted a logical extension of the gray-level image gradient computation. Our team investigated and reported the development and implementation of a new robust color GVF snake based upon a weighted chromatic gradient operator which achieved superb performance utilizing LUV color space and robust estimation. Please see [43,45] for a detailed description of the approach.

Shape signature module
In order to address the challenges of discriminating among a set of disorders which exhibited such a high degree of similarity in terms of their visual appearance under the microscope, our team experimented with a wide range of different shape characterizations including bending energies [46] and chain code [47]. Although each individual method offered some advantages, systematic experiments showed that the best results were obtained by implementing a modified version of the Elliptic Fourier Descriptor which allowed recognition of individual cells irrespective of variations in rotation, translation and magnification. This feature was extremely important because of the rotational and scale differences that often exist between a given novel query imaged cell and those cells contained within the gold-standard repository of cases. Through the use of statistical optimization methods we identified 10 harmonics as a reliable trade-off between computation and reliability and accuracy [42,48]. Figure 5 shows representative input image with nuclear boundary rendered in white along with corresponding forward and reverse transforms while varying the number of harmonics used to encode the contour.

Texture signatures
During the course of numerous observations and discussions with hematopathologists it became evident that in order for the system that we were designing to achieve a significant level of reliability the driving software would require some means for gauging differences in the underlying granularity of the nucleus and prominence of the nucleoli. To address this issue, we implemented a multidimensional texture operator based upon a modified version of the multi-scale simultaneous autoregressive (MRSAR) model introduced by Mao and Jain [49]. The algorithm that we used was based upon a second-order model described by five parameters at 3 different resolution levels (5×5, 7×7, and 9×9 neighborhoods) giving rise to a 15dimensional feature vector. In Fig. 6 we have included a few representative nuclear profiles to illustrate what aspect of the samples are actually being measured. Please see [42,50] for a complete description of the algorithms.

Similarity metrics
Perhaps the greatest challenge that we confronted while developing the prototype system was that there was no obvious way to combine the disparate feature measurements for color, shape and texture that  were generated so as to allow one to reliably judge the level of similarity of the spatial and spectral signatures exhibited by the query cells and "ground-truth" database of cases.
In the first generation prototype we utilized a downhill simplex approach to determine the optimal weighting factors for each of the feature measures for a test ensemble of imaged specimens. An interesting finding from these early studies was that it appeared that the most informative aspects of the phenotype related to the underlying granularity or chromatin patterns of individual nuclei [42].
There are quite a few similarity metrics can be used to judge the similarity of the spatial and spectral signatures. These include Lp distance, such as L 1 or L 2 distance, the χ 2 distance, Kullback-Leibler (KL) divergence, Jensen-Shannon divergence [51] and the Earth Mover's distance [52].
The L 2 distance is just simple Euclidean distance which is generally applied to measure the distance between two vectors on Euclidean space. L 1 norm is the absolute value of the difference between two vectors and it is more robust than L 2 norm although it might be more difficult for optimization. The χ 2 distance is defined as the sum of the squared difference between two feature vectors over the sum of the two feature vectors. It has proved to be an effective distance function for texture similarity measurement. KL divergence is a non-symmetric similarity measurement of the difference between two signatures or probability distribution. JS divergence is based on KL divergence and its square root results in to a real metric. It is actually a smoothed and symmetric version of the KL divergence. The Earth Mover distance is a cross-bin distance function which does not require the two feature vectors to be aligned.

Algorithm robustness
Due to complex nature of image features, highdimensionality is an important factor to consider when evaluating a content-based image retrieval system. As data dimensionality increases, the amount of test data needed to generate a statistically sound system can be prohibitively high. Leave-one-out and cross validation are among the most used methods in such situations. More advanced methods such as bootstrapping are also useful for some applications [53]. When having a very small sample set is available, leave-one-out method will exclude one sample from the data during classifier training and then use the "unseen" data for testing, while repeating this strategy through the dataset to assess system stability. Similarly, an n-fold cross validation method will divide the data randomly into n groups, withholding one group as test set in each test while assessing the average performance of n tests. It should be noted that even when such mechanisms are properly implemented the classifier may fail to generalize to discriminate data collected under conditions which vary only slightly from training data.
The average five-class classification accuracy using a mixed set of 3,691 benign, chronic lymphocytic leukemia (CLL), mantle cell lymphoma (MCL), follicular center cell lymphoma (FCC), and acute leukemia acute lymphocytic leukemia (ALL) and acute myelogenous leukemia (AML) samples was 93.18% based on ten-fold cross validation in this closed dataset.
After repeated refinements to the system it was clear that using a maximal margin classifier, support vector machine (SVM), was the optimal approach for classifying the cells. Throughout the course of those studies the largest source of errors was caused by ambiguity between MCL and CLL. Please see [43] for a detailed treatment of the algorithmic development and comparative performance studies.

Tissue microarray analysis
As the number of new techniques and technologies used in medical applications continues to increase our efforts have been redirected from the specific problems associated with developing an image guided decision support system for assessing hematologic malignancies and towards the development of a generalizable set of automated imaging, computational and data management tools for assessing large-scale cohorts of tissue samples. For example, tissue microarray (TMA) technology was recently developed to enable investigators to extract small cylinders of tissue from histological sections and arrange them in a matrix configuration on a recipient paraffin block such that hundreds can be analyzed simultaneously [54][55][56]. Over the years, TMA technology has been validated for use in cancer research by investigators who systematically compared interpretations using TMAs with those rendered using whole tissue sections or through validation with cDNA microarray findings. Together, these studies encompassed a broad range of cancer types including breast cancer [57][58][59][60][61][62][63]; prostate cancer [64][65][66][67]; gastric cancer [68]; colorectal cancer [69][70][71]; lymphomas [72][73][74][75][76], multiple myeloma [77]; soft tissue sarcoma [78]; renal cell carcinoma [79]; bladder tumor [80]; glioma [81,82]; melanoma [83]; and lung tumor [84,85]. Some of the protocols used in array preparation have since been refined to accommodate specific types of specimens, e.g., cell lines and to improve the reliability of the method [86][87][88][89][90][91][92].
Quantitatively assessing biomarker expression and staining characteristics of tissue microarrays typically involves some degree of manual or semi-quantitative evaluation of the specimens while they are subjectively scored [67,92]. Unfortunately, such approaches are prone to inter-and intra-observer error. To address this issue, several investigators have begun to develop improved methods for performing the quantitative evaluation of TMA's [93][94][95]. For example, the AQUA (Automated Quantitative Analysis) system was designed to utilize molecular based methods to assess protein expression while reducing the variability of pathologist-based evaluations of samples [96]. Unfortunately, in spite of the significant progress achieved in this area thus far most of the systems that have been developed are limited by the fact that tissue they are closed and proprietary; do not exploit the potential of advanced computer vision techniques; and/or do not conform with emerging data standards.
Capitalizing on the experience that our team gained while developing the image guided decision support (IGDS) system (described in Section 2), we began designing, developing and evaluating a deployable system for performing automated analysis of tissue microarrays (TMA) in collaborative, multi-user environments. Through competitive extramural funding from National Institutes of Health we have modified and optimized many of the governing algorithms that had been developed to perform image analysis of imaged hematopathology specimens so that they could reliably distinguish among the cellular components and tissue types within imaged tissue microarrays [97,98]. Figure 7 shows a representative tissue microarray exhibiting bowing of rows and columns and missing (detached) tissue discs. Unfortunately, such artifacts are not uncommon because of the tremendous amount of care and skill that is required to assemble and construct tissue microarrays. To address this issue software was developed to correct for such aberrations through the use of a modified Hough transform which automatically locates, delineates and indexes each individual disc while inserting blank place holders in the database for detached discs [99][100][101][102].  An even greater challenge that our team confronted while developing the TMA analysis system was caused by the fact that tissue arrays are often prepared using two or more different histological stains simultaneously, which complicates the attempts to automate quantification of the staining characteristics of these specimens. To address this issue, we developed a robust color decomposition module to perform reliable polar transformations and peak detection in multidimensional color spaces. The resulting color decomposition algorithm was subsequently tested using breast cancer tissue arrays which had been stained with anti-Smad antibodies and a range of other imaged specimens [100,101]. The algorithm was later modified to automatically drive the robotic stage of a microscope while randomly sampling the color at over 500,000 pixel locations to provide input to the decomposition module. Results from application of the methods are shown for a representative specimen in Fig. 8.

Delineation of tumor boundaries at the tissue-level
One of the greatest obstacles for automated quantitative analysis of TMA specimens stems from the fact that each individual disc (histospot) within a given tissue microarray is a complex heterogeneous tissue sample. Since many assays are designed to assess the concentration of target molecules within a specific tissue type, a proper evaluation of imaged TMA's must not only reflect the total amount of signal within each tissue disc (histospot) but must also describe the spatial distribution of the signal within the tissue of interest.
Towards that end, our team investigated and reported the use of texton distributions as a means for performing tissue-level segmentation of imaged specimens.
The algorithm was designed to sample specimens at multiple scales while generating texton histograms for regions of interest. Because of the high dimensionality of the distribution space and non-separable nature of the data our team devised a fast, new strategy for computing regional histograms using an integral histogram technique which is only linearly dependent upon the size of the histogram and independent of the region size. As an extension of this work we developed a new region covariance descriptor which provided superior performance for texture classification while providing a natural way of fusing features [103]. The results of using texton-based segmentation to delineate epithelial/tumor regions within imaged breast cancer tissue discs are shown in Fig. 9.
When the segmentation results were compared with those which had been hand-drawn by a board-certified anatomic pathologist for 300 cancer tissue discs the average false positive rate was 4.13% and the average false negative rate was 2.15% [104].

Characterizing expression signatures at the multiple levels of granularity
It is now largely accepted that staining and expression signatures can provide valuable prognostic and clinical insight regarding therapy planning and for predicting response to treatment for certain patient populations [105][106][107]. As an extension of some of the early work in the field, our team has been investigating the use of texton signatures as a means for classifying cancer and disease progression in imaged cancer tissue arrays.
Utilizing an ensemble of 3,744 breast cancer samples we performed a series of experiments to determine the preferred filter banks and optimal number of modes 288 Fig. 9. Segmentation results for representative breast cancer tissue discs showing segmentation probabilistic maps. Darker region represents higher probability of epithelial tissue. The algorithm uses texton-based texture descriptor based on region covariance. that were necessary for differentiating among the expression patterns of 3 different types of breast cancer. Systematic studies showed that the discriminatory power of the texton signatures was highly dependent upon the choice of filter banks utilized to generate the histograms and the statistical approach used to render classifications. Gentle Boosting using an eight-node CART decision tree as the weak learner provided an overall accuracy of 86.02% using only 30% of the specimens as the training set [108]. One interesting, but unexpected outcome from these studies was that by storing the raw filter responses of the staining maps the resulting reference library is not spatially constrained and can be used to conduct analysis at multiple levels of granularity, i.e., at the whole disc, tumor, individual tissue, cell or sub-cellular level [97]. Figure 10 shows the results from studies in which the analysis concentrated on three different sub-regions within the imaged breast cancer specimens (tumor region alone; whole tissue sample; and non-tumor region alone). To obtain these results the tumor and non-tumorous regions were automatically delineated using the segmentation algorithms described in Section 3.2. Comparative performance analysis was carried out using three different algorithms: K nearest neighbors (K = 5), soft margin support vector machines (SVM) and boosting. It was clear that the maximal margin classifiers, such as boosting and SVM, performed significantly better than simple classifiers such as KNN. In addition these experiments showed that using the tumor region mask provided appreciable improvements in classification accuracy of the specimens.

D.J. Foran et al. / Automated image interpretation and computer-assisted diagnostics
In a set of parallel experiments our team investigated the use of the color decomposition and texton analysis for assessing human epidermal growth factor receptor 2 immunohistochemical assays [109,110]. In those studies it was shown that the algorithms were able to reliably characterize the underlying expression patterns of immunostained specimens. Figure 11 shows some representative examples in which the computer-based analysis was used to identify specimens exhibiting similar staining characteristics.

Multi-spectral imaging
Although the majority of clinical and research scientists still utilize traditional microscopes and imaging devices to conduct their experiments, new technologies such as virtual microscopes and multispectral imaging cameras continue to gain wider acceptance. Towards that end our team has begun to explore the utility of multispectral imaging (MSI) technology using "off-the-shelf" devices which can be configured for bright-field or fluorescence microscopy in the visible wavelength range of 420-720 nm and up to 950 nm for fluorescence-only systems. Figure 12 shows a composite RGB image of single histospot from a breast tissue microarray and the corresponding multispectral images corresponding to the most salient wavelengths.

MSI in bright-field analysis
Multi-spectral imaging devices are capable of capturing data across a range of frequencies within the electromagnetic spectrum. Although several uses have been identified and reported in pathology [111][112][113], it is generally difficult to determine, a priori, which applications would benefit most significantly from this relatively new technology. In a recent study, Boucheron et al. found minimal (<1%) performance improvement using multispectral imaging over a standard RGB imaging for pixel-level nuclear classification of routine H&E images [114]. In a separate study, Roula et al. reported improved classification capacity using MSI for H&E slides of radical prostatectomies [115]. The lack of clear consensus over the utility of multi-spectral imaging in bright-field microscopy prompted our team to take a closer look at the underlying principles.
As a first step we conducted comparative performance studies to determine, empirically, whether or not there exists a significant benefit to utilizing multi-spectral imaging technology to characterize hematoxylin stained breast tissue acquired using bright-field microscopy. During those experiments breast TMA's were imaged using a Nuance VIS-FLEX MSI camera at 20X (Cri Inc., Woburn, MA 01801 USA). Each resulting image "cube" has a spatial resolution of 1392×1040, sampled at 31 spectral bands across the wavelength interval 420-720 nm. The prototype software that was used to conduct the experiments has three modules. The first extracts a region of interest, using adaptive thresholding to remove background noise and morphological processing to smooth the region. The second performs texture feature extraction using a local binary pattern to extract rotation-invariant, uniform patterns for each specimen at each of the 31 spectral channels. The third module performs feature selection and classification. For each spectral band, exhaustive feature selection was used to search for the combination of channels that yielded the best classification accuracy. AdaBoost [116] using a linear perceptron least-square classifier as weak learner was applied as a strong classifier to select the set of representative color channels. The selected color channels (spectral bands) carry the greatest discriminatory power and are used to produce a majority vote in the final classification.
During those studies 92 breast TMA discs were used to evaluate performance. A sensitivity of 0.91 and specificity of 0.89 were achieved when multispectral data was used whereas a sensitivity of 0.83 and specificity of 0.85 was achieved using RGB data. See Qi X, et al. for a full description of the technique [117]. Encouraged by these results our team is currently testing and optimizing the underlying MSI analysis algorithms for a wider range of specimens and stains.

Metamers
As an extension of these experiments our team is developing a metamer-based metric which can be used to determine, a priori, which applications would benefit most from the use of muli-spectral imaging technology. Metamers are considered as different spectral power distributions which induce the same CIE XYZ tristimulus value under a given illuminant. There are an infinite number of spectral distributions which give rise to each color [118]. For two pixels from a cube to be considered metamers, the spectrum of each pixel, subtracted in a pair-wise fashion and multiplied by the appropriate color-matching coefficient [113], must fall within a small range around zero. In a recent set of experiments our team utilized the fundamentals of colorimetry in an attempt to narrow the uncertainty of knowing when a multi-spectral image captured potentially discriminatory information which was absent in the standard RGB image counterpart and we applied a standard condition for detecting metamerism. In this work, we used a linear algebra framework to investigate the relationship between the spectral image and its standard-image counterpart. Figure 13 (data courtesy of [113]) shows the plots for some routine stains. Our preliminary feasibility studies indicate that those stains which absorb heavily in regions where the colormatching coefficients are small are most likely to benefit from MSI technology, however, this line of thought warrants closer examination [119,120]. In the next phase of our experiments we will investigate the use of metamers for objectively determining those applications for which MSI would be most appropriate and test and optimize those metrics for the full range of specimens and stains under study.

Quantum dot antigen-antibody conjugates
In a parallel set of experiments, our team has begun to investigate the use of quantum dots (QD's) technologies to accentuate visible differences among tagged biomarkers. Quantum dots (QD's) are small light-emitting particles that have excellent proper- ties for fluorescent labeling of biologic molecules and exhibit superior signal brightness, resistance to photo-bleaching, and narrow, well defined emission spectra. Antibody-conjugated quantum dots have already been used for quantitative molecular profiling of protein expression in formalin-fixed, paraffinembedded (FFPE) sections [121]. A similar approach using QD-labeled oligonucleotide probes for multiplexed fluorescent in situ hybridization(FISH) has been reported and shown to be a reproducible measurement of RNA levels in clinical samples [111]. One advantage this approach offers over other techniques, such as quantitative RT-PCR, is that it preserves spatial information allowing one to determine which cell populations are expressing the molecule of interest. A high signal for an mRNA using RT-PCR of a FFPE sample cannot distinguish whether it is being expressed by the tumor cells or the stromal cells in the specimen.
Our team has been working on the development of a reliable clustering algorithm which can be used to preprocess multispectral datasets of the imaged specimen immediately after acquisition for the purposes of quickly and automatically detecting the most informative wavelengths. To investigate the feasibility of using these tools we have devised a set of well-defined experiments which focused on a clinically salient application involving the evaluation of non-Hodgkin's lymphoma histology specimens. The aim of these experiments was to determine the staining and expression patterns of FDC markers CD21, CD23, CD32, CD35 using FISH or QD or combination of those two imaging techniques.
First FISH slides were blocked in PBS and washed. Primary antibody was added and washed. Secondary antibody with QD was added; incubated for 1 hour, then washed. DAPI was added and slides coverslipped.
Multispectral images were acquired, after automatic removal of auto-fluorescence, at resolution of 1392 × 1042 × 31, where the 31 spectral bands cover wavelengths 420-720 nm. Composite images of each pure FISH and QD signals were created and the expression pattern of each biomarker was digitized and stored.
In a second set of experiments we evaluated the MSI setup using frozen sections of tonsil which were stained for chromosomes 11 and 14. The multiple myeloma cell line MM1.S was stained with FISH probes specific for chromosomes 11 and 14. The FISH labeled cells were stained for the surface expression of CD20. This setup allowed simultaneous detection of chromosomes 11 and 14 in the nucleus and surface expression of CD20. Slides were evaluated for the surface expression of either CD21 or CAN. Similar to the results of MM1.S experiments, both the chromosome FISH probes and the surface expression of CD21 and CNA could be detected demonstrating the utility of these protocols with MSI [122].

Automated co-registration of consecutive imaged histologic sections
During the course of our interactions with enduser oncologists and pathologists, it has become apparent that the capacity to perform quick, reliable coregistration of digitized, histologic cross-sections has a wide range of useful applications including generating 3D image stacks for visualizing micro-structures; performing 3D modeling of tumor environment; correlating image features and localization of biomarkers across adjacent sections; and matching sections which have been prepared using complementary immunostains.
Driven by these incentives our team recently reported a new method for performing fast, robust image co-registration, which combines landmark and region-based strategies [123,124]. Most existing methods are highly sensitive to variations in image quality, degree of deformation, and require fine-tuning of parameter settings. The algorithm that our team developed is computationally efficient and operates in a completely unsupervised fashion. Due to the relatively small number of landmarks needed to drive the algorithm, it runs faster than several popular nonlinear registration algorithms reported in the literature, including the finite element method (FEM), BSpline fitting and Daemon's algorithm. The algorithm begins by automatically detecting landmarks followed by executing a coarse to fine estimation of the non-linear mapping among those points. Multiple resolution orientation histograms and intensity templates are combined to obtain a fast representative local descriptor of the detected landmarks. A quick estimator, RANSAC (RANdom SAmple Consensus), is utilized to reject outliers during the initial landscape correspondence. The final refined inliers are used to robustly estimate a Thin Spline Transform (TPS) to complete the final nonlinear registration. The method was shown to provide sub-voxel accuracy for co-registering 3D radiology datasets. We have already performed 3D experiments comparing the performance of the newly developed algorithm with those achieved using the Insight Segmentation and Registration Toolkit (ITK) and the algorithms developed at the French national institute for research in computer science and control (INRIA). The specific algorithms within the ITK and MedINRIA that we used to conduct performance studies were Bspline and Daemon's method. In the case of ITK we utilized the open source code implementation. For MedINRIA we used the software, which is freely available for download. In each case we systematically adjusted their parameters to achieve best performance. Figure 14 lists the comparative results for the three algorithms [124].
The algorithm was tested using a wide-range of datasets exhibiting nonlinear deformation. Figure 15 shows representative examples using imaged TMA, Brain CT and rat lung during respiration. In each case the first row represents the fixed image and the second row is the moving image. The last row represents the transformation recovery results. In these examples the more similar the result image and fixed image the better the registration.
In order to evaluate the performance of the tissue disc alignment protocol pairs of consecutive sections were extracted from a mixed set of tissue arrays, which had been stained for the same marker. Tissue discs were rotated out of registration after which the non-linear co-registration algorithm was applied. The target disc was subsequently rendered in red whereas the localization test disc was rendered in green. The accuracy of the match resulting from the registration process was visualized by superposing the two discs (localization and target) as shown in Fig. 16 with regions of absolute alignment rendered in yellow.
During these studies the registration algorithm was shown to compensate for partial loss of tissue across imaged sections because of the elimination of outliers and robust matching of landmarks. Additional experiments were conducted in which the original target disc was graphically rotated by 30 degrees, scaled by 70%, and translated by 10 pixels to generate a test disc to assess performance.
Through the systematic application of the deformable co-registration algorithms across a given series of thin cross-section of tissues constituting the tumor or tumors under study, a 3-dimensional representation of the tumor environment can be built, visualized and interrogated.

Data organization and management
Advances in pathology imaging technology and CAD have spurred the challenges of data management and data exchange for pathology applications. Traditional database systems are not optimized to handle images and their related data, especially large amount of image metrics generated in CAD. Researchers needed to develop suitable systems that help organize their own data structure and support their specific queries. To address the organizational and data management challenges of the PathMiner project, our team developed and tested an Intelligent Archival (IA) subsystem which enables individuals from disparate clinical and research sites to populate databases with new cases including correlated image metrics and imaged specimens in multi-user environments [100,101,125]. A server-side JAVA application automatically checks for potential conflicts of new entries, and populates the database while keeping an entry in the database to indicate the location of the digitized cancer specimens. The database can be accessed through redundant application servers which are mirrored for added stability. The software has been developed using a modular design [125].
In order to further this type of efforts to be used widely by the community, metadata and identifiers are being developed to meet all HIPAA require- ments for sharing data anonymized for research [126]. The constituent entities of the Tissue Microarray Repository subsystem are being designed in keeping with emerging guidelines from the caBIG initiative and the Association for Pathology Informatics. The overarching idea is to keep the definitions generalizable and to provide the underlying structure which can support a broader range of imaging applications.

The ImageMiner data model and pathology analytical imaging standards (PAIS) model
The data model underlying the ImageMiner database [125] was developed based upon direc-tion providing by a panel of consulting oncologists and pathologists [100]. The data model is designed to house both quantitative and qualitative information derived from the physical and digital specimens, including clinical data and research data. Version 1.0 of the data model consists of 58 classes and 262 data elements (attributes). It has undergone review by the NCI Enterprise Vocabulary Services program to ensure compliance with caBIG ® standards and has been loaded into the Cancer Data Standards Repository (caDSR). The model can be viewed and retrieved via caBIG CDE Browser (https://cdebrowser.nci.nih.gov/CDEBrowser/).
Our team is currently working to extend and harmonize the analysis results component of the ImageMiner data model with the PAIS (Pathology Analytical Imaging Standards) model [127] to support markup and annotations in TMA, pathology, and microscopy imaging applications, including multispectral data, while maintaining interoperability with corresponding standards in the radiology domain. As such, PAIS is being developed in keeping with the Annotation and Image Markup (AIM) model [128,129], which is under development in the caBIG ® In-Vivo Imaging Workspace to support radiologic image annotation and markup in health care and clinical trial environments. PAIS has been optimized for representing fine-grained markups and annotations and provides additional information for data provenance, such as algorithms and parameters used for image segmentation. For additional details on the PAIS standards please see [130,131].

High-throughput analysis of imaged specimens on a grid
In an attempt to address the challenges of highthroughput analysis, several investigators have begun to exploit distributed computing technologies. For example, our own team recently demonstrated the use of a high-performance computing system for automatic analysis of imaged histopathology breast tissue specimens [132]. Gurcan et al. reported the successful application of distributed computing in a pilot project to support automated characterization of neuroblastoma using the Shimada classification system [133]. The ImageMiner system that we are developing is a logical extension of our early successful efforts developing network-based clinical decision support systems [42,98,104,134,135] and largescale, feasibility studies that we conducted on IBM's World Community Grid in July 2006, using more than 100,000 imaged tissue samples [43,98]. World Community Grid afforded our team the collective computational power of approximately 250,000 computers world-wide which translated into the computation equivalent of what it would take a standard desktop 2,900 years to accomplish. Figure 17 shows the graphical logo for the project on a representative client computer which was participating in the project.
One of the most difficult tasks for our team was keeping up with the pace at which World Community Grid was processing the data. To meet the demands of the project our team purchased a high-throughput whole slide scanning virtual microscope. In addition, we worked out the details of training the computer to automatically delineate each tissue disc from the imaged arrays, perform the color decomposition operations, and package each staining map into work units before submitting them to the Grid. It was also necessary for us to work out the details of performing  quality control on the results that were returned to our laboratory and correlating those results with their corresponding clinical profiles. Figure 18 shows the workflow of data throughout the "Help Defeat Cancer" project. WCG enabled our team to conduct large-scale feasibility studies demonstrating the use of texton histogram signatures for characterizing and classifying staining and expression signatures in image cancer tissue microarrays which resulted in the generation of a reference library of expression signatures for more than 100,000 tissue samples.

Man-machine studies and practical uses
Over the course of the past 12 months, the TMA computational and imaging tools have been migrated to the Histopathology & Imaging Shared Core Resources at The Cancer Institute of New Jersey. They have been successfully used to analyze microarrays consisting of cancers of the breast, head & neck, and prostate. As part of a recent study the automated software was used to quantify Beclin1 expression which was shown to be predictive of autophagy [136]. Our team has also conducted a series of man-machine performance studies. In the first experiments we utilized the TMA analysis tools to evaluate IHC staining intensity on imaged breast cancer TMA specimens comprised of 1407 tissue cores. The results showed that the computer software algorithms achieved similar interpretations to those provided by a panel of 3 board-certified pathologists and was consistent with inter-pathologist concordance. These results were presented at the 2010 Annual Conference of the United States and Canadian Academy of Pathology [137]. In the next phase of the project our team plans to conduct a much more comprehensive set of performance experiments. To facilitate those studies we have already deployed a suite of grid-enabled TMA analysis tools to Emory University and Ohio State University and are in the process of installing and testing the software at strategic sites at University of Pennsylvania, and University of Pittsburgh Medical Center each of whom have agreed to participate in the performance studies and serve as adopter sites for the project going forward.

Future directions
Advances in digital imaging have made it possible for pathologist to generate high-resolution image data as part of their routine investigative and diagnostic activities. While the resulting data sets offer unparalleled opportunities for visualization there still remains a lack of satisfactory imaging and computational tools which allow for automated high-throughput analysis of specimens. Future advances in pathology will rely on the availability of reliable methods and algorithms that can keep pace with emerging technologies such as virtual microscopy and multi-spectral cameras which currently overwhelm a traditional outfitted clinical or research department. Our work and the collected efforts of a growing number of engineers, physicians and scientists throughout the community is directed towards the design, development, and evaluation of tools which will reduce learning curves and facilitate more efficient use of these resources.