Colorimetric Analysis Using Scene-Adaptive Color Conversion Matrix of Calibrated CIS

The RGB signals of different CISs (color image sensors) do not register the same values for the same viewing scene owing to their different spectral sensitivity and white balance mechanisms. Thus, CISs must be characterized based on CIE standard observation for colorimetric purposes. One general method for characterizing CISs is least square polynomial modeling to derive the colorimetric transfer matrix between RGB outputs and CIE XYZ tristimulus inputs. However, the transfer matrix that is obtained under the standard CIE illumination is unable to estimate various conditions of a CIS that is operated under various illuminations with varying chromaticity and luminance. Therefore, repeated experiments are necessary to obtain accurate colorimetric analysis results. This paper presents a scene-adaptive colorimetric analysis method using images captured by a general consumer camera under various environments.


Introduction
Generally, the RGB outputs generated by CISs (color image sensors) are device-dependent.This means that the RGB signals do not correspond to device-independent tristimulus based on CIE CMFs (color matching functions) [1].This is because the spectral sensitivity of CISs is not matched to the standard CMFs, and the characteristics of color sensors that are utilized in consumer cameras usually depend on the surrounding environment when capturing a scene [2].Therefore, it is necessary to determine the transfer characteristics defining the relationship between RGB signals and standard color stimuli for post-color matching and enhancement processes such as color reproduction, color correction, and HDR (high dynamic range) rendering to facilitate accurate scene rendering in general display devices [3][4][5][6][7][8].
The derivation of a transform relationship between RGB signals and CIE  stimulus is known as CIS characterization [9], and it can be performed by spectral-sensitivitybased and color-target-based methods.The color-targetbased method uses reference colors obtained from their  values on a color chart and thus is relatively simple and practical when compared to the spectral-sensitivity-based method that requires spectral analysis of the camera.The polynomial regression method based on least squares fitting has been widely adopted by many color researchers for calculation of the transfer matrix from the captured RGB values and their  values [9][10][11][12][13].However, camera characterization under the specific standard illuminant does not provide good estimations for the different ambient light conditions in which an image is captured.Accordingly, for more accurate color analysis, flexible and rigorous experiments in various white balance conditions should be performed.
Generally, most color images from CISs should be calibrated chromatically to remove specific color cast.The changed transfer matrix for any other white balance condition can be obtained based on the surrounding illumination and the phosphor primaries of a camera [12,13].However, color measurement performance can be enhanced using a recalculated transfer matrix based on illuminant estimation techniques.Commonly, the illuminant estimation is referred to as white point estimation.It is assumed that the CIS is then calibrated to produce similar RGB values for white under any illuminant.However, it is uncertain which illuminant will be used and how similar the RGB values will be to each other.In theory, color correction with unknown white balance is not problematic, because the calibrating coefficients used to scale the sensor parameters are absorbed into the color transformation required for the color correction [14].
In this paper, a novel colorimetric analysis method is proposed using images captured by a consumer digital camera.In general cases, it is hard to estimate the internal calibration state of the camera sensor and surrounding illumination.For this reason, without the illumination estimation process, the color conversion function for various RGB levels is derived using only two images captured on the AWB (auto white balance) mode and the preset mode.The difference of the two images is used for the internal calibration process that corrects the color balance.It is possible to estimate the color conversion of the image sensor through the color transfer function.In simulation results, performance comparisons are made with the characterized cameras based on the single matrix and the multimatrices.This demonstrates that the proposed camera characterization method performs effectively relative to previous methods.

Image Calibration
In handling captured images, an important issue concerns the color balance status of the image sensor.In most consumer digital cameras, the sensor color channels are calibrated according to sensor response to the intensity and wavelength of the scene.The AWB techniques find the optimal color balanced condition for varying surroundings, by using various illumination estimation methods such as grey world, white patch based, neural network based, and bootstrapping methods [14].Moreover, conventional digital cameras adopt a knee curve for dynamic range compression and the nonsaturated highlight range [15].
Figure 1 shows the difference between the sensor responses of Nikon and Canon cameras.Both cameras are balanced for identical illumination of A and D65 illuminants for the same scene.This allows a comparison between different digital cameras as regards color balance transformation.Sensor responses are plotted in the relative RGB space and show how the RGB values are calibrated between white balanced and unbalanced images.For some color patches, the RGB values are synthesized with the different sensor characteristics of two cameras.It can be seen that the RGB channel gains are controlled differently for different signal levels and that the RGB responses vary significantly even though light conditions are consistent.These are affected by the camera-specific color constancy algorithms.In other words, the color sensors of two cameras perform differently even for the case of color balanced for the same illumination.The internal calibrating parameters are absorbed into the color transformation and cause nonlinear transformation owing to the gain, gamma, and knee control properties.

Colorimetric Analysis
Color imaging systems using CISs can be characterized by the colorimetric method.The resulting camera characterization generates a transfer matrix for the colorimetric analysis of captured images.The color-target-based method uses reference colors and their corresponding  values to determine an approximated linear transformation between CIE  values and camera RGB signals.The transfer characterization matrix is derived through polynomial regression based on measurements of known color samples.The color targetbased characterization using polynomial regression is generally adopted owing to its simplicity and effectiveness.
The transformation processing between  tristimulus for  color test targets and corresponding camera RGB signals is shown in the following: . ( The camera characterization matrix  c is determined to minimize the color differences over all test targets.Because 3×  RGB values should correspond to 3 ×   tristimulus through a 3 × 3 matrix, the 3 × 3  c matrix is derived by using the least squares method [16].
Nevertheless, this method is accurate only under certain standard illuminants because the characteristics of CISs are dependent on the surrounding illumination.In addition, the internal AWB process controls the variations of chromaticity and intensity for correct imaging.The cameras have different colorimetric characteristics according to white balance conditions.Accordingly, the transfer matrix should be recalculated for these inconsistent cases and thus illuminationadaptive measuring methods have been proposed.
A recently proposed method uses the estimated phosphor primaries and reference white points to calculate the tristimulus constant matrix for certain illumination conditions [12].However, this requires phosphor chromaticity information for the RGB and white reference to be previously measured.An alternative method is color measurement based on multicharacterization and illuminant estimation [13].First, the representative camera transfer matrix is measured for A, D50, and D65 illuminants.Then, the CCT (correlated color temperature) of an illuminant is estimated to select the correct representative transfer matrix for the illuminant.Finally, the tristimulus weighting factors from ALC (auto luminance control) functions are modified to control signal gains for sustained luminance levels.This enables more precise estimation results through the selective  c .

Colorimetric Analysis Based on a Calibrated Color Conversion Matrix
The characteristics and calibration of CISs are dependent on the surrounding illuminants.To obtain a precise camera characteristic that changes under different ambient conditions, it is necessary to know calibrating parameters such as the status of the ALC registers and balanced white points.However, the characteristics of balanced images and the sensor mechanism are unknown for consumer cameras; thus, it is very difficult to find out the condition of the calibrated sensors in an arbitrary ambient situation.The previous analytical methods are applicable only under restricted measuring circumstances, in which it is possible to understand the sensor's signal processing by interfacing with the internal registers.Thus, in this section, an ambient-independent method is proposed for enhanced colorimetric measurements.It approaches the resulting color transformation based on the characterizations for the uncalibrated and calibrated images in each preset and AWB mode.First, RGB input signals are converted to normalized values that are compensated linearly with the inverse of the camera gamma: where       are the gamma compensated and normalized outputs from the RGB inputs,  is the bit-depth ( = 8), and       are gamma values for each RGB channel.
Then, the camera characterization for the transfer function should be performed under the reference illuminant.The color transfer matrix is calculated for a camera set by the preset mode.Herein, the 5000 K illuminant is adopted for the reference, and RGB channel gains are set internally such that RGB outputs are in a ratio of 1 : 1 : 1 for white and grey samples under that illuminant.Thus, the preset mode means the reference mode for the D50 illuminant.The transfer characterization of a CIS can be conducted using polynomial regression with the least squares method.For the reference illuminant,  tristimulus are estimated from the derived 3 × 3 matrix ( c D50 ) and RGB signals in the preset mode as per the following: However, for the arbitrary illuminant condition, the single characterization does not perform an adequate estimation of the reflected color signal.The transfer matrix should be modified to correspond to the changed surroundings.As shown in Figure 1, the calibrated images represent the converted color balances and it is possible to find the weight of primaries and channel gains by using the difference between the uncalibrated and calibrated images.The difference between the uncalibrated image in the preset (or reference) mode and the calibrated image in the specific AWB condition can be used to determine a color conversion for different signal levels.The transfer gains of each , , and  for the lower part ( lg ) and upper part ( ug ) are calculated approximately as per the following: where "̂" denotes the average value, subscript "low AWB" and "low ref" refer to the lower part in the AWB mode and the reference mode, respectively,  c is the average value of total reference mode inputs, and "up AWB" and "up ref" refer to the upper part in each mode.The calculated data are sampled at a rate of 10 : 1 to reduce computing burden.
According to the separated signals, the approximation function for transfer gains consists of two parts.The bi-segmental channel transfer function is derived to reflect the nonlinear transformation property as per the following: where  AWB is the estimated AWB output,  lc is the average value of reference values less than  c ,  uc is the average value of reference values more than  c , and  max and  max are the maximum values in the reference and AWB modes, respectively.
The value of  AWB has been estimated nonlinearly using a quadratic function for the estimation of calibrated outputs corresponding to the region of reference inputs more than  lc , while a transfer function has been derived linearly for the relatively narrow region of lower values less than  lc .The construction of the channel transfer function using bisegmental color gains,  lg and  ug , is illustrated in Figure 2.
Then, the new color transfer matrix of the CIS is reconstructed using a color conversion matrix ( CC ) between the uncalibrated and calibrated images as per (6).Finally, the tristimulus of color objects in a scene can be estimated using the generated transfer matrix.When the illuminant changes, the color conversion matrix controls 3 channel gains according to the image calibration by AWB processing.The proposed colorimetric analysis flow is shown in Figure 3(b) in comparison with the multi- c method of Figure 3(a).The multi- c method requires finding characteristic transfer matrices under various standard illuminations and estimating the illuminant.On the other hand, the adaptive- c gives estimated results for illuminants and color information by obtaining the color conversion matrix using images captured in two different modes without the illuminant estimation and the AWB information of the image sensor.

Simulation Results
The proposed algorithm has been tested on images containing Macbeth 24 color samples and various objects under the representative D50, D65, and A illuminants using a Macbeth lighting booth.A schematic diagram of the camera characterization and color estimation is shown in Figure 4.The viewing conditions considered for the verification of the method are restricted within the range of indoor environments and three illuminants.A CMOS CIS (TOSHIBA TCM8230MD) module including a control interface was used for the experiments.To evaluate the multi- c method compared with the proposed method, first the illuminant was estimated using a single characterization matrix (D50) for higher luminance pixels corresponding to the upper 10% of  signals.Then, the most close matching of the three illuminants was selected and the corresponding characterization matrix was used for color estimations.For the consideration of the preset mode, the internal parameters of the camera were fixed so that the RGB outputs maintained the white balance state under the 5000 K illuminant.
Tables 1, 2, and 3 show the relative performance of the color and illuminant estimations in color differences between the measured and predicted values and a comparison of the processing steps between multi- c and adaptive- c .The average errors were computed in CIE   V  color space.Comparisons of average color differences for color samples and illuminants for the three methods are shown in Tables 1  and 2, respectively.The results show that the adaptive- c method performs better than the multi- c method by 16% and 65% for the estimation of color samples and illuminants, respectively.The meaningful results are shown in color estimations under illuminant D65 and white estimations for D50, D65, and A illuminants.In comparison with the conventional multicharacterization for typical illuminants through the illuminant estimation process, the proposed method requires only a single characterization, no ALC information, and no  c selection based on illuminant estimation; however, it does require acquiring two-mode images.
In Table 3, the multi- c method requires finding the characteristic transfer matrices for various standard illuminations.Additionally, the illuminant estimation for  c selection using pixel samples of higher levels is inaccurate.However, the adaptive- c demonstrates satisfactory results for illuminants and color information by obtaining the color conversion matrix using images captured in two different modes without any other interfacing with the CIS.

Conclusions
In color imaging systems, colorimetric measuring is necessary to correctly reproduce color images.This is generally performed by techniques such as chromatic adaptation transformation that requires absolute luminance and chrominance information.
It is also indispensable to estimate illuminants and object colors for the correction of degraded effects by tonal mapping and color distortion in rendering high dynamic range   (2) Characterizing a CIS for 3 standard illuminants (A, D50, D65).
Δ  V  Color samples 0.0142 0.0133 Illuminants (white samples) 0.0158 0.0056 scenes on low dynamic range displays.However, it is difficult to obtain accurate colorimetric measurement because the AE and AWB of cameras operate according to the exposed environment.The status of sensor calibration is unknown because it differs in many imaging systems.In general, color conversion techniques have been developed to convert between different illuminants based strictly on the physical quantities, and thus the measured results depend on the chosen illuminant.In this paper, to solve this problem, the transfer matrix according to white-balance conditions is estimated but does not use the internal CIS register values.The difference between the uncalibrated and calibrated images is solely used to derive the color conversion matrix.This method of color analysis can be consistent with the assumption of known color-processing and illuminants.Experimental results show the proposed method is valid in terms of measuring performance.The prediction of luminance and chromaticity of scenes is applicable to CISs in automatic systems responding to various environments.

Figure 1 :
Figure 1: Comparisons of relative RGB transform relationships obtained from color balanced images by two different cameras for the same illuminant: (a) under illuminant A, (b) under illuminant D65.

Figure 2 :
Figure 2: Illustration of the estimation of the channel transfer function using lower and upper color gains,  lg and  ug .

Figure 3 :Figure 4 :
Figure 3: Block diagrams of color analysis processes: (a) the multi- c method and (b) the scene-adaptive- c method.

Table 1 :
Comparisons of estimation performance among single- c , multi- c , and adaptive- c in color differences.

Table 2 :
Comparisons of illuminant estimation among single- c , multi- c , and adaptive- c in color differences.      are the calibrated inputs in the AWB mode and       are the uncalibrated inputs in the reference mode.      are values for general nonstandard illuminant environments.

Table 3 :
Comparisons between multi- c and adaptive- c in terms of processing steps and results.Multi- c based on illuminant estimationAdaptive- c based on color conversion estimation Processing steps(1) Set the CIS : AWB off, manual RGB gain (1 : 1 : 1), gamma correction off.