The Very High Temperature Reactor Methods Development group at the Idaho National Laboratory identified the need for a defensible and systematic uncertainty and sensitivity approach in 2009. This paper summarizes the results of an uncertainty and sensitivity quantification investigation performed with the SUSA code, utilizing the International Atomic Energy Agency CRP 5 Pebble Bed Modular Reactor benchmark and the INL code suite PEBBEDTHERMIX. Eight model input parameters were selected for inclusion in this study, and after the input parameters variations and probability density functions were specified, a total of 800 steady state and depressurized loss of forced cooling (DLOFC) transient PEBBEDTHERMIX calculations were performed. The six data sets were statistically analyzed to determine the 5% and 95% DLOFC peak fuel temperature tolerance intervals with 95% confidence levels. It was found that the uncertainties in the decay heat and graphite thermal conductivities were the most significant contributors to the propagated DLOFC peak fuel temperature uncertainty. No significant differences were observed between the results of Simple Random Sampling (SRS) or Latin Hypercube Sampling (LHS) data sets, and use of uniform or normal input parameter distributions also did not lead to any significant differences between these data sets.
Title 10 Part 50 (10 CFR 50.46) of the United States Code of Federal Regulations first allowed “Best Estimate” calculations rather than conservative code models of safety parameters in nuclear power plants in the 1980s, stipulating, however, that uncertainties be identified and quantified [
In general, code uncertainty refers to uncertainty in the ability of a computer software product, coupled with a specific model, to accurately describe the actual physical system of interest. The computer model is an integration of the mathematical model, the numerical techniques used to solve those equations, and the representation of the physical model by the input geometry and material specifications. Each element contributes to the total uncertainty in the output parameter of interest, usually referred to as the Figure of Merit (FOM) in nuclear safety studies. The mathematical model consists of one or more governing equations that describe the balance between the creation, destruction, and flow of some quantity of interest (e.g., heat, coolant mass, or neutron flux) within a homogeneous control volume. It also consists of one or more subgrid equations that relate these gross phenomena to more complex physics that are neglected at the scale of the homogeneous control volume (e.g., neutron streaming between pebbles, heat conduction from the kernels to the pebble surface, etc.).
A further complication is that very few computer codes solve the analytic form of its governing equations. Instead, the differential operators in these equations are expanded as a truncated series and cast as a set of difference equations solved over a discrete mesh. If the equations are well posed, the solution is unique and refining the mesh reduces the error between the solutions of the discretized equation and the original differential equation. Unfortunately, unlimited mesh refinement is not possible and one must tolerate some truncation error. Furthermore, in many complex fluid system simulation codes, the combination of governing equations and subgrid correlations yields illposed systems of differential equations that do not converge to the analytical solution upon refinement of the mesh.
Another important source of uncertainty is that the input model is a simplification of the actual physical geometry. For example, the distribution of fuel pebbles in a HTGR core is neither regular nor uniform, but to model individual pebbles is computationally prohibitive. Complex geometrical detail in some of the prismatic HTGR designs can likewise be very difficult to model accurately. The fourth major source of input uncertainty is the material neutronic and thermophysical properties. For core analysis, these include thermal properties such as conductivity and heat capacity, fluid properties such as density and viscosity, and neutronic properties such as cross sections. Knowledge of these parameters for each material of interest may be limited in the range of conditions found in a typical HTGR. Such uncertainty can be reduced through material testing and measurement, but the amount of testing is often limited by cost and schedule constraints and must be propagated through the calculations. In some cases, the natural variability of a given parameter under even the best experimental conditions may be large enough to inject uncertainty that cannot be ignored. Finally, when modeling of an actual operating reactor is considered, it is well known that the operational conditions (power level, inlet temperature, measured mass flow rates) can also have associated uncertainty ranges.
Of the four types of uncertainty sources indicated here, the uncertainties in material properties can usually be addressed by relatively simple manipulation of the corresponding values in the input decks, and geometry simplifications can be benchmarked against higher fidelity codes (e.g., 2D versus 3D effects). In contrast, propagation of uncertainties in mathematical models and solver techniques are much more challenging, and in most cases not yet attempted in industry. Developments in uncertainty methodology are therefore currently focused on crosssection, model and material input data uncertainties, and specifically on the propagation of uncertainties through sets of coupled neutronic and thermalfluid calculations.
Several large international uncertainty quantification programs have been developed in recent years, of which the LWR Uncertainty Analysis in Modeling (UAM) benchmark [
The Very High Temperature Reactor (VHTR) Methods Development group at the Idaho National Laboratory (INL) identified the need for a defensible and systematic uncertainty and sensitivity approach that conforms to the code scaling, applicability, and uncertainty (CSAU) process in 2009. The Gesellschaft für Anlagen und Reaktorsicherheit (GRS) has incorporated a stochastic sampling CSAU approach that is particularly well suited for coupled reactor physics and thermal fluids core analysis into the Software for Uncertainty and Sensitivity Analyses (SUSA) code [
This paper summarizes the results of an uncertainty quantification investigation performed with SUSA, utilizing a typical HTGR benchmark (the International Atomic Energy Agency CRP5 Pebble Bed Modular Reactor 400 MW Exercise 2) and the INL code suite PEBBEDTHERMIX. For this study, the effects of uncertainties in the crosssection data and the propagated celltolatticetocore uncertainties have not yet been included—the focus here is on the effect of model and material input uncertainties for a coupled transient case. The CRP on HTGR UAM [
Two major approaches to perform uncertainty propagation in a statistically rigorous manner can be identified [
Methods based on the propagation of
Methods based on the propagation of
(The direct perturbation method is also identified as a possible third, but expensive, approach in [
The two approaches can be summarized as follows.
Statistical methods (input uncertainty propagation).
Assign subjective probability ranges and distributions to (an unlimited number of) input parameters.
Propagate the combined uncertainty through the core models to determine the statistical distribution properties of the Figure of Merit (FOM). Typically lower and upper tolerance intervals can be obtained within some defined confidence level, using an ordered statistics approach.
Disadvantages include the subjective selection of input parameter distributions types and ranges, and the fact that at least 59 model calculations are required to obtain acceptable statistical information for an onesided 95% tolerance limit with 95% confidence.
Deterministic methods (output uncertainty propagation).
Use of a relevant set of experimental data to establish a database of uncertain data for a large number of input parameters.
Create hypercubes characterizing physical parameters and their dependencies for a wide variety of plant conditions, transients, and so forth.
Perform a single calculation utilizing all input parameters to determine the error bands enveloping the output FOM.
Disadvantages include the availability of large operational/experimental datasets for the construction of the hypercubes, and the inability of this method to distinguish the major contributors to the overall uncertainty.
Typically, because of the deterministic method requirement to have a large and comprehensive experimental database available, LWR and BWR uncertainty studies can use this method (especially for thermal fluid uncertainty studies). However, in the HTGR domain, very limited experimental and operational data exists, and the use of statistical uncertainty methods is currently the only viable approach for coupled uncertainty propagation.
The first step in the GRS method is to select the set of uncertain input parameters that will be used to evaluate the desired FOM. It is important to note that the stochastic sampling methodology is independent of the number of input parameters selected, and that all of them are sampled simultaneously as a single set. Information from the manufacture of nuclear power plant components as well as from experiments and previous calculations are used to define the mean value and probability distribution or standard deviation of uncertain parameters. Normal (and in some cases uniform) distributions are used in the absence of more knowledge about the input parameters. Once these distributions and dependencies have been established, the analyst can:
generate a random sample of size
perform the corresponding
calculate quantitative uncertainty statements, for example, 5% and 95% tolerance intervals within a specified confidence level, usually 95% (denoted here as 95%/95%);
calculate quantitative sensitivity measures to identify those uncertain parameters that contribute most to the uncertainty of the results.
The number of code calculations is determined by the requirement to estimate a tolerance interval for the quantity of interest with a specified confidence level. Wilks’ formulae [
where
Figure
Example of the GRS methodology applied to the PEBBEDTHERMIX CRP5 PBMR 400 exercise 2 benchmark calculation.
The THERMIXKONVEK code was developed in Germany during the German HTGR program for the thermal fluid analysis of pebble bed HTRs [
An existing PEBBEDTHERMIX model of the CRP5 PBMR 400 MW benchmark was used as the starting point of the uncertainty study.
The typical FOM for a DLOFC event is the peak fuel temperature, that is, the maximum spatial and temporal temperature reached in the fuel spheres. As a demonstration of the uncertainty methodology applied to a typical HTGR problem, eight input parameters were selected for this study, as shown in Table
PEBBED CRP5 PBMR 400 DLOFC uncertainty study input parameters.
Parameter  Mean value  2 Standard deviations (2 
PDF type 

Reactor power  400 MW  ±8 MW (2%)  Normal and uniform 
Reactor inlet gas temperature (RIT)  500°C  ±10°C (2%)  Normal and uniform 
Decay heat multiplication factor  1.0  ±0.057 (5.7%)  Normal and uniform 
Fuel specific heat multiplication factor  1.0  ±0.06 (6%)  Normal and uniform 
Reflector specific heat multiplication factor  1.0  ±0.10 (10%)  Normal and uniform 
Fuel conductivity multiplication factor  1.0  ±0.14 (14%)  Normal and uniform 
Pebble bed effective conductivity multiplication factor  1.0  ±0.08 (8%)  Normal and uniform 
Reflector conductivity multiplication factor  1.0  ±0.10 (10%)  Normal and uniform 
Since the selection of the input parameters and their distributions is one of the known weak points of the stochastic sampling methodology, the results of a separate TINTE study, performed for the PBMR 400 MW design [
The variations on the power and reactor inlet gas temperature were applied directly on the absolute value of the input variable itself (e.g., 2% on 400 MW), in contrast to the decay heat, specific heat, and thermal conductivity, where the variations were applied as multiplication factors on the complex correlations that are used to calculate these variables. For example, the specific heat capacity of the reflector graphite material is a thirdorder polynomial function of temperature
The sampled multiplicative factor
It should be noted that the decay heat is an almost linearly dependent variable of the long term steady state reactor power, and as such it is not an independent input variable to this uncertainty quantification. While the SUSA code allows for input parameter dependencies (correlations) to be specified as part of the input preparation, this option was not selected for this study. It was rather decided to see if this input parameter correlation can be observed in the output data sensitivity analysis.
The mean and two standard deviations
A second potential weak point of the statistical method is the justification for the selection of the probability density function (PDF) types. Typical thermal physical properties, such as specific heat and thermal conductivity, can be obtained from the manufacturers, and are usually specified as normal PDFs with mean standard deviation values. More complex variable PDFs (e.g., variations in the core bypass flows) can be biased/skewed to one side, for example, gap widths grow larger or shrink over time as the graphite reflectors swell and shrink with fast fluence exposure. In cases where no definitive uncertainty information exists, a uniform or a normal/Gaussian PDF can be used with or without truncated tails. For this study, both normal and uniform PDFs were selected to assess if this factor plays a significant role in the DLOFC peak fuel temperature uncertainty. Figure
PDFs for the specific heat and thermal conductivity correlation input parameters.
A total of six case sets (consisting of four sets of 100 and two sets of 200 model runs each) were performed for this study as described below and summarized in Table
Number of model runs: the number of model runs was doubled from 100 to 200 to investigate if a larger population sample produced significantly different statistical indicators.
Sampling methodology: SUSA is capable of using either the simple random sampling (SRS) [
Distribution type: two of the 100 model run sets were designed to quantify possible differences that could result when the input parameter PDFs are changed from uniform to normal distribution types.
Number of parameters sampled: a final point of interest was the uncertainty contribution of a few dominant input parameters compared to the combination of all eight input parameters. To this end, two sets of 100 model runs each were performed, varying only the material correlations and only the power, RIT, and decay heat correlation, respectively.
PEBBEDTHERMIX CRP5 DLOFC uncertainty study cases.
Number of model runs  Input parameter sampling method  Input parameter distribution type  Model input parameters varied 

100  Latin Hypercube  Uniform  Power, RIT, decay heat only 
100  Latin Hypercube  Uniform  Specific heat and thermal conductivity only 
100  Latin Hypercube  Uniform  All 
100  Latin Hypercube  Gaussian/normal  All 
200  Latin Hypercube  Gaussian/normal  All 
200  Simple Random  Gaussian/normal  All 
In the discussions that follow, the notation will be of the format “number, sampling method, distribution type”; for example, the first entry in Table
The SUSA generated input data can be verified for conformance to the user’s specifications using scatter plots, as shown in Figure
Sampled values of the total power (MW) for the 200 LHS Normal set.
For the next step of the uncertainty quantification procedure, the SUSAgenerated data for the eight input parameters were used to create PEBBED and THERMIX model input files for the steady state and DLOFC calculations. The six sets listed in Table
The time behavior of the maximum fuel temperature during the DLOFC transient is shown in Figures
The PBMR core design leads to the typical HTGR loss of cooling behavior, that is, a slow increase in the maximum fuel temperature over several hours, with the peak fuel temperature reached 40–60 hours into the transient.
The shapes of the curves in the first 30 cases are similar but the gradients are not. Although the same physical phenomena are present in all the DLOFC events, the rate of energy deposition (correlated to the decay heat) and energy removal (correlated to the fuel and reflector specific heat and thermal conductivities) differ for each of these cases, according to the sampled input values.
Changes in the eight input parameters have opposite effects on the maximum fuel temperature: an increase in the decay heat will increase the fuel temperature, but an increase in the fuel graphite conductivity will remove heat faster from the core, and therefore lead to a lower fuel temperature. Since each DLOFC case consists of a random sampled set of the eight input parameters, the low fuel temperature curves can be the result of a few parameters sampled low (or high) simultaneously, and an average fuel temperature curve could be caused by a cancellation of effects. These factors also cause the shift in time when the peak fuel temperature values are reached.
The spread in maximum fuel temperatures between the first 30 cases is not constant with time. For example, it starts off with less than 5°C in the first hour and increases to 98°C for the 200 LHS Normal set, as shown in Figure
The 95%/95% twosided tolerance intervals are not compared at a fixed time point, but rather at the varying time point where a specific case reaches its peak DLOFC fuel temperature. This study therefore compares the
The temperature variation bandwidth for the 3 cases shown here seem to be quite different. The two sets that consisted of 100 runs each produced significantly larger variations than the 200 LHS Normal set (e.g., 141°C versus 94°C), and there is also a smaller difference between the 100 LHS Normal and 100 SRS Uniform sets (131°C versus 141°C). It is however important to note that the figures just show the first 30 model runs of each set for clarity sake. For the 200 LHS Normal set, the remaining 170 model runs sample a larger portion of the “true” unknown distribution, with a resultant larger bandwidth, as shown in Figure
The primary FOM for this study (DLOFC peak fuel temperature) results are presented in Figure
A more pronounced visual difference can be observed in Figure
DLOFC maximum fuel temperature versus time for the first 30 cases of the 100 SRS Uniform set.
DLOFC maximum fuel temperature versus time for the first 30 cases of the 100 LHS Normal set.
DLOFC maximum fuel temperature versus time for the first 30 cases of the 200 LHS Normal set.
DLOFC maximum fuel temperature versus time for all 200 cases of the 200 LHS Normal set.
Peak DLOFC fuel temperature of the 200 LHS Normal set at 50 hours.
Normalized peak DLOFC fuel temperature histograms for the 200 LHS and SRS Normal sets.
Normalized peak DLOFC fuel temperature histograms for the 100 LHS Normal and Uniform sets.
For the uncertainty quantification step, SUSA can perform several statistical correlation fitness tests on the output data to determine the properties of the unknown FOM distribution. For example, the KolmogorovSmirnov (KS) test [
An illustration of the KS test is shown in Figures
PDF and fitted normal distribution results for the 100 LHS Normal set: KS level of significance = 0.9919.
PDF and fitted uniform distribution results for the 100 LHS Normal set: KS Level of Significance = 0.0026.
A second example of the Lilliefors and KS test results for the 100 LHS Uniform set is presented in Table
Lilliefors and KS test results for the 100 LHS Uniform set.
Test applied for Figure of Merit (DLOFC peak fuel temperature) fit  Level of Significance for Distribution Type  

Normal  Lognormal  Exponential  Gamma  
Lilliefors  0.64  0.79  0.01  — 
KS  0.92  0.97  —  0.95 
PEBBED CRP5 DLOFC uncertainty study results.
No of model runs  Input data  Output FOM (DLOFC Peak fuel temperature (°C))  

Sampling method  Distribution type  Parameters varied 

Population mean value 
 
99  LHS  Uniform  Power, RIT, decay heat  1545  1603  1664 
100  LHS  Uniform  Specific heat, thermal conductivity  1555  1605  1652 
100  LHS  Uniform  All  1513  1605  1686 
99  LHS  Normal  All  1536  1604  1675 
200  LHS  Normal  All  1537  1604  1658 
199  SRS  Normal  All  1531  1604  1680 
A summary of the mean and 95%/95% twosided tolerance intervals at the time when the peak fuel temperature are reached are shown in Table
The following observations can be made from this data.
The mean values for all six datasets are almost identical (e.g., 2°C variation on 1604°C), that is, regardless of the sampling method, parameters included, or distribution types, these six independent random sets predict the same mean DLOFC peak fuel temperature (1604°C).
Even before an analytical sensitivity study is performed to determine which of the factors are responsible for most of the variations in the output data, the first two datasets shown here already show that the power, inlet gas temperature, and decay heat variations contribute significantly to the variation seen in the DLOFC peak fuel temperature. On their own, these three small input variations produced lower and upper tolerance intervals ranging between 1545°C and 1664°C (a spread of 119°C), while the much larger uncertainty variations in the five material correlations lead to values of 1555°C and 1652°C (a smaller spread of 97°C). Both these sets can be compared with the 100 LHS Uniform set where all eight input variables were included and 95%/95% twosided tolerance intervals of 1513°C and 1686°C were obtained.
The use of an uniform distribution will result in the sampling of high and low input values more frequently compared to a normal distribution, since the probability of sampling a high, mean, or low value is identical for a uniform distribution, but there is a lower probability to sample from the low and high tails of the normal distribution. This effect could partly explain the lower and higher tolerance intervals on the peak fuel temperature (1513°C versus 1536°C, and 1686°C versus 1675°C) for the 100 LHS Uniform set, compared to the 99 LHS Normal set values. The difference between the two lower and upper tolerance intervals predictions is however minimal: only 23°C and 11°C on a mean value of 1604°C, respectively. The use of normal distributions for input parameter variations, as is most commonly applied when no other information is available, could therefore lead to slightly lower estimates of the tolerance limits, compared to the use of uniform distributions. This observation might however only be valid for this specific HTGR design, code and model combination and these sampled datasets.
An interesting current issue in the uncertainty quantification community revolves around the issue of applying stratified sampling techniques (like Latin Hypercube) to improve the coverage of the input sample set ([
It should again be noted however that these differences are small compared to the mean peak DLOFC fuel temperature (only 1.4% on 1604°C), so that the conclusions reached in a Sandia Laboratory study [
Model calculations are crucial timeconsuming factors for the statistical uncertainty method. This study did not observe significant differences between the tolerance intervals obtained with sets consisting of 100 or 200 model runs, that is, covering the Wilks’ formula range from the second (93 runs) to the fifth (181 runs) order. The Wilks’ formula second order application (93 runs) therefore seems to be sufficient for this core design, model and transient. This conclusion is supported by the ATHLET PWR study [
A single example of the time dependent nature of the data shown in Table
It has been shown in this section that the input uncertainties in only eight parameters already lead to 95%/95% twosided tolerance intervals of 1531°C and 1680°C on a mean value of 1604°C (the data for the 200 SRS Normal set has been used here). These values represents an uncertainty band/spread of approximately 4.6% around the mean value of 1604°C for the peak fuel temperature during a DLOFC transient in the PBMR design. A more complete study, taking into account all known input uncertainties could possibly lead to a larger uncertainty bandwidth. These uncertainties need to be taken into account during the HTGR reactor safety margin design process.
Time dependent minima, maxima, mean and 5/95 percentiles for the 200 LHS Normal set maximum fuel temperature.
This section presents selected results from the SUSA sensitivity analysis. An overview of the definitions, uses, and advantages of typical sensitivity parameters (regression coefficients, correlation measurements, partial and empirical coefficients, etc.) can be found in [
The Kendall rank correlation coefficients and the empirical correlations ratios shown in Figure
Indices for data in Figure
Input variable number  Description 

1  Reactor power 
2  Reactor inlet gas temperature (RIT) 
3  Decay heat multiplication factor 
4  Fuel specific heat multiplication factor 
5  Reflector specific heat multiplication factor 
6  Fuel conductivity multiplication factor 
7  Pebble bed effective conductivity multiplication factor 
8  Reflector conductivity multiplication factor 
Kendall rank correlation coefficients for the 100 LHS Uniform (a) and 200 LHS Normal (b) sets. (c) Partial correlation coefficients for the Kendall correlation of the 200 LHS Normal set. Empirical correlation ratios for the 100 LHS Uniform (d) and 200 LHS Normal (e) datasets.
All data shown here is for their effects on the FOM, the DLOFC peak fuel temperature. The
Apart from a change in order between input parameters 7 and 8, the Kendall rank correlation coefficients results shown Figures
The empirical correlation ratios presented in Figures
As a final example, the time dependent empirical correlation ratios for the 200 LHS Normal data set shown in Figure
Empirical correlation ratios (peak fuel temperature) variations versus time for the 200 LHS Normal set.
This report summarizes the results of an uncertainty and sensitivity quantification study performed with the GRS code SUSA, utilizing a typical high temperature reactor benchmark (the IAEA CRP5 PBMR 400 MW Exercise 2) and the INL suite of codes PEBBEDTHERMIX. The following steps were performed as part of the uncertainty and sensitivity analysis.
Eight PEBBEDTHERMIX model input parameters were selected for inclusion in the uncertainty study: the total reactor power, inlet gas temperature, decay heat, and the specific heat capacity and thermal conductivity of the fuel, pebble bed, and reflector graphite.
The input parameters variations and probability density functions were specified, and a total of 800 PEBBEDTHERMIX model calculations were performed, divided into 4 sets of 100 and 2 sets of 200 steady state and DLOFC transient calculations each.
The DLOFC peak fuel temperature was supplied to SUSA as model output parameters of interest. Using both the Simple Random and the Latin Hypercube Sampling techniques, the Wilks formulation was applied to the 6 datasets, and the 5% and 95% tolerance limits were determined with 95% confidence levels.
A SUSA sensitivity study was performed to obtain correlation data between the input and output parameters, and to identify the primary contributors to the output data uncertainties.
It was found that the uncertainties in the decay heat, pebble bed, and reflector thermal conductivities were responsible for significant contributions to the propagated uncertainty in the DLOFC peak fuel temperature. No significant differences were observed between the results of SRS or LHS sampled datasets, and the same conclusion was made from a comparison between the results of sets that used uniform input parameter distributions as opposed to normal distributions. The 95%/95% twosided tolerance intervals values of 1531°C and 1680°C represent an uncertainty band/spread of approximately 4.6% around the mean value of 1604°C for the peak fuel temperature during a DLOFC transient in the PBMR 400 MW design.
Possible future investigations in the HTGR uncertainty assessment program at INL include the following.
Clarify the approach on complex nonstatistical uncertainties: bypass flows through the reflectors, including radial and axial power peaking factors, control rod worths, and so forth.
The propagation of the uncertainties in the cross section data from the basic nuclear ENDF libraries to the coupled transient solutions represents a significant challenge. In this regard, it is currently planned to utilize the Generalized Perturbation Theory (GPT) and stochastic sampling (XSUSA) capabilities of the upcoming SCALE 6.2 release [
This work is supported by the U.S. Department of Energy, Assistant Secretary for the Office of Nuclear Energy, under DOE Idaho Operations Office Contract DEAC0705ID14517.