Safe operation and industrial improvements are coming from the technology development and operational experience (OE) feedback. A long life span for many industrial facilities makes OE very important. Proper assessment and understanding of OE remains a challenge because of organization system relations, complexity, and number of OE events acquired. One way to improve OE events understanding is to focus their investigation and analyze in detail the most important. The OE ranking method is developed to select the most important events based on the basic event parameters and the analytical hierarchy process applied at the level of event groups. This paper investigates further how uncertainty in the model affects ranking results. An analysis was performed on the set of the two databases from the 20 years of nuclear power plants in France and Germany. From all uncertainties the presented analysis selected ranking indexes as the most relevant for consideration. Here the presented analysis of uncertainty clearly shows that considering uncertainty is important for all results, especially for event groups ranked closely and next to the most important one. Together with the previously performed sensitivity analysis, uncertainty assessment provides additional insights and a better judgment of the event groups’ importance in further detailed investigation.
Collecting and understanding operating experience are an important part of keeping continuous, reliable, and safe operation of any complex industrial facility including nuclear power plants. This operating experience is taken from the facility to the national and international levels (e.g., [
In [
This paper investigates how the uncertainty of ranking indexes of relative importance (i.e., weightings) influences the ranking results. The described method is applied separately for 20 years of events from the nuclear power plants operation in France and Germany. The focus is only on the uncertainty for each data source, with no aim of comparing them because of significant differences in technology, data collection, and other factors.
Here, we first briefly present the event groups ranking method and then proceed to analyze uncertainty. The ranking method description focuses on the most critical elements for the uncertainty. As mentioned before, a complete description of the method is available in [
Ranking is performed at the level of event groups (EG) using selected sets of parameters. For the brevity, only about half of the more important groups are selected and presented in the paper. Table
Selected event groups for ranking uncertainty assessment with respected number of events and trend value.
ID | FR | DE | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
|
G & R | IRS | Pr. | T5y | U |
|
IRS | Pr. | T5y | U | |
01.2 | 161 |
|
5 | 11 | 0.98 | N | 102 | 0 | 3 | 0.33 | Y |
|
|||||||||||
02.3 | 271 |
|
2 | 15 | 0.35 | Y | 6 | 0 | 0 | 0.50 | N |
|
|||||||||||
03 | 44 |
|
7 | 8 | 0.20 | Y | 33 | 1 | 1 | 0.50 | Y |
|
|||||||||||
04.1 | 89 |
|
16 | 10 | 0.18 | Y | 44 | 1 | 0 | 0.76 | N |
04.2 | 20 |
|
4 | 0 | 0.83 | N | 6 | 1 | 0 | 0.23 | Y |
04.3 | 70 |
|
2 | 5 | 0.52 | N | 38 | 0 | 0 | 0.01 | Y |
|
|||||||||||
05.1 | 64 |
|
43 | 21 | 0.42 | Y | 1 | 0 | 0 | 0.50 | N |
05.2 | 48 |
|
1 | 6 | 0.18 | Y | 149 | 0 | 8 | 0.75 | Y |
|
|||||||||||
07.1 | 1415 |
|
43 | 97 | 0.97 | Y | 168 | 3 | 19 | 0.54 | Y |
|
|||||||||||
08.2 | 18 |
|
3 | 5 | 0.50 | N | 104 | 0 | 2 | 0.32 | Y |
08.3 | 320 |
|
31 | 24 | 0.25 | Y | — | — | — | — | — |
|
|||||||||||
09.1 | 105 |
|
16 | 15 | 0.21 | Y | 167 | 1 | 13 | 0.81 | Y |
09.3 | 130 |
|
16 | 35 | 0.20 | Y | 8 | 0 | 0 | 0.50 | N |
|
|||||||||||
10.2 | 5 |
|
0 | 0 | 0.53 | N | 33 | 0 | 0 | 0.12 | Y |
|
|||||||||||
12 | 236 |
|
48 | 65 | 0.40 | Y | 4 | 0 | 1 | 0.50 | N |
|
|||||||||||
13.2 | — | — | — | — | — | — | 351 | 5 | 11 | 0.93 | Y |
13.3 | — | — | — | — | — | — | 458 | 6 | 10 | 0.94 | Y |
13.4 | 1054 |
|
46 | 190 | 0.17 | Y | — | — | — | — | — |
|
|||||||||||
14 | 470 |
|
12 | 48 | 0.97 | N | 172 | 5 | 22 | 0.04 | Y |
|
|||||||||||
15 | 111 |
|
6 | 15 | 0.00 | Y | 42 | 1 | 2 | 0.76 | N |
|
|||||||||||
16.1 | 1194 |
|
14 | 39 | 0.87 | Y | 69 | 3 | 2 | 0.97 | N |
16.2 | 1681 |
|
5 | 34 | 0.00 | Y | 68 | 1 | 1 | 0.89 | N |
16.3 | 1726 |
|
28 | 61 | 0.02 | Y | 102 | 0 | 1 | 0.60 | N |
|
|||||||||||
17.1 | 393 |
|
13 | 22 | 0.16 | Y | — | — | — | — | — |
17.2 | 219 |
|
9 | 20 | 0.47 | Y | — | — | — | — | — |
17.3 | 162 |
|
14 | 5 | 0.26 | Y | — | — | — | — | — |
17.4 | — | — | — | — | — | — | 119 | 0 | 0 | 0.21 | Y |
|
|||||||||||
18.1 | 108 |
|
22 | 15 | 0.25 | Y | 23 | 0 | 1 | 0.93 | N |
|
|||||||||||
19.3 | 35 |
|
2 | 6 | 0.50 | N | 12 | 1 | 3 | 0.55 | Y |
|
|||||||||||
21 | 78 |
|
25 | 30 | 0.40 | Y | 82 | 3 | 0 | 0.93 | N |
FR: French dataset; DE: German dataset; ID: number of the event groups (described in Table
The ranking index value is calculated for each parameter and event group as a ratio of the number of events for that parameter in the group and the maximal number of events for the same parameter considering all the event groups:
For the trend, RI value is calculated to present an unchanged number of events with 0.5, increasing number of events (parameter value up to 1), and decreasing number of events (values down to 0) over a certain period.
The final total ranking value for each EG is calculated as the sum of all parameters ranking values multiplied by respect weights:
These weights could be judged by experts directly or more transparently and consistently by using some methods like the analytical hierarchy process (AHP, [
Ranking indexes weights for each dataset.
Country ranking index | Frequency | Generic and recurring | IRS | Precursor | Trend |
---|---|---|---|---|---|
France (FR) | 0.07 | 0.19 | 0.15 | 0.36 | 0.23 |
Germany (DE) | 0.08 |
|
0.20 | 0.40 | 0.32 |
IRS: events reported to the IAEA/NEA international reporting system.
ID and description for selected event groups with presence in the respected country database.
ID | Description | FR | DE |
---|---|---|---|
01.2 | Loss of offsite power-loss of electrical bus bar | N | Y |
|
|||
02.3 | Reactivity events-transient pressure or temperature changes | Y | N |
|
|||
03 | Loss of heat removal system | Y | Y |
|
|||
04.1 | Leakages from the primary circuit | Y | N |
04.2 | Leakages from primary to secondary circuit | N | Y |
04.3 | Primary coolant leakage outside cont. | Y | Y |
|
|||
05.1 | Essential water intake clogging | Y | N |
05.2 | Failures in the component cooling or service water systems operating trains | Y | Y |
|
|||
07.1 | Scram with influence on reactivity | Y | Y |
|
|||
08.2 | Other instrumentation and control failures affecting the reactor protection system | N | Y |
08.3 | Instrumentation and control failures related to other safety systems | Y | — |
|
|||
09.1 | Complete emergency diesel generator failure | Y | Y |
09.3 | Incipient emergency diesel generator failure | Y | N |
|
|||
10.2 | Control rod events without influence on reactivity | N | Y |
|
|||
12 | Common cause failure events | Y | N |
|
|||
13.2 | Failure of 1 safety system redundancy | — | Y |
13.3 | Failure of a safety system component | — | Y |
13.4 | Failure of safety features redundancies | Y | — |
|
|||
14 | Actuation of safety features | Y | Y |
|
|||
15 | Lubricant | Y | N |
|
|||
16.1 | Circuit misalignment (human/organization) | Y | N |
16.2 | Risk analysis (human/organization) | Y | N |
16.3 | Inadequate maintenance quality (human/organization) | Y | N |
|
|||
17.1 | Failure after release from maintenance or design modification | Y | — |
17.2 | Failure caused by plant design modification | Y | — |
17.3 | Failure caused by using temporary devices | Y | — |
17.4 | Failure related to NPP modification | — | Y |
|
|||
18.1 | Event external to the site | Y | N |
|
|||
19.3 | Other reportable internal fires | N | Y |
|
|||
21 | Design/construction | Y | N |
Y or N designates existence of the particular event group in the FR and DE country database.
Applying (
Statistics for combined uncertainty results in France data with AHP RI weights point results (with triangle distribution, graphically presented in Figure
EG ID | Q1 | Median | AHP point | Mean | Q3 |
---|---|---|---|---|---|
FR02.3 | 11% | 19% |
|
19% | 22% |
FR03 | 11% | 20% |
|
22% | 25% |
FR04.1 | 16% | 28% |
|
28% | 32% |
FR05.1 | 21% | 30% |
|
32% | 39% |
FR05.2 | 5% | 17% |
|
19% | 24% |
FR07.1 | 35% | 42% |
|
44% | 51% |
FR08.3 | 24% | 34% |
|
35% | 42% |
FR09.1 | 14% | 26% |
|
26% | 32% |
FR09.3 | 19% | 30% |
|
29% | 33% |
FR12 | 35% | 43% |
|
44% | 50% |
FR13.4 | 68% | 76% |
|
73% | 83% |
FR15 | 13% | 26% |
|
28% | 33% |
FR16.1 | 30% | 38% |
|
41% | 47% |
FR16.2 | 22% | 37% |
|
40% | 47% |
FR16.3 | 45% | 53% |
|
54% | 61% |
FR17.1 | 28% | 35% |
|
36% | 37% |
FR17.2 | 14% | 20% |
|
20% | 22% |
FR17.3 | 20% | 27% |
|
27% | 29% |
FR18.1 | 17% | 28% |
|
28% | 35% |
FR21 | 20% | 28% |
|
28% | 34% |
EG: event group (description is available in Table
Statistics for combined uncertainty results for German data with AHP RI weights point results (with triangle distribution, graphically presented in Figure
EG ID | Q1 | Median | AHP point | Mean | Q3 |
---|---|---|---|---|---|
DE01.2 | 18% | 27% |
|
27% | 34% |
DE03 | 14% | 20% |
|
20% | 23% |
DE04.2 | 14% | 25% |
|
26% | 31% |
DE04.3 | 9% | 28% |
|
29% | 39% |
DE05.2 | 24% | 25% |
|
24% | 28% |
DE07.1 | 50% | 59% |
|
58% | 64% |
DE08.2 | 16% | 25% |
|
26% | 33% |
DE09.1 | 26% | 36% |
|
34% | 39% |
DE10.2 | 8% | 25% |
|
26% | 34% |
DE13.2 | 42% | 47% |
|
48% | 60% |
DE13.3 | 44% | 49% |
|
54% | 66% |
DE14 | 84% | 89% |
|
85% | 93% |
DE17.4 | 13% | 27% |
|
26% | 33% |
DE19.3 | 16% | 21% |
|
21% | 25% |
EG: event group (description is available in Table
Most generally, uncertainty consists of the aleatory and epistemic part, which includes parameter and model uncertainties [
The sampling-based approach is selected because of its effectiveness and wide use. The basic idea is that analysis results
In this uncertainty study, event groups ranking is quantified as a statistical distribution resulting from the ranking indexes weights defined with statistical distributions instead as point value. This will allow examining event groups ranking sensitivity and defining need for further assessment (i.e., better uncertainty method regarding input statistical distribution, etc.).
A next critical step for the uncertainty analysis is the determination of distributions for the parameters sampling. Without sufficient base for direct determination of specific distributions from the nature of input parameters, they are selected based on the accepted referenced approach and suitability for this stage of ranking uncertainty study. Two different distributions could be found related to the uncertainty where AHP is applied: triangular [
For the triangular distribution, separate simulation was performed for each RI with minimal sampling values set to 0 and maximum set to 1 keeping the original AHP value as the most likely. Figure
Sampling distribution of inputs: “Frequency” RI weight and other RI weights distribution (created using (
In the case of uniform distribution of the input variable after several iterations ±33% change from the original AHP weights is selected for further analysis. Higher value would be in fact sensitivity analysis and smaller value would not influence the results enough.
For this exercise, ranking values were quantified, based on the AHP determined RI weights using Microsoft Excel with add-on Monte Carlo simulation and statistical tool Quantum XL [
Sampling distribution of results for four highest ranked event groups for German data using input from the “Frequency” RI weight triangle distribution sampling: on the left with 10 and on the right with 100 thousand samples.
Finally, after all simulations for all RIs are performed, cumulative ranking values distributions are presented with statistical analysis and box plot [
Based on the approach described in the previous section for each country dataset, one simulation was performed for every RI (i.e., five for France and four for Germany datasets). The following subsections present the most important results and findings.
Results from each simulation are generating ranking values distribution for all selected event groups. This is a vast volume of results and for brevity graphical results are presented only for highest ranked event groups. Most of the results are presented for simulations with triangle distribution. For France datasets, Figure
Final results for the distribution of all ranking values for selected event groups are presented with a box plot in Figure
Box plot of the French event groups ranking cumulative uncertainty results with triangular distribution for all ranking indexes weights inputs. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range). Q1 and Q3 (represented by the boxes) with Mean and Median values are also presented in Table
Box plot of the German event groups ranking uncertainty results with triangular distribution for all ranking indexes weights inputs. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range). Q1 and Q3 (represented by the boxes) with Mean and Median values are also presented in Table
French input data sampling distribution for all RIs weights (i.e., sampled and corrected using (
French data results distribution for eight highest ranked event groups with all RIs weights triangle distribution inputs separate sampling: figures from (a) to (e) for sampling “Frequency,” “G&R,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 1400 to around 2200 and in “G&R” case almost 6000). Respected inputs samplings are presented in Figure
German input data sampling distribution for all RIs weights (i.e., sampled and corrected using (
German data results distribution for four highest ranked event groups with all RIs weights triangle distribution sampling: figures from (a) to (d) for sampling “Frequency,” “IRS,”,” “Precursor” and “Trend,” RI, respectively (max number of samples for peak RI values is from 1400 to over 9000). Respected inputs samplings are presented in Figure
For uniform distributions, simulations with ±33% sampling of RI weights produce a much smaller influence to the final ranking values for all event groups. Because of brevity only final box plot graphs are presented for each country dataset in Figures
Box plot of the French event groups ranking cumulative uncertainty results with uniform ±33% distribution of all ranking indexes weights. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range).
Box plot of the German event groups ranking uncertainty results with uniform ±33% distribution of all ranking indexes weights. Resulting ranking could be analyzed with point values of Mean and Median or uncertainty range with box. Outliers are defined by their distance from the edge of the box: suspected 1.5 and unsuspected 3 IRQ (interquartile range).
Resulting combined ranking values uncertainty for both countries shows numerically and graphically how ranking is affected by RI weights uncertainty. For both countries, an overlap (by looking at values between Q1 and Q3) is smaller for higher ranked event groups. For France data the two highest ranked event groups (13.4 and 16.3) are clearly separated and then after that the following five (7.1, 8.3, 12, 16.1, and 16.2). For Germany data, event group 14 is clearly the highest ranked and the following three event groups (7.1, 13.2, and 13.3) are distinctly higher ranked than the rest of the event groups.
As a form of simulation results confirmation, it is valuable to point out that the difference between Median, Mean, and baseline AHP RI weights ranking results is very small. The difference for France data is between −1% and 4% and between −2% and 5% for German data (Tables
By looking at the resulting ranking values distributions, it is clear that the difference between Q1 and Q3 varies between event groups. The same is true for the spread symmetry (regarding the size and side).
In the case of uniform ±33% distribution of the RI weights, Mean and Median results are the same and equal to the baseline AHP RI weights ranking results. Q1 and Q3 values are very close, with the maximal difference ~3% of absolute value which is less than 10% of relative value even for the low ranked event groups. Compared to the triangle distribution case, outliers and suspected outliers are almost nonexistent. These results do not show any significant change and they clearly do not have value compared to the triangular simulations.
The presented results, from triangular simulations, seem mostly useful to prove the robustness of ranking results obtained by only using AHP point estimate for RI weights. Based on the two countries datasets, it also seems that uncertainty results are helping find better separation between more important event groups and the rest of them. Finally, for two closely ranked event groups, it seems easier to select more important by using uncertainty results (e.g., FR5.1 versus FR8.3 and DE1.2 versus DE4.2).
A comparison between two countries datasets has a lot of limitations due to the various differences (i.e., technology, regulation, safety culture, etc.). This does not prevent making observations about uncertainty results. It seems that uncertainty results are consistent between these two datasets with all mentioned differences regarding the background.
Uncertainty assessment is important for enhancing the model results interpretation and usability. Critical parameters uncertainty influencing the model results can be assessed by a sampling-based approach. AHP determined weights for ranking indexes are selected as the most important epistemic source of uncertainty in the event groups ranking method. Triangular distribution with a most likely value determined by the AHP and limits between 0 and 1 is selected as a sampling input. Dependencies between all ranking indexes were treated with simple corrections and separate simulation for each ranking index. Combined distributions for all event groups ranking values are generated from all simulations. The uncertainty assessment approach applied to two countries datasets proves both consistent and valuable results.
An alternative uniform sampling distribution proved to be without sufficient merit to be considered for use.
Event groups ranking with analytical hierarchy based ranking indexes weights could be improved with sampling-based uncertainty assessment because it confirms most of the results and provides help for better distinguishing closely ranked event groups.
This appendix provides abbreviations, nomenclature, and some additional details about ranking method, input data, and assessment results. This information is not essential to follow the paper; however, it could help understand the complete picture of the presented uncertainty approach and results.
This is a short description of the application of the analytical hierarchy method for the determination of ranking indexes relative importance in the event group ranking assessment (modified from [
The main advantage of AHP is that it can be used to determine the relative importance for any number of parameters requiring only their pairwise comparison. Pairwise comparison (relative importance
Pairwise relative importance could be estimated using a different scale and the most popular is from 1 to 9 (i.e., 1 meaning both RIs are valued as equally important) with respect to reciprocal values. The additional significant advantage of AHP is that the consistency of the comparison can be quantified with a so-called consistency ratio and then iteratively improved if needed.
A further advantage of using AHP is that a multilayered set of parameters can be used and expert judgments from more than one expert can be combined. These features were not applied in the ranking application described in this study. In the future, using more than one expert opinion might be interesting to analyze sensitivity and further improve the ranking method.
Tables
Analytical hierarchy process
German dataset
European Commission
Event group
French dataset
International Atomic Energy Agency
International reporting system
Joint Research Centre-Institute for Energy and Transport
Nuclear Energy Agency
Operating experience
Ranking index.
Reciprocal matrix with all relative RI importance
Relative importance of
Correction (i.e., multiplier) for other RI weights
Subset of events in the FR database comprising generic and recurring events
Event group
Event group identification number
Interquartile range (bottom and top of the box are the 25th and 75th percentile)
Subset of events reported to the IRS (for FR, only the first-time event for each type is reported)
Maximum eigenvalue
Number of events
Data points beyond 3 IQR from the edge of the
Precursor events
1st Quartile
3rd Quartile
Ranking index value
Data points between 1.5 and 3 IRQ from the edge of the
Trend parameter value (i.e., 0.5 for unchanged number of events, up to 1 for increasing trend, and down to 0 for decreasing trend).
Event groups considered in the presented uncertainty assessment
Weight for
Weight for parameter (i.e., relative importance)
Sampled weight for
Vector of local weightings.
The authors declare that there is no conflict of interests regarding the publication of this paper.
Critical data used as the base and input for this work are produced by the work performed for the European Clearinghouse EC JRC-IET, by Dr. Michael Maqua from the