The purpose of this paper is to examine various DoE methods for uncertainty quantification of production forecast during reservoir management. Considering all uncertainties for analysis can be time consuming and expensive. Uncertainty screening using experimental design methods helps reducing number of parameters to manageable sizes. However, adoption of various methods is more often based on experimenter discretions or company practices. This is mostly done with no or little attention been paid to the risks associated with decisions that emanated from that exercise. The consequence is the underperformance of the project when compared with the actual value of the project. This study presents the analysis of the three families of designs used for screening and four DoE methods used for response surface modeling during uncertainty analysis. The screening methods (sensitivity by one factor at-a-time, fractional experiment, and Plackett-Burman design) were critically examined and analyzed using numerical flow simulation. The modeling methods (Box-Behnken, central composite, D-optima, and full factorial) were programmed and analyzed for capabilities to reproduce actual forecast figures. The best method was selected for the case study and recommendations were made as to the best practice in selecting various DoE methods for similar applications.
The increasing necessity for systematic and consistency of methodologies for analyzing uncertainties and their associated effects on investment decisions has become very important in management decisions. This is one of the major reasons why the use of experimental design has been on the increase in the exploration and production (E&P) industry. To be able to harness the benefits associated with design of experiment (DoE), users must apply it wisely, failure of which can result in waste of time and costly decisions. Important areas of application of DoE in petroleum engineering include uncertainty screening and quantification. With DoE one can significantly reduce the number of simulations required to assess uncertainties since considering large number of simulations is not feasible due to limited available time and resources. Previous screening methodologies [
Recent research and applied reservoir characterization and modelling projects have focused on the application of experimental design (ED) and response surface techniques [
The underlining assumption using 2-level factorial designs is that the 3-factor interactions higher order factors are negligible. Consequently, most of the experiments are omitted from the analysis and therefore seldom used for construction of response surfaces. For illustration, if we consider 7 parameters system that require a 27 = 128-run full design. Among the 127 degrees of freedom in the design, only 7 are used to estimate the linear terms and 21 to estimate the 2-factor interactions. The remaining 119 degrees of freedom are associated with three-factor interactions higher order factors that are supposed to be negligible. In practice, if the number of factors ranges from 2 to 15, fractional factorial can be used to screen the significant few by estimating the main effects and interactions. There are various possible fraction designs: 64-, 32-, or 16-run designs. The 16-run design usually is preferred since this is the cheapest design with resolution IV that is available for uncertainties higher than eight.
Two-level Plackett-Burman (PB) designs have been used in the context of classical statistical experiments for screening purposes. However, PB designs cannot be used when higher number of levels is required or where simulator proxy’s development remains the priority of the study. One variable at a time is a screening process involving varying a parameter values between its minimum (−1) and maximum (+1) ranges within the parameter space. The degree of the displacement from the base case of resultant response indicates the extent of influence the parameter has on the measured response. One variable at a time method is a static sensitivity common in classical laboratory experiments. However, its application for screening large number of uncertainties has been reported [
Three-level experimental design methods are high resolution algorithms for response surface construction. Central composite design (CCD) consists of a factorial design with the corners at +1 of the cube, augmented by additional “star” and “centre” points, which allow the estimation of the second-order polynomial equation [
There are various response modeling techniques described in the literature [
Fetel and Caumon [
Whenever experimental design is used, RSM are usually constructed with regression, interpolation, and neural network [
This study critically examined three common families of experimental designs used for screening during uncertainty analysis in simulation for reservoir management. These include (a) sensitivity by one factor at-a-time, (b) fractional experiment, and (c) Placket-Burman design. The selection of “heavy-hitters” was based on the dynamic sensitivity with responses measured at the end-of-simulation. However, the potential of missing out some parameters that are time dependent as pointed out by Amudo et al., 2008 [
There are total of 10 major uncertainties in this study. Table
Experimental range in terms of multipliers on the base case uncertain parameters.
S/N | Parameters | Keywords | Minimum value | Base case | Maximum value |
---|---|---|---|---|---|
1 | Oil viscosity | OVISC | 0.90 | 1 | 1.10 |
2 | Horizontal permeability | PERMX | 0.57 | 1 | 1.29 |
3 | Vertical permeability | PERMZ | 0.50 | 1 | 6.00 |
4 | Porosity | PORO | 0.90 | 1 | 1.10 |
5 | Critical gas saturation | SGCR | 0.50 | 1 | 1.50 |
6 | Critical water saturation | SWCR | 0.53 | 1 | 1.07 |
7 | Fault transmissibility multiplier | MULTFLT | 0.50 | 1 | 2.00 |
8 | Water relative permeability | KRW(SORW) | 0.36 | 1 | 1.25 |
9 | Initial water saturation | SWI | 0.65 | 1 | 0.90 |
10 | Aquifer pore volume | AQUIPV | 0.85 | 1 | 1.35 |
Experiments were performed using history matched model. The response is the cumulative oil production (FOPT) after 15 and 30 years of forecast. Simulation runs vary according to design methods and at the end of simulation, responses are available for statistical analysis. Figure
A typical production profile showing the end of history match (calibrated model) and beginning of production forecast (start date for all experiments).
The three screening designs examined in this study include (a) sensitivity by one factor at-a-time, (b) fractional experiment, and (c) Placket-Burman design. Using the parameters in Table
DoE matrix for fractional factorial method.
Run | A: OVISC | B: PERMX | C: PERMZ | D: PORO | E: SGCR | F: SWCR | G: MULTFLT | H: KRW | J: ISW | K: AQUPV | Response, FOPT (STB) | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
15 years | 30 years | |||||||||||
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
PB design table for 10 parameters.
Run | A: OVISC | B: PERMX | C: PERMZ | D: PORO | E: SGCR | F: SWCR | G: MULTFLT | H: KRW | J: ISW | K: AQUPV | FOPT (15 yrs) | FOPT (30 yrs) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
One variable at a time design table for 10 parameters.
Runs | OVISC | PERMX | PERMZ | PORO | SGCR | SWCR | MULTFLT | KRW | SWI | AQUIPV | FOPT (15 yrs) | FOPT (30 yrs) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
29761616 | 35245672 |
|
|
|
|
|
|
|
|
|
|
|
29116404 | 34489452 |
|
|
|
|
|
|
|
|
|
|
|
29343824 | 34495768 |
|
|
|
|
|
|
|
|
|
|
|
29058182 | 34226452 |
|
|
|
|
|
|
|
|
|
|
|
28566772 | 33322416 |
|
|
|
|
|
|
|
|
|
|
|
29484880 | 34739616 |
|
|
|
|
|
|
|
|
|
|
|
29502398 | 35138476 |
|
|
|
|
|
|
|
|
|
|
|
29016448 | 34228136 |
|
|
|
|
|
|
|
|
|
|
|
28963490 | 34109836 |
|
|
|
|
|
|
|
|
|
|
|
28046878 | 32737122 |
|
|
|
|
|
|
|
|
|
|
|
28873100 | 33538296 |
|
|
|
|
|
|
|
|
|
|
|
28597504 | 33661924 |
|
|
|
|
|
|
|
|
|
|
|
28902034 | 34034268 |
|
|
|
|
|
|
|
|
|
|
|
30595368 | 36193200 |
|
|
|
|
|
|
|
|
|
|
|
27848692 | 32568350 |
|
|
|
|
|
|
|
|
|
|
|
28692348 | 33769084 |
|
|
|
|
|
|
|
|
|
|
|
28460282 | 33083528 |
|
|
|
|
|
|
|
|
|
|
|
28978374 | 33975740 |
|
|
|
|
|
|
|
|
|
|
|
28928676 | 33991664 |
|
|
|
|
|
|
|
|
|
|
|
26487010 | 30639428 |
The “1”, “−1,” and “0” represent absolute high, low, and base case values of the parameters. The contribution of each uncertainty factor was estimated following the significant test for the regression used in the analysis of variance (ANOVA) for all the methods. The two hypotheses used for the test are as follows. The null hypothesis: all treatments are of equal effects: The alternative hypothesis: some treatment is of unequal effects:
To reject the null hypothesis
The effects of the different attributes on the response for Table
Figures
Pareto charts from fractional experiment showing key parameters impacting reserves after (a) 15-year and (b) 30-year forecasts.
Pareto charts from Placket-Burman experiment showing key parameters impacting reserves after (a) 15-year and (b) 30-year forecasts.
Pareto chart from one parameter at-a-time experiment showing key parameters impacting reserves after (a) 15-year forecast and (b) 30-year forecast.
Figure
Figure
Figure
To determine whether there is a linear relationship between the response and various “heavy-hitters,” test for significance of regression was performed by ANOVA. This study utilized the computed Fisher variable (
Table
Analysis of variance for Box-Behnken model associated with fractional factorial screening design.
Source | Sum of squares | DF | Mean square |
|
Prob > |
|
---|---|---|---|---|---|---|
Model |
|
9 |
|
1031.25 | <0.0001 | Significant |
A-PORO |
|
1 |
|
881.42 | <0.0001 | |
B-PERMX |
|
1 |
|
1013.13 | <0.0001 | |
C-PERMZ |
|
1 |
|
420.69 | <0.0001 | |
D-ISW |
|
1 |
|
994.28 | <0.0001 | |
E-OVISC |
|
1 |
|
1806.44 | <0.0001 | |
B2 |
|
1 |
|
104.26 | <0.0001 | |
C2 |
|
1 |
|
61.71 | <0.0001 | |
D2 |
|
1 |
|
3160.24 | <0.0001 | |
E2 |
|
1 |
|
202.82 | <0.0001 | |
Residual |
|
36 |
|
|||
Lack of fit |
|
31 |
|
|||
Pure error |
|
5 | 0 | |||
Cor. total |
|
45 |
In general, 10 different response surface correlations were developed. The regression equations were based on the outcome from the three screening methods earlier discussed. Four methods were considered for response surface modelling: Box-Behnken design (BBD), central composite design (CCD), full factorial design (FFD), and D-optimal design (DOD). Thus, model equations (
Tables
Constants in (
|
Box-Behnken | Central composite | D-optimal | Full factorial |
---|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Constants in (
|
Box-Behnken | Central composite | D-optimal | Full factorial |
---|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Constants in (
|
Box-Behnken method | D-optimal method |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Figures
Comparison of the actual and predicted reserves by (a) FRFRSMs, (b) PBRSMs, and (c) OVAATRSMs.
Parity plots do not reveal much information. However, the plots for the Box-Behnken method generally demonstrate that the method predict the actual value more accurately. On these plots the vast majority of the points are along the
The validation using cross plots only assess model efficiencies within the experimental range of parameters. In order to determine model predictability elsewhere this study performed a “blind test” using parameter values outside the range already defined in Table
Figures
Comparison of the predictions using (a) FRFRSMs, (b) PBRSMs, and (c) OVAATRSMs; (d) the simulated reserves upon reduction of OVISC by 30, 50, and 70%.
The performance indices used are summarized in Table
Performance indices for model evaluation.
Name of measure | Formula |
---|---|
Absolute deviation |
|
|
|
Average absolute deviation |
|
|
|
Root mean square error |
|
|
|
Average absolute percentage relative Error |
|
|
|
Maximum error |
|
|
|
|
|
Standard deviation |
|
Summary of the statistical analysis of RSMs from fractional factorial screening.
Performance index | Box-Behnken | Full factorial* | D-optimal | Central composite |
---|---|---|---|---|
RMSE |
|
|
|
|
AAD |
|
|
|
|
AD |
|
|
|
|
|
|
|
|
|
SD |
|
|
|
|
AAPRE |
|
|
|
|
|
|
|
|
|
Adj. |
|
|
|
|
Pred. |
|
|
|
|
Exp. runs |
|
|
|
|
*2- level full factorial was used.
Summary of the statistical analysis of RSMs from Placket-Burman screening.
Performance index | Box-Behnken | Full factorial | D-optimal | Central composite |
---|---|---|---|---|
RMSE |
|
|
|
|
AAD |
|
|
|
|
AD |
|
|
|
|
|
|
|
|
|
SD |
|
|
|
|
AAPRE |
|
|
|
|
|
|
|
|
|
Adj. |
|
|
|
|
Pred. |
|
|
|
|
Exp. runs |
|
|
|
|
Summary of the statistical analysis of RSMs from OVAAT screening.
Performance index | D-optimal | Box-Behnken |
---|---|---|
RMSE | 345823.8 | 282381.6 |
AAD | 281176.5 | 215245.9 |
AD | 588.2 |
|
|
2.1 | 2.3 |
SD | 1.1 | 0.7 |
AAPRE | 0.9 | 0.6 |
|
97.2 | 98.2 |
Adj. |
96.3 | 97.7 |
Pred. |
95.1 | 96.7 |
Exp. runs | 37.0 | 62.0 |
As shown in Table
Summary table for model ranking.
Ranking measures | Box-Behnken | Response surface methods | Full factorial | |
---|---|---|---|---|
Central composite | D-optimal | |||
Fractional factorial screening | ✓ | ✓ | ✓ | Infeasible |
Placket-Burman screening | ✓ | ✓ | ✓ | ✓ |
OVAAT screening | ✓ | Infeasible | ✓ | Infeasible |
Error analysis | Best | Average | Average | Better |
Blind test | Fail | ✓ | Fail | ✓ |
Correlation |
✓ | ✓ | ✓ | ✓ |
✓ Pass
Infeasible: not practicable due to large number of experimental runs.
If it is desired to develop risk curves from where P90/P50/P10 values for the response can be determined alongside its representative models, the analysis favours the selection of Box-Behnken method. There are however three possible response surfaces depending on the screening method. These are response surface equations (
If it is desired to utilized the proxy equation to simulate and evaluate future development strategies such as the needs for acquiring additional information, undertaking stimulation or any of EOR such as in situ combustion, a more efficient model capable of estimating reservoir performance within acceptable margin of error is desired. Both central composite (CCD) and full factorial methods are adequate.
However, in this analysis, full factorial method performed far better than the CCD. FFD tends to be impractical for large number of uncertainties. We have demonstrated in this study that a 2-level factorial experiment can be used for constructing response surface of quality as comparable as that from 3-level full factorial. However, we highly recommended consideration for resolution of the factorial fractions to be used.
The Monte Carlo technique [
Figure
Impact of different RSMs on uncertainty assessment
The costs and benefits of performing 46 and 62 experiments instead of the desired 29 PB experiments were determined by constructing multiple risk curves with equal or about the same P50 (mean) values for all models.
Figure
Risk curves for Case 1: (a) Box-Behnken equations (
The risk curves obtained by CCD developed from Placket-Burman and fractional factorial are a little at variance from each other due to combination that possesses dissimilar occurrence probability.
In summary, both fractional screening design and PB design tend to give approximate result. Based on the analysis, there is no significant added advantage in performing 46 experiments as required by fractional factorial screening method. PB with fewer experimental runs is therefore desirable.
It was found that 2000 equiprobable realisations iteratively built in Excel were enough to stabilize the resulting forecast distributions. One critical measure used to generate each realisation of the model is that the deterministic reserve value was fairly maintained at simulation base case value throughout the process. Cumulative probability distribution for the forecast reserves is shown in Figure
Reserves distribution for (a) Box-Behnken RSM equation (
Ranking of uncertainty impact on production forecast.
Stochastic model profiles corresponded with P10/P50/P90 for Box-Behnken, central composite, and full factorial PB associated methods.
The major objective of this study was to investigate the implications of various experimental design assumptions usually made while performing uncertainty analysis after reservoir simulation for reservoir management. The three families of screening designs considered are fractional factorial, Placket-Burman, and one variable at-a-time. The four response surface methods considered for the regression modeling are full factorial, central composite, Box-Behnken, and D-optimal. A total of 9 response surface models developed were validated and subjected to statistical error analysis. The models were ranked using some criteria and based on case objectives, selection of appropriate model was made. The risk curves generated were used to provide information about the costs and benefits of conducting additional experiments due to differences in the number of factors emanated from using different screening methods. Also the risk curve was employed to identify stochastic models associated with the P10/P50/P90 realizations.
This study examined three screening designs (Placket-Burman, fractional factorial, and one variable at-a-time) and four response surface methodologies (Box-Behnken, central composite, D-optimal, and full factorial) commonly used for uncertainty analysis. In all screening methods, years of production forecast played important role on associated number of “heavy-hitters.” One variable at-a-time was identified with largest number of parameters and hence considered not ideal because of the attendant large number of simulation runs. Unlike Placket-Burman, a low resolution fractional factorial in addition to main effects, considered significance of factor interactions, required more simulation runs, but can prevent exclusion of some factors with minimal main effect but significant interaction effect. Nevertheless, the analysis performed in this study using Monte Carlo simulation shows that there was no added advantage using fractional factorial in lieu of Placket-Burman for screening.
The “best” model for uncertainty quantification must be selected based on the reservoir management objectives. Box-Behnken method was adequate to determine P10/P50/P90 and associated models. On the other hand, to evaluate future development strategies such as EOR, stimulation, and the needs for acquiring additional information, full factorial and central composite designs are more efficient predictors within acceptable margin of error. A full 2-level factorial or high resolution fractional factorial method was equally adequate for the construction of response surfaces for uncertainty quantification where 3-level full factorial was not feasible.
The authors declare that there is no conflict of interests regarding the publication of this paper.
The authors acknowledged Schlumberger for providing the software at AUST for simulation. The authors also wish to acknowledge Petroleum Technology Development Fund (PTDF) for supporting this research.