A New Robust Dynamic Data Envlopment Analysis Approach for Sustainable Supplier Evaluation

Supplier selection is one of the intricate decisions of managers in modern business era.There are different methods and techniques for supplier selection. Data envelopment analysis (DEA) is a popular decision-making method that can be used for this purpose. In this paper, a new dynamic DEA approach is proposed which is capable of evaluating the suppliers in consecutive periods based on their inputs, outputs, and the relationships between the periods classified as desirable relationships, undesirable relationships, and free relationships with positive and negative natures. To this aim various social, economic, and environmental criteria are taken into account. A new method for constructing an ideal decision-making unit (DMU) is proposed in this paper which differs from the existing ones in the literature according to its capability of considering periods with unit efficiencies which do not necessarily belong to a unique DMU. Furthermore, the new ideal DMU has the required ability to rank the suppliers with the same efficiency ratio. In the concerned problem, the supplier that has unit efficiency in each period is selected to construct an ideal supplier. Since it is possible to have more than one supplier with unit efficiency in each period, the ideal supplier can be made with different scenarios with a given probability. To deal with such uncertain condition, a new robust dynamic DEA model is elaborated based on a scenario-based robust optimization approach. Computational results indicate that the proposed robust optimization approach can evaluate and rank the suppliers with unit efficiencies which could not be ranked previously. Furthermore, the proposed ideal DMU can be appropriately used as a benchmark for other DMUs to adjust the probable improvement plans.


Introduction
Supplier selection is an important strategic decision of managers for the economic and industry.In recent decades scholars and practitioners have paid special attention to this issue.To name a few relevant samples we can refer to Khan et al. [1] who analyzed the suppliers regarding their ability in transferring the technology.However, they merely considered economic criteria in evaluation of four levels of technology transfer among suppliers of auto industries in Pakistan.Nowadays, the decision-maker duty (in supplier selection) has become more and more intricate.This means that they must care specifically about sustainability criteria while supplier selection.Sustainable supplier evaluation and selection concept is resulted from incorporating environmental and social responsibility factors into economic factors when making decisions regarding supply chain management (SCM) [2].In recent years, sustainability factors have played pivotal role in supplier evaluation and selection process [3].Ratan et al. [4] discussed that sustainability principles force companies to select the suppliers which develop products and services, preserve environmental resources and look after manpower and communities.Beamon [5] introduced ethical and social responsibilities criteria as fundamental requirements of sustainable SCM for future decades.
A wide range of multiple criteria models and approaches such as fuzzy, AHP, ANP, TOPSIS and DEA have been proposed to deal with supplier selection issue over the last two decades.Some of the following researchers applied fuzzy, AHP/ANP-based methods to deal with multi-criteria supplier selection problems (e.g.[6][7][8][9][10][11][12]). Some other researchers used TOPSIS based methods to evaluate the suppliers (e.g.[9,13,14]).For instance, using fuzzy inference system, Amindoust et al. [15] proposed a ranking model based on 2 Advances in Operations Research fuzzy inference system for sustainable supplier selection.Wen et al. [3] introduced a model for sustainable supplier evaluation using intuitionistic fuzzy sets' group decisionmaking models.
Another applied method is DEA which is capable to evaluate the suppliers based on weighted ratio of outputs to input.According to Kumar et al. [16], DEA is an applicable and effective tool for supplier selection problem.In the DEA approach, the importance or weights of inputs and outputs are determined through model itself in a pairwise comparison manner without any human's interference [17].Different models of DEA have been developed in the literature to evaluate the potential suppliers.For this purpose, Weber et al. (2000) proposed a hybrid multi-objective programming (MOP)-DEA model.Farzipoor Saen [18] proposed a DEA model for ranking suppliers in the presence of imprecise data, weight restriction, and nondiscretionary factors.Also, Farzipoor Saen [19] suggested a DEA model for supplier selection in the presence of undesirable outputs and imprecise data.Noorizadeh et al. [20] introduced a model for supplier selection in the presence of dual-role factors, nondiscretionary inputs, and weight restrictions.To help managers for ranking and selecting the best suppliers in the presence of undesirable outputs and stochastic data, Azadi and Farzipoor Saen [21] developed a new slack-based measure model.Azadi et al. [22] developed a chance-constrained DEA (CCDEA) model for supplier selection in the presence of stochastic data and nondiscretionary factors.Kumar et al. [16] proposed a unified green DEA (GDEA) model for selecting the best suppliers using a comprehensive environment friendly approach.

Literature Review
Reviewing the relevant literature reveals that traditional models of DEA evaluate efficiency of DMUs merely in one specific past period.Hence, Dynamic DEA (D-DEA) was an appropriate approach which was initially developed by Sengupta [23] and, nowadays, it is used for evaluating DMUs in different periods [24].In the literature related to dynamic DEA, Färe and Grosspkof [25] proposed a dynamic production frontier using an intermediate output which relates annual production processes.Tone and Tsutsui [26] introduced a new dynamic slack-based measure (DSBM) model to assess DMUs in different periods, using carryover variables (links).They introduced four types of carryovers (links) as desirable, undesirable, discretionary, and non-discretionary (fixed) links.Nevertheless, one of the deficiencies of the existing dynamic DEA models is their inability in introducing a strictly efficient DMU with efficiency score of unity in all periods.In fact, strictly efficient DMUs are defined as those that have unit efficiency in all periods.If efficiency score of a DMU in one of the periods has less than unity, it won't be considered as a strictly efficient unit.This deficiency can be seen in the works by Yousefi et al. [27] and Cook et al. [28].To overcome this problem, we propose a new method for constructing the ideal DMU (IDMU) where in each period a different DMU with unit efficiency is selected.In fact, the ideal DMU is constructed by a combination of different strictly efficient DMUs.In other words, in this paper, we extend the dynamic DEA model to evaluate suppliers in different periods based on their inputs, outputs and the relationships between the periods classified as desirable relationships, undesirable relationships and free relationships with positive and negative natures.The proposed model is used to evaluate suppliers based on sustainable supplier criteria such as social, economic and environmental criteria.
In a methodological point of view, the recent hybridapproach studies by Tavana et al., 2017;Shabanpour et al., 2017a;Shabanpour et al., 2017b;Yousefi et al., 2016 andYousefi et al., 2015 have played fundamental roles in creating the main idea and the contributions of this study.Tavana et al., [29] developed a hybrid DEA framework for Sustainable Supplier Evaluation.They combined the goal programming and dynamic DEA model to assess efficiency of suppliers over several periods.Their approach enables the decisionmaker to provide improved solutions for inefficient suppliers based on the extent to which the suppliers achieve future goals (benchmarks).Merging dynamic DEA with ANN models, Shabanpour et al., [30] created a novel framework for assessing and forecasting prospective efficiency of green suppliers.Likewise, another relevant survey was conducted by Shabanpour et al., [31].They applied robust values to define managerial goals (improvement solutions) for evaluating suppliers.Given the fact that the management goals are inherently uncertain as well as based on human interference, they created a new robust double-frontier DEA model to assess and rank sustainable suppliers.Yousefi et al., [32] developed a scenario-based robust DEA technique to deal with sustainable suppliers' evaluation.Yousefi et al., [33], furthermore, combined goal programming and network DEA and proposed a novel DEA framework to evaluate supply chains.Their approach has the potential of predicting the DMUs' efficiencies in prospective periods.Accordingly, the decision-maker can not only evaluate suppliers/supply chains but also rank them based on their efficiency trend in several time periods.Tanskanen et al. [34] consider relationship strategies for levels of suppliers with respect to sustainable criteria.Varoutsa and Scapens [35] evaluate the supply chain's agents inside the organization.Patala et al. [36] consider sustainable criteria such as economic, environmental and social criteria in supplier evaluation.
The ideal DMU, as another assessment method, has been used in different performance evaluation problems over the past decade.Wang et al. [37] created an interval DEA model in which efficiency was calculated within the range of an interval.The upper bound of the interval was set to one and the lower bound was established by introducing a virtual IDMU, whose performance was superior to any DMU.Jahanshahloo et al. [38] developed two ranking methods using positive IDMU.They ranked 20 Iranian bank branches by two ranking methods.Hatami-Marbini et al. [39] provided a four-phase fuzzy DEA framework based upon the theory of displaced ideal.They made two hypothetical DMUs namely the ideal and nadir DMUs as reference points to rank the DMUs.Jahanshahloo et al. [40] proposed an interval DEA model to attain an efficiency interval, including evaluations from both the optimistic and the pessimistic perspectives.In their method, the lower bounds of the DMUs are increased to obtain the maximum value.The derived points from this method were called ideal points.Then, the ideal points are employed to rank DMUs.Wang et al. [41] developed new DEA models for cross-efficiency evaluation by introducing a virtual IDMU and a virtual anti-ideal DMU (ADMU).The purpose of their study was to measure the cross-efficiencies in a neutral and more logical way.
This paper aims to evaluate the suppliers of a home appliance company based on sustainable suppliers' criteria using a new robust dynamic DEA model which is capable to evaluate and rank the suppliers with unit efficiencies which could not be ranked in the previously developed approaches.Furthermore, a new method is developed to construct an ideal DMU.The proposed ideal DMU is made up of a combination of DMUs with unit efficiency in each period.It is possible to have more than one DMU with unit efficiency in each period thus resulting in different combinations or scenarios.Therefore, a two-step scenario-based robust approach is employed to deal with these scenarios all with unit efficiencies.The proposed ideal DMU can be appropriately used as a benchmark for other DMUs to adjust the probable improvement plans.
The main contributions of this paper are summarized as follows: (i) Presenting a new method for constructing an ideal DMU in the dynamic DEA model (since it is unlikely to find a DMU with unit efficiency in all periods, in each period a different DMU with unit efficiency can be considered) (ii) Presenting a two-step robust method to deal with different scenarios for ideal DMU (it is possible to have more than one DMU with unit efficiency in each period and therefore different combinations of DMUs result in different scenarios) (iii) Presenting robust ranks and improvement plans for all efficient and inefficient units Correspondingly, the following research questions are expected to be addressed in this paper: (i) What is the efficiency of suppliers in different periods with respect to sustainable criteria?
(ii) What is the rank of suppliers when more than one supplier with unit efficiency exists?
(iii) How can an ideal DMU be constructed to present better improvement methods when no DMU exist with unit efficiency in all periods?
(iv) What if when multiple DMUs have unit efficiency in a period?
The rest of the paper is organized as follows.In Section 2 the proposed dynamic DEA model is presented; first the deterministic model (Model D) is introduced and then the standard form of the model is written (Model S).Section 3 introduces the proposed two-step robust method which are proposed for the dynamic DEA; For step "a," the proposed robust input/output-oriented dynamic DEA model is defined (Model RI a /RO a ) along with the corresponding linear form (Model LRI a /LRO a ).Afterwards for step "b," the proposed robust input/output-oriented dynamic DEA model is defined (Model RI b /RO b ) along with the corresponding linear form (Model LRI b /LRO b ).In Section 4 the proposed robust dynamic DEA models are investigated on a case study for supplier selection.Finally, in Section 5 conclusions are brought along with future research directions.

Problem Statement and Formulation
The purpose of this paper is to evaluate suppliers of a company in consecutive periods based on sustainable criteria within three categories: (1) social, (2) economic, and (3) environmental.In each period, there are a number of inputs and outputs for each supplier.Also some materials may be transferred from one period to the next period(s) such as backorders and uncashed checks.These make relationships between periods which some of them are desirable and some are undesirable.In the context of DEA, each supplier is considered as a DMU.Since the suppliers are evaluated in multiple periods, dynamic version of DEA is appropriate.Dynamic DEA aims at evaluating  DMUs during  periods with respect to relationships between periods.For every DMU, in each period, n inputs and  outputs are considered.Also, three types of relationships such as desirable, undesirable and free are considered which ensure the link between periods.Relationships with desirable and undesirable nature need to be maximized and minimized, respectively.Free relationships are those that lack any essence and their nature cannot be recognized some of them have positive nature and some have negative nature.Figure 1 graphically illustrates the proposed dynamic DEA.
In the dynamic DEA, usually a strictly efficient unit called ideal DMU is specified to be considered as a benchmark so that inefficient DMUs try to reach it.In fact, the ideal DMU is the one that in all periods has unit efficiency and it is a strictly efficient unit.In most cases such a DMU which is efficient in all periods does not exist and according to the existing definition no ideal DMU can be recognized.To overcome this problem, we introduce a new method for constructing the ideal DMU where in each period a different DMU with unit efficiency in that period is selected.Obviously, the proposed ideal DMU is virtual and does not exist in reality.To clarify, consider 3 DMUs in 5 periods (Figure 2).According to the existing method for constructing the ideal DMU, no DMU is selected as an ideal DMU whereas according to the proposed method in this paper, the ideal DMU is composed of DMU 1, DMU 2, DMU 3, DMU 2, and DMU 1, respectively.
In the dynamic DEA, the efficient frontier is constructed in a pairwise comparison between units where units with maximum ratio of outputs to inputs are selected to construct the efficient frontier as presented by dotted line in Figure 3.The improvement method is presented for inefficient units to push them towards this efficient frontier.This issue can be mentioned as another shortcoming of the existing dynamic DEA models where the benchmark(s) as well as improvement plans are merely introduced for inefficient units.Actually, those models do not present improvement methods for  benchmarks themselves.The proposed ideal DMU, shown by an asterisk in Figure 3, can be introduced as a benchmark for both inefficient units and efficient units.The boundary which is presented by the solid line in Figure 3 is the frontier which has been constructed by the ideal DMU.As mentioned earlier, the proposed ideal DMU consists of periods with unit efficiencies which do not necessarily belong to a DMU.It is possible to have more than one DMU with unit efficiency in a specific period.Therefore, different scenarios may exist for the ideal DMU.To deal with these scenarios, we employ a scenario-based robust optimization approach and propose a robust dynamic DEA model to present improvement plans for all DMUs.Even when all DMUs obtain similar efficiency, the proposed model can rank those units by considering an absolutely efficient unit called ideal DMU.Actually, the ideal DMU can be used as a unique benchmark based on which improvement methods can be presented for other DMUs.As a matter of fact, initially DMUs are evaluated based on a dynamic DEA approach and then different scenarios are constructed for the ideal DMU (ideal supplier) through a combination of efficient DMUs in each period.
Altogether, the proposed ideal DMU addresses the following concerns about the existing DEA models: (i) Presenting the improvement methods for efficient units (in existing models the improvement methods can only be presented for inefficient units) (ii) Considering requirements and opinions of experts in presenting the improvement methods (these opinions are considered in inputs and outputs values of the ideal DMU).
(iii) Modifying the benchmarks in case these units are not acceptable from DM's perspective.

Acronyms for Models
: Undesirable relationship for the jth DMU in period p. (i=1,..,  ), (j=1,. .., m), (p=1,. .., P).  Note that the right hand sides of the above constraints, i.e.,   ,   , and     u  , are positive values.The left hand side of the mentioned constraints, i.e.,   ,   ,    ,    , and    , is connected together through  p j .The continuity of the flow representing the relationship between pth and (p+ )th periods is ensured through (1), where ∝ is a general index which can be d, u, or  representing desirable, undesirable, and free relationships.These constraints are important in the proposed dynamic DEA model, since they connect period  to period p+ and ensure having a series of time periods.
. .e Standard Mathematical Model for the Deterministic Dynamic DEA (Model S).After writing the standard form for the constraints of model (D), the final standard model (S) whose constraints have equal sign is obtained as follows: In model (S) like model (D), two alternative cases are considered to calculate the efficiency of the under investigated DMU.One is input-oriented (S-1-IN) and the other is outputoriented (S-1-OUT) which are described in the followings.As a matter of fact, the choice of these objective functions depends on the DEA approach, i.e., whether it is inputoriented or output-oriented, which is used for presenting improvement plans.
Objective function (S-1-IN) represents the total efficiency of the input-oriented model.This objective function is based on the nonredial input-oriented model which considers undesirable relationships, i.e.,    and s − ip , along with surplus of inputs, i.e.,  −  , which should be simultaneously minimized.If all these variables become zero, the efficiency of the considered DMU in period p is one.Obviously, a DMU with overall efficiency equal to one is the one which has unit efficiency in all periods.This objective function calculates the weighted mean of the efficiencies in all periods whose value is between 0 and 1, i.e., (0 ≤ Φ *  ≤ 1), (0 ≤  *  ≤ 1).The optimal value for the efficiency of period p in input-oriented model is according to Objective function (S-1-OUT) represents the total efficiency of the output-oriented model.The optimal value for the efficiency of period  in output-oriented model is according to The denominator of the objective function deals with the slack of outputs,  +  , free relationships with positive nature, s + ip , and desirable relationships,    .If these values become zero, the denominator becomes one and therefore the efficiency of the considered DMU in period  is one.If these slacks get values more than one, the denominator becomes more than one.Therefore, the efficiency of the considered DMU in period  is less than one.Consequently, the total efficiency in the objective function gets a value between zero and one, i.e., (0 ≤ Ψ *  ≤ 1), (0 ≤  *  ≤ 1).In Constraint (S-2), s − ip represents the slack for the th input in period .The left hand side of this constraint is the inputs of the underinvestigated DMU in period .If the value of s − ip be zero, it means that the supplier does not have excess consumption for that input in period .
In constraint (S-3), s + ip represents the surplus for the th output in period .In the rest of the constraints,    ,    ,    , respectively, represent the slack of the desirable relationship, surplus of the undesirable relationship, and the deviation of the free relationship.Note that the auxiliary variables used to standardize constraints have negative natures.For inputs it means excess consumption, for outputs it means shortage in production, for desirable relationships it means shortage in this relationship, for undesirable relationships it means excess in this relationship, and for free relationships it means deviation in this relationship.Constraint (S-6) contains a free of sign variable.The deviation of the free relationship can either be stated as slack or surplus.Therefore, to deal with the free of sign variable s f ip , two positive variables, s f− ip and s f+ ip , are defined and the following constraints are considered: Consequently, the following constraints are substituted for Constraint (S-6):

Robust Dynamic DEA Model
As mentioned previously, one of the deficiencies of the existing dynamic DEA is the lack of a strictly efficient unit as a unit that can be introduced as a benchmark.To overcome this deficiency, we introduce a new method for constructing the ideal DMU which is constructed by making use of the results obtained from the dynamic DEA.The proposed ideal DMU is made up of different periods each of which contains DMUs with unit efficiency.As a matter of fact, in each period, DMUs are evaluated and DMU(s) with unit efficiency are selected to construct the ideal DMU.Since more than one DMU with unit efficiency may exist in each period, different combinations of DMUs may be generated for the ideal DMU.Each of these combinations is called a "scenario".More than one DMU with unit efficiency in each period leads to different combinations or scenarios for the ideal DMU whose probabilities of occurrence are considered the same in this paper.Different scenarios for the ideal DMU result in different improvement plans.By taking the advantages of the scenario-based robust optimization method and applying it for the studying dynamic DEA, we evaluate and rank the suppliers with respect to these scenarios for the ideal DMU.This process is done in two steps.The first step (step a) formulates the robust optimization model where one of the scenarios is under investigation.The second step (step b) formulates the robust optimization model where other DMUs along with the selected scenario unit are under investigation.
The procedure for the proposed supplier evaluation and rank model is summarized in the following procedure.

Procedure: Supplier Evaluation and Rank through the Proposed Robust Dynamic DEA Begin
(1) Determine inputs, outputs, desirable, and undesirable relationships for suppliers in each period.
(2) Consider each supplier as a DMU and employ the dynamic (input/ouput-oriented) DEA model to determine the efficiency values of each supplier in each period.
(3) If there is a DMU with unit efficiency values in all periods consider it as a strictly efficient unit.
(4) Otherwise, build a virtual ideal DMU whose periods belong to DMUs with unit efficiency thus leading to different scenarios for the ideal DMU.
(5) In step "a," evaluate and rank scenarios (with equal probability and based on the punishment and encouragement values considered for each scenario) using the proposed linear (input/output-oriented) robust dynamic DEA model (LRI a /LRO a ).The best scenario is considered as a unique benchmark for presenting improvement plans.
(6) In step "b," consider the selected scenario from step "a" along with other suppliers (resulting in m+ number of DMUs) and evaluate the suppliers through models (LRI b /LRO b ).

End.
Notations Used in the Proposed Robust Method ^: The probability of occurrence for scenario s    : The unit cost for th input of sth scenario in period p.    : The unit cost for th undesirable relationship of sth scenario in period p.

𝑣 𝑠
: The unit cost for th free relationship of sth scenario in period p.
ℎ   : The unit revenue for th output of sth scenario in period p.
: The unit revenue for th desirable relationship of sth scenario in period p.
: The unit revenue for ith free relationship with positive nature of sth scenario in period p.
. .Step a: A Scenario Unit Is under Investigation.In this section the efficiency of scenarios are investigated through models RI a or RO a , depending on the decision-maker's approach which could be input-oriented or output-oriented.
At each time one of the scenarios is under investigation and the efficiencies of scenarios for the ideal DMU are calculated and they are ranked.The high ranked scenario is selected based on which the improvement plan is presented.As a matter of fact, the best scenario has more distance from other DMUs (see Figure 3) and the improvement plan for other DMUs is presented in the worst case.Therefore, the proposed improvement plan is robust against different scenarios which could be considered for the ideal DMU.

. . . Robust Optimization for Input Oriented DEA Model
Step a (RI a ) (RI a -2) The objective function (RI a -1) consists of three terms.The first term calculates the average efficiency of scenarios.The second term calculates the deviation of efficiency of scenarios from the average value.The third term calculates the total profit or loss resulted from scenarios.In fact, in each scenario, outputs, desirable relationships and free relationships with positive natures, yield return.Whilst inputs, undesirable relationships, and free relationships with negative natures result in cost.Note that in this term, the values of returns are subtracted from the values of costs.Therefore, if this term is negative it means that the considering scenario is profitable; otherwise it makes losses.
It is worth noting that since each scenario is a combination of DMUs with unit efficiency selected from different periods, the efficiency of each scenario is one.Therefore, the average efficiency of all scenarios is one and consequently the standard deviation is zero.

( ) Linear Robust Input-Oriented Model
Step a (LRI a ).To deal with the absolute function and make the model linear, two positive variables  +  and  −  are defined.
(LRI a -1) Other constraints of model (RI a ) hold.

. . . Robust Optimization for Output-Oriented DEA Model
Step a (RO a ).The robust optimization for the outputoriented model differs with the input oriented model in the objective function and also in the average and the linearized constraints.Other constraints are the same for both.
Other constraints of model (RI a ) hold.
Like the input-oriented model, the objective function (RO a -1) consists of three terms.The first term minimizes the average efficiency of scenarios with weight importance of .The second term minimizes the standard deviation of efficiencies with weight importance of 1-.Finally, the third term calculates the total profit or loss resulted from scenarios.

( ) Linear
Other constraints of model (RI a ) hold.

. . Step b: A DMU from Other DMUs Is under Investigation.
In the previous section the under investigation DMU was one of the scenarios and we evaluated scenarios and selected the suitable one.In fact, model (RI a /RO a ) calculates benefit or loss resulted from each scenario and we can select the best one accordingly.In this section, other DMUs are evaluated along with the selected scenario and therefore the number of DMUs increases by one (i.e., m+1).Actually the best scenario is considered in model (RI b /RO b ).

. . . Robust Optimization for Output-Oriented DEA Model
Step b (RO b ) Other min Other constraints of model (RI b ) hold.

Case Study Implementation and Performance Evaluation
In this section, the performance of the proposed robust dynamic DEA approach is investigated via a case study taken from NANIWA (http://www.naniwa.ir)appliances production plant.The mentioned firm aims at evaluating its 35 suppliers in 4 time periods based on environmental, social, and economic criteria.For the purpose of evaluation, for each supplier as a DMU, 4 inputs, 3 desirable and undesirable relationships, and 4 outputs are considered.

Inputs
(1) Price offered by suppliers (as an economic criterion): It is the money (in $1000) that is paid to suppliers for each unit of products.(2) Cost of recoverable packages (as an environmental criterion): This input is the cost (in $100) that is imposed to the company by the supplier for using recoverable packages.It is worth noting that it is mandatory for Naniwa Company to ship their products in suitable pallets with recoverable packages to prevent damaging the environment.

(3) Final transportation cost (as an economic criterion):
This is the final cost (in $100) which is imposed by the supplier to the firm for transportation of shipped pallets.The farthest the supplier is and the less accessible the paths are, the more cost imposed.This cost is the main concern of the decision-makers in the company.(4) Work safety and labor health (as a social criterion): This is the cost that each supplier pays for dangers that exist and accidents which happen in the workplace.The less damage and casualties are, the less cost is paid by each supplier.

Relationships
(1) Used technology in the production line of suppliers (as a desirable relationship and an economic criterion): The used technology is scored by expert's opinion using 9-point Likert spectrum according to Table 1.Likert scale is used to convert qualitative factors into quantitative values [42].There are variety of scales which can be rated as 1 to 5, 1 to 7, and 1 to 9. Valuation of factors in this scale is performed according to concept of each factor [43].
(2) Green research and development (as a free relationship and an environmental criterion): The corresponding budget per year (in $100).Green research and development is a dual-role factor which plays the role of both undesirable and desirable factors.Green research and development can be considered as an undesirable criterion since it is cost of performing green researches.On the other hand, green research and development is a desirable criterion, because it implies innovations in manufacturing green products and services and environmental efficiency enhancement.
(3) Shortages (as an undesirable relationship and an economic criterion): The amount of shipments (in terms of pallets) that have not delivered in the past period and should be met in the next period by the supplier.

Outputs
(1) Obtaining ISO certificates and observance of standards: this kind of output is scored by expert's opinion using 9-point spectrum with respect to qualitative and environmental criteria and standards and also workplace standards.Table 2 shows the 9-point spectrum.
(2) Quality (as an economic criterion): Quality of products is evaluated by Likert scale.In Table 3, using a 9-point Likert scale, valuation of quality of parts supplied by suppliers is presented.
(3) Supply capacity (g 2): maximum amount of materials that supplier can send to Naniwa Co.
(4) Efficiency of energy consumption (as an environmental criterion): Efficiency of the energy consumption is the third output which is an environmental criterion.
To determine an appropriate scale for evaluating Advances in Operations Research 13 Using 100-point scale.Table 4 shows the energy consumption of the suppliers.
Figure 4 illustrates the proposed dynamic DEA approach on the case study to evaluate suppliers in periods 2011 to 2014.The inputs, outputs, and relationships between periods is illustrated in this figure.
The inputs, relationships, and outputs data for 35 suppliers in 4 time periods are presented in Table 5 in the Appendix.
In Table 6, the DMUs (i.e., 35 suppliers) are evaluated by making use of the existing input-oriented dynamic DEA model on data shown in the Table 5 presented in the Appendix.The DEA model is selected according to the DM's approach for which he/she want to present their proposed improvement plans.Table 6 shows the efficiency values for 35 suppliers in 4 periods from 2011 to 2014.
The results of Table 6 show that none of the suppliers have unit efficiency values in all 4 periods.Therefore, none of them are strictly efficient.To construct a strictly efficient unit as a benchmark for other units, in each period the DMU with unit efficiency is selected as a candidate.As a result, 8 scenarios are generated for ideal DMU which are different combinations of DMUs (suppliers) with unit efficiency in each period.These resulting scenarios are presented in Table 7 with probability of 0.125 for each one.If these scenarios are taken into consideration and evaluated along with other 35 suppliers, the efficiency of scenarios becomes one because they consist of DMUs with unit efficiencies in all periods.Therefore, we can claim that the existing dynamic DEA model cannot evaluate and rank the scenarios.In the proposed model (RI a ), since all scenarios have unit efficiency, the average efficiency is also one and the deviation of efficiencies is zero.Therefore, we set  = 1, 1 −  = 0. To deal with this difficulty, a punishment (Cost) and encouragement (Income) value is considered for the inputs, outputs, and relationships of each scenario.
The costs and incomes associated with inputs, outputs, and relationships are presented in Table 8 with the following notations: : Unit cost for input j for all scenarios (j: price, packaging, transportation cost, and workforce health cost): this is a penalty that is imposed to a scenario by the decisionmaker for each unit of inputs.
ℎ   : Unit income for output j for all scenarios (j: quality, production capacity, the number of acquired ISO and standards, and efficiency of the energy consumption): this is the encouragement that is considered for a scenario for each unit of outputs.
1 : Unit cost of shortages in shipped pallets for all scenarios (undesired relationships): this is the penalty that is decided by the decision-maker for each unit of this undesired relationship.In this paper for backlogs or goods shipped with a delay a penalty is considered for the supplier in that period.
1 : Unit cost of green research and development for all scenarios (free relationships): if the free relationship be an undesired relationship a penalty value will be assigned for each unit of this relationship.In our case study, if the green R&D is considered as an undesired relationship for the under investigation DMU, a penalty is considered for that supplier for each unit of this relationship.
1 : Unit income of green research and development for all scenarios (free relationships): if the free relationship is a desired relationship an encouragement value will be assigned for each unit of this relationship.In our case study, if the green R&D is considered as a desired relationship for the under investigation DMU, an encouragement value is considered for that supplier for each unit of this relationship.
1 : Unit income of technological power for all scenarios (desirable relationships).
As mentioned earlier, the existing dynamic DEA model cannot rank different scenarios for the ideal supplier.In this paper, first we evaluate and rank these scenarios through the proposed input-oriented robust dynamic DEA model (RI a ).In Table 9, the punishment and encouragement for each scenario resulting from model (RI a ) is presented.As presented in Table 9, the second scenario is the best scenario since it has the maximum value of benefit.Generally, if the objective function of model (RI a ) is positive, the scenario will generate loss while it is negative the scenario will make benefits.If all scenarios generate loss, the scenario with minimum loss will be selected.The values of landa obtained from model (RI a ) admit that the scenarios can be considered as a benchmark.In the next step, the second scenario which is the best scenario is considered as the ideal DMU and it is evaluated along with other DMUs (suppliers) through model (RI b ).The ideal DMU which is actually the ideal scenario is considered as a unique benchmark which owns both the property of a real supplier in that it consists of some periods and the property of a virtual supplier in that it is strictly efficient and consists of periods with unit efficiency.Therefore, the proposed ideal DMU can present improvement plans for all suppliers and can rank the suppliers with same efficiency.The results obtained from model (RI b ) are reported in Table 10.As presented in Table 10, the ideal DMU gives the ability of ranking the suppliers that had the same rank in Table 6.From Table 6 we can see that suppliers 2 and 13, 19 and 3, 8 and 31, 6 and 4, two by two have the same rank because the total efficiency of these DMUs are equal.Using the proposed model (RI b ) for evaluating the suppliers and constructing an ideal DMU help us to rank the suppliers.This ranking considers the standard deviation of efficiencies in calculating the efficiency of each supplier.Furthermore, the proposed ideal DMU gives us the opportunity of presenting improvement methods for all DMUs.As a matter of fact, the improvement methods are presented based on their distance from the ideal DMU (supplier).The high ranked scenario resulting from model (RI a ), which is the second scenario, has maximum distance from other suppliers and it is the worst case that can be happen for an ideal scenario based on which improvement plans can be presented.Through model (RI b ) other suppliers can be ranked based on their distance from this worst case ideal scenario.Therefore, we could claim that the resulting ranks and improvement plans cannot get worse when each of the other scenarios, which could be happen with a given probability, is considered thus resulting in a robust solution.The proposed robust method is implemented in two steps and it is is able to obtain a solution which is immunized against different scenarios exist in constructing the ideal DMU.In the first step, at each time one of the scenarios is under investigation.The high ranked scenario which has more distance from other DMUs is selected as a unique benchmark based on which the improvement plan is presented in the worst case.In fact, in the second step other DMUs along with the worst case scenario are under investigation and improvement plans are proposed based on their distance from the ideal DMU.Therefore, the proposed method can rank the suppliers with the same efficiency and propose an improvement plan which is robust against different scenarios which could be considered for the ideal DMU.

Conclusions and Future Research Directions
For future study on this work, apart from the uncertainty in different scenarios for ideal DMU, one can consider the input, output, and relationship values as uncertain parameters which can vary in a convex uncertainty set.Then a hybrid robust optimization approach, i.e., a hybridization of the scenario-based and robust counterpart methods, can be developed to deal with these uncertainties.

𝑓𝑖𝑗𝑝:- 7 )
Free relationship for the jth DMU in period p. (i=1,..,  ), (j=1,. .., m), (p=1,. .., P).    : Benchmark for the jth inefficient DMU in period p. o: Index for the under investigation DUM  −  : Weight of the th input  +  : Weight of the th output   : Weight of the pth period  *  : Efficiency of the under investigated DMU in period p (input-oriented model) Φ * 0 : Total efficiency of the under investigated DMU (input-oriented model)  *  : Efficiency of the under investigated DMU in period p (output-oriented model) Ψ *  : Total efficiency of the under investigated DMU (output-oriented model) . .e Proposed Mathematical Model for the Deterministic Dynamic DEA (Model D) 1, . . .,   ;  = 1, . . ., ) 1, . . .,   ;  = 1, . . ., ) (D-5) z f iop : free of sign ( = 1, . . .,   ;  = 1, . . ., ) j = 1, . . ., m : p = 1, . . ., P) (DObjective function (D-1-IN) and (D-1-OUT) maximizes the efficiency of the under investigation DMU.(D-1-IN) is the objective function of the input-oriented model and (D-1-OUT) is the objective function of the output-oriented model.At each time either objective function (D-1-IN) or (D-1-OUT) is considered.Constraint set (D-2) ensures that the th input of the under investigation DMU in period  be greater than or equal to weighted sum of input  in period  for all DMUs.Constraint set (D-3) ensures that the th output of the under investigation DMU in period  be less than or equal to weighted sum of output  in period  for all DMUs.Constraint set (D-4) indicates that the value of the th desirable relationship of the under investigation DMU in period  be less than or equal to weighted sum of the desirable relationship  in period  for all DMUs.Constraint set (D-5) indicates that the value of the th undesirable relationship of the under investigation DMU in period  be greater than or equal to weighted sum of the undesirable relationship  in period  for all DMUs.Constraint set (D-6) defines the free relationship variables which are free of sign.Constraint set (D-7) defines the weight variables of improvement plans.
constraints of model (RI b ) hold.( ) Linear Robust Output-Oriented Model Step b (LRO b ).By considering two positive variables  +  and  −  and substitut-Advances in Operations Research ing them with the absolute function, the model is linearized.

Figure 4 :
Figure 4: Demonstration of the proposed dynamic DEA on the considered case study.

z ijp d ,z ijp u ,z ijp f Period p Y ijp Period p+1 Y ijp+1 X ijp x ijp+1
,  : Total number of desirable, undesirable, and free relationships, respectively.  : th input of the jth DMU in period p (i= 1, 2, 3, . . . . ..., n) D: Deterministic model S: Standard model RI a : Robust Input-oriented model step a RI b : Robust Input-oriented model step b RO a : Robust Output-oriented model step a RO b : Robust Output-oriented model step b LRI a : Linear Robust Input-oriented model step a LRI b : Linear Robust Input-oriented model step b LRO a : Linear Robust Output-oriented model step a LRO b : Linear Robust Output-oriented model step b Before describing the models, the used notations can be described as follows.Notations m: Index of DMUs n: Index of inputs s: Index of outputs p: Index of time periods   ,  : th output of the jth DMU in period p (i= 1, 2, 3, . . . . .., s) Robust Output-Oriented Model Step a (LRO a ).By considering two positive variables  +  and  −  and substituting it with the absolute function, the model is linearized.

Table 4 :
Spectrum for efficiency of the energy consumption.
Existing dynamic DEA models just present improvement plans and do not have the ability of ranking DMUs.The proposed model apart from having the ability of ranking DMUs can present improvement plans.In most DEA models, when different time periods are considered in evaluating DMUs, no DMUs can be introduced as a strictly efficient unit which is efficient in all periods.Our contribution is introducing a new method for constructing the ideal DMU such that apart from the previous definition for ideal DMU which considers one of the DMUs which has unit efficiency in all periods as an ideal DMU, a virtual DMU is considered as an ideal DMU.The new proposed ideal DMU has the ability of presenting an improvement method for all suppliers and also ranking the suppliers with the same efficiency.The proposed ideal DMU introduces a strictly efficient unit by building a combination of DMUs with unit efficiency in each period.It is possible that multiple DMUs have unit efficiency in a period.Therefore, different combinations of these units lead to different scenarios for the ideal DMU each of which can happen with a specific probability.The existing dynamic DEA model cannot evaluate and rank the scenarios which all have unit efficiency.To deal with these scenarios, a scenario-based robust optimization model for the dynamic DEA is developed which is capable of ranking the scenarios based on a punishment and encouragement value assigned to each scenario.

Table 5 :
The inputs, relationships and outputs data for 35 suppliers in 4 time periods.

Table 8 :
Costs and incomes associated with inputs, outputs and relationships.

Table 9 :
Evaluating different scenarios for the Ideal DMU (Model RI a ).

Table 10 :
Evaluating suppliers along with the ideal supplier (Model RI b ).