Introduction

Decision-making is a frequent and essential activity for any organization [1]. We frequently understand that, independent of organization size, importance or economic sector, decision-making is a day-to-day task necessary for operational, tactical and strategic purposes. According to [2], this task is crucial because it involves fundamental decisions that shape the course of a firm. Developing tools such as explicit models, whether formalized or not, can help decision makers answer the questions that arise during this process [3]. These tools help to illustrate possible solutions and generate a recommendation to increase the coherence between the development of a process and the goals of the stakeholders [4], [5]. The word recommendation is used to make both the analyst and the person or group of people who make the decision aware that they are free to follow the suggestions or not [5]. Collaborative decision-making aims to reach a high level of consensus to achieve unity and ownership, thus avoiding a simple vote [3]. Hence, to improve decision-making transparency, auditability, and analytic rigor, it is necessary to use a method that supports this process [6], [7]. One of the options cited by several researchers is Multi-Criteria Decision Analysis (MCDA) [4], [8], [9], [10], [11], [12], [13], which was developed in the 1960s [14]. Since then, this technique has been the subject of numerous research projects in fields such as medicine, engineering, finance, and economics [15], [16], [17], [18], [19], [20].

One aspect of MCDA that some authors have explored is the handling of uncertainty in data and parameter values [13], [21], [22], [23], [24], [25], [26], [27], [28], [29]. Potential sources of uncertainty come from the definitions of alternative performance and the criteria weights, which is one of the most challenging steps in the MCDA process [21], [25]. Some methods have been developed to address these uncertainties [22], [23], [26], [27], [28], [30], [31]. While most of the methodologies employ statistical tools to evaluate uncertainty in some of the input data, they do not account for the thresholds included in most MCDA concerning all the uncertainties of the input data. Additionally, few methodologies can assess the quality criteria (performance values and weights), which are considered key aspects of decision-making problems [32].

Although many MCDA methods, including uncertainties, have arisen during the past few years for a wide range of applications [33], [34], [35], [36], [37], [38], [39], [40], [41], it seems necessary to determine which type of problems require the inclusion of uncertainties as input data when assessing an MCDA. Additionally, MCDA problem characterization, which could define each factor if deserving of uncertainty inclusion, is lacking, as well as sufficiently flexible tools to evaluate these factors.

This paper aims to identify which type of MCDA problems should include uncertainties in a decision-making process.

Methods

Case Studies

For this study, we used two real-world decision-making case studies and a hypothetical case with differing numbers of alternatives, criteria, and surveys. The first case study is the decision-making process for Stormwater Harvesting (SWH) at the Pontificia Universidad Javeriana Bogotá (PUJ) [39]. The second case study is the selection of the target community of the PUJ Social Program PROSOFI (from its Spanish acronym) [1]. The third case is an example from the book “Decisiones con múltiples objetivos e incertidumbres” [42] related to the selection of alternatives for sustainable development management of a small catchment.

Case Study 1: PUJ Campus Stormwater Harvesting

For this case study, six scenarios for SWH were proposed on the campus of PUJ. The authors suggested eight criteria to evaluate each scenario. The University’s Physical Resources Division (PRD) defined the criteria based on a set proposed by [33]. Within the criteria set, technical aspects, such as hydraulic and quality performances, management aspects, such as compatibility with the University’s Master Plan, and financial aspects, such as net present value (NPV) of the investment, were considered. We used a survey applied to four leaders from the PRD and five experts on water management to define the criteria weights (table 1). For more details of this study, see [39].

Galarza-Molina et al. [39] built the scenarios or possible alternatives based on a division of the PUJ campus in twelve basins. They proposed the use of SUDS (Sustainable Urban Drainage Systems) [43] for the collection and treatment of the stormwater in the basins, including bioretention gardens, permeable paving, and constructed wetlands.

The input scenarios emerged from an ideal scenario where all the water harvested in the campus would be used (maximum supply volume) with drinking water quality (maximum water quality). The other scenarios were created by decreasing the water quality (fewer possible uses) and the supply (fewer basins included), obtaining high, medium and low water demands and high, medium and low water supplies. The above considerations allowed sizing of the SUDS considered and the pumping and treatment systems. Table 2 shows the description of the scenarios.

As a result, [39] constructed table 3, with eight criteria and six scenarios (8 × 6 matrix) with minimum, average and maximum values per criterion and scenario.

Case Study 2: Selection of PROSOFI Target Community

For this case study, [1] defined the 21 alternatives (19 Bogotá urban districts plus two municipalities near Bogotá) along with criteria and weighting factors based on a participatory project of the university. [1] developed a survey applied to 143 faculty members to determine the criteria and weighting factors. The survey asked which essential aspects should be considered for the selection of the PROSOFI target community and which of these aspects were more relevant than others. [1] analyzed the results from the survey (111 persons, including directors, faculty members, and administrative staff) to find inconsistencies, using a script that compared the answers of each person. Table 4 shows the selection criteria for each key aspect [1].

Galarza Molina et al. [1] constructed the matrix of alternatives using the defined criteria and the evaluation of alternatives. They obtained information from the 2005 Bogotá census [44], Bogotá’s Chamber of Commerce [45], the Undersecretary’s office for planning Economic and Social Affairs [46] and the experience of experts. As a result, this case study has a 17 × 21 matrix (see table 5, one part of the decision matrix).

Case Study 3: Book Example

With the aim of comparing different types of decision problems, we used an example from the book Decisiones con múltiples objetivos e incertidumbres by [42]. The decision-making process here selects the best alternative for sustainable development management in a 250-ha catchment. The authors proposed seven criteria to evaluate the alternatives (table 6).

Table 7 shows the probability distributions and weights of the criteria and the matrix of alternatives (7 × 7 matrix). In this table are the minimum, average and maximum values per criterion and scenario. [42] determined the average values of the evaluation matrix and the criteria weights. For the minimum and maximum values, we defined them with variations of 1,7% or 7,8%, and the probability distributions of the criteria were chosen arbitrarily.

Summary of the Case Studies

Table 8 illustrates the differences among the three case studies.

Decision-Making Tools

ELECTRE II is part of the family of methods collectively known as ELECTRE (ELimination Et Choix Traduisant la REalité – ELimination and Choice Expressing REality). Proposed in the late 1960s by [47], it is considered the first method dealing with problem ranking [48]. We chose this method because (i) it enables a dialogue between various stakeholders involved in the decision-making process but provides input/voice differences between them by means of a criteria weighting procedure [33]; (ii) it allows for pairwise comparison and direct ranking using binary outranking relations, which can be particularly useful for negotiation processes; (iii) it handles criteria measured in different units without having to use numerical-scale conversions with identical ranges (a point of contrast with ELECTRE I) [48], [33]; and (iv) it uses only concordance and discordance levels, a practical benefit compared with the more elaborated ELECTRE III, which uses three categories of threshold—indifference, preference, and veto [48], [33] .

The CRIDE (multiCRIteria DEcision support tool —CRIDE is a Celtic word that means heart— [49]) tool, based on the multi-criteria analysis (MCA) method ELECTRE II, incorporates Monte Carlo simulations to include uncertainties such as expert opinion variability and imprecision of the different criteria involved in the decision. Galarza-Molina et al. [39] used a Monte Carlo simulation in CRIDE to create random values in the evaluation matrix, the criteria weights, and the concordance and discordance levels using normal or uniform probability distributions.

Initiated Tests

We undertook the following analyses based on recommendations in [32]: (i) We used a robustness test to measure the tool capacity to produce results with the same trends, regardless of the number of criteria used in the matrix of alternatives [32]. We ran CRIDE without one criterion for each execution (the criteria were removed one by one), and then we evaluated the information that gives each criterion by estimating its incidence on the results variability by counting the number of noncoincidences per execution and dividing each one by the total sum of noncoincidences. (ii) The redundancy test consists of evaluating which criterion is unnecessary [50]; we used ELECTRE II to remove pairs of criteria and computed the redundancy indicator as the sum of the differences between the alternative positions obtained with ELECTRE II and CRIDE. (iii) We used the sensitivity analysis to produce slight weight variations (i. e., we used the average criterion for CRIDE).

The data used for each case study for ELECTRE II were (i) the average criteria weights, (ii) the average values of the matrix of alternatives and (iii) concordance levels of c_min,1 = 0,7 > c_min,2 = 0,6 > c_min,3 = 0,50. We defined the discordance levels (two per criteria) as the average between the maximum differences of each criterion. We chose the following thresholds arbitrarily: 20th percentile for the first level (d_max,1) and 35th percentile for the second level (d_max,2).

In contrast, the data used for CRIDE were (i) all the results from the criteria weight survey, (ii) the minimum and maximum values of the matrix of alternatives and (iii) the random concordance values. For the discordance levels, we used the same definition described previously.

Results and Discussion

Figure 1 shows the general results generated by ELECTRE II and CRIDE for the three cases in this study. The CRIDE results are called the “reference execution”, which means that the results were obtained by applying CRIDE with all possible variations.

The results for SWH (figure 1a) were almost the same, except for the third and fourth places. The tools suggested that the more suitable solutions were S5 and S6 (first and second places). According to figure 1b, most PROSOFI alternatives changed their ranking positions, except Fontibón and Teusaquillo. The PROSOFI results from CRIDE (figure 1b) suggest that the more suitable positions, in decreasing preference order, are Santa Fe, Zipaquirá, Candelaria, Mártires and San Cristobal, while ELECTRE II suggested a ranking of Usme, Soacha, Zipaquirá, Santa Fe and Ciudad Bolívar. The common alternatives between these results were Zipaquirá and Santa Fe. For the third case study using the Book example (figure 1c), the results show that the more suitable solutions are alternatives A4 and A3 (first and second places), the worst one was A7, and the alternatives A1, A5 and A6 did not change their ranking places. Initially, these results showed that the alternatives’ positions of the PROSOFI case study and the Book example tended to have more changes in their ranking places than the SWH case study. In other words, for PROSOFI and the Book example, it seems that uncertainties could be relevant in the decision-making process. It was necessary to know the level of complexity of each case to generalize this result by testing for robustness, redundancy, and sensitivity.

Robustness Test

Figure 2 illustrates the results of the robustness test for the SWH (figure 2a) and Book example (figure 2b) case studies, and figure 3 and table 9 show the PROSOFI case study. The noncoincidence factors are in parenthesis: high values indicate more changes in the alternatives’ ranking places. The tables in figure 2 indicate the ranking places that do not change for each execution. For the SWH case study (figure 2a), the results indicate that it is necessary to include criteria 7 (net present value) and 8 (project’s internal rate of return [IRR] vs. another project’s IRR) to determine the best alternative. However, if the objective is to rank the first three places, it is necessary to consider criteria 4 (frequency and need of operation and maintenance), 7 and 8 (financial criteria), as these criteria have the highest noncoincidence factors (28,57%). For the Book example case study (figure 2b), it is necessary to include criterion 7 (investment) (noncoincidence factor of 19,35%) to determine the best alternative, while if the objective is the ranking of the first three suitable alternatives, all criteria appear to be necessary.

For the PROSOFI case study (figure 3 and table 9), the robustness test shows that (i) to determine the best alternative, it is necessary to take into account criteria 1 (travel time between PUJ campus and PROSOFI community) and 2 (availability of public transportation), with noncoincidence factors of 5,83% and 6,73%, respectively. (ii) To determine the first three alternatives, one must account for all the criteria except criteria 4 (previous presence of The Society of Jesus), 16 (existence of micro, small and medium enterprises) and 17 (displaced population). (iii) To determine the first five alternatives, one must account for all the criteria except criterion 4; (iv) the execution “Without Criterion 10” (private schools) has the lowest noncoincidence factor (4,04%), which indicates that this execution produces fewer options (see table 9).

These results show that all the criteria for the PROSOFI and Book example provide relevant information. In contrast, for SWH, some criteria could be removed without affecting the ranking results.

Redundancy Test

The second test considered was the redundancy of the criteria set. For the SWH case study (figure 4a), the pairs of criteria that produce the same results are 2-3 and 2-4, with a redundancy indicator equal to 8, and pairs 4-8, 5-6, 5-7, and 5-8, with a redundancy indicator equal to 10.

For the PROSOFI case study, ELECTRE II was executed without each pair of criteria. As a result, of all the combinations of the 17 criteria, we analyzed 136 pairs (figure 4b): Table 10 shows the 85 pairs that produce the same redundancy indicator results.

For the Book example case study (figure 4c), the pairs that produce the same results are 1-2, 3-6, 3-7 and 5-6, with a redundancy indicator equal to 14; 1-3 and 1-6, with a redundancy indicator equal to 20; 1-4 and 1-7, with a redundancy indicator equal to 16; 2-3 and 4-7, with a redundancy indicator equal to 6; 2-4, 2-5, 2-6, 2-7, 3-5, 4-5 and 4-6, with a redundancy indicator equal to 8; and 5-7 and 6-7, with a redundancy indicator equal to 12.

The possible redundant pairs (table 10) were removed, and CRIDE was executed to compare the results obtained with the reference execution of each case study (dashed lines in figure 5). Then, we evaluated noncoincidence factors. The pairs that present the same coincidence factor and produce the same results are possible candidates for redundant criteria.

According to figure 5a (SWH), the pairs of criteria 5-7 and 5-8 produced the same results with a noncoincidence factor of 25%. Criteria 7 (net present value) and 8 (Project’s IRR vs. another project’s IRR) are the criteria that evaluate the alternatives at the financial aspect. Therefore, we can conclude that these criteria are redundant when applied in the present case study.

For the PROSOFI case study (figure 5b), it was difficult to identify the pairs that present the same noncoincidence factor and produce the same results: from numerical results obtained and saved in files (not shown in this paper), it can be observed that none of the pairs produced the same results, with the exception of the pairs of criteria 3-17 and 16-17, which produced the same six first ranking results (criterion 3: previous presence of the School of Engineering; criterion 16: existence of micro, small and medium enterprises). Therefore, criteria 3 and 16 would be redundant if we want to know the first six ranking results.

For the Book example case study (figure 5c), none of the pairs produced the same results. Therefore, all the criteria for the alternatives ranking are needed.

On the other hand, it was interesting to understand why criteria 3 and 16 (figure 6) were redundant for the PROSOFI case study. At first glance, there is no clear relation between these criteria, such as in the SWH with criteria 7 and 8 (because both are financial aspects). A Pearson correlation test (after verifying normal distribution and homogeneity of variance for both criteria) shows no significant correlation (p-value > 0,05) between criterion 3 and criterion 16.

Sensitivity Analysis

Finally, we performed a sensitivity analysis (figure 7). For the SWH case study, the results did not show significant variations compared to the reference execution (figure 7a), which means that the decision-maker can use the average weight of one of the criteria to know the first three positions for a multi-criteria decision-making ranking problem. If criteria 2, 3, 4, 5 and 8 (with noncoincidence factors of 20%) were fixed, alternatives S2 and S3 would change their positions.

For the PROSOFI case study (figure 7b), the results show the fixed weights of criteria 3 (previous presence of the School of Engineering), 9 (public schools), 11 (relation: public schools/total Schools) and 13 (population density) change second and third places: the noncoincidence factors are 8,57% (criterion 3) and 4,76% (criteria 9, 11 and 13). The ranking places that do not change are 1st, 4th, 5th, 8th, 19th, 20th and 21st.

In contrast, for the Book example case study (figure 7c), if any criterion was fixed, all the ranking results would change (with noncoincidence factors of 14,29%). This case study is sensitive to slight changes in the criteria weights, in contrast to the other case studies.

The first test (robustness) showed that for the PROSOFI and Book example, all the criteria provide relevant information because they have a high incidence on the variability of the results. Almost the same result occurred with the second test (redundancy): Most of the criteria appear to have a high incidence on the variability of the results. For the last test (sensitivity analysis), it was observed that the Book example is sensitive to slight changes on the weight criteria, in contrast to case studies (a) SWH and (b) PROSOFI. These results show that including uncertainties in the matrix of alternatives for a case study with more alternatives, criteria, and surveys can affect the variability of the results.

The above findings suggest that it is better to include uncertainty (e. g., CRIDE) in the decision-making process for case studies with a substantial number of alternatives and criteria. Additionally, it is important not to include criteria that could introduce redundant information as a basis to simplify decision-making problems, as was the case for the SWH case study (e. g., criteria 7 and 8 of the SWH case study).

Conclusions

This paper aimed to identify when it is useful to include uncertainties in a decision-making process for MCDA problems, based on ELECTRE II and CRIDE tools and applying robustness, redundancy and sensitivity tests for two real-world decision-making case studies and a hypothetical case, with differing numbers of alternatives, criteria, and surveys.

The results show that for a simple decision-making problem such as the SWH case study, it is not necessary to include uncertainties, and a simple MCDA tool, as ELECTRE II, seems to be sufficient. This statement implies that, for this case, time and money could be saved during the collection of data because there is no need to consider variations. However, for the other problems analyzed, with more alternatives, criteria, and surveys, the inclusion of uncertainties seems to be necessary, and a tool such as CRIDE could be used to manage this scenario. This implies that, for these cases, a more comprehensive collection of information to fill the matrix of alternatives is needed. Some trends relating to the incidence of the variability and the difficulty of the MCDA problems were observed: the incidence of the variability seems to be higher for more difficult MCDA problems (PROSOFI case study: 21 alternatives, 17 criteria and 93 surveys) compared with more straightforward problems (SWH case study: 6 alternatives, 8 criteria and 9 surveys). However, more tests with more case studies are required to generalize this result.

In addition, the approach developed in CRIDE allows for the evaluation of the established criteria set by testing robustness, redundancy, and sensitivity: as evidenced in the present study, by using CRIDE outputs, the setting of the criteria for a particular problem, considered one of the critical aspects of decision-making, could be used instead of conventional statistical tests (e. g., correlation tests, ANOVA, PCA, etc.). However, a more detailed and formal methodological proposition for achieving this task is still required.

References

Notes

* Research article.

Author notes

^a Corresponding author. E-mail: sgalarza@javeriana.edu.co

Additional information

How to cite this article: S. Galarza-Molina and A. Torres, “Including uncertainties in multi-criteria decision-making: A superfluous task or a requirement?,” Ing. Univ. vol. 23, no. 1, 2019 [Online]. https://doi.org/10.11144/Javeriana.iyu23-1.iumd

Descargar