Including uncertainties in multi-criteria decision-making : a superfluous task or a requirement ? 1

Objective: This paper aims to assess cases where it is necessary to include uncertainties in decision-making input data. Materials and Methods: Three case studies with different numbers of alternatives and different numbers and natures of criteria were evaluated. The CRIDE tool [39] was used to include uncertainties in the input data. Results and Discussion: The results obtained showed that for more difficult decision-making problems, the inclusion of variations in input data could change the final decision, while for less challenging problems, it is unnecessary to take uncertainties into account. Conclusions: These findings could be useful for decision-makers in obtaining more accurate results or in saving time and money related to input data acquisition.


Introduction
Decision-making is a frequent and essential activity for any organization [1]. We frequently understand that, independent of organization size, importance or economic sector, decisionmaking is a day-to-day task necessary for operational, tactical and strategic purposes. According to [2], this task is crucial because it involves fundamental decisions that shape the course of a firm. Developing tools such as explicit models, whether formalized or not, can help decision makers answer the questions that arise during this process [3]. These tools help to illustrate possible solutions and generate a recommendation to increase the coherence between the development of a process and the goals of the stakeholders [4], [5]. The word recommendation is used to make both the analyst and the person or group of people who make the decision aware that they are free to follow the suggestions or not [5]. Collaborative decision-making aims to reach a high level of consensus to achieve unity and ownership, thus avoiding a simple vote [3]. Hence, to improve decision-making transparency, auditability, and analytic rigor, it is necessary to use a method that supports this process [6], [7]. One of the options cited by several researchers is Multi-Criteria Decision Analysis (MCDA) [4], [8]- [13], which was developed in the 1960s [14]. Since then, this technique has been the subject of numerous research projects in fields such as medicine, engineering, finance, and economics [15]- [20].
One aspect of MCDA that some authors have explored is the handling of uncertainty in data and parameter values [13], [21]- [29]. Potential sources of uncertainty come from the definitions of alternative performance and the criteria weights, which is one of the most challenging steps in the MCDA process [21], [25]. Some methods have been developed to address these uncertainties [22], [23], [26]- [28], [30], [31]. While most of the methodologies employ statistical tools to evaluate uncertainty in some of the input data, they do not account for the thresholds included in most MCDA concerning all the uncertainties of the input data. Additionally, few methodologies can assess the quality criteria (performance values and weights), which are considered key aspects of decisionmaking problems [32].
Although many MCDA methods, including uncertainties, have arisen during the past few years for a wide range of applications [33]- [41], it seems necessary to determine which type of problems require the inclusion of uncertainties as input data when assessing an MCDA. Additionally, MCDA problem characterization, which could define each factor if deserving of uncertainty inclusion, is lacking, as well as sufficiently flexible tools to evaluate these factors. This paper aims to identify which type of MCDA problems should include uncertainties in a decision-making process.

Case Studies
For this study, we used two real-world decision-making case studies and a hypothetical case with differing numbers of alternatives, criteria, and surveys. The first case study is the decision-making process for Stormwater Harvesting (SWH) at the Pontificia Universidad Javeriana Bogotá (PUJ) [39]. The second case study is the selection of the target community of the PUJ Social Program PROSOFI (from its Spanish acronym) [1]. The third case is an example from the book "Decisiones con múltiples objetivos e incertidumbres" [42] related to the selection of alternatives for sustainable development management of a small catchment.

Case Study 1: PUJ Campus Stormwater Harvesting
For this case study, six scenarios for SWH were proposed on the campus of PUJ. The authors suggested eight criteria to evaluate each scenario. The University's Physical Resources Division (PRD) defined the criteria based on a set proposed by [33]. Within the criteria set, technical aspects, such as hydraulic and quality performances, management aspects, such as compatibility with the University's Master Plan, and financial aspects, such as net present value (NPV) of the investment, were considered. We used a survey applied to four leaders from the PRD and five experts on water management to define the criteria weights (table 1). For more details of this study, see [39]. Galarza-Molina et al. [39] built the scenarios or possible alternatives based on a division of the PUJ campus in twelve basins. They proposed the use of SUDS (Sustainable Urban Drainage Systems) [43] for the collection and treatment of the stormwater in the basins, including bioretention gardens, permeable paving, and constructed wetlands.
The input scenarios emerged from an ideal scenario where all the water harvested in the campus would be used (maximum supply volume) with drinking water quality (maximum water quality). The other scenarios were created by decreasing the water quality (fewer possible uses) and the supply (fewer basins included), obtaining high, medium and low water demands and high, medium and low water supplies. The above considerations allowed sizing of the SUDS considered and the pumping and treatment systems. Table 2 shows the description of the scenarios. 12 basin runoff collection sites with landscape irrigation (maximum supply/low quality). S4 9 basin runoff collection sites with potability (medium supply/maximum quality) S5 9 basin runoff collection sites without potability: floor cleaning, sanitary discharge and landscape irrigation (medium supply /medium quality). S6 9 basin runoff collection sites with landscape irrigation (medium supply/low quality).

Source: Galarza-Molina et al. [39]
As a result, [39] constructed table 3, with eight criteria and six scenarios (8 × 6 matrix) with minimum, average and maximum values per criterion and scenario.     means that performance of the alternative with respect to the specific criteria increases as the criterion value also increases.
 means that performance of the alternative with respect to the specific criteria decreases as the criterion value increases.

Case Study 2: Selection of PROSOFI Target Community
For this case study, [1] defined the 21 alternatives (19 Bogotá urban districts plus two municipalities near Bogotá) along with criteria and weighting factors based on a participatory project of the university. [1] developed a survey applied to 143 faculty members to determine the criteria and weighting factors. The survey asked which essential aspects should be considered for the selection of the PROSOFI target community and which of these aspects were more relevant than others. [1] analyzed the results from the survey (111 persons, including directors, faculty members, and administrative staff) to find inconsistencies, using a script that compared the answers of each person. Table 4 shows the selection criteria for each key aspect [1].

Case Study 3: Book Example
With the aim of comparing different types of decision problems, we used an example from the book Decisiones con múltiples objetivos e incertidumbres by [42]. The decision-making process here selects the best alternative for sustainable development management in a 250ha catchment. The authors proposed seven criteria to evaluate the alternatives (table 6).

Source: adapted from Smith et al. [42]
Summary of the Case Studies Table 8 illustrates the differences among the three case studies. Source: author's own elaboration

Decision-Making Tools
ELECTRE II is part of the family of methods collectively known as ELECTRE (ELimination Et Choix Traduisant la REalité -ELimination and Choice Expressing REality). Proposed in the late 1960s by [47], it is considered the first method dealing with problem ranking [48]. We chose this method because (i) it enables a dialogue between various stakeholders involved in the decision-making process but provides input/voice differences between them by means of a criteria weighting procedure [33]; (ii) it allows for pairwise comparison and direct ranking using binary outranking relations, which can be particularly useful for negotiation processes; (iii) it handles criteria measured in different units without having to use numerical-scale conversions with identical ranges (a point of contrast with ELECTRE I) [48], [33]; and (iv) it uses only concordance and discordance levels, a practical benefit compared with the more elaborated ELECTRE III, which uses three categories of threshold-indifference, preference, and veto [48], [33] .
The CRIDE (multiCRIteria DEcision support tool -CRIDE is a Celtic word that means heart- [49]) tool, based on the multi-criteria analysis (MCA) method ELECTRE II, incorporates Monte Carlo simulations to include uncertainties such as expert opinion variability and imprecision of the different criteria involved in the decision. Galarza-Molina et al. [39] used a Monte Carlo simulation in CRIDE to create random values in the evaluation matrix, the criteria weights, and the concordance and discordance levels using normal or uniform probability distributions.

Initiated Tests
We undertook the following analyses based on recommendations in [32]: (i) We used a robustness test to measure the tool capacity to produce results with the same trends, regardless of the number of criteria used in the matrix of alternatives [32]. We ran CRIDE without one criterion for each execution (the criteria were removed one by one), and then we evaluated the information that gives each criterion by estimating its incidence on the results variability by counting the number of noncoincidences per execution and dividing each one by the total sum of noncoincidences. (ii) The redundancy test consists of evaluating which criterion is unnecessary [50]; we used ELECTRE II to remove pairs of criteria and computed the redundancy indicator as the sum of the differences between the alternative positions obtained with ELECTRE II and CRIDE. (iii) We used the sensitivity analysis to produce slight weight variations (i. e., we used the average criterion for CRIDE).
The data used for each case study for ELECTRE II were (i) the average criteria weights, (ii) the average values of the matrix of alternatives and (iii) concordance levels of cmin,1 = 0,7 > cmin,2 = 0,6 > cmin,3 = 0,50. We defined the discordance levels (two per criteria) as the average between the maximum differences of each criterion. We chose the following thresholds arbitrarily: 20th percentile for the first level (dmax,1) and 35th percentile for the second level (dmax,2).
In contrast, the data used for CRIDE were (i) all the results from the criteria weight survey, (ii) the minimum and maximum values of the matrix of alternatives and (iii) the random concordance values. For the discordance levels, we used the same definition described previously. Figure 1 shows the general results generated by ELECTRE II and CRIDE for the three cases in this study. The CRIDE results are called the "reference execution", which means that the results were obtained by applying CRIDE with all possible variations.

Results and Discussion
The results for SWH (figure 1a) were almost the same, except for the third and fourth places. The tools suggested that the more suitable solutions were S5 and S6 (first and second places). According to figure 1b, most PROSOFI alternatives changed their ranking positions, except Fontibón and Teusaquillo. The PROSOFI results from CRIDE (figure 1b) suggest that the more suitable positions, in decreasing preference order, are Santa Fe, Zipaquirá, Candelaria, Mártires and San Cristobal, while ELECTRE II suggested a ranking of Usme, Soacha, Zipaquirá, Santa Fe and Ciudad Bolívar. The common alternatives between these results were Zipaquirá and Santa Fe. For the third case study using the Book example ( figure 1c), the results show that the more suitable solutions are alternatives A4 and A3 (first and second places), the worst one was A7, and the alternatives A1, A5 and A6 did not change their ranking places. Initially, these results showed that the alternatives' positions of the PROSOFI case study and the Book example tended to have more changes in their ranking places than the SWH case study. In other words, for PROSOFI and the Book example, it seems that uncertainties could be relevant in the decision-making process.
It was necessary to know the level of complexity of each case to generalize this result by testing for robustness, redundancy, and sensitivity.

Source: author's own elaboration
For the PROSOFI case study (figure 3 and table 9), the robustness test shows that (i) to determine the best alternative, it is necessary to take into account criteria 1 (travel time between PUJ campus and PROSOFI community) and 2 (availability of public transportation), with noncoincidence factors of 5,83 % and 6,73 %, respectively. (ii) To determine the first three alternatives, one must account for all the criteria except criteria 4 (previous presence of The Society of Jesus), 16 (existence of micro, small and medium enterprises) and 17 (displaced population). (iii) To determine the first five alternatives, one must account for all the criteria except criterion 4; (iv) the execution "Without Criterion 10" (private schools) has the lowest noncoincidence factor (4,04 %), which indicates that this execution produces fewer options (see table 9).  Table 9. The number of places to be maintained without criterion each time for PROSOFI case study; the gray column indicates the execution "Without Criterion 10" (private schools)

Social Dynamics and Social Organization Cultural and ethnic elements
These results show that all the criteria for the PROSOFI and Book example provide relevant information. In contrast, for SWH, some criteria could be removed without affecting the ranking results.

Redundancy Test
The second test considered was the redundancy of the criteria set. For the SWH case study (figure 4a), the pairs of criteria that produce the same results are 2-3 and 2-4, with a redundancy indicator equal to 8, and pairs 4-8, 5-6, 5-7, and 5-8, with a redundancy indicator equal to 10.
For the PROSOFI case study, ELECTRE II was executed without each pair of criteria. As a result, of all the combinations of the 17 criteria, we analyzed 136 pairs (figure 4b): Table 10 shows the 85 pairs that produce the same redundancy indicator results.

Source: author's own elaboration
The possible redundant pairs (table 10) were removed, and CRIDE was executed to compare the results obtained with the reference execution of each case study (dashed lines in figure 5). Then, we evaluated noncoincidence factors. The pairs that present the same coincidence factor and produce the same results are possible candidates for redundant criteria.
According to figure 5a (SWH), the pairs of criteria 5-7 and 5-8 produced the same results with a noncoincidence factor of 25 %. Criteria 7 (net present value) and 8 (Project's IRR vs. another project's IRR) are the criteria that evaluate the alternatives at the financial aspect. Therefore, we can conclude that these criteria are redundant when applied in the present case study.

Source: author's own elaboration
For the PROSOFI case study (figure 5b), it was difficult to identify the pairs that present the same noncoincidence factor and produce the same results: from numerical results obtained and saved in files (not shown in this paper), it can be observed that none of the pairs produced the same results, with the exception of the pairs of criteria 3-17 and 16-17, which produced the same six first ranking results (criterion 3: previous presence of the School of Engineering; criterion 16: existence of micro, small and medium enterprises). Therefore, criteria 3 and 16 would be redundant if we want to know the first six ranking results.
For the Book example case study (figure 5c), none of the pairs produced the same results. Therefore, all the criteria for the alternatives ranking are needed.
On the other hand, it was interesting to understand why criteria 3 and 16 ( figure 6) were redundant for the PROSOFI case study. At first glance, there is no clear relation between these criteria, such as in the SWH with criteria 7 and 8 (because both are financial aspects). A Pearson correlation test (after verifying normal distribution and homogeneity of variance for both criteria) shows no significant correlation (p-value > 0,05) between criterion 3 and criterion 16.

Sensitivity Analysis
Finally, we performed a sensitivity analysis ( figure 7). For the SWH case study, the results did not show significant variations compared to the reference execution (figure 7a), which means that the decision-maker can use the average weight of one of the criteria to know the first three positions for a multi-criteria decision-making ranking problem. If criteria 2, 3, 4, 5 and 8 (with noncoincidence factors of 20 %) were fixed, alternatives S2 and S3 would change their positions.

Source: author's own elaboration
The first test (robustness) showed that for the PROSOFI and Book example, all the criteria provide relevant information because they have a high incidence on the variability of the results. Almost the same result occurred with the second test (redundancy): Most of the criteria appear to have a high incidence on the variability of the results. For the last test (sensitivity analysis), it was observed that the Book example is sensitive to slight changes on the weight criteria, in contrast to case studies (a) SWH and (b) PROSOFI. These results show that including uncertainties in the matrix of alternatives for a case study with more alternatives, criteria, and surveys can affect the variability of the results.
The above findings suggest that it is better to include uncertainty (e. g., CRIDE) in the decision-making process for case studies with a substantial number of alternatives and criteria. Additionally, it is important not to include criteria that could introduce redundant information as a basis to simplify decision-making problems, as was the case for the SWH case study (e. g., criteria 7 and 8 of the SWH case study).

Conclusions
This paper aimed to identify when it is useful to include uncertainties in a decision-making process for MCDA problems, based on ELECTRE II and CRIDE tools and applying robustness, redundancy and sensitivity tests for two real-world decision-making case studies and a hypothetical case, with differing numbers of alternatives, criteria, and surveys.
The results show that for a simple decision-making problem such as the SWH case study, it is not necessary to include uncertainties, and a simple MCDA tool, as ELECTRE II, seems to be sufficient. This statement implies that, for this case, time and money could be saved during the collection of data because there is no need to consider variations. However, for the other problems analyzed, with more alternatives, criteria, and surveys, the inclusion of uncertainties seems to be necessary, and a tool such as CRIDE could be used to manage this scenario. This implies that, for these cases, a more comprehensive collection of information to fill the matrix of alternatives is needed. Some trends relating to the incidence of the variability and the difficulty of the MCDA problems were observed: the incidence of the variability seems to be higher for more difficult MCDA problems (PROSOFI case study: 21 alternatives, 17 criteria and 93 surveys) compared with more straightforward problems (SWH case study: 6 alternatives, 8 criteria and 9 surveys). However, more tests with more case studies are required to generalize this result.
In addition, the approach developed in CRIDE allows for the evaluation of the established criteria set by testing robustness, redundancy, and sensitivity: as evidenced in the present study, by using CRIDE outputs, the setting of the criteria for a particular problem, considered one of the critical aspects of decision-making, could be used instead of conventional statistical tests (e. g., correlation tests, ANOVA, PCA, etc.). However, a more detailed and formal methodological proposition for achieving this task is still required.