Including Uncertainties in Multi-Criteria Decision-Making: A Superfluous Task or a Requirement?*
Incluyendo incertidumbres en la toma de decisiones multi-criterio: ¿una tarea superflua o un requerimiento?
Ingeniería y Universidad, vol. 23, no. 1, 2019
Pontificia Universidad Javeriana
Sandra Galarza-Molina a email@example.com
Universidad Javeriana, Colombia
Universidad Javeriana, Colombia
Date received: 04 April 2018
Date accepted: 06 December 2018
Date published: 24 June 2019
Abstract: Objective: This paper aims to assess cases where it is necessary to include uncertainties in decision-making input data. Materials and Methods: Three case studies with different numbers of alternatives and different numbers and natures of criteria were evaluated. The CRIDE tool was used to include uncertainties in the input data. Results and Discussion: The results obtained showed that for more difficult decision-making problems, the inclusion of variations in input data could change the final decision, while for less challenging problems, it is unnecessary to take uncertainties into account. Conclusions: These findings could be useful for decision-makers in obtaining more accurate results or in saving time and money related to input data acquisition.
Keywords: Multicriteria decision-making, uncertainties, robustness, sensitivity and redundancy analysis.
Resumen: Objetivo: El objetivo de este artículo es evaluar en qué casos es necesario incluir incertidumbres en la información necesaria para la toma de decisiones. Materiales y métodos: Tres casos de estudio con un número diferente de alternativas y diferente número y natrualeza de criterios fueron evaluados. La herramienta CRIDE se utilizó para incluir las incertidumbres en la información de entrada. Resultados y discusión: Los resultados obtenidos muestran que para problemas de toma de decisión más complejos, la inclusión de variaciones en los datos de entrada pueden incidir en la decisión final, mientras que para problemas menos difíciles es innecesario tomar en cuenta dichas incertidumbres. Conclusiones: Estos hallazgos pueden resultar útiles para los tomadores de decisiones en cuanto a la obtención de resultados más precisos o con el propósito de ahorrar tiempo y dinero relacionado con la adquisición de la información necesaria para la toma de decisiones.
Palabras clave: toma de decisiones multicriterio, incertidumbres, análisis de robustez, sensibilidad y redundancia.
Decision-making is a frequent and essential activity for any organization . We frequently understand that, independent of organization size, importance or economic sector, decision-making is a day-to-day task necessary for operational, tactical and strategic purposes. According to , this task is crucial because it involves fundamental decisions that shape the course of a firm. Developing tools such as explicit models, whether formalized or not, can help decision makers answer the questions that arise during this process . These tools help to illustrate possible solutions and generate a recommendation to increase the coherence between the development of a process and the goals of the stakeholders , . The word recommendation is used to make both the analyst and the person or group of people who make the decision aware that they are free to follow the suggestions or not . Collaborative decision-making aims to reach a high level of consensus to achieve unity and ownership, thus avoiding a simple vote . Hence, to improve decision-making transparency, auditability, and analytic rigor, it is necessary to use a method that supports this process , . One of the options cited by several researchers is Multi-Criteria Decision Analysis (MCDA) , , , , , , , which was developed in the 1960s . Since then, this technique has been the subject of numerous research projects in fields such as medicine, engineering, finance, and economics , , , , , .
One aspect of MCDA that some authors have explored is the handling of uncertainty in data and parameter values , , , , , , , , , . Potential sources of uncertainty come from the definitions of alternative performance and the criteria weights, which is one of the most challenging steps in the MCDA process , . Some methods have been developed to address these uncertainties , , , , , , . While most of the methodologies employ statistical tools to evaluate uncertainty in some of the input data, they do not account for the thresholds included in most MCDA concerning all the uncertainties of the input data. Additionally, few methodologies can assess the quality criteria (performance values and weights), which are considered key aspects of decision-making problems .
Although many MCDA methods, including uncertainties, have arisen during the past few years for a wide range of applications , , , , , , , , , it seems necessary to determine which type of problems require the inclusion of uncertainties as input data when assessing an MCDA. Additionally, MCDA problem characterization, which could define each factor if deserving of uncertainty inclusion, is lacking, as well as sufficiently flexible tools to evaluate these factors.
This paper aims to identify which type of MCDA problems should include uncertainties in a decision-making process.
For this study, we used two real-world decision-making case studies and a hypothetical case with differing numbers of alternatives, criteria, and surveys. The first case study is the decision-making process for Stormwater Harvesting (SWH) at the Pontificia Universidad Javeriana Bogotá (PUJ) . The second case study is the selection of the target community of the PUJ Social Program PROSOFI (from its Spanish acronym) . The third case is an example from the book “Decisiones con múltiples objetivos e incertidumbres”  related to the selection of alternatives for sustainable development management of a small catchment.
Case Study 1: PUJ Campus Stormwater Harvesting
For this case study, six scenarios for SWH were proposed on the campus of PUJ. The authors suggested eight criteria to evaluate each scenario. The University’s Physical Resources Division (PRD) defined the criteria based on a set proposed by . Within the criteria set, technical aspects, such as hydraulic and quality performances, management aspects, such as compatibility with the University’s Master Plan, and financial aspects, such as net present value (NPV) of the investment, were considered. We used a survey applied to four leaders from the PRD and five experts on water management to define the criteria weights (table 1). For more details of this study, see .
Galarza-Molina et al.  built the scenarios or possible alternatives based on a division of the PUJ campus in twelve basins. They proposed the use of SUDS (Sustainable Urban Drainage Systems)  for the collection and treatment of the stormwater in the basins, including bioretention gardens, permeable paving, and constructed wetlands.
The input scenarios emerged from an ideal scenario where all the water harvested in the campus would be used (maximum supply volume) with drinking water quality (maximum water quality). The other scenarios were created by decreasing the water quality (fewer possible uses) and the supply (fewer basins included), obtaining high, medium and low water demands and high, medium and low water supplies. The above considerations allowed sizing of the SUDS considered and the pumping and treatment systems. Table 2 shows the description of the scenarios.
As a result,  constructed table 3, with eight criteria and six scenarios (8 × 6 matrix) with minimum, average and maximum values per criterion and scenario.
Case Study 2: Selection of PROSOFI Target Community
For this case study,  defined the 21 alternatives (19 Bogotá urban districts plus two municipalities near Bogotá) along with criteria and weighting factors based on a participatory project of the university.  developed a survey applied to 143 faculty members to determine the criteria and weighting factors. The survey asked which essential aspects should be considered for the selection of the PROSOFI target community and which of these aspects were more relevant than others.  analyzed the results from the survey (111 persons, including directors, faculty members, and administrative staff) to find inconsistencies, using a script that compared the answers of each person. Table 4 shows the selection criteria for each key aspect .
Galarza Molina et al.  constructed the matrix of alternatives using the defined criteria and the evaluation of alternatives. They obtained information from the 2005 Bogotá census , Bogotá’s Chamber of Commerce , the Undersecretary’s office for planning Economic and Social Affairs  and the experience of experts. As a result, this case study has a 17 × 21 matrix (see table 5, one part of the decision matrix).
Case Study 3: Book Example
With the aim of comparing different types of decision problems, we used an example from the book Decisiones con múltiples objetivos e incertidumbres by . The decision-making process here selects the best alternative for sustainable development management in a 250-ha catchment. The authors proposed seven criteria to evaluate the alternatives (table 6).
Table 7 shows the probability distributions and weights of the criteria and the matrix of alternatives (7 × 7 matrix). In this table are the minimum, average and maximum values per criterion and scenario.  determined the average values of the evaluation matrix and the criteria weights. For the minimum and maximum values, we defined them with variations of 1,7% or 7,8%, and the probability distributions of the criteria were chosen arbitrarily.
Summary of the Case Studies
Table 8 illustrates the differences among the three case studies.
ELECTRE II is part of the family of methods collectively known as ELECTRE (ELimination Et Choix Traduisant la REalité – ELimination and Choice Expressing REality). Proposed in the late 1960s by , it is considered the first method dealing with problem ranking . We chose this method because (i) it enables a dialogue between various stakeholders involved in the decision-making process but provides input/voice differences between them by means of a criteria weighting procedure ; (ii) it allows for pairwise comparison and direct ranking using binary outranking relations, which can be particularly useful for negotiation processes; (iii) it handles criteria measured in different units without having to use numerical-scale conversions with identical ranges (a point of contrast with ELECTRE I) , ; and (iv) it uses only concordance and discordance levels, a practical benefit compared with the more elaborated ELECTRE III, which uses three categories of threshold—indifference, preference, and veto ,  .
The CRIDE (multiCRIteria DEcision support tool —CRIDE is a Celtic word that means heart— ) tool, based on the multi-criteria analysis (MCA) method ELECTRE II, incorporates Monte Carlo simulations to include uncertainties such as expert opinion variability and imprecision of the different criteria involved in the decision. Galarza-Molina et al.  used a Monte Carlo simulation in CRIDE to create random values in the evaluation matrix, the criteria weights, and the concordance and discordance levels using normal or uniform probability distributions.
We undertook the following analyses based on recommendations in : (i) We used a robustness test to measure the tool capacity to produce results with the same trends, regardless of the number of criteria used in the matrix of alternatives . We ran CRIDE without one criterion for each execution (the criteria were removed one by one), and then we evaluated the information that gives each criterion by estimating its incidence on the results variability by counting the number of noncoincidences per execution and dividing each one by the total sum of noncoincidences. (ii) The redundancy test consists of evaluating which criterion is unnecessary ; we used ELECTRE II to remove pairs of criteria and computed the redundancy indicator as the sum of the differences between the alternative positions obtained with ELECTRE II and CRIDE. (iii) We used the sensitivity analysis to produce slight weight variations (i. e., we used the average criterion for CRIDE).
The data used for each case study for ELECTRE II were (i) the average criteria weights, (ii) the average values of the matrix of alternatives and (iii) concordance levels of cmin,1 = 0,7 > cmin,2 = 0,6 > cmin,3 = 0,50. We defined the discordance levels (two per criteria) as the average between the maximum differences of each criterion. We chose the following thresholds arbitrarily: 20th percentile for the first level (dmax,1) and 35th percentile for the second level (dmax,2).
In contrast, the data used for CRIDE were (i) all the results from the criteria weight survey, (ii) the minimum and maximum values of the matrix of alternatives and (iii) the random concordance values. For the discordance levels, we used the same definition described previously.
Results and Discussion
Figure 1 shows the general results generated by ELECTRE II and CRIDE for the three cases in this study. The CRIDE results are called the “reference execution”, which means that the results were obtained by applying CRIDE with all possible variations.
The results for SWH (figure 1a) were almost the same, except for the third and fourth places. The tools suggested that the more suitable solutions were S5 and S6 (first and second places). According to figure 1b, most PROSOFI alternatives changed their ranking positions, except Fontibón and Teusaquillo. The PROSOFI results from CRIDE (figure 1b) suggest that the more suitable positions, in decreasing preference order, are Santa Fe, Zipaquirá, Candelaria, Mártires and San Cristobal, while ELECTRE II suggested a ranking of Usme, Soacha, Zipaquirá, Santa Fe and Ciudad Bolívar. The common alternatives between these results were Zipaquirá and Santa Fe. For the third case study using the Book example (figure 1c), the results show that the more suitable solutions are alternatives A4 and A3 (first and second places), the worst one was A7, and the alternatives A1, A5 and A6 did not change their ranking places. Initially, these results showed that the alternatives’ positions of the PROSOFI case study and the Book example tended to have more changes in their ranking places than the SWH case study. In other words, for PROSOFI and the Book example, it seems that uncertainties could be relevant in the decision-making process. It was necessary to know the level of complexity of each case to generalize this result by testing for robustness, redundancy, and sensitivity.
Figure 2 illustrates the results of the robustness test for the SWH (figure 2a) and Book example (figure 2b) case studies, and figure 3 and table 9 show the PROSOFI case study. The noncoincidence factors are in parenthesis: high values indicate more changes in the alternatives’ ranking places. The tables in figure 2 indicate the ranking places that do not change for each execution. For the SWH case study (figure 2a), the results indicate that it is necessary to include criteria 7 (net present value) and 8 (project’s internal rate of return [IRR] vs. another project’s IRR) to determine the best alternative. However, if the objective is to rank the first three places, it is necessary to consider criteria 4 (frequency and need of operation and maintenance), 7 and 8 (financial criteria), as these criteria have the highest noncoincidence factors (28,57%). For the Book example case study (figure 2b), it is necessary to include criterion 7 (investment) (noncoincidence factor of 19,35%) to determine the best alternative, while if the objective is the ranking of the first three suitable alternatives, all criteria appear to be necessary.
For the PROSOFI case study (figure 3 and table 9), the robustness test shows that (i) to determine the best alternative, it is necessary to take into account criteria 1 (travel time between PUJ campus and PROSOFI community) and 2 (availability of public transportation), with noncoincidence factors of 5,83% and 6,73%, respectively. (ii) To determine the first three alternatives, one must account for all the criteria except criteria 4 (previous presence of The Society of Jesus), 16 (existence of micro, small and medium enterprises) and 17 (displaced population). (iii) To determine the first five alternatives, one must account for all the criteria except criterion 4; (iv) the execution “Without Criterion 10” (private schools) has the lowest noncoincidence factor (4,04%), which indicates that this execution produces fewer options (see table 9).
These results show that all the criteria for the PROSOFI and Book example provide relevant information. In contrast, for SWH, some criteria could be removed without affecting the ranking results.
The second test considered was the redundancy of the criteria set. For the SWH case study (figure 4a), the pairs of criteria that produce the same results are 2-3 and 2-4, with a redundancy indicator equal to 8, and pairs 4-8, 5-6, 5-7, and 5-8, with a redundancy indicator equal to 10.
For the PROSOFI case study, ELECTRE II was executed without each pair of criteria. As a result, of all the combinations of the 17 criteria, we analyzed 136 pairs (figure 4b): Table 10 shows the 85 pairs that produce the same redundancy indicator results.
For the Book example case study (figure 4c), the pairs that produce the same results are 1-2, 3-6, 3-7 and 5-6, with a redundancy indicator equal to 14; 1-3 and 1-6, with a redundancy indicator equal to 20; 1-4 and 1-7, with a redundancy indicator equal to 16; 2-3 and 4-7, with a redundancy indicator equal to 6; 2-4, 2-5, 2-6, 2-7, 3-5, 4-5 and 4-6, with a redundancy indicator equal to 8; and 5-7 and 6-7, with a redundancy indicator equal to 12.
The possible redundant pairs (table 10) were removed, and CRIDE was executed to compare the results obtained with the reference execution of each case study (dashed lines in figure 5). Then, we evaluated noncoincidence factors. The pairs that present the same coincidence factor and produce the same results are possible candidates for redundant criteria.
According to figure 5a (SWH), the pairs of criteria 5-7 and 5-8 produced the same results with a noncoincidence factor of 25%. Criteria 7 (net present value) and 8 (Project’s IRR vs. another project’s IRR) are the criteria that evaluate the alternatives at the financial aspect. Therefore, we can conclude that these criteria are redundant when applied in the present case study.
For the PROSOFI case study (figure 5b), it was difficult to identify the pairs that present the same noncoincidence factor and produce the same results: from numerical results obtained and saved in files (not shown in this paper), it can be observed that none of the pairs produced the same results, with the exception of the pairs of criteria 3-17 and 16-17, which produced the same six first ranking results (criterion 3: previous presence of the School of Engineering; criterion 16: existence of micro, small and medium enterprises). Therefore, criteria 3 and 16 would be redundant if we want to know the first six ranking results.
For the Book example case study (figure 5c), none of the pairs produced the same results. Therefore, all the criteria for the alternatives ranking are needed.
On the other hand, it was interesting to understand why criteria 3 and 16 (figure 6) were redundant for the PROSOFI case study. At first glance, there is no clear relation between these criteria, such as in the SWH with criteria 7 and 8 (because both are financial aspects). A Pearson correlation test (after verifying normal distribution and homogeneity of variance for both criteria) shows no significant correlation (p-value > 0,05) between criterion 3 and criterion 16.
Finally, we performed a sensitivity analysis (figure 7). For the SWH case study, the results did not show significant variations compared to the reference execution (figure 7a), which means that the decision-maker can use the average weight of one of the criteria to know the first three positions for a multi-criteria decision-making ranking problem. If criteria 2, 3, 4, 5 and 8 (with noncoincidence factors of 20%) were fixed, alternatives S2 and S3 would change their positions.
For the PROSOFI case study (figure 7b), the results show the fixed weights of criteria 3 (previous presence of the School of Engineering), 9 (public schools), 11 (relation: public schools/total Schools) and 13 (population density) change second and third places: the noncoincidence factors are 8,57% (criterion 3) and 4,76% (criteria 9, 11 and 13). The ranking places that do not change are 1st, 4th, 5th, 8th, 19th, 20th and 21st.
In contrast, for the Book example case study (figure 7c), if any criterion was fixed, all the ranking results would change (with noncoincidence factors of 14,29%). This case study is sensitive to slight changes in the criteria weights, in contrast to the other case studies.
The first test (robustness) showed that for the PROSOFI and Book example, all the criteria provide relevant information because they have a high incidence on the variability of the results. Almost the same result occurred with the second test (redundancy): Most of the criteria appear to have a high incidence on the variability of the results. For the last test (sensitivity analysis), it was observed that the Book example is sensitive to slight changes on the weight criteria, in contrast to case studies (a) SWH and (b) PROSOFI. These results show that including uncertainties in the matrix of alternatives for a case study with more alternatives, criteria, and surveys can affect the variability of the results.
The above findings suggest that it is better to include uncertainty (e. g., CRIDE) in the decision-making process for case studies with a substantial number of alternatives and criteria. Additionally, it is important not to include criteria that could introduce redundant information as a basis to simplify decision-making problems, as was the case for the SWH case study (e. g., criteria 7 and 8 of the SWH case study).
This paper aimed to identify when it is useful to include uncertainties in a decision-making process for MCDA problems, based on ELECTRE II and CRIDE tools and applying robustness, redundancy and sensitivity tests for two real-world decision-making case studies and a hypothetical case, with differing numbers of alternatives, criteria, and surveys.
The results show that for a simple decision-making problem such as the SWH case study, it is not necessary to include uncertainties, and a simple MCDA tool, as ELECTRE II, seems to be sufficient. This statement implies that, for this case, time and money could be saved during the collection of data because there is no need to consider variations. However, for the other problems analyzed, with more alternatives, criteria, and surveys, the inclusion of uncertainties seems to be necessary, and a tool such as CRIDE could be used to manage this scenario. This implies that, for these cases, a more comprehensive collection of information to fill the matrix of alternatives is needed. Some trends relating to the incidence of the variability and the difficulty of the MCDA problems were observed: the incidence of the variability seems to be higher for more difficult MCDA problems (PROSOFI case study: 21 alternatives, 17 criteria and 93 surveys) compared with more straightforward problems (SWH case study: 6 alternatives, 8 criteria and 9 surveys). However, more tests with more case studies are required to generalize this result.
In addition, the approach developed in CRIDE allows for the evaluation of the established criteria set by testing robustness, redundancy, and sensitivity: as evidenced in the present study, by using CRIDE outputs, the setting of the criteria for a particular problem, considered one of the critical aspects of decision-making, could be used instead of conventional statistical tests (e. g., correlation tests, ANOVA, PCA, etc.). However, a more detailed and formal methodological proposition for achieving this task is still required.
 S. Galarza Molina, A. Torres, S. Fajardo y B. Pérez Muzuzu, “Herramienta de análisis multi-criterio como soporte para el diseño del programa social de la Facultad de Ingeniería,” Estud. Gerenc., vol. 27, no. 121, pp. 175–194, 2011. Available: http://bit.ly/2HSTND9
 K. M. Eisenhardt and M. J. Zbaracki, “Strategic decision making,” Strat. Manage. J., vol 13, no. S2, 17–37, 1992. Available: http://bit.ly/2ERk6HV
 D. Ben-Arieh and Zhifeng Chen, “Linguistic-labels aggregation and consensus measure for autocratic decision making using group recommendations,” IEEE Trans. Syst. Man Cybern. A, Syst. Hum., vol. 36, no. 3, pp. 558–568, 2006. doi: 10.1109/TSMCA.2005.853488
 R. Lahdelma, P. Salminen, and J. Hokkanen, “Using multicriteria methods in environmental planning and management,” Environ. Manag., vol. 26, no. 6, pp. 595–605, 2000. doi: 10.1007/s002670010118
 B. Roy, “Paradigms and challenges,” in Int. Ser. Oper. Res. Manag. Sci., J. Figueira, S. Greco, and M. Ehrgott, Eds. Boston: Springer Science + Business Media, 2005 pp. 3–24. https://doi.org/10.1007/0-387-23081-5_1
 K. Brown, W. Adger, E. Tompkins, P. Bacon, D. Shim, and K. Young, “Trade-off analysis for marine protected area management,” Ecol. Econ., vol. 37, no. 3, pp. 417–434, 2001. https://doi.org/10.1016/S0921-8009(00)00293-7
 A. Joubert, A. Leiman, H. de Klerk, S. Katua, and J. Aggenbach, “Fynbos (fine bush) vegetation and the supply of water: A comparison of multi-criteria decision analysis and cost-benefit analysis,” Ecol. Econ., vol. 22, no. 2, pp. 123–140, 1997. https://doi.org/10.1016/S0921-8009(97)00573-9
 J. Gough and J. Ward, “Environmental decision-making and lake management,” J. Environ. Manag., vol. 48, no. 1, pp. 1–15, 1996. https://doi.org/10.1006/jema.1996.0063
 D. Dunning, Q. Ross, and M. Merkhofer, “Multiattribute utility analysis for addressing Section 316(b) of the Clean Water Act,” Environ. Sci. Policy, vol. 3, pp. 7–14, 2000. https://doi.org/10.1016/S1462-9011(00)00022-8
 M. Flug, H. Seitz, and J. Scott, “Multicriteria decision analysis applied to Glen Canyon Dam,” J. Water Resour. Plan. Manag., vol. 126, no. 5, pp. 270–276, 2000. https://doi.org/10.1061/(ASCE)0733-9496(2000)126:5(270)
 R. Nayak and R. Panda, “Integrated management of a canal command in a river Delta using multi-objective techniques,” Water Resour. Manag., vol. 15, no. 6, pp. 383–401, 2001. https://doi.org/10.1023/A:1015593417769
 A. Joubert, T. Stewart, and R. Eberhard, “Evaluation of water supply augmentation and water demand management options for the City of Cape Town,” J. Multi-Criteria Decis. Ana., vol. 12, no. 1, pp. 17–25, 2003. https://doi.org/10.1002/mcda.342
 I. Durbach and T. Stewart, “Modeling uncertainty in multi-criteria decision analysis,” Eur. J. Oper. Res., vol. 223, no. 1, pp. 1–14, 2012. https://doi.org/10.1016/j.ejor.2012.04.038
 B. Roy, “Classement et choix en présence de points de vue multiples,” Rev. Fr. Informat. Rech. O., vol. 2, no. 8, pp. 57–75, 1968. Available: http://www.numdam.org/article/RO_1968__2_1_57_0.pdf
 N. Belacel, “Multicriteria assignment method PROAFTN: Methodology and medical application,” Eur. J. Oper. Res., vol. 125, no. 1, pp. 175–183, 2000. https://doi.org/10.1016/S0377-2217(99)00192-7
 S. Hajkowicz and K. Collins, “A review of multiple criteria analysis for water resource planning and management,” Water Resour. Manag., vol. 21, no. 9, pp. 1553–1566, 2006. https://doi.org/10.1007/s11269-006-9112-5
 C. Chen and W. Hung, “Applying ELECTRE and maximizing deviation method for stock portfolio selection under fuzzy environment,” in Opportunities and Challenges for Next-Generation Applied Intelligence, B. Chien and T. Hong, Ed. Heidelberg: Springer, 2009, pp. 85–91. https://doi.org/10.1007/978-3-540-92814-0_14
 C. Weinhardt and S. Seifert, “Developments in GDN research: Introduction,” Group Decis. Negotiat., vol. 19, no. 2, pp. 107–109, 2009. https://doi.org/10.1007/s10726-009-9185-7
 F. Uzoka, J. Osuji, and O. Obot, “Clinical decision support system (DSS) in the diagnosis of malaria: A case comparison of two soft computing methodologies,” Expert Syst. Appl., vol. 38, no. 3, pp. 1537–1553, 2011. https://doi.org/10.1016/j.eswa.2010.07.068
 F. Cismondi et al., “Multi-stage modeling using fuzzy multi-criteria feature selection to improve survival prediction of ICU septic shock patients,” Expert Syst. Appl., vol. 39, no. 16, pp. 12332–12339, 2012. https://doi.org/10.1016/j.eswa.2012.04.027
 B. Roy and P. Vincke, “Multicriteria analysis: Survey and new directions,” Eur. J. Oper. Res., vol. 8, no. 3, pp. 207–18, 1981. https://doi.org/10.1016/0377-2217(81)90168-5
 H. Barron and C. Schmidt, “Sensitivity analysis of additive multiattribute value models,” Oper. Res., vol. 36, no. 1, pp. 122–127, 1988. https://doi.org/10.1287/opre.36.1.122
 B. Mareschal, “Weight stability intervals in multicriteria decision aid,” Eur. J. Oper. Res., vol. 33, no. 1, pp. 54–64, 1988. https://doi.org/10.1016/0377-2217(88)90254-8
 J. Dyer, P. Fishburn, R. Steuer, J. Wallenius, and S. Zionts, “Multiple criteria decision making, multiattribute utility theory: The next ten years,” Manag. Sci., vol. 38, no. 5, pp. 645–654, 1992. https://doi.org/10.1287/mnsc.38.5.645
 O. Larichev and H. Moshkovich, “ZAPROS-LM: A method and system for ordering multiattribute alternatives,” Eur. J. Oper. Res., vol. 82, no. 3, pp. 503–521, 1995. https://doi.org/10.1016/0377-2217(93)E0143-L
 E. Triantaphyllou and A. Sánchez, “A sensitivity analysis approach for some deterministic multi-criteria decision-making methods,” Decis. Sci., vol. 28, no. 1, pp. 151–194, 1997. https://doi.org/10.1111/j.1540-5915.1997.tb01306.x
 M. Bender and S. Simonovic, “A fuzzy compromise approach to water resource systems planning under uncertainty,” Fuzzy Sets Syst., vol. 115, no. 1, pp. 35–44, 2000. https://doi.org/10.1016/S0165-0114(99)00025-1
 G. Fu, “A fuzzy optimization method for multicriteria decision making: An application to reservoir flood control operation,” Expert Syst. Appl., vol. 34, no. 1, pp. 145–149, 2008. https://doi.org/10.1016/j.eswa.2006.08.021
 J. Skorupski, “Multi-criteria group decision making under uncertainty with application to air traffic safety,” Expert Syst. Appl., vol. 41, no. 16, pp. 7406–7414, 2014. https://doi.org/10.1016/j.eswa.2014.06.030
 R. Mosadeghi, J. Warnken, R. Tomlinson, and H. Mirfenderesk, “Uncertainty analysis in the application of multi-criteria decision-making methods in Australian strategic environmental decisions,” J. Environ. Plan. Manag., vol. 56, no. 8, pp. 1097–1124, 2013. https://doi.org/10.1080/09640568.2012.717886
 K. Hyde, H. Maier, and C. Colby, “Reliability-based approach to multicriteria decision analysis for water resources,” J. Water Resour. Plan. Manag., vol. 130, no. 6, pp. 42–438, 2004. doi: 10.1061/(ASCE)0733-9496(2004)130:6(429)
 P. Moura, S. Barraud, M. Baptista, and F. Malard, “Multicriteria decision-aid method to evaluate the performance of stormwater infiltration systems over the time,” Water Sci. Technol., vol. 64, no. 10, p. 1993–2000, 2011. doi: 10.2166/wst.2011.154
 C. Martin, Y. Ruperd, and M. Legret, “Urban stormwater drainage management: The development of a multicriteria decision aid approach for best management practices,” Eur. J. Oper. Res., vol. 181, no. 1, pp. 338–349, 2007. https://doi.org/10.1016/j.ejor.2006.06.019
 A. Shanian, A. Milani, C. Carson, and R. Abeyaratne, “A new application of ELECTRE III and revised Simos’ procedure for group material selection under weighting uncertainty,” Knowl.-Based Syst., vol. 21, no. 7, pp. 709–720, 2008. https://doi.org/10.1016/j.knosys.2008.03.028
 C. Giannoulis and A. Ishizaka, “A web-based decision support system with ELECTRE III for a personalised ranking of British universities,” Decis. Support Syst., vol. 48, no. 3, pp. 488–497, 2010. https://doi.org/10.1016/j.dss.2009.06.008
 H. Chen, M. Wood, C. Linstead, and E. Maltby, “Uncertainty analysis in a GIS-based multi-criteria analysis tool for river catchment management,” Environ. Model. Softw., vol. 26, no. 4, pp. 395–405, 2011. https://doi.org/10.1016/j.envsoft.2010.09.005
 N. Chen and Z. Xu, “Hesitant fuzzy ELECTRE II approach: A new way to handle multi-criteria decision making problems,” Inf. Sci., vol. 292, pp. 175–197, 2015. DOI: 10.1016/j.ins.2014.08.054
 F. Dong, Y. Liu, H. Su, Z. Liang, R. Zou, and H. Guo, “Uncertainty-based multi-objective decision making with hierarchical reliability analysis under water resources and environmental constraints,” Water Resour. Manag., vol. 30, no. 2, pp. 805–822, 2015. https://doi.org/10.1007/s11269-015-1192-7
 S. Galarza-Molina, A. Torres, P. Moura, and J. Lara-Borrero, “CRIDE: A case study in multi-criteria analysis for decision-making support in rainwater harvesting,” Int. J. Inf. Technol. Decis. Making, vol. 14, no. 1, pp. 43–67, 2015. https://doi.org/10.1142/S0219622014500862
 L. Scholten, N. Schuwirth, P. Reichert, and J. Lienert, “Tackling uncertainty in multi-criteria decision analysis – An application to water supply infrastructure planning,” Eur. J. Oper. Res., vol. 242, no. 1, pp. 243–260, 2015. https://doi.org/10.1016/j.ejor.2014.09.044
 A. Afsordegan, M. Sánchez, N. Agell, S. Zahedi, and L. Cremades, “Decision making under uncertainty using a qualitative TOPSIS method for selecting sustainable energy alternatives,” Int. J. Environ. Sci. Technol., vol. 13, no. 6, pp. 1419–1432, 2016. https://doi.org/10.1007/s13762-016-0982-7
 R. Smith, G. Poveda, O. Mesa, D. Valencia, I. Dyner y P. Jaramillo, Decisiones con multiples objetivos e incertidumbre, 2nd ed. Medellín: Universidad Nacional de Colombia, 2000.
 P. Martin, Sustainable Urban Drainage Systems. London: Construction Industry Research and Information Association, 2001.
 Departamento Administrativo Nacional de Estadística (DANE), Censo general 2005 a nivel nacional, DANE, 2005. Available: http://bit.ly/2JT1xaB
 Cámara de Comercio de Bogotá, Observatorio de Seguridad en Bogotá no. 37. Balance enero-junio de 2009. Bogotá: Cámara de Comercio de Bogotá, 2009. Available: http://bit.ly/2Kvegzw
 Subsecretaría de Planeación Socioeconómica, Encuesta Calidad de Vida 2007. Bogotá 2007. Bogotá: Subsecretaría de Planeación Socioeconómica, 2007. Available: http://bit.ly/2Kq4GOz
 B. Roy and P. Bertier, “La méthode ELECTRE II: Une méthode de classement en présence de critères multiples,” in Note de Travail, no. 142, SEMA (Metra International), Direction Scientifique. Paris: Grupo Metra, 1971, pp. 291–302.
 J. Figueira, V. Mousseau, and B. Roy, “ELECTRE methods,” in Mult. Criteria Decis. Anal. State Art Surveys, J. Figueira, G. Salvatore, and M. Ehrgott, Ed. Boston: Springer Science & Business Media, 2005, pp. 133–162. https://doi.org/10.1007/0-387-23081-5_4
 D. Davis, The Development of Celtic Linguistics, 1850-1900. London: Routledge, 2002.
 Nera.com, 2018. [Online]. Available: http://bit.ly/2WzAOG7
a Corresponding author. E-mail: firstname.lastname@example.org
How to cite this
article: S. Galarza-Molina
and A. Torres, “Including uncertainties in multi-criteria decision-making: A
superfluous task or a requirement?,” Ing. Univ. vol. 23, no. 1, 2019