477 Ingeniería y Universidad Ing. Univ. 0123-2126 2011-2769 Pontificia Universidad Javeriana Colombia revistascientificasjaveriana@gmail.com 47760079002 https://doi.org/10.11144/Javeriana.iyu23-1.iumd Artículos Including Uncertainties in Multi-Criteria Decision-Making: A Superfluous Task or a Requirement?* Incluyendo incertidumbres en la toma de decisiones multi-criterio: ¿una tarea superflua o un requerimiento? http://orcid.org/0000-0002-0002-1915 Galarza-Molina Sandra a sgalarza@javeriana.edu.co http://orcid.org/0000-0001-8693-8611 Torres Andrés Universidad Javeriana, Colombia Universidad Javeriana Colombia Universidad Javeriana, Colombia Universidad Javeriana Colombia a Corresponding author. E-mail: sgalarza@javeriana.edu.co Enero-Junio 2019 23 1 1 25 04 04 2018 06 12 2018 24 06 2019 https://creativecommons.org/licenses/by/4.0/ Esta obra está bajo una Licencia Creative Commons Atribución 4.0 Internacional. Abstract

Objective: This paper aims to assess cases where it is necessary to include uncertainties in decision-making input data. Materials and Methods: Three case studies with different numbers of alternatives and different numbers and natures of criteria were evaluated. The CRIDE tool was used to include uncertainties in the input data. Results and Discussion: The results obtained showed that for more difficult decision-making problems, the inclusion of variations in input data could change the final decision, while for less challenging problems, it is unnecessary to take uncertainties into account. Conclusions: These findings could be useful for decision-makers in obtaining more accurate results or in saving time and money related to input data acquisition.

Resumen

Objetivo: El objetivo de este artículo es evaluar en qué casos es necesario incluir incertidumbres en la información necesaria para la toma de decisiones. Materiales y métodos: Tres casos de estudio con un número diferente de alternativas y diferente número y natrualeza de criterios fueron evaluados. La herramienta CRIDE se utilizó para incluir las incertidumbres en la información de entrada. Resultados y discusión: Los resultados obtenidos muestran que para problemas de toma de decisión más complejos, la inclusión de variaciones en los datos de entrada pueden incidir en la decisión final, mientras que para problemas menos difíciles es innecesario tomar en cuenta dichas incertidumbres. Conclusiones: Estos hallazgos pueden resultar útiles para los tomadores de decisiones en cuanto a la obtención de resultados más precisos o con el propósito de ahorrar tiempo y dinero relacionado con la adquisición de la información necesaria para la toma de decisiones.

Keywords Multicriteria decision-making uncertainties robustness sensitivity and redundancy analysis Palabras clave toma de decisiones multicriterio incertidumbres análisis de robustez sensibilidad y redundancia How to cite this article S. Galarza-Molina and A. Torres, “Including uncertainties in multi-criteria decision-making: A superfluous task or a requirement?,” Ing. Univ. vol. 23, no. 1, 2019 [Online]. https://doi.org/10.11144/Javeriana.iyu23-1.iumd Introduction

Decision-making is a frequent and essential activity for any organization [1]. We frequently understand that, independent of organization size, importance or economic sector, decision-making is a day-to-day task necessary for operational, tactical and strategic purposes. According to [2], this task is crucial because it involves fundamental decisions that shape the course of a firm. Developing tools such as explicit models, whether formalized or not, can help decision makers answer the questions that arise during this process [3]. These tools help to illustrate possible solutions and generate a recommendation to increase the coherence between the development of a process and the goals of the stakeholders [4], [5]. The word recommendation is used to make both the analyst and the person or group of people who make the decision aware that they are free to follow the suggestions or not [5]. Collaborative decision-making aims to reach a high level of consensus to achieve unity and ownership, thus avoiding a simple vote [3]. Hence, to improve decision-making transparency, auditability, and analytic rigor, it is necessary to use a method that supports this process [6], [7]. One of the options cited by several researchers is Multi-Criteria Decision Analysis (MCDA) [4], [8], [9], [10], [11], [12], [13], which was developed in the 1960s [14]. Since then, this technique has been the subject of numerous research projects in fields such as medicine, engineering, finance, and economics [15], [16], [17], [18], [19], [20].

One aspect of MCDA that some authors have explored is the handling of uncertainty in data and parameter values [13], [21], [22], [23], [24], [25], [26], [27], [28], [29]. Potential sources of uncertainty come from the definitions of alternative performance and the criteria weights, which is one of the most challenging steps in the MCDA process [21], [25]. Some methods have been developed to address these uncertainties [22], [23], [26], [27], [28], [30], [31]. While most of the methodologies employ statistical tools to evaluate uncertainty in some of the input data, they do not account for the thresholds included in most MCDA concerning all the uncertainties of the input data. Additionally, few methodologies can assess the quality criteria (performance values and weights), which are considered key aspects of decision-making problems [32].

Although many MCDA methods, including uncertainties, have arisen during the past few years for a wide range of applications [33], [34], [35], [36], [37], [38], [39], [40], [41], it seems necessary to determine which type of problems require the inclusion of uncertainties as input data when assessing an MCDA. Additionally, MCDA problem characterization, which could define each factor if deserving of uncertainty inclusion, is lacking, as well as sufficiently flexible tools to evaluate these factors.

This paper aims to identify which type of MCDA problems should include uncertainties in a decision-making process.

Methods Case Studies

For this study, we used two real-world decision-making case studies and a hypothetical case with differing numbers of alternatives, criteria, and surveys. The first case study is the decision-making process for Stormwater Harvesting (SWH) at the Pontificia Universidad Javeriana Bogotá (PUJ) [39]. The second case study is the selection of the target community of the PUJ Social Program PROSOFI (from its Spanish acronym) [1]. The third case is an example from the book “Decisiones con múltiples objetivos e incertidumbres” [42] related to the selection of alternatives for sustainable development management of a small catchment.

Case Study 1: PUJ Campus Stormwater Harvesting

For this case study, six scenarios for SWH were proposed on the campus of PUJ. The authors suggested eight criteria to evaluate each scenario. The University’s Physical Resources Division (PRD) defined the criteria based on a set proposed by [33]. Within the criteria set, technical aspects, such as hydraulic and quality performances, management aspects, such as compatibility with the University’s Master Plan, and financial aspects, such as net present value (NPV) of the investment, were considered. We used a survey applied to four leaders from the PRD and five experts on water management to define the criteria weights (table 1). For more details of this study, see [39].

SWH – Criteria and weights as defined by survey participants Table 1 SWH – Criteria and weights as defined by survey participants Source: Galarza-Molina et al. [39]

Galarza-Molina et al. [39] built the scenarios or possible alternatives based on a division of the PUJ campus in twelve basins. They proposed the use of SUDS (Sustainable Urban Drainage Systems) [43] for the collection and treatment of the stormwater in the basins, including bioretention gardens, permeable paving, and constructed wetlands.

The input scenarios emerged from an ideal scenario where all the water harvested in the campus would be used (maximum supply volume) with drinking water quality (maximum water quality). The other scenarios were created by decreasing the water quality (fewer possible uses) and the supply (fewer basins included), obtaining high, medium and low water demands and high, medium and low water supplies. The above considerations allowed sizing of the SUDS considered and the pumping and treatment systems. Table 2 shows the description of the scenarios.

Proposed stormwater harvesting scenarios Table 2 Proposed stormwater harvesting scenarios Source: Galarza-Molina et al. [39]

As a result, [39] constructed table 3, with eight criteria and six scenarios (8 × 6 matrix) with minimum, average and maximum values per criterion and scenario.

SWH – Matrix of alternatives Table 3 SWH – Matrix of alternatives Source: Galarza-Molina et al. [39]

⇑ means that performance of the alternative with respect to the specific criteria increases as the criterion value also increases.

⇓ means that performance of the alternative with respect to the specific criteria dicreases as the criterion value increases.

 Case Study 2: Selection of PROSOFI Target Community

For this case study, [1] defined the 21 alternatives (19 Bogotá urban districts plus two municipalities near Bogotá) along with criteria and weighting factors based on a participatory project of the university. [1] developed a survey applied to 143 faculty members to determine the criteria and weighting factors. The survey asked which essential aspects should be considered for the selection of the PROSOFI target community and which of these aspects were more relevant than others. [1] analyzed the results from the survey (111 persons, including directors, faculty members, and administrative staff) to find inconsistencies, using a script that compared the answers of each person. Table 4 shows the selection criteria for each key aspect [1].

PROSOFI – Aspects and criteria for the selection of the target community Table 4 PROSOFI – Aspects and criteria for the selection of the target community Source: Galarza Molina et al. [1]

* Pontificia Universidad Javeriana is part of the Society of Jesus.

** Colombia has six socioeconomic strata, where 6 an 1 are the most and least affluent, respectively.

*** SISBEN: Colombian system for identification of potential beneficiaries of social programs.

Galarza Molina et al. [1] constructed the matrix of alternatives using the defined criteria and the evaluation of alternatives. They obtained information from the 2005 Bogotá census [44], Bogotá’s Chamber of Commerce [45], the Undersecretary’s office for planning Economic and Social Affairs [46] and the experience of experts. As a result, this case study has a 17 × 21 matrix (see table 5, one part of the decision matrix).

One part of the PROSOFI matrix of alternatives Table 5 One part of the PROSOFI matrix of alternatives Source: Galarza Molina et al. [1]

Case Study 3: Book Example

With the aim of comparing different types of decision problems, we used an example from the book Decisiones con múltiples objetivos e incertidumbres by [42]. The decision-making process here selects the best alternative for sustainable development management in a 250-ha catchment. The authors proposed seven criteria to evaluate the alternatives (table 6).

Book example criteria Table 6 Book example criteria Source: adapted from Smith et al. [42]

Table 7 shows the probability distributions and weights of the criteria and the matrix of alternatives (7 × 7 matrix). In this table are the minimum, average and maximum values per criterion and scenario. [42] determined the average values of the evaluation matrix and the criteria weights. For the minimum and maximum values, we defined them with variations of 1,7% or 7,8%, and the probability distributions of the criteria were chosen arbitrarily.

Book example - Matrix of alternatives, criteria weights, and probability distributions ($: Colombian pesos) Table 7 Book example - Matrix of alternatives, criteria weights, and probability distributions ($: Colombian pesos) Source: adapted from Smith et al. [42]

⇑ means that performance of the alternative for the specific criteria increases as the criterion value also increases.

⇓ means that performance of the alternative for the specific criteria dicreases as the criterion value increases.

# means the number of new employees (criterion 3) or the number of new tourism businesses (criterion 6).

 Summary of the Case Studies

Table 8 illustrates the differences among the three case studies.

Summary of three case studies Table 8 Summary of three case studies Source: author’s own elaboration

Decision-Making Tools

ELECTRE II is part of the family of methods collectively known as ELECTRE (ELimination Et Choix Traduisant la REalité – ELimination and Choice Expressing REality). Proposed in the late 1960s by [47], it is considered the first method dealing with problem ranking [48]. We chose this method because (i) it enables a dialogue between various stakeholders involved in the decision-making process but provides input/voice differences between them by means of a criteria weighting procedure [33]; (ii) it allows for pairwise comparison and direct ranking using binary outranking relations, which can be particularly useful for negotiation processes; (iii) it handles criteria measured in different units without having to use numerical-scale conversions with identical ranges (a point of contrast with ELECTRE I) [48], [33]; and (iv) it uses only concordance and discordance levels, a practical benefit compared with the more elaborated ELECTRE III, which uses three categories of threshold—indifference, preference, and veto [48], [33] .

The CRIDE (multiCRIteria DEcision support tool —CRIDE is a Celtic word that means heart— [49]) tool, based on the multi-criteria analysis (MCA) method ELECTRE II, incorporates Monte Carlo simulations to include uncertainties such as expert opinion variability and imprecision of the different criteria involved in the decision. Galarza-Molina et al. [39] used a Monte Carlo simulation in CRIDE to create random values in the evaluation matrix, the criteria weights, and the concordance and discordance levels using normal or uniform probability distributions.

 Initiated Tests

We undertook the following analyses based on recommendations in [32]: (i) We used a robustness test to measure the tool capacity to produce results with the same trends, regardless of the number of criteria used in the matrix of alternatives [32]. We ran CRIDE without one criterion for each execution (the criteria were removed one by one), and then we evaluated the information that gives each criterion by estimating its incidence on the results variability by counting the number of noncoincidences per execution and dividing each one by the total sum of noncoincidences. (ii) The redundancy test consists of evaluating which criterion is unnecessary [50]; we used ELECTRE II to remove pairs of criteria and computed the redundancy indicator as the sum of the differences between the alternative positions obtained with ELECTRE II and CRIDE. (iii) We used the sensitivity analysis to produce slight weight variations (i. e., we used the average criterion for CRIDE).

The data used for each case study for ELECTRE II were (i) the average criteria weights, (ii) the average values of the matrix of alternatives and (iii) concordance levels of cmin,1 = 0,7 > cmin,2 = 0,6 > cmin,3 = 0,50. We defined the discordance levels (two per criteria) as the average between the maximum differences of each criterion. We chose the following thresholds arbitrarily: 20th percentile for the first level (dmax,1) and 35th percentile for the second level (dmax,2).

In contrast, the data used for CRIDE were (i) all the results from the criteria weight survey, (ii) the minimum and maximum values of the matrix of alternatives and (iii) the random concordance values. For the discordance levels, we used the same definition described previously.

 Results and Discussion

Figure 1 shows the general results generated by ELECTRE II and CRIDE for the three cases in this study. The CRIDE results are called the “reference execution”, which means that the results were obtained by applying CRIDE with all possible variations.

The results for SWH (figure 1a) were almost the same, except for the third and fourth places. The tools suggested that the more suitable solutions were S5 and S6 (first and second places). According to figure 1b, most PROSOFI alternatives changed their ranking positions, except Fontibón and Teusaquillo. The PROSOFI results from CRIDE (figure 1b) suggest that the more suitable positions, in decreasing preference order, are Santa Fe, Zipaquirá, Candelaria, Mártires and San Cristobal, while ELECTRE II suggested a ranking of Usme, Soacha, Zipaquirá, Santa Fe and Ciudad Bolívar. The common alternatives between these results were Zipaquirá and Santa Fe. For the third case study using the Book example (figure 1c), the results show that the more suitable solutions are alternatives A4 and A3 (first and second places), the worst one was A7, and the alternatives A1, A5 and A6 did not change their ranking places. Initially, these results showed that the alternatives’ positions of the PROSOFI case study and the Book example tended to have more changes in their ranking places than the SWH case study. In other words, for PROSOFI and the Book example, it seems that uncertainties could be relevant in the decision-making process. It was necessary to know the level of complexity of each case to generalize this result by testing for robustness, redundancy, and sensitivity.

Results obtained from CRIDE and ELECTRE II: (a) SWH results; (b) PROSOFI results; (c) Book example results Figure 1 Results obtained from CRIDE and ELECTRE II: (a) SWH results; (b) PROSOFI results; (c) Book example results Source: author’s own elaboration

Results obtained from CRIDE and ELECTRE II: (a) SWH results; (b) PROSOFI results; (c) Book example results Figure 1 (Cont.) Results obtained from CRIDE and ELECTRE II: (a) SWH results; (b) PROSOFI results; (c) Book example results Source: author’s own elaboration

Results obtained from CRIDE and ELECTRE II: (a) SWH results; (b) PROSOFI results; (c) Book example results Figure 1 (Cont.) Results obtained from CRIDE and ELECTRE II: (a) SWH results; (b) PROSOFI results; (c) Book example results Source: author’s own elaboration

Robustness Test

Figure 2 illustrates the results of the robustness test for the SWH (figure 2a) and Book example (figure 2b) case studies, and figure 3 and table 9 show the PROSOFI case study. The noncoincidence factors are in parenthesis: high values indicate more changes in the alternatives’ ranking places. The tables in figure 2 indicate the ranking places that do not change for each execution. For the SWH case study (figure 2a), the results indicate that it is necessary to include criteria 7 (net present value) and 8 (project’s internal rate of return [IRR] vs. another project’s IRR) to determine the best alternative. However, if the objective is to rank the first three places, it is necessary to consider criteria 4 (frequency and need of operation and maintenance), 7 and 8 (financial criteria), as these criteria have the highest noncoincidence factors (28,57%). For the Book example case study (figure 2b), it is necessary to include criterion 7 (investment) (noncoincidence factor of 19,35%) to determine the best alternative, while if the objective is the ranking of the first three suitable alternatives, all criteria appear to be necessary.

Alternatives’ positions without a criterion each time: (a) Stormwater harvesting, (b) Book example. The noncoincidence factors are in parentheses Figure 2 Alternatives’ positions without a criterion each time: (a) Stormwater harvesting, (b) Book example. The noncoincidence factors are in parentheses Source: author’s own elaboration

Alternatives’ positions without a criterion each time: (a) Stormwater harvesting, (b) Book example. The noncoincidence factors are in parentheses Figure 2 (Cont.) Alternatives’ positions without a criterion each time: (a) Stormwater harvesting, (b) Book example. The noncoincidence factors are in parentheses Source: author’s own elaboration

For the PROSOFI case study (figure 3 and table 9), the robustness test shows that (i) to determine the best alternative, it is necessary to take into account criteria 1 (travel time between PUJ campus and PROSOFI community) and 2 (availability of public transportation), with noncoincidence factors of 5,83% and 6,73%, respectively. (ii) To determine the first three alternatives, one must account for all the criteria except criteria 4 (previous presence of The Society of Jesus), 16 (existence of micro, small and medium enterprises) and 17 (displaced population). (iii) To determine the first five alternatives, one must account for all the criteria except criterion 4; (iv) the execution “Without Criterion 10” (private schools) has the lowest noncoincidence factor (4,04%), which indicates that this execution produces fewer options (see table 9).

PROSOFI alternatives’ positions without a criterion each time; noncoincidence factors are in parentheses Figure 3 PROSOFI alternatives’ positions without a criterion each time; noncoincidence factors are in parentheses Source: author’s own elaboration

The number of places to be maintained without criterion each time for PROSOFI case study; the gray column indicates the execution “Without Criterion 10” (private schools) Table 9 The number of places to be maintained without criterion each time for PROSOFI case study; the gray column indicates the execution “Without Criterion 10” (private schools) Source: author’s own elaboration

These results show that all the criteria for the PROSOFI and Book example provide relevant information. In contrast, for SWH, some criteria could be removed without affecting the ranking results.

 Redundancy Test

The second test considered was the redundancy of the criteria set. For the SWH case study (figure 4a), the pairs of criteria that produce the same results are 2-3 and 2-4, with a redundancy indicator equal to 8, and pairs 4-8, 5-6, 5-7, and 5-8, with a redundancy indicator equal to 10.

For the PROSOFI case study, ELECTRE II was executed without each pair of criteria. As a result, of all the combinations of the 17 criteria, we analyzed 136 pairs (figure 4b): Table 10 shows the 85 pairs that produce the same redundancy indicator results.

For the Book example case study (figure 4c), the pairs that produce the same results are 1-2, 3-6, 3-7 and 5-6, with a redundancy indicator equal to 14; 1-3 and 1-6, with a redundancy indicator equal to 20; 1-4 and 1-7, with a redundancy indicator equal to 16; 2-3 and 4-7, with a redundancy indicator equal to 6; 2-4, 2-5, 2-6, 2-7, 3-5, 4-5 and 4-6, with a redundancy indicator equal to 8; and 5-7 and 6-7, with a redundancy indicator equal to 12.

Redundancy indicator vs. criteria pairs; the dots indicate pairs that have the same redundancy indicator: (a) SWH; (b) PROSOFI; (c) Book example Figure 4 Redundancy indicator vs. criteria pairs; the dots indicate pairs that have the same redundancy indicator: (a) SWH; (b) PROSOFI; (c) Book example Source: author’s own elaboration

Redundancy indicator vs. criteria pairs; the dots indicate pairs that have the same redundancy indicator: (a) SWH; (b) PROSOFI; (c) Book example Figure 4 (Cont.) Redundancy indicator vs. criteria pairs; the dots indicate pairs that have the same redundancy indicator: (a) SWH; (b) PROSOFI; (c) Book example Source: author’s own elaboration

Redundancy indicator vs. criteria pairs; the dots indicate pairs that have the same redundancy indicator: (a) SWH; (b) PROSOFI; (c) Book example Figure 4 (Cont.) Redundancy indicator vs. criteria pairs; the dots indicate pairs that have the same redundancy indicator: (a) SWH; (b) PROSOFI; (c) Book example Source: author’s own elaboration

The possible redundant pairs (table 10) were removed, and CRIDE was executed to compare the results obtained with the reference execution of each case study (dashed lines in figure 5). Then, we evaluated noncoincidence factors. The pairs that present the same coincidence factor and produce the same results are possible candidates for redundant criteria.

According to figure 5a (SWH), the pairs of criteria 5-7 and 5-8 produced the same results with a noncoincidence factor of 25%. Criteria 7 (net present value) and 8 (Project’s IRR vs. another project’s IRR) are the criteria that evaluate the alternatives at the financial aspect. Therefore, we can conclude that these criteria are redundant when applied in the present case study.

The 85 criteria pairs that produce similar results for the PROSOFI case study Table 10 The 85 criteria pairs that produce similar results for the PROSOFI case study Source: author’s own elaboration

For the PROSOFI case study (figure 5b), it was difficult to identify the pairs that present the same noncoincidence factor and produce the same results: from numerical results obtained and saved in files (not shown in this paper), it can be observed that none of the pairs produced the same results, with the exception of the pairs of criteria 3-17 and 16-17, which produced the same six first ranking results (criterion 3: previous presence of the School of Engineering; criterion 16: existence of micro, small and medium enterprises). Therefore, criteria 3 and 16 would be redundant if we want to know the first six ranking results.

For the Book example case study (figure 5c), none of the pairs produced the same results. Therefore, all the criteria for the alternatives ranking are needed.

On the other hand, it was interesting to understand why criteria 3 and 16 (figure 6) were redundant for the PROSOFI case study. At first glance, there is no clear relation between these criteria, such as in the SWH with criteria 7 and 8 (because both are financial aspects). A Pearson correlation test (after verifying normal distribution and homogeneity of variance for both criteria) shows no significant correlation (p-value > 0,05) between criterion 3 and criterion 16.

Alternatives’ positions without a pair of criteria each time (redundancy test); noncoincidence factors are in parentheses: (a) Stormwater harvesting, (b) PROSOFI, (c) Book example Figure 5 Alternatives’ positions without a pair of criteria each time (redundancy test); noncoincidence factors are in parentheses: (a) Stormwater harvesting, (b) PROSOFI, (c) Book example Source: author’s own elaboration

Alternatives’ positions without a pair of criteria each time (redundancy test); noncoincidence factors are in parentheses: (a) Stormwater harvesting, (b) PROSOFI, (c) Book example Figure 5 (Cont.) Alternatives’ positions without a pair of criteria each time (redundancy test); noncoincidence factors are in parentheses: (a) Stormwater harvesting, (b) PROSOFI, (c) Book example Source: author’s own elaboration

Alternatives’ positions without a pair of criteria each time (redundancy test); noncoincidence factors are in parentheses: (a) Stormwater harvesting, (b) PROSOFI, (c) Book example Figure 5 (Cont.) Alternatives’ positions without a pair of criteria each time (redundancy test); noncoincidence factors are in parentheses: (a) Stormwater harvesting, (b) PROSOFI, (c) Book example Source: author’s own elaboration

Minimum and maximum values for criteria 3 (previous presence of others PUJ schools) and 16 (existence of micro, small and medium enterprises) Figure 6 Minimum and maximum values for criteria 3 (previous presence of others PUJ schools) and 16 (existence of micro, small and medium enterprises) Source: author’s own elaboration

Sensitivity Analysis

Finally, we performed a sensitivity analysis (figure 7). For the SWH case study, the results did not show significant variations compared to the reference execution (figure 7a), which means that the decision-maker can use the average weight of one of the criteria to know the first three positions for a multi-criteria decision-making ranking problem. If criteria 2, 3, 4, 5 and 8 (with noncoincidence factors of 20%) were fixed, alternatives S2 and S3 would change their positions.

For the PROSOFI case study (figure 7b), the results show the fixed weights of criteria 3 (previous presence of the School of Engineering), 9 (public schools), 11 (relation: public schools/total Schools) and 13 (population density) change second and third places: the noncoincidence factors are 8,57% (criterion 3) and 4,76% (criteria 9, 11 and 13). The ranking places that do not change are 1st, 4th, 5th, 8th, 19th, 20th and 21st.

In contrast, for the Book example case study (figure 7c), if any criterion was fixed, all the ranking results would change (with noncoincidence factors of 14,29%). This case study is sensitive to slight changes in the criteria weights, in contrast to the other case studies.

Alternatives’ positions leaving each criterion fixed (Sensitivity analysis): (a) Stormwater harvesting, (b) PROSOFI, (c) Book example; noncoincidence factors are in parentheses Figure 7 Alternatives’ positions leaving each criterion fixed (Sensitivity analysis): (a) Stormwater harvesting, (b) PROSOFI, (c) Book example; noncoincidence factors are in parentheses Source: author’s own elaboration

Alternatives’ positions leaving each criterion fixed (Sensitivity analysis): (a) Stormwater harvesting, (b) PROSOFI, (c) Book example; noncoincidence factors are in parentheses Figure 7 (Cont.) Alternatives’ positions leaving each criterion fixed (Sensitivity analysis): (a) Stormwater harvesting, (b) PROSOFI, (c) Book example; noncoincidence factors are in parentheses Source: author’s own elaboration

Alternatives’ positions leaving each criterion fixed (Sensitivity analysis): (a) Stormwater harvesting, (b) PROSOFI, (c) Book example; noncoincidence factors are in parentheses Figure 7 (Cont.) Alternatives’ positions leaving each criterion fixed (Sensitivity analysis): (a) Stormwater harvesting, (b) PROSOFI, (c) Book example; noncoincidence factors are in parentheses Source: author’s own elaboration

The first test (robustness) showed that for the PROSOFI and Book example, all the criteria provide relevant information because they have a high incidence on the variability of the results. Almost the same result occurred with the second test (redundancy): Most of the criteria appear to have a high incidence on the variability of the results. For the last test (sensitivity analysis), it was observed that the Book example is sensitive to slight changes on the weight criteria, in contrast to case studies (a) SWH and (b) PROSOFI. These results show that including uncertainties in the matrix of alternatives for a case study with more alternatives, criteria, and surveys can affect the variability of the results.

The above findings suggest that it is better to include uncertainty (e. g., CRIDE) in the decision-making process for case studies with a substantial number of alternatives and criteria. Additionally, it is important not to include criteria that could introduce redundant information as a basis to simplify decision-making problems, as was the case for the SWH case study (e. g., criteria 7 and 8 of the SWH case study).

 Conclusions

This paper aimed to identify when it is useful to include uncertainties in a decision-making process for MCDA problems, based on ELECTRE II and CRIDE tools and applying robustness, redundancy and sensitivity tests for two real-world decision-making case studies and a hypothetical case, with differing numbers of alternatives, criteria, and surveys.

The results show that for a simple decision-making problem such as the SWH case study, it is not necessary to include uncertainties, and a simple MCDA tool, as ELECTRE II, seems to be sufficient. This statement implies that, for this case, time and money could be saved during the collection of data because there is no need to consider variations. However, for the other problems analyzed, with more alternatives, criteria, and surveys, the inclusion of uncertainties seems to be necessary, and a tool such as CRIDE could be used to manage this scenario. This implies that, for these cases, a more comprehensive collection of information to fill the matrix of alternatives is needed. Some trends relating to the incidence of the variability and the difficulty of the MCDA problems were observed: the incidence of the variability seems to be higher for more difficult MCDA problems (PROSOFI case study: 21 alternatives, 17 criteria and 93 surveys) compared with more straightforward problems (SWH case study: 6 alternatives, 8 criteria and 9 surveys). However, more tests with more case studies are required to generalize this result.

In addition, the approach developed in CRIDE allows for the evaluation of the established criteria set by testing robustness, redundancy, and sensitivity: as evidenced in the present study, by using CRIDE outputs, the setting of the criteria for a particular problem, considered one of the critical aspects of decision-making, could be used instead of conventional statistical tests (e. g., correlation tests, ANOVA, PCA, etc.). However, a more detailed and formal methodological proposition for achieving this task is still required.

 References [1] S. Galarza Molina, A. Torres, S. Fajardo y B. Pérez Muzuzu, “Herramienta de análisis multi-criterio como soporte para el diseño del programa social de la Facultad de Ingeniería,” Estud. Gerenc., vol. 27, no. 121, pp. 175–194, 2011. Available: http://bit.ly/2HSTND9 Galarza Molin S. Torres A. Fajardo S. Pérez Muzuzu B. Herramienta de análisis multi-criterio como soporte para el diseño del programa social de la Facultad de Ingeniería Estud. Gerenc. 2011 27 121 175 194 Available: http://bit.ly/2HSTND9 [2] K. M. Eisenhardt and M. J. Zbaracki, “Strategic decision making,” Strat. Manage. J., vol 13, no. S2, 17–37, 1992. Available: http://bit.ly/2ERk6HV Eisenhardt K.M. Zbarack M. J. Strategic decision making Strat. Manage. J., 1992 13 S2 17 37 Available: http://bit.ly/2ERk6HV [3] D. Ben-Arieh and Zhifeng Chen, “Linguistic-labels aggregation and consensus measure for autocratic decision making using group recommendations,” IEEE Trans. Syst. Man Cybern. A, Syst. Hum., vol. 36, no. 3, pp. 558–568, 2006. doi: 10.1109/TSMCA.2005.853488 Ben-Arieh D. Chen Zhifeng Linguistic-labels aggregation and consensus measure for autocratic decision making using group recommendations IEEE Trans. Syst. Man Cybern. A, Syst. Hum. 2006 36 3 558 568 10.1109/TSMCA.2005.853488 [4] R. Lahdelma, P. Salminen, and J. Hokkanen, “Using multicriteria methods in environmental planning and management,” Environ. Manag., vol. 26, no. 6, pp. 595–605, 2000. doi: 10.1007/s002670010118 Lahdelma R. Salminen P. Hokkanen J. Using multicriteria methods in environmental planning and management Environ. Manag 2000 26 6 595 605 10.1007/s002670010118 [5] B. Roy, “Paradigms and challenges,” in Int. Ser. Oper. Res. Manag. Sci., J. Figueira, S. Greco, and M. Ehrgott, Eds. Boston: Springer Science + Business Media, 2005 pp. 3–24. https://doi.org/10.1007/0-387-23081-5_1 Roy B. Figueira J. Greco S. Ehrgott M. Int. Ser. Oper. Res. Manag. Sci. 2005 3 24 Boston Springer Science + Business Media Paradigms and challenges https://doi.org/10.1007/0-387-23081-5_1 [6] K. Brown, W. Adger, E. Tompkins, P. Bacon, D. Shim, and K. Young, “Trade-off analysis for marine protected area management,” Ecol. Econ., vol. 37, no. 3, pp. 417–434, 2001. https://doi.org/10.1016/S0921-8009(00)00293-7 Brown K. Adger W. Tompkins E. Bacon P. Shim D. Young K. Trade-off analysis for marine protected area management Ecol. Econ 2001 37 3 417 434 https://doi.org/10.1016/S0921-8009(00)00293-7 10.1016/S0921-8009(00)00293-7 [7] A. Joubert, A. Leiman, H. de Klerk, S. Katua, and J. Aggenbach, “Fynbos (fine bush) vegetation and the supply of water: A comparison of multi-criteria decision analysis and cost-benefit analysis,” Ecol. Econ., vol. 22, no. 2, pp. 123–140, 1997. https://doi.org/10.1016/S0921-8009(97)00573-9 Joubert A. Leiman A. de Klerk H. Katua S. Aggenbach J. Fynbos (fine bush) vegetation and the supply of water: A comparison of multi-criteria decision analysis and cost-benefit analysis, Ecol. Econ 1997 22 2 123 140 https://doi.org/10.1016/S0921-8009(97)00573-9 10.1016/S0921-8009(97)00573-9 [8] J. Gough and J. Ward, “Environmental decision-making and lake management,” J. Environ. Manag., vol. 48, no. 1, pp. 1–15, 1996. https://doi.org/10.1006/jema.1996.0063 Gough J. Ward J. Environmental decision-making and lake management J. Environ. Manag 1996 48 1 1 15 https://doi.org/10.1006/jema.1996.0063 10.1006/jema.1996.0063 [9] D. Dunning, Q. Ross, and M. Merkhofer, “Multiattribute utility analysis for addressing Section 316(b) of the Clean Water Act,” Environ. Sci. Policy, vol. 3, pp. 7–14, 2000. https://doi.org/10.1016/S1462-9011(00)00022-8 Dunning D. Ross Q. Merkhofer M. Multiattribute utility analysis for addressing Section 316(b) of the Clean Water Act Environ. Sci. Polic 2000 3 7 14 https://doi.org/10.1016/S1462-9011(00)00022-8 10.1016/S1462-9011(00)00022-8 [10] M. Flug, H. Seitz, and J. Scott, “Multicriteria decision analysis applied to Glen Canyon Dam,” J. Water Resour. Plan. Manag., vol. 126, no. 5, pp. 270–276, 2000. https://doi.org/10.1061/(ASCE)0733-9496(2000)126:5(270) Flug M. Seitz H. Scott J. Multicriteria decision analysis applied to Glen Canyon Dam J. Water Resour. Plan. Manag 2000 126 5 270 276 https://doi.org/10.1061/(ASCE)0733-9496(2000)126:5(270) 10.1061/(ASCE)0733-9496(2000)126:5(270) [11] R. Nayak and R. Panda, “Integrated management of a canal command in a river Delta using multi-objective techniques,” Water Resour. Manag., vol. 15, no. 6, pp. 383–401, 2001. https://doi.org/10.1023/A:1015593417769 Nayak R. Panda R. Integrated management of a canal command in a river Delta using multi-objective techniques Water Resour. Manag 2001 15 6 383 401 https://doi.org/10.1023/A:1015593417769 10.1023/A:1015593417769 [12] A. Joubert, T. Stewart, and R. Eberhard, “Evaluation of water supply augmentation and water demand management options for the City of Cape Town,” J. Multi-Criteria Decis. Ana., vol. 12, no. 1, pp. 17–25, 2003. https://doi.org/10.1002/mcda.342 Joubert A. Stewart T. Eberhard R. Evaluation of water supply augmentation and water demand management options for the City of Cape Town J. Multi-Criteria Decis. Ana 2003 12 1 17 25 https://doi.org/10.1002/mcda.342 10.1002/mcda.342 [13] I. Durbach and T. Stewart, “Modeling uncertainty in multi-criteria decision analysis,” Eur. J. Oper. Res., vol. 223, no. 1, pp. 1–14, 2012. https://doi.org/10.1016/j.ejor.2012.04.038 Durbach I. Stewart T. Modeling uncertainty in multi-criteria decision analysis Eur. J. Oper. Res. 2012 223 1 1 14 https://doi.org/10.1016/j.ejor.2012.04.038 10.1016/j.ejor.2012.04.038 [14] B. Roy, “Classement et choix en présence de points de vue multiples,” Rev. Fr. Informat. Rech. O., vol. 2, no. 8, pp. 57–75, 1968. Available: http://www.numdam.org/article/RO_1968__2_1_57_0.pdf Roy B. Classement et choix en présence de points de vue multiples Rev. Fr. Informat. Rech. O. 1968 2 8 57 75 Available: http://www.numdam.org/article/RO_1968__2_1_57_0.pdf [15] N. Belacel, “Multicriteria assignment method PROAFTN: Methodology and medical application,” Eur. J. Oper. Res., vol. 125, no. 1, pp. 175–183, 2000. https://doi.org/10.1016/S0377-2217(99)00192-7 Belacel N. Multicriteria assignment method PROAFTN: Methodology and medical application Eur. J. Oper. Res. 2000 125 1 175 183 https://doi.org/10.1016/S0377-2217(99)00192-7 10.1016/S0377-2217(99)00192-7 [16] S. Hajkowicz and K. Collins, “A review of multiple criteria analysis for water resource planning and management,” Water Resour. Manag., vol. 21, no. 9, pp. 1553–1566, 2006. https://doi.org/10.1007/s11269-006-9112-5 Hajkowicz S. Collins K. A review of multiple criteria analysis for water resource planning and management Water Resour. Manag. 2006 21 9 1553 1566 https://doi.org/10.1007/s11269-006-9112-5 10.1007/s11269-006-9112-5 [17] C. Chen and W. Hung, “Applying ELECTRE and maximizing deviation method for stock portfolio selection under fuzzy environment,” in Opportunities and Challenges for Next-Generation Applied Intelligence, B. Chien and T. Hong, Ed. Heidelberg: Springer, 2009, pp. 85–91. https://doi.org/10.1007/978-3-540-92814-0_14 Chen C. Hung W. Chien B. Hong T. Applying ELECTRE and maximizing deviation method for stock portfolio selection under fuzzy environment Opportunities and Challenges for Next-Generation Applied Intelligence 2009 85 91 Heidelberg Springer https://doi.org/10.1007/978-3-540-92814-0_14 10.1007/978-3-540-92814-0_14 [18] C. Weinhardt and S. Seifert, “Developments in GDN research: Introduction,” Group Decis. Negotiat., vol. 19, no. 2, pp. 107–109, 2009. https://doi.org/10.1007/s10726-009-9185-7 Weinhardt C. Seifert S. Developments in GDN research: Introduction Group Decis. Negotiat 2009 19 2 107 109 https://doi.org/10.1007/s10726-009-9185-7 10.1007/s10726-009-9185-7 [19] F. Uzoka, J. Osuji, and O. Obot, “Clinical decision support system (DSS) in the diagnosis of malaria: A case comparison of two soft computing methodologies,” Expert Syst. Appl., vol. 38, no. 3, pp. 1537–1553, 2011. https://doi.org/10.1016/j.eswa.2010.07.068 Uzoka F. Osuji J. Obot O. Clinical decision support system (DSS) in the diagnosis of malaria: A case comparison of two soft computing methodologies,” Expert Syst. Appl 2011 38 3 1537 1553 https://doi.org/10.1016/j.eswa.2010.07.068 10.1016/j.eswa.2010.07.068 [20] F. Cismondi et al., “Multi-stage modeling using fuzzy multi-criteria feature selection to improve survival prediction of ICU septic shock patients,” Expert Syst. Appl., vol. 39, no. 16, pp. 12332–12339, 2012. https://doi.org/10.1016/j.eswa.2012.04.027 Cismondi F. Multi-stage modeling using fuzzy multi-criteria feature selection to improve survival prediction of ICU septic shock patients Expert Syst. Appl. 2012 39 16 12332 12339 https://doi.org/10.1016/j.eswa.2012.04.027 10.1016/j.eswa.2012.04.027 [21] B. Roy and P. Vincke, “Multicriteria analysis: Survey and new directions,” Eur. J. Oper. Res., vol. 8, no. 3, pp. 207–18, 1981. https://doi.org/10.1016/0377-2217(81)90168-5 Roy B. Vincke P. Multicriteria analysis: Survey and new directions Eur. J. Oper. Res. 1981 8 3 207 218 https://doi.org/10.1016/0377-2217(81)90168-5 10.1016/0377-2217(81)90168-5 [22] H. Barron and C. Schmidt, “Sensitivity analysis of additive multiattribute value models,” Oper. Res., vol. 36, no. 1, pp. 122–127, 1988. https://doi.org/10.1287/opre.36.1.122 Barron H. Schmidt C. Sensitivity analysis of additive multiattribute value models Oper. Res. 1988 36 1 122 127 https://doi.org/10.1287/opre.36.1.122 10.1287/opre.36.1.122 [23] B. Mareschal, “Weight stability intervals in multicriteria decision aid,” Eur. J. Oper. Res., vol. 33, no. 1, pp. 54–64, 1988. https://doi.org/10.1016/0377-2217(88)90254-8 Mareschal B. Weight stability intervals in multicriteria decision aid Eur. J. Oper. Res 1988 33 1 54 64 https://doi.org/10.1016/0377-2217(88)90254-8 10.1016/0377-2217(88)90254-8 [24] J. Dyer, P. Fishburn, R. Steuer, J. Wallenius, and S. Zionts, “Multiple criteria decision making, multiattribute utility theory: The next ten years,” Manag. Sci., vol. 38, no. 5, pp. 645–654, 1992. https://doi.org/10.1287/mnsc.38.5.645 Dyer J. Fishburn P. Steuer R. Wallenius J. Zionts S. Multiple criteria decision making, multiattribute utility theory: The next ten years Manag. Sci. 1992 38 5 645 654 https://doi.org/10.1287/mnsc.38.5.645 10.1287/mnsc.38.5.645 [25] O. Larichev and H. Moshkovich, “ZAPROS-LM: A method and system for ordering multiattribute alternatives,” Eur. J. Oper. Res., vol. 82, no. 3, pp. 503–521, 1995. https://doi.org/10.1016/0377-2217(93)E0143-L Larichev O. Moshkovich H. ZAPROS-LM: A method and system for ordering multiattribute alternatives, Eur. J. Oper. Res. 1995 82 3 503 521 https://doi.org/10.1016/0377-2217(93)E0143-L 10.1016/0377-2217(93)E0143-L [26] E. Triantaphyllou and A. Sánchez, “A sensitivity analysis approach for some deterministic multi-criteria decision-making methods,” Decis. Sci., vol. 28, no. 1, pp. 151–194, 1997. https://doi.org/10.1111/j.1540-5915.1997.tb01306.x Triantaphyllou E. Sánchez A. A sensitivity analysis approach for some deterministic multi-criteria decision-making methods Decis. Sci. 1997 28 1 151 194 https://doi.org/10.1111/j.1540-5915.1997.tb01306.x 10.1111/j.1540-5915.1997.tb01306.x [27] M. Bender and S. Simonovic, “A fuzzy compromise approach to water resource systems planning under uncertainty,” Fuzzy Sets Syst., vol. 115, no. 1, pp. 35–44, 2000. https://doi.org/10.1016/S0165-0114(99)00025-1 Bender M. Simonovic S. A fuzzy compromise approach to water resource systems planning under uncertainty Fuzzy Sets Syst 2000 115 1 35 44 https://doi.org/10.1016/S0165-0114(99)00025-1 10.1016/S0165-0114(99)00025-1 [28] G. Fu, “A fuzzy optimization method for multicriteria decision making: An application to reservoir flood control operation,” Expert Syst. Appl., vol. 34, no. 1, pp. 145–149, 2008. https://doi.org/10.1016/j.eswa.2006.08.021 Fu G. A fuzzy optimization method for multicriteria decision making: An application to reservoir flood control operation Expert Syst. Appl. 2008 34 1 145 149 https://doi.org/10.1016/j.eswa.2006.08.021 10.1016/j.eswa.2006.08.021 [29] J. Skorupski, “Multi-criteria group decision making under uncertainty with application to air traffic safety,” Expert Syst. Appl., vol. 41, no. 16, pp. 7406–7414, 2014. https://doi.org/10.1016/j.eswa.2014.06.030 Skorupski J. Multi-criteria group decision making under uncertainty with application to air traffic safety Expert Syst. Appl 2014 41 16 7406 7414 https://doi.org/10.1016/j.eswa.2014.06.030 10.1016/j.eswa.2014.06.030 [30] R. Mosadeghi, J. Warnken, R. Tomlinson, and H. Mirfenderesk, “Uncertainty analysis in the application of multi-criteria decision-making methods in Australian strategic environmental decisions,” J. Environ. Plan. Manag., vol. 56, no. 8, pp. 1097–1124, 2013. https://doi.org/10.1080/09640568.2012.717886 Mosadeghi R. Warnken J. Tomlinson R. Mirfenderesk H. Uncertainty analysis in the application of multi-criteria decision-making methods in Australian strategic environmental decisions J. Environ. Plan. Manag. 2013 56 8 1097 1124 https://doi.org/10.1080/09640568.2012.717886 10.1016/j.eswa.2014.06.030 [31] K. Hyde, H. Maier, and C. Colby, “Reliability-based approach to multicriteria decision analysis for water resources,” J. Water Resour. Plan. Manag., vol. 130, no. 6, pp. 42–438, 2004. doi: 10.1061/(ASCE)0733-9496(2004)130:6(429) Hyde K Maier H. Colby C. Reliability-based approach to multicriteria decision analysis for water resources J. Water Resour. Plan. Manag 2004 130 6 42 438 10.1061/(ASCE)0733-9496(2004)130:6(429) [32] P. Moura, S. Barraud, M. Baptista, and F. Malard, “Multicriteria decision-aid method to evaluate the performance of stormwater infiltration systems over the time,” Water Sci. Technol., vol. 64, no. 10, p. 1993–2000, 2011. doi: 10.2166/wst.2011.154 Moura P. Barraud S. Baptista M. Malard F. Multicriteria decision-aid method to evaluate the performance of stormwater infiltration systems over the time Water Sci. Technol. 2011 64 10 1993 2000 10.2166/wst.2011.154 [33] C. Martin, Y. Ruperd, and M. Legret, “Urban stormwater drainage management: The development of a multicriteria decision aid approach for best management practices,” Eur. J. Oper. Res., vol. 181, no. 1, pp. 338–349, 2007. https://doi.org/10.1016/j.ejor.2006.06.019 Martin C. Ruperd Y. Legret M. Urban stormwater drainage management: The development of a multicriteria decision aid approach for best management practices Eur. J. Oper. Res 2007 181 1 338 349 https://doi.org/10.1016/j.ejor.2006.06.019 10.1016/j.ejor.2006.06.019 [34] A. Shanian, A. Milani, C. Carson, and R. Abeyaratne, “A new application of ELECTRE III and revised Simos’ procedure for group material selection under weighting uncertainty,” Knowl.-Based Syst., vol. 21, no. 7, pp. 709–720, 2008. https://doi.org/10.1016/j.knosys.2008.03.028 Shanian A. Milan A. Carson C. Abeyaratne R. A new application of ELECTRE III and revised Simos’ procedure for group material selection under weighting uncertainty Knowl.-Based Syst. 2008 21 7 709 720 https://doi.org/10.1016/j.knosys.2008.03.028 10.1016/j.knosys.2008.03.028 [35] C. Giannoulis and A. Ishizaka, “A web-based decision support system with ELECTRE III for a personalised ranking of British universities,” Decis. Support Syst., vol. 48, no. 3, pp. 488–497, 2010. https://doi.org/10.1016/j.dss.2009.06.008 Giannoulis C. Ishizaka A. A web-based decision support system with ELECTRE III for a personalised ranking of British universities Decis. Support Syst 2010 48 3 488 497 https://doi.org/10.1016/j.dss.2009.06.008 10.1016/j.dss.2009.06.008 [36] H. Chen, M. Wood, C. Linstead, and E. Maltby, “Uncertainty analysis in a GIS-based multi-criteria analysis tool for river catchment management,” Environ. Model. Softw., vol. 26, no. 4, pp. 395–405, 2011. https://doi.org/10.1016/j.envsoft.2010.09.005 Chen H. Wood M. Linstead C. Maltby E. Uncertainty analysis in a GIS-based multi-criteria analysis tool for river catchment management Environ. Model. Softw. 2011 26 4 395 405 https://doi.org/10.1016/j.envsoft.2010.09.005 10.1016/j.envsoft.2010.09.005 [37] N. Chen and Z. Xu, “Hesitant fuzzy ELECTRE II approach: A new way to handle multi-criteria decision making problems,” Inf. Sci., vol. 292, pp. 175–197, 2015. DOI: 10.1016/j.ins.2014.08.054 Chen N Xu Z. Hesitant fuzzy ELECTRE II approach: A new way to handle multi-criteria decision making problems Inf. Sci. 2015 292 175 197 10.1016/j.ins.2014.08.054 [38] F. Dong, Y. Liu, H. Su, Z. Liang, R. Zou, and H. Guo, “Uncertainty-based multi-objective decision making with hierarchical reliability analysis under water resources and environmental constraints,” Water Resour. Manag., vol. 30, no. 2, pp. 805–822, 2015. https://doi.org/10.1007/s11269-015-1192-7 Dong F. Liu Y. Su H. Liang Z. Zou R. Guo H. Uncertainty-based multi-objective decision making with hierarchical reliability analysis under water resources and environmental constraints Water Resour. Manag 2015 30 2 805 822 https://doi.org/10.1007/s11269-015-1192-7 10.1007/s11269-015-1192-7 [39] S. Galarza-Molina, A. Torres, P. Moura, and J. Lara-Borrero, “CRIDE: A case study in multi-criteria analysis for decision-making support in rainwater harvesting,” Int. J. Inf. Technol. Decis. Making, vol. 14, no. 1, pp. 43–67, 2015. https://doi.org/10.1142/S0219622014500862 Galarza-Molina S. Torres A. Moura P. Lara-Borrero J. CRIDE: A case study in multi-criteria analysis for decision-making support in rainwater harvesting Int. J. Inf. Technol. Decis. Making 2015 14 1 43 67 https://doi.org/10.1142/S0219622014500862 10.1142/S0219622014500862 [40] L. Scholten, N. Schuwirth, P. Reichert, and J. Lienert, “Tackling uncertainty in multi-criteria decision analysis – An application to water supply infrastructure planning,” Eur. J. Oper. Res., vol. 242, no. 1, pp. 243–260, 2015. https://doi.org/10.1016/j.ejor.2014.09.044 Scholten L. Schuwirth N. Reichert P. Lienert J. Tackling uncertainty in multi-criteria decision analysis – An application to water supply infrastructure planning Eur. J. Oper. Res. 2015 242 1 243 260 https://doi.org/10.1016/j.ejor.2014.09.044 10.1016/j.ejor.2014.09.044 [41] A. Afsordegan, M. Sánchez, N. Agell, S. Zahedi, and L. Cremades, “Decision making under uncertainty using a qualitative TOPSIS method for selecting sustainable energy alternatives,” Int. J. Environ. Sci. Technol., vol. 13, no. 6, pp. 1419–1432, 2016. https://doi.org/10.1007/s13762-016-0982-7 Afsordegan A. Sánchez M. Agell N. Zahedi S. Cremades L. Decision making under uncertainty using a qualitative TOPSIS method for selecting sustainable energy alternatives Int. J. Environ. Sci. Technol 2016 13 6 1419 1432 https://doi.org/10.1007/s13762-016-0982-7 10.1007/s13762-016-0982-7 [42] R. Smith, G. Poveda, O. Mesa, D. Valencia, I. Dyner y P. Jaramillo, Decisiones con multiples objetivos e incertidumbre, 2nd ed. Medellín: Universidad Nacional de Colombia, 2000. Smith R. Poveda G. Mesa O. Valencia D. Dyer I. Jaramillo P. Decisiones con multiples objetivos e incertidumbre 2000 Medellín Universidad Nacional de Colombia 2nd ed. [43] P. Martin, Sustainable Urban Drainage Systems. London: Construction Industry Research and Information Association, 2001. Martin P. Sustainable Urban Drainage Systems 2001 London Construction Industry Research and Information Association [44] Departamento Administrativo Nacional de Estadística (DANE), Censo general 2005 a nivel nacional, DANE, 2005. Available: http://bit.ly/2JT1xaB Departamento Administrativo Nacional de Estadística (DANE) Censo general 2005 a nivel nacional 2005 Available: http://bit.ly/2JT1xaB [45] Cámara de Comercio de Bogotá, Observatorio de Seguridad en Bogotá no. 37. Balance enero-junio de 2009. Bogotá: Cámara de Comercio de Bogotá, 2009. Available: http://bit.ly/2Kvegzw Cámara de Comercio de Bogotá Observatorio de Seguridad en Bogotá no. 37. Balance enero-junio de 2009 2009 Bogotá Cámara de Comercio de Bogotá Available: http://bit.ly/2Kvegzw [46] Subsecretaría de Planeación Socioeconómica, Encuesta Calidad de Vida 2007. Bogotá 2007. Bogotá: Subsecretaría de Planeación Socioeconómica, 2007. Available: http://bit.ly/2Kq4GOz Subsecretaría de Planeación Socioeconómica Encuesta Calidad de Vida 2007. Bogotá 2007 2007 Bogotá Subsecretaría de Planeación Socioeconómica Available: http://bit.ly/2Kq4GOz [47] B. Roy and P. Bertier, “La méthode ELECTRE II: Une méthode de classement en présence de critères multiples,” in Note de Travail, no. 142, SEMA (Metra International), Direction Scientifique. Paris: Grupo Metra, 1971, pp. 291–302. Roy B. Bertier P. La méthode ELECTRE II: Une méthode de classement en présence de critères multiples Note de Travail 1971 142 291 302 Paris Grupo Metra SEMA (Metra International), Direction Scientifique [48] J. Figueira, V. Mousseau, and B. Roy, “ELECTRE methods,” in Mult. Criteria Decis. Anal. State Art Surveys, J. Figueira, G. Salvatore, and M. Ehrgott, Ed. Boston: Springer Science & Business Media, 2005, pp. 133–162. https://doi.org/10.1007/0-387-23081-5_4 Figueira J. Mousseau V. Roy B. Figueira J. Salvatore G. Ehrgott M. ELECTRE methods Mult. Criteria Decis. Anal. State Art Surveys 2005 133 162 Boston Springer Science & Business Media https://doi.org/10.1007/0-387-23081-5_4 10.1007/0-387-23081-5_4 [49] D. Davis, The Development of Celtic Linguistics, 1850-1900. London: Routledge, 2002. Davis D. The Development of Celtic Linguistics, 1850-1900 2002 London Routledge [50] Nera.com, 2018. [Online]. Available: http://bit.ly/2WzAOG7 Nera.com 2018 Available: http://bit.ly/2WzAOG7 Notes

Research article.