Assessing Information Integration Processes Using Between- or Within-Subjects Designs: Some More Evidence*
Evaluando los procesos de integración de la información utilizando diseños entre- o dentro- sujetos: más evidencia
Universitas Psychologica, vol. 18, no. 1, 2019
Pontificia Universidad Javeriana
Gérard Chasseigne a firstname.lastname@example.org
François-Rabelais University, Francia
Maria Teresa Muñoz Sastre
Jean-Jaurès University, Francia
Paul Clay Sorum
Albany Medical College, Estados Unidos
Institute of Advanced Studies, Francia
Date received: 30 December 2017
Date accepted: 05 March 2018
Abstract: Within-subject designs (WSDs) remain unappreciated in psychology although many experimental tactics can reduce or eliminate the demand and order effects that WSDs tend to create. Comparative studies conducted in the Information Integration Theory (IIT) framework have shown that patterns of results observed using WSDs can largely be replicated using between-subject designs (BSDs). In order to add evidence to these findings, three additional studies were conducted in order to complement data obtained in previous studies. One of these studies was about health risk perception and tested the possibility to find evidence for a disjunctive rule of information integration using a BSD. The other two studies focused on the valuation process of IIT. The new findings regarding the disjunctive rule added support to the view that equivalent results can be obtained either with a highly economical repeated-measures design or with a much costlier independent factorial group arrangement. However, when the focus was on the valuation process and not on the integration process, ratings obtained in the BSD condition seemed to be restricted to a limited range of values by comparison with ratings obtained in the WSD condition. An explanation in terms of context effect is offered.
Keywords human judgment, information integration theory, within-subject designs, between-subject designs, health risk assessment.
Resumen: Los diseños dentro sujetos (WSD) no se aprecian en la psicología, aunque existen muchas tácticas experimentales que pueden reducir o eliminar la demanda y los efectos de orden que tienden a crear los WSD. Los estudios comparativos realizados en el marco de la Teoría de Integración de la Información (IIT) han demostrado que los patrones de resultados observados con WSD se pueden replicar en gran medida utilizando diseños entre sujetos (BSD). Para agregar evidencia a estos hallazgos, se realizaron tres estudios adicionales para complementar los datos obtenidos en estudios anteriores. Uno de estos estudios fue sobre la percepción de riesgo en salud y evaluó la posibilidad de encontrar evidencia de regla disyuntiva de integración de la información utilizando un BSD. Los otros dos estudios se centraron en el proceso de valoración de IIT. Los nuevos hallazgos con respecto a la regla disyuntiva sustentaron la opinión de que se pueden obtener resultados equivalentes tanto con un diseño altamente económico de medidas repetidas o con un acuerdo de grupo factorial independiente mucho más costoso. Sin embargo, cuando la atención se centró en el proceso de valoración y no en el proceso de integración, las calificaciones obtenidas en la condición BSD parecían estar restringidas a un rango limitado de valores en comparación con las calificaciones obtenidas en la condición WSD. Se ofrece una explicación en términos del efecto de contexto.
Palabras clave: juicio humano, Teoría de Integración de la Información, diseños dentro sujetos, diseños entre sujetos, evaluación de riesgo en salud.
Despite their interest, within-subjects designs (WSDs) remain unwelcome in psychology. They are in bad repute for several reasons. Firstly, they would create demand effects. In WSDs, participants are presented with several stimuli in close succession. As these stimuli obviously vary as a function of one or several parameters, (a) participants’ attention would be attracted by the information that varies from one stimulus to the other and (b) this information would, as a result, be given more weight during their judgment process as compared with the information that does not vary. This belief is still very strong despite evidence showing that when the judgment task is borrowed from daily life, such a possibility is unlikely (Mullet et al., 2012, 2016). In field contexts, participants know in advance what information is, from their own perspective, important or not important. Huge variations in a factor that they consider as irrelevant, given the judgment task at hand, would not make any change. Nevertheless, in laboratory settings, such a possibility exists when the task is, from the participants’ perspective, mostly meaningless. In this case, however, the real issue is: Why present people with material that does not make sense to them (Mullet, 2012)?
Secondly, they would create order effects. As in WSDs, participants are presented with stimuli in close succession, the first stimulus and the immediate participant’s responses to it would contaminate the way the second one is perceived, processed and responded to. This second belief is also still very strong despite evidence showing that many experimental precautions can reduce or eliminate order effects. For example, a prior session of familiarization with the stimuli can eliminate warm-up, practice and learning effects (Anderson, 2001, 2002).
Thirdly, findings from WSDs would not be generalizable to the real world because the structure of the world is of a between-subjects kind. This third belief has been strong among authors working in the decision-making field although it is not only false but absurd. The flow of life makes us pass without a break from one situation to another, which is for obvious reasons closely similar to the previous one. To illustrate, think of doctors who receive, in close succession, several patients in consultation. These doctors’ diagnoses and recommendations can in no way be made independently of professional training and professional experience; that is, of memory for past and current experiences. Think also in evolutionary terms. If the world structure was of a between-subjects type; that is, if it prevented living organisms from becoming aware of variations in the environment, no evolution would have been possible.
Six previous studies conducted in the Information Integration Theory framework
Methodological studies conducted in the Information Integration Theory framework (Anderson, 1996, 2016, 2018) have shown that findings obtained using WSDs can be replicated using BSDs. Howe and Loftus (1992) used scenarios that depicted a fight between two persons. Two factors were considered in these scenarios: (a) level of intent to harm from the part of the aggressor, and (b) consequences of the fight (e.g., severe injury). In the WSD condition, twelve participants were presented with the whole set of eight scenarios, in close succession. They were to judge of the level of blame the aggressor deserved in each case. The BDS condition involved 96 participants, split into eight groups of twelve, who were presented with only one scenario. In both conditions, intent had the most substantial effect, (c) consequences had a weaker, although significant effect and no Intent x Consequence interaction was detected. In other words, the Blame = Intent + Consequences rule suggested by Leon (1980) was supported by both sets of empirical findings.
Mullet and Chasseigne (2017) also found support for additive-type integration rules using scenarios in which participants expressed willingness to forgive offenses under various circumstances. Each of their 376 participants was presented with one scenario depicting a situation in which Person A severely and intentionally hurt Person B, and then has or has not apologized for the harmful act. Intent, consequences, and apologies had significant effects, and more importantly, none of the two- and three-way interactions was significant. In other words, the Willingness to Forgive = Intent + Consequences – Apologies rule suggested by Girard and Mullet (1997) was supported by empirical findings resulting from the use of a BSD. Apologies had, however, a weaker effect in the BSD condition than in the original WSD study.
Howe and Loftus (1996) used scenarios that depicted legal punishments that varied as a function of certainty and severity. Their set of 216 participants was split into nine groups of 24. Each participant was presented with only one scenario and asked to judge the level of deterrence that could be expected from this kind of punishment. In this BSD condition, certainty and severity had a significant effect and, more importantly, the Certainty x Severity interaction was significant. The pattern of mean responses was fan-shaped and was open to the right. In other words, the Deterrence = Certainty x Severity rule previously found by these authors using a WSD was also found using a BSD.
Mullet and Chasseigne (2017) also found support for multiplicative-type integration rules using scenarios in which participants expressed probable consumers’ intoxication levels as a function of the type of drink consumed (beer, wine or fortified wine) and the number of glasses of this drink (one, three or five). The size of the glass was kept constant. Their 376 participants were split into nine independent groups and each participant was presented with only one scenario. Type of drink and the number of glasses had significant effects, and more importantly, the Type x Number interaction was significant. The pattern of mean responses was fan-shaped and was open to the right. In other words, the Intoxication = Type of drink x Number of glasses multiplicative-type rule suggested by Muñoz Sastre, Mullet, and Sorum (2000) was also found using a BSD.
The design used by Howe and Loftus in their two studies (1992, 1996) did not allow testing averaging information integration rules because no scenario in which a piece of information was missing was part of their designs. Mullet and Chasseigne (2017) tested the possibility to detect averaging rules of information integration using complete and incomplete information BSDs. They presented two-third of their 316 participants with one of 12 scenarios depicting a situation in which two persons share variable levels of passion the one for the other, experience variable levels of intimacy, and have or have not decided to marry (Sternberg, 1997). The remaining participants were presented with one of six scenarios depicting a similar situation but in which the commitment information was missing. In both conditions, participants were asked to assess the level of true love experienced by these persons. In the complete information condition, passion, intimacy, and commitment had significant effects, and none of the interactions was significant. The effects of passion and intimacy were, however, significantly weaker in this condition than in the incomplete information condition. This set of findings is the signature of an equal-weight averaging process of information integration. In other words, the Love = (w Passion + w’ Intimacy + w” Commitment) / w + w’ + w”) information integration rule suggested by Falconi and Mullet (2003) can also be found using a BSD. Ratings were, however, higher in the BSD condition than in the Falconi and Mullet’ (2003) study.
In a subsequent study, Mullet and Chasseigne (2017) also tested the possibility to detect more complex, differential-weight averaging rules of information integration using, as in the previous study, complete and incomplete information BSDs. They presented two-third of their 316 participants with one of 12 scenarios depicting a person regarding the level of agreeableness, openness and emotional stability (McCrae & Costa, 1996). The remaining participants were presented with one of six scenarios depicting a similar situation but in which the agreeableness information was missing. In both conditions, participants were asked to assess the level of attractiveness of the person depicted by the two or three adjectives. In the complete information condition, agreeableness, openness, and emotional stability had significant effects, and all two- and three-way interactions were significant. This result is consistent with the view that a differential-equal weighting scheme has been implemented. The effects of openness and emotional stability were, however, significantly weaker in this condition than in the incomplete information condition. This second result is consistent with the view that an averaging scheme has been implemented. Taken together, both sets of findings are the signature of a differential-weight averaging process of information integration. In other words, the Attractiveness = (w a Agreeableness + w o’ Openness + w s” Stability) / w a + w o’ + w s”) information integration rule suggested by Cretenet, Mullet, and Dru (2015, see also Mullet, Cretenet, & Dru, 2014), in which the weight attributed to each factor depends on the level of this factor, can also be found using a BSD.
The present set of studies
The present set of studies was aimed at completing the set of studies conducted by Howe and Loftus (1992, 1996) and by Mullet and Chasseigne (2017). As in these works, findings obtained using BSDs will be compared with findings obtained through the use of WSDs. The first study was about health risk perception and the disjunctive rule of information integration. The second study was also about health risk perception but focused on the valuation process of information integration.
Disjunction . In studies about risk perception, a pattern opposite to the fan-shaped and open-to-the right one observed when a multiplicative rule is operative has been repeatedly observed (e.g., Hermand, Mullet & Lavieville, 1997). This pattern corresponds to the use of a disjunctive rule of information integration. In order to test whether such a pattern can be observed in a BSD condition, nine scenarios depicting a person’s levels of daily intake of alcohol and tobacco were presented to 376 participants.
An example of one story is the following: “Each morning, Christopher smokes some cigarettes. At midday, he is used to have lunch in the company of his colleagues and to drink some glasses of wine. At the end of the day, he has usually smoked about one pack of cigarettes and drunk equivalent of one liter of red wine. To what extent do you think that he runs the risk of having a type of cancer associated with this daily consumption?” Ratings were provided on a 16-point scale anchored by not a risk at all (0) and very high risk (15).
The panel on the left of Figure 1 shows the main results observed in the BSD condition. Curves are ascending and separated. In other words, both circumstantial factors had the expected effect on the judged level of cancer risk. The pattern of curves is fan-shaped to the left, that is, an interaction was present, which, in this case, is the signature of a disjunctive process of information integration. Participants considered that indulging in only one of these two risky behaviors represented a high health risk. This pattern is highly similar to the one shown in the right panel that presents results borrowed from the study by Hermand et al. (1997) in which the same scenarios were presented, but a WSD was used. (Responses from 20 participants also aged 21-26, from both genders, were randomly selected.)
An ANOVA on the whole set of data was performed with a design of Condition (BDS vs. WDS) x Tobacco Consumption x Alcohol Intake, 2 x 3 x 3. The Condition effect was not significant. The Tobacco x Alcohol interaction was significant and concentrated in its bilinear component, F(1, 359) = 24.79, p < .001. The Condition x Tobacco x Alcohol interaction was not significant, which is consistent with the view that the disjunctive rule of information integration holds similarly in both conditions.
This finding was, in a certain way, implied by the findings reported by Howe and Loftus (1996) and by Mullet and Chasseigne (2017). If the responses scale were inverted (as if participants were expected to assess levels of healthiness), and if the values on the x-axis were also inverted, the patterns of responses observed would be, as shown in Figure 2, utterly similar to the ones observed each time a multiplicative rule of information integration is applied.
Valuing . According to Anderson (1996), a series of three functions – Valuation, Integration, and Response -- converts a set of external or internal stimuli into a single judgment. The Valuation Function converts stimuli (e.g., number of glasses of some drink or feeling of unease) into concurrent psychological representations (e.g., elevated risk of cancer). In other words, this function assigns a value to each stimulus, and this value depends on the type and intensity of the stimulus and on the goal pursued (e.g., assessing health risk). Through psychological Integration, these values are weighted and combined into an overall implicit response. Finally, the judgment is generated employing the Response function.
In most IIT studies, the researchers’ focus is on the integration process. In the study by Hermand et al. (1997) reported above, the focus was on the disjunctive rule of information integration, and the levels of the tobacco intake factor, for example, were selected in order to maximize the chances of correctly diagnosing the integration rule. In some studies, however, the focus is on the valuation process. In these studies, one factor —the factor of interest — has many levels and the other factor, the role of which is purely technical, has only two or three levels. For example, in a study conducted by Muñoz Sastre, Mullet, and Sorum (1999), one of the factors —the tobacco consumption factor— had six levels and the other factor —type of tobacco — had three levels. The objective of the study was to examine whether lay people considered that the risk of cancer was a linear function of tobacco consumption. An example of a scenario used was the following: “Sydney smokes 35 cigarettes daily. He usually smokes light tobacco. To what extent do you think that he runs the risk of having a lung cancer associated with this daily consumption?” Ratings were provided on a 16-point scale anchored by no risk at all (0) and very high risk (15).
The left panel of Figure 3 shows the main results observed in the BSD condition. Curves are ascending: The more the number of cigarettes smoked daily, the higher the risk. The right panel shows the pattern observed in the WSD condition. Both patterns are similar, but curves are steeper in the right panel, and their slope is negatively accelerated.
An ANOVA on the whole set of data was performed with a design of Condition (BDS vs. WDS) x Type of Tobacco x Number of Cigarettes, 2 x 3 x 6. The Condition x Number interaction was significant, F(5, 700) = 17.46, p < 0.001.
As this was the first instance of an apparent discrepancy between results obtained in BSD and WSD conditions, a second study was run that also focused on the valuation process. The scenarios used were borrowed from Muñoz Sastre, Pecarisi, Legrain, Mullet and Sorum (2007) who examined the acceptability of induced abortion among adolescents as a function of the fetus’ age. An example of scenario used is the following: “Pauline is 15 years old. She has always wanted to go to college. She is one-month pregnant and has told her doctor she wants an abortion. Her parents consent, as does her boyfriend. To what extent do you think that abortion is an acceptable procedure for Pauline in this case?” Ratings were provided on a 16-point scale anchored by not at all acceptable (0) and completely acceptable (15). The main results are shown in Figure 4. Curves are descending: the older the fetus, the less acceptable the procedure. Curves were, however, less steep and much higher than the ones shown in the original study. For example, at four months, the mean acceptability rating reported in the original paper was 5.15 (on a 0-15 scale). It was higher than 10 in the present study.
Our new findings regarding the disjunctive rule of information integration rule add support to the view already expressed by Howe and Loftus (1996) that equivalent results can be obtained either with a highly economical repeated-measures design or with a much costlier independent factorial group arrangement. There is, however, one caveat to this optimistic stance. When the focus of the study is on the valuation process and not on the integration process, ratings obtained in the BSD condition seems to be restricted to a limited range of values by comparison with ratings obtained in the WSD condition.
There is no independent touchstone that would allow deciding which set of findings —the one that results from the application of a BSD or the one that results from the application of a WSD— is true and which is false. In other words, there is no independent touchstone that would allow deciding which methodology is, given the purpose of the study, appropriate and which is flawed. In view of Figures 3 and 4, it seems fair, however, to consider that, when the purpose of a study is to examine the values that people attach to different levels of a factor, using WSDs is a better option than using BSDs. This view rests on the empirical fact that discrimination between levels of the factor of interest —tobacco intake — is more fine-grained if a WSD rather than a BSD is used.
Findings observed in the two valuation studies are reminiscent of Birnbaum’s (1999) results. This author had participants assess the subjective size of numbers using a 10-point scale, ranging from “very very small” to “very very large”. Some participants judged the subjective size of the number 9 and other participants judged the subjective size of the number 221. Mean ratings in the 9 condition were significantly higher than ratings in the 221 condition.
According to Birnbaum (1999), the reason behind this perplexing finding was that when presented alone, the number 221 recalls a context of three-digit numbers (ranging from 0 to 999) among which, its first cipher being 2, it may seem quite small. In contrast, the number 9 recalls a context of one-digit numbers (ranging from 0 to 9) among which it may seem quite large. When presented together or in close succession in the same session both numbers —9 and 221— were, of course, judged entirely different and 221 was judged much higher than 9. In our two valuation studies, different contexts were, probably in the same way as in Birnbaum’s study, evoked depending, for example, whether the number of cigarettes was only 5, or 15, or as high as 30. Our results were not as perplexing as the ones reported in Birnbaum’s, but an evident shrinkage in subjective values associated with tobacco consumption levels was observed with BSD.
As a result, neither Birnbaum’s perplexing finding nor the new findings reported here in the BSD condition must be interpreted as additional demonstrations of cognitive biases in people’s judgment capacities (Fox, 1992) but as an additional demonstration that the exclusive use of BSD can lead researchers to gather confusing data, which can lead them and the scientific community as a whole to enduring misconceptions regarding human performance (Mullet, 2012).
Anderson, N. H. (1996). A functional theory of cognition. Mahwah, NJ: Erlbaum.
Anderson, N. H. (2001). Scientific psychology series. Empirical direction in design and analysis. Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
Anderson, N. H. (2002). Methodology and statistics in single-subject experiments. In J. Wixted, (Ed.), Stevens' handbook of experimental psychology (Vol. 4, pp. 301-337). London: John Wiley & Sons, Inc.
Anderson, N. H. (2016). Information integration theory: Unified psychology based on three mathematıcal laws. Universitas Psychologica, 15(3), 1-7. http://doi.org/10.11144/Javeriana.upsy15-3.iitu
Anderson, N. H. (2018). Moral science. New York, NY: Psychology Press.
Birnbaum, M. H. (1999). How to show that 9 > 221: Collect judgments in a between-subjects design. Psychological Methods, 4(3), 243-249. http://doi.org/10.1037/1082-989X.4.3.243
Cretenet, J., Mullet, E., & Dru, V. (2015). Motor and cognitive integration: Effect of bilateral behaviors on judgment. Acta Psychologica, 161, 64-72. http://doi.org/10.1016/j.actpsy.2015.08.004
Falconi, A., & Mullet, E. (2003). Cognitive algebra of love through the adult life. International Journal of Aging and Human Development, 57(3), 277-290. http://doi.org/10.2190/NPQH-MDLX-F48U-AA35
Fox, R. (1992). Prejudice and the unfinished mind: A new look at an old failing. Psychological Inquiry, 3(2), 137-152. http://doi.org/10.1207/s15327965pli0302_12
Girard, M., & Mullet, E. (1997). Forgiveness in adolescents, young, middle-aged, and older adults. Journal of Adult Development, 4(4), 209-220. https://doi.org/10.1007/BF02511412
Hermand, D., Mullet, E., & Lavieville, S. (1997). Perception of the Combined Effects of Smoking and Alcohol on Cancer Risks in Never Smokers and Heavy Smokers. Journal of Health Psychology, 2(4), 481-491. https://doi.org/10.1177/135910539700200405
Howe, E. S., & Loftus, T. C. (1992). Integration of intention and outcome information by students and circuit court judges: Design economy and individual differences. Journal of Applied Social Psychology, 22(2),102-116. http://doi.org/10.1111/j.1559-1816.1992.tb01524.x
Howe, E. S., & Loftus, T. C. (1996). Integration of certainty, severity, and celerity information in judged deterrence value: Further Evidence and Methodological Equivalence1. Journal of Applied Social Psychology, 26(3), 226-242. http://doi.org/10.1111/j.1559-1816.1996.tb01848.x
Leon, M. (1980). Integration of intent and consequence information in children’s moral judgments. In F. Wilkening, J. Becker, & T. Trabasso (Eds.), Information integration by children (pp. 71–97). Hillsdale, NJ: Lawrence Erlbaum Associates.
McCrae, R. R., & Costa, P. T., Jr. (1996). Toward a new generation of personality theories: Theoretical contexts for the five-factor model. In J. S. Wiggins (Ed.), The five-factor model of personality: Theoretical perspectives (pp. 51-87). New York, NY: Guilford Press.
Mullet, E. (2012). The superfluous postulate of human rationality. Rivista Internazionale di Scienze Sociali, 120(3), 269-288. Retrieved from https://EconPapers.repec.org/RePEc:vep:journl:y:2012:v:120:i:3:p:269-288
Mullet, E., & Chasseigne, G. (2017). Assessing information integration processes: A comparison of findings obtained with between-subjects designs versus within-subjects designs. Quality & Quantity, 52(4), 1977-1988. http://doi.org/10.1007/s11135-017-0592-6
Mullet, E., Cretenet, J., & Dru, V. (2014). Motor influences on judgment: Motor and cognitive integration. British Journal of Psychology, 105(1), 69-91. http://doi.org/10.1111/bjop.12022
Mullet, E., López López, W., Kpanake, L., Mukashema, I., Armange, R., Kamble, S., … Neto, F. (2016). Functional Measurement in the field of ethics in politics. Universitas Psychologica, 15(3). http://doi.org/10.11144/Javeriana.upsy15-3.fmf
Mullet, E., Sorum, P. C., Teysseire, N., Nann, S., Morales, G. E., Ahmed, R., … Muñoz Sastre, M. T. (2012). Functional Measurement in the field of empirical bioethics. Psicologica: International Journal of Methodology and Experimental Psychology, 33, 665-681. Retrieved from https://files.eric.ed.gov/fulltext/EJ980500.pdf
Muñoz Sastre, M. T., Mullet, E., & Sorum, P. (1999). Relationship between cigarette dose and perceived risk of lung cancer. Preventive Medicine, 28(6), 566-571. https://doi.org/10.1006/pmed.1999.0482
Muñoz Sastre, M. T., Mullet, E., & Sorum, P. C. (2000). Self-assessment of inebriation from external indices. Addictive Behaviors, 25(5), 663-681.
Muñoz Sastre, M. T., Pecarisi, C., Legrain, E., Mullet, E., & Sorum, P. C. (2007). Acceptability in France of induced abortion for adolescents. American Journal of Bioethics, 7(8), 26-32. https://doi.org/10.1080/15265160701462368
Sternberg, R. J. (1997). Construct validation of a triangular love scale. European Journal of Social Psychology, 27(3), 313-335. http://doi.org/10.1002/(SICI)1099-0992(199705)27:3<313::AID-EJSP824>3.0.CO;2-4
Corresponding author. Email:
How to cite: Chasseigne, G., Muñóz-Sastre, M. T., Sorum,
P. C., & Mullet, E. (2019). Assessing information integration processes using
between- or within-subjects designs: Some more evidence. Universitas Psychologica,
18 (1). https://doi.org/10.11144/Javeriana.upsy18-1.aiip