The Openness / Intellect Model on the M 5-50 : Supporting the Flexibility of IPIP-Based Instruments

**Ph.D. y Master en Counseling Psychology. Facultad de Filosofía y Ciencias Humanas, Dirección de Postgrado. Correo electrónico: alexvuyk@ku.edu ***School of Education, Department of Educational Psychology. Correo elctrónico: c981w725@ku.edu; kmulton@ku.edu; pbingram@ku.edu ABSTRACT This study examined ways to improve fit and interpretive capacity of the M5-50, an IPIP-based personality instrument, using the Openness/ Intellect model (O/I) given a history of poor performance of the M5-50 Openness scale (Socha, Cooper, & McCord, 2010). With participants from Amazon’s MTurk (n = 305), theoretical models for the five-factor model, Openness as a 10-item single factor, and the O/I model were tested and fitted poorly. After removing one problematic item, the O/ I model showed acceptable fit. Findings indicate that the O/I model improved psychometric validity and interpretive capacity for the M5-50. The flexibility and open access of IPIP-based instruments such as the M5-50 makes the IPIP an optimal choice for instrument adaptation and construction in Latin America.

To cite this article: Vuyk, M. A., Ingram IV, P. B., Multon, K. D., Warlick, C. A. (2016).The Openness/Intellect Model on the M5-50: Supporting the flexibility of IPIP-Based Instruments.Universitas Psychologica,15 (4).http://dx.doi.org/10.11144/Javeriana.upsy15-4.omsfResearchers seeking to advance the five-factor model (FFM) have been advocating for the use of the International Personality Item Pool ([IPIP]; Goldberg, 1999), a scientific collaboratory to develop personality measures, because of its free and publicly accessible nature.The IPIP database (available at http://ipip.ori.org ) holds 2413 items in English measuring 230 personality constructs in 302 scales (Goldberg, 1999) that can be combined based on the researcher's needs.It has a rich history of being able to replicate the same personality structures seen within proprietary instruments such as the NEO Personality Inventory (Baldasaro, Shanahan, & Bauer, 2013;Gow, Whiteman, Pattie, & Deary, 2005).The IPIP is, therefore, a viable alternative to explore and expand understanding of personality.
Psychological research in Latin America continues to grow and Benito (2012) suggested international integration as a particular area with room for improvement.Research in Latin America should focus on topics that can be contrasted globally and use standard methods that are comparable (Benito, 2012).Debate is still ongoing as to whether instruments for Latin America should be adapted from existing ones or whether they should be constructed from scratch to reflect the culture and context (Fernández, Pérez, Alderete, Richaud, & Fernández Liporace, 2010).Test construction requires a solid theory to support it and Latin American psychology has not yet developed to a point of having sufficiently tested theories (Fernández et al., 2010).However, adaptation also has caveats: certain constructs might not be cross-culturally comparable and on multiple occasions the adaptation process is not as rigorous as it should be.A compromise can be to adapt tests following rigorous methodological procedures and to resort to test construction only when the specific test characteristics make adaptation more difficult than developing a new test, or when constructs are nonequivalent across cultures.The five-factor model of personality is cross-culturally valid (McCrae & Terracciano, 2005) and thus adaptation seems like a plausible alternative.The IPIP appears to be amenable to adaptation with four versions in Spanish and three versions in Portuguese (Goldberg, 1999).Equal instruments promote international integration as results can be compared and aggregated in meta-analyses.
Developing countries consistently encounter problems with a lack of resources and low allocation of funds to research.Concentrating on open-access publication policies, journals, and research repositories would allow improved dissemination of scientific content in such a climate (Benito, 2012).Similarly, we could argue that large-scale research costs can be prohibitive with proprietary instruments.A quick review of personality instruments used in Latin American papers show that various versions of the NEO Personality Inventory in Spanish are a popular choice (Contreras-Torres, Espinosa-Méndez, & Esguerra-Pérez, 2008;Gázquez, Pérez, Mercader, & Inglés, 2014;Morán, Méndez, González, Landero-Hernández, & Menezes, 2014;Ortiz et al., 2012).When funding is a concern, turning to open-access instruments such as the IPIP can allow many studies to be conducted that would otherwise not have been possible.Even more, IPIP-based instruments are flexible and can be modified based on needs.Thus, turning to the IPIP to measure personality could be beneficial for Latin American researchers.
The IPIP is flexible and permits organization of items as needed to measure the intended constructs.McCord (2002) designed two instruments intended to measure the FFM using items derived from the IPIP.The M5-50 is a brief, 50-item instrument which measures the five domains: Neuroticism, Extraversion, Conscientiousness, Agreeableness, and Openness to Experience, and was standardized and normed for use with an adult population.The M5-336 measures FFM domains and facets, and it has been part of several studies looking at the construct validity of the Openness to Experience domain (Proctor & McCord, 2009).Although the items of the M5-50 items are nested within the M5-336, psychometric information might not be congruent between different test versions (Cronbach & Meehl, 1955).Even though items of the M5-50 were supposedly selected because they possessed the best psychometric values on the IPIP (Socha et al., 2010), further testing of the M5-50 is imperative before clinicians use it.
Moreover, the M5-50 has demonstrated problems finding a well-fitted CFA model for its FFM structure (Socha et al., 2010).Although Socha and colleagues made an argument that the M5-50 was approaching reasonable fit, low fit indices suggest inherent problems to the construction of the instrument (Brown, 2006).In addition to a CFA, Socha and colleagues (2010) also performed an exploratory factor analysis (EFA) because of the failures in the M5-50 to meet the appropriate model fit guidelines.However, this analysis also encountered problems.Several items intended to be on the Agreeableness domain did not load onto any domains.Although Openness did not demonstrate the problems in the same way that Agreeableness did, this may have been a result of the rotation method used, or possibly the tendency for factor structures to emerge irregularly in Openness (Gomez, 2006).Regardless, these difficulties in establishing appropriate fit statistics within the M5-50 make it difficult to generalize construct validity of domains and left the M5-50, as well as the Openness domain, in need of further evaluation.
These results are not surprising as the most pronounced and consistent area of problems for the FFM has been the Openness to Experience domain (Caspi, Roberts, & Shiner, 2005).Structural problems with Openness appeared in a variety of forms.Problems included genderbased factor invariance larger than in the other domains (Gomez, 2006), issues related to subordinate factors emerging instead of a sole, unified construct (Herzhoff & Tackett, 2012), and arguments that Openness exists as an artifact of the correlational matrix from which it was derived, instead of an actual, emergent construct (McKenzie, 1998).These problems with Openness coincided with the reemergence of the Openness/Intellect debate.
The Openness/Intellect debate began at the time of the emergence of this domain initially known as Factor Five, as researchers argued that this Factor Five in questionnaire measures represented openness to new experiences and imagination, while in lexical measures (e.g., adjective lists) it represented intellect and a desire for knowledge (Trapnell, 1994).This discrepancy appeared in the multiple names given to Factor Five, such as Openness to Experience, Intellect, Openness to Ideas, Openness to Aesthetics, or Creative Mentality (Johnson, 1994;McCrae, 1994).Factor analyses of several personality instruments showed that Openness and Intellect appeared to be distinctive, while still connected (DeYoung, Quilty, & Peterson, 2007;Johnson, 1994).Studies showed a relationship of only Openness with creativity (Nusbaum & Silvia, 2011) and implicit learning (Kaufman et al., 2010).They also showed a relationship of only Intellect with working memory (DeYoung, Shamosh, Green, Braver, & Gray, 2009) and fluid intelligence (Nusbaum & Silvia, 2011).These studies support the preference of some scholars to rename the factor Openness/Intellect or refer to it as the O/ I Model .Still, most FFM instruments such as the M5-50 conceive and measure Openness as a single domain, instead of a compound comprised of two distinct but related parts (DeYoung, Quilty, Peterson, & Gray, 2014).
The problematic model fit seen on the M5-50 CFA and the lack of information regarding the method on the EFA conducted by Socha et al. (2010)

Participants and Procedures
A total of 305 participants (163 males, 53.4%), whose ages ranged from 18 to 71 (M = 34.38,SD = 11.76), were recruited for this study.Participants endorsed a range of education levels: less high school (n = 2), high school or GED (n = 39), some college (n = 81), associate's graduate degree (n = 36), and doctoral degree (n = 6).Participants were recruited through MTurk, crowdsourcing software designed by Amazon.In MTurk, the researchers posted a request for completion of the study, including details of compensation and the psychometric nature of the assessment, as well as an informed consent statement.The MTurk request linked to a battery of personality instruments on the online platform Qualtrics.Critics have acknowledged that MTurk was a valid method for research in social and behavioral sciences (Shapiro, Chandler, & Mueller, 2013).For a complete review of how the MTurk system works, see Mason and Suri (2012).This study was conducted as part of a larger examination into personality, with several brief measures included in the survey.Participants received a compensation of US$ 3.00 for successfully completing the measures.Instruments appeared in a randomized order in Qualtrics and contained attention checks.At the end of the survey on Qualtrics, participants received a unique code that they entered in MTurk as proof of completion.Average time of completion was 52 minutes and 11 seconds.

Measure
The M5-50 (McCord, 2002) is comprised of 50 items drawn from the IPIP database and is conceptualized to measure the five broad domains of the NEO Personality Inventory.Each of the five domains on the M5-50 is measured using 10 items drawn from the IPIP.Participants respond to each item on a 5-point Likert scale ranging from 1 (inaccurate) to 5 (accurate).Half of the items for each scale are reverse-coded.Items belonging to the Openness domain scale can be found in Table 1.For a comprehensive discussion on the IPIP, see Goldberg et al. (1999).

Data Analyses
Confirmatory factor analyses were used to test in all models was set by fixing the factor variance.To assess model fit, we followed Little's (2013) guidelines of poor fit if CFI < 0.9 and RMSEA/SRMR < 0.1, acceptable fit if CFI > 0.9 and RMSEA/SRMR < 0.08, or very good fit if CFI > 0.95 and RMSEA/ SRMR < 0.05.First, we tested the fit of the M5-50 as conceptualized by McCord (2002) and described by Socha et al. (2010), with each of the 50 items loading on their corresponding domains.The second analysis exclusively tested the fit of the Openness domain.This followed Gignac, Bates, and Jang's (2007) proposition of analyzing each personality domain separately, given the fit problems that arise from domain cross-loadings and correlated residuals.The third analysis tested the fit of the Openness domain as a single construct but eliminated item 20 (i.e., tendency to vote for conservatives) as suggested in Ingram et al. (2013), as it appeared to be an item that created problems with factor structure.The fourth analysis tested the fit of the O/I model, specifying Openness and Intellect as two constructs rather than as a single Openness domain, grouped with Openness or Intellect, first according to the EFA results of Ingram et al. (2013) and second according to conceptual similarity to O/I model concepts when there was ambiguity in EFA results (e.g., those included into the third factor).The fifth planned analysis tested the O/I model as two constructs but eliminated item 20, again based on the suggestion of Ingram et al. (2013).Lastly, the sixth analysis incorporated the O/I model with item 20 removed into the M5-50 to examine changes made to the instrument as a result of modifications to Openness.

Power Analyses
Power analyses were conducted using the Rbased software created by Preacher and Coffman (2006) based on the test of poor fit that MacCallum, Browne, and Sugawara (1996) proposed.The test of poor fit allows us to know whether, given the sample size, we would be able to reject a hypothesis of a bad fit in the sample if the population fit is good and vice versa (MacCallum et al., 1996).Based on models, estimated degrees of freedom would fall among values of 20 and 50.We specified a null hypothesis of a bad fit with an RMSEA value that we set at 0.1 and higher, and an alternative hypothesis of a good fit with an RMSEA value that we set at 0.05 and lower following Little's (2013) fit guidelines.For 20 degrees of freedom, 0.8 power, and 0.05 alpha level, the power analysis indicated that the minimum sample size would be 188; for 50 degrees of freedom, 0.8 power, and 0.05 alpha level, the minimum sample size would be 99 (Preacher & Coffman, 2006).The study sample of N = 305 exceeds minimum values and would permit models to be more complex than those specified in this study.

Results
Intercorrelations among Openness domain items can be found in Table 1.Missing data appeared to be completely at random on 2 of 305 participants as it did not relate to any other variables, and thus was estimated using full-information maximum likelihood (FIML).As described by Little (2013), FIML is a model-based approach that utilizes all the data available to predict what the parameter estimates would have been if the missing values had been present.
The CFA with the proposed five-factor M5-50 model indicated poor fit, consistent with the M5-50 literature (Socha et al., 2010) and with the assertion that CFA might not be suited for five-factor personality models as multiple crossloadings might blur results (Gignac et al., 2007).The second planned CFA tested the fit of the Openness domain as originally conceptualized by McCord (2002) as a single latent factor with 10 items.If cross-loadings prevented the fit of the entire personality test, then testing individual domains should at least show good fit as described by Gignac et al. (2007), yet the second model also showed poor fit.To run the third CFA, we removed item 20 as it was duplicated and demonstrated problems in the past (Ingram et al., 2013) and tested the Openness domain with nine items, which demonstrated poor fit as well.
The fourth planned CFA tested the fit of the Openness/Intellect model in the original 10 items.Previously, data fit the O/I model better than it did Openness as a single factor in an EFA (Ingram et al., 2013).Still, the O/I model with the 10 items presented poor fit.Thus, the fifth planned CFA included the recommendation of Ingram et al. (2013) of removing item 20.This model fit the data acceptably.Even though fit was not optimal, it showed a significant improvement over the previous models, suggesting that item 20 should be removed from the M5-50.The sixth, and final, planned CFA model tested the incorporation of the O/I model into the full structure of the M5-50 to allow a comparison of fit between the initial five factor model and the O/I approach.This did not fit the data well; however, it showed an incremental improvement over the initial structure (i.e., Model 1) of the M5-50.Our hypothesis was confirmed as model number five was the best-fitting theory-driven model.See Figure 1 for factor loadings, factor correlations, and item variances.Table 2 provides essential fit indices of all models for ease of comparison, including the Akaike Information Criterion (AIC).AIC is a χ 2 -based fit index that adjusts for lack of parsimony (Akaike, 1987); it represents one way of comparing non-nested models, with lower values indicating better, more parsimonious models.

Discussion
The purpose of this study was to examine the structure of the M5-50 when incorporating the O/ I model to examine if doing so would improve fit statistics that were previously observed (i.e., Socha et al., 2010), given the IPIP's flexibility to modify and refine instruments based on scientific outcomes.Just as CFA fit statistics previously seen for the M5-50 with traditional FFM scoring were not ideal (Socha et al., 2010), the traditional single factor scoring of Openness proved highly problematic in this study as well.Although Socha et al.'s low fit encompassed the five domains in the M5-50, the model statistics in this study imply that Openness as a single factor is problematic and likely contributed to the low fit indices.Future studies could re-evaluate specific design features of the M5-50 as high factor loadings alone may not be the most important inclusion criterion to select IPIP items, which was the favored criterion as noted by Socha and colleagues (2010).Lastly, when attempting to bring the Openness domain into line with the O/I model to increase interpretability, model fit was able to be improved for the M5-50.This supports scoring the M5-50 Openness domain items differently according to the O/I model, particularly with the advised removal of item 20, and provides an alternative scoring approach that Socha and colleagues conclude may be needed to develop the M5-50's utility.And, just as in the work by Ingram et al. (2013), there is further support that the removal of item 20 might also improve the fit and appropriateness of the traditional FFM scoring for the M5-50.
The factor structure of the M5-50 demonstrated improved fit with the O/I model in this study.In the fifth model, which included two correlated factors corresponding to Openness and Intellect and eliminated item 20, item loadings on factors were all significant and ranged from 0.29 to 1.14.The Openness factor had higher item loadings than the Intellect factor.In the Openness factor, three items (2, 15, and 29) showed strong loadings between 0.89 and 1.14, while item 1 had a weaker loading of 0.35.In the Intellect factor, two items (5 and 22) had stronger loadings while items 8, 37, and 43 had weaker loadings ranging between 0.29 and 0.41.This difference in loadings makes sense because the Openness domain as conceptualized in the FFM has six facets; items in the Openness domain of the M5-50 represent a summative and comprehensive interpretation all of those facets within one construct.
Two limitations related to the sample in this study must be mentioned.First, the MTurk system recruits participants broadly and it is possible that participants are outside of the validation sample for the M5-50.Also, although the average time of completion was within the expected range and no attention checks were missed, a portion of participants completed the survey in significantly less time than expected.Despite these two factors, these limitations are not considered to have invalidated the findings of this study since the first model tested produced similar fit statistics to those seen before (Socha et al., 2010) and indications of an O/I model had been found in a separate sample (Ingram et al., 2013).Findings are likely to be generalizable to a more representative sample.In fact, being able to replicate models using crowd-sourcing software reaffirms the robustness of the IPIP, even when using the MTurk system.
The findings of this study also have general implications for personality assessment.The decision to analyze just the Openness domain was made in order to minimize the number of intercorrelations, a frequent problem in FFM instruments (Gignac et al., 2007).Thereby, this study supports the notion that examinations of structural characteristics for domains and facets should be made separately before they are incorporated into more complex models.Most importantly, however, is the exemplification made with this study of the ease with which the IPIP can be used for the design and adaptation of instrument proxies.With adaptations able to be conducted on any instrument stemming from the IPIP database, items based on the IPIP provide a free and powerful alternative to proprietary personality instruments which require expressed permission of the copyright holders and license fees per use.In light of the debate on whether test adaptation or test construction is preferred for Latin American research (Fernández et al., 2011), the flexibility of the IPIP makes it feasible for both test adaptation and test construction.Test adaptation can be done by translating and validating existing scales, while test construction can be done by translating culturally relevant items and combining them into a new instrument.This allows supporters of each position to work with the IPIP.
Presently, several translations of IPIP items into Spanish and Portuguese exist, yet more work remains.Spanish translations include items in the IPIP-VIA version measuring 24 character strengths, two translations of items in the IPIP version of the 16PF test subscales, and one translation of the 50-item inventory measuring Big Five structure (Goldberg, 1999).In Portuguese, one translation involved the 100item inventory measuring the Big Five structure, while two others translated the 50-item Big Five inventory (Goldberg, 1999).Translated items can be found in journal publications or by contacting authors.It would be convenient to have these existing translated items, as well as any further translations or adaptations, available on the IPIP website for ease of access.This would simplify retrieval for Latin American researchers and make it more convenient for the field to use this free personality test items repository.
led other researchers to examine the instrument more comprehensively.Due to the historic difficulties of the Openness domain in FFM instruments, Ingram, Boan-Lenzo and Vuyk (2013) tested the factor solution within degree (n = 23), bachelor's degree (n = 118), the fit of six proposed models.The scale | Universitas Psychologica | V. 15 | No. 4 | Octubre-Diciembre | 2016 | Openness on the M5-50 and determined that the O/I model emerged more than a single latent factor.When testing the Openness domain assuming its oblique nature, one item was a Heywood case and overstated the relevance of Liberalism, which is a secondary facet within Openness.This item was excluded from further analyses and doing so produced a more interpretable extraction matrix.Congruent with the O/I model, Openness and Intellect emerged as distinct factors within the M5-50's Openness domain and comprised a majority of the items of the domain.Remaining items grouped onto a third, barely emergent factor, representing secondary facets of the Openness domain.Building upon the work of Ingram et al. (2013) and exemplifying the versatility of the IPIP, the goal of this study is to examine the feasibility of incorporating the O/I model onto an IPIP-based instrument representing the traditional factor structure of the FFM.The O/I model was not included as a potential alternative model for testing CFA fit in Socha et al. (2010) study.Given the tendency of the traditional Openness domain in the M5-50 to become differentiated into the O/I model using exploratory factor analyses(Ingram et al., 2013) confirmatory factor analyses using the proposed interpretive models will expand knowledge of the M5-50's psychometric properties.It was hypothesized that, when examining Openness as conceptualized by the O/ I model as having intrinsic distinctive aspects, the Openness domain would improve its fit indices.

Figure 1
Figure 1 Factor loadings, covariances, and residual variances for the O/I model without item 20.