Keywords
rubric
writing
university students
How to Cite
Abstract
This article describes the process of construction and validation of a rubric to measure the writing of first-year university students. 117 essays written by Spanish Philology students (75 women and 42 men) of the University of Costa Rica were collected. Two evaluators, based on a rubric, rated the compositions after which different raters agreement coefficients were calculated to assess the technical quality of the rubric. Among the most important results, a 90% of absolute agreement was observed [± 3 errors] in most of the areas evaluated. On the other hand, small and moderate correlations between the aspects included in the rubric were observed. From these results the utility of using text linguistics as a theoretical basis for constructing written expression rubrics are discussed.
Beauvais, C., Olive, T. & Passerault, J. M. (2011). Why Are Some Texts Good and Others Not? Relationship Between Text Quality and Management of the Writing Processes. Journal of Educational Psychology, 103(2), 415–428.
Behizadeh, N., & Engelhard Jr, G. (2011). Historical view of the influences of measurement and writing theories on the practice of writing assessment in the United States. Assessing writing, 16(3), 189-211.
Bejar, I. I. (2012). Rater cognition: Implications for validity. Educational Measurement: Issues and Practice, 31(3), 2-9.
Breland, H. M., Bridgeman, B & Fowles, M. E. (1999). Writing assessment in admission to higher education: Review and framework. New York: College Entrance Examination Board.
Brown, G. T. (2009). The reliability of essay score: the necessity of rubrics and moderation. In L. Meyer et al. (Eds.), Tertiary Assessment and Higher Education Student Outcomes: Policy, Practice, and Research (pp. 43-50). Wellington, New Zealand: Ako Aotearoa.
Calsamiglia, H., & Tusón, A. (2007). Las cosas del decir. España: Ariel.
Carroll, J. (1993). Human Cognitive Abilities. New York: Cambridge University Press.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4), 213.
Crossley, S. A, & McNamara, D. S (2010). Cohesion, Coherence, and Expert Evaluations of Writing Proficiency. Proceedings of the 32nd annual conference of the Cognitive Science Society, 984-989.
De Beaugrande, R. (1984). Text Production. Recuperado de http://www.beaugrande.com/text_production. htm
East, M. (2009). Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing. Assessing Writing, 14(2), 88-115.
Eckes, T. (2005). Examining raters effects in TestDaF writing and speaking performance assessments: a many-facet Rasch analysis. Language assessment quarterly 2(3), 197-221.
Eckes, T. (2009). Many-facet Rasch measurement. In S. Takala (Ed.), Reference supplement to the manual for relating language examinations to the Common European Framework of Reference for Languages: Learning, teaching, assessment (Section H). Strasbourg, France: Council of Europe/Language Policy Division.
Halliday, M. & Matthiessen, C. (2004). An Introduction to Functional Grammar. London: Hodder Arnold.
Jeffery, J. V. (2009). Constructs of writing proficiency in US state and national writing assessments: Exploring variability. Assessing Writing, 14(1), 3-24.
Johnson, R. L. (2009). Assessing performance: Designing, scoring, and validating performance tasks. New York: Guilford Press.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: reliability, validity and educational consequences. Educational Research Review 2(2), 130-144.
Knoch, U. (2007). ‘Little coherence, considerable strain for reader’: A comparison between two ratings scales for the assessment of coherence. Assessing writing, 12(2), 108-128.
Lomas, C. (1999). Cómo enseñar a hacer cosas con las palabras. España: Editorial Paidós.
Louwerse, M., Graesser, A., McNamara, D., Jeuniaux, P., & Yang, F. (2006). Coherence is also in the eye of the beholder. In M. Silva, & A. Cox (Eds.), Proceedings of the Cognitive Science Workshop “What have eye movements told us so far, and what is next?”. Recuperado de 129.219.222.66:8080/SoletlabWeb/ pdf/CoherenceEyeBeholder.pdf
McNamara, D., Kintsh, E., Butler, N., & Kintsh, W. (1996). Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and instruction, 14(1), 1-43.
McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic Features of Writing Quality. Writing Communication 27(1), 57-88.
Montero, E. (2007). Teoría G: un futuro paradigma para el análisis de pruebas psicométricas. Actualidades en Psicología, 21, 117-144.
Moreira, T. (2008). Construcción y validación de pruebas de expresión escrita en la Universidad de Costa Rica. Avances en medición, 6(1), 85-100.
Núñez, R., & del Teso, E. (1996). Semántica y pragmática del texto común. España: Ediciones Cátedra.
Olive, T. (2004). Working Memory in Writing: Empirical Evidence From the Dual-Task Technique. European Psycologis, 9(1), 32-42.
Olive, T., Kellogg, R. T., & Piolat, A. (2008). Verbal, visual, and spatial working memory demands during text composition. Applied Psycholinguistics, 29(4), 669-687.
Parodi, G., & Núñez, P. (2002). En búsqueda de un modelo cognitivo/textual para la evaluación del texto escrito. En M. Martínez (Ed.), Propuesta de intervención pedagógica para la comprensión y producción de textos académicos (pp. 65-98). Cátedra UNESCO MECEAL: Lectura y Escritura. Recuperado de http:// www.unesco-lectura.univalle.edu.co/articulos.html
Revelle, W. (s. f.). An introduction to psychometric theory with applications in R. Recuperado de http://personality-project.org/r/book/.
Rodino, A., & Ross, R. (1997). Problemas de Expresión Escrita del Estudiante Universitario Costarricense. Costa Rica: EUNED.
Sánchez, C. (2004a). Historiografía de la enseñanza de la redacción en Costa Rica: los libros de texto. Revista de Filología y Lingüística, 20(1), 219-246.
Sánchez, C. (2004b). La puntuación y las unidades textuales: una perspectiva discursiva para el estudio de los problemas de su uso y para su enseñanza. Revista Educación, 28(2), 233-254.
Sánchez, C. (2005a). Los problemas de redacción de los estudiantes costarricenses: una propuesta de revisión desde la lingüística del texto. Filología y Lingüística, 21(1), 267-295.
Sánchez, C. (2005b). Los conectores discursivos: su empleo en redacciones de estudiantes universitarios costarricenses. Filología y Lingüística, 21(2), 169-199.
Sánchez, C. (2006a). Historia de un desencuentro: investigación y enseñanza de la redacción en Costa Rica. Revista de Filología y Lingüística, 22(1), 223- 245.
Sánchez, C. (2006b). ¿Cuestión de método? Sobre los cursos remediales universitarios de expresión escrita. Revista Educación, 30(1), 65-81.
Sánchez, C. (2007). Los objetivos de la instrucción gramatical en la enseñanza del español como lengua materna. Filología y Lingüística, 33(1), 167-190.
Sasaki, M., & Hirose, K. (1999). Development of an analytic rating scale for Japanese L1 writing. Language Testing, 16(4), 457-478.
This journal is registered under a Creative Commons Attribution 4.0 International Public License. Thus, this work may be reproduced, distributed, and publicly shared in digital format, as long as the names of the authors and Pontificia Universidad Javeriana are acknowledged. Others are allowed to quote, adapt, transform, auto-archive, republish, and create based on this material, for any purpose (even commercial ones), provided the authorship is duly acknowledged, a link to the original work is provided, and it is specified if changes have been made. Pontificia Universidad Javeriana does not hold the rights of published works and the authors are solely responsible for the contents of their works; they keep the moral, intellectual, privacy, and publicity rights. Approving the intervention of the work (review, copy-editing, translation, layout) and the following outreach, are granted through an use license and not through an assignment of rights. This means the journal and Pontificia Universidad Javeriana cannot be held responsible for any ethical malpractice by the authors. As a consequence of the protection granted by the use license, the journal is not required to publish recantations or modify information already published, unless the errata stems from the editorial management process. Publishing contents in this journal does not generate royalties for contributors.