The goal of this study was comparing methodology and result presentation quality in articles from Latin America and other regions. This type of comparison was not found in other sources of information, which presented an additional incentive to this research. Therefore, this is a novel study that critically analyses methodology and result presentation of scientific papers on the subject of postpartum depression (PPD) prevalence.

The quality of a scientific article is directly proportional both to the original research methodological quality and the results presentation. The scientific quality of a given research and the respective article presenting its results, is fundamentally dependent on its external validity (the power of generalizing results) and its internal validity (whether what supposed to be measured is actually being measured) (Hoppen, 1998). Some characteristics that indicate the high quality of a research report are the relevance of the theme, described in a well-defined objective; the theories and studies that support the relevance of the research, based on a contemporary literature review and judged with strict selection criteria (Trzesniak, 2014); consistent methodology, tested in previous research when possible; data collection and analysis performed in a way which could as much as possible reduce bias; and the potential of generating concise results (Volpato, 2015).

What makes a scientific article different from other publications?

Evaluation before approval is certainly the biggest difference between an article and other types of scientific production. Specialists evaluate the text and provide their contributions prior to publication, which both, validates the research and assists with improving the text through expertise.

Several scholars have contributed to deepen discussions about the evaluation of the quality of research reports published in scientific articles. In the field of Psychology, the American Psychological Association (APA) has published guidelines for the comprehension of the process of scientific texts evaluation for 85 years. In 1929, the first Publishing Manual was published, a seven pages document with a group of procedures to increase reading comprehension of scientific papers. Created under the responsibility of the United States National Research Council, the writers were psychologists, anthropologists, and publishing professionals (American Psychological Association, 2010).

Versions and revisions of this manual were published in 1952, 1957 and 1967. Due to the growing complexity of scientific communication, the following editions were published in 1974, 1983, 1994, and 2001. Known mostly by its simplicity and consistency of its quotation style and reference normalization, the Publication Manual of the American Psychological Association has also established norms for the use of language that have had long-term effects. The guidelines for non-sexist language in APA journals were particularly influent, published for the first time in 1974, it provided effective alternatives to commonly used sexist language (American Psychological Association, Task Force on Sexual Bias in Graduate Education, 1975). The guidelines to reduce the language bias were updated throughout the years and, nowadays, they offer effective guidelines to write about race, ethnicity, gender, age, sexual orientation, disabilities and wealth (American Psychological Association, 2010).

The Publication Manual of the American Psychological Association is in its sixth edition and guides authors regarding the style that must be followed when writing scientific articles prior to submission to the journals published by the Association (American Psychological Association, 2010). The latest version included procedures for systematic review, with or without meta-analysis, besides innovations concerning digital technologies, with emphasis on the role of Digital Object Identifier (DOI), as a trustworthy mechanism to find information. New ethical guidelines concerning the determination of main authors and collaborations; duplication of the same publication; plagiarism and auto plagiarism; participants concealment; research instruments validity; data availability for third party verification; clarity and accuracy verification procedures of empirical researches; title simplification to make it friendlier in e-publishing; new guidelines for language bias reduction and language standardization; new guidelines for the presentation of statistical analysis presentation and abbreviations, were also introduced. It is worth pointing out the increased coverage of the publishing process, including the role of peer-review, ethical, legal requirements, publications standard policies, and guidance on how to interact with editors while the article is in press. The Publication Manual of the American Psychological Association is certainly the most important publishing style manual in the Psychology field.

The attention to quality of scientific articles in Psychology has also led several scholars to create tools to help authors and editors in the task of writing, formatting, evaluating scientific articles, and searching for instruments for scientific production evaluation. (Buela-Casal, 2003; Buela-Casal & López, 2005; Koller, Couto, & Hohendorff, 2014; Sabadini, Sampaio, & Koller, 2009; Trzesniak, Plata-Caviedes, & Córdoba-Salgado, 2012; Witter, 2006).

Important instruments are being developed to guide and improve the writing of articles that report epidemiological studies. Vilas Boas and Silvany Neto (2012) have pointed out the Quorum for quality studies presentation, including systematic revisions; the Consort for randomized essays; the STARD for diagnostic studies; the Moose for meta-analysis of observational studies and the Strobe for the primary observational studies. The authors say that the use of methodological quality evaluation instruments while developing epidemiological researches can improve the results presentation in these studies. They present the Questionário de Avaliação de Pesquisas Epidemiológicas Observacionais Analíticas - Qualiepi (Observational Epidemiological Analytical Research Evaluation Questionnaire). In the mid to long term, society will benefit from these results, inasmuch as public policies guide their clinical and public health practices grounded in more accurate scientific knowledge (Vilas Boas & Silvany Neto, 2012).

It is important to look for a “golden standard” in scientific articles. In order to do so, researchers must improve the application of study design. Only then the reports will be able to support decisions in the clinical practice and also help designing public policies.

Current Study

The goal of this study was to compare the methodological quality of the studies and the results presentation of scientific articles from Latin America and other regions that included a systematic revision about the prevalence of PPD. The study aimed to answer two questions: 1. Do scientific articles with data collected in Latin America and other regions present the same quality? 2. Do scientific articles published in journals in Latin America and other regions present the same quality? We understand that comparing the results of research conducted in Latin America and elsewhere is an opportunity to promote the scientific works developed by authors who have been struggling to generate quality knowledge.

Method

The design of the study consisted of a documentary research, with critical analysis of existing literature. Two independent reviewers performed the definition of the databases and search strategy, the classification of the studies according to the inclusion criteria, and the data collection. The postpartum depression theme was chosen because it was the subject of a systematic review that was being conducted at the time, allowing the use of data on the quality of the articles for both purposes.

The hypothesis of the study is that the methodological quality and presentation of the results of scientific articles published in Latin-American and non-Latin-American journals have an equivalent score on the Loney Scale. The quality of the articles was measured according to the scale developed by Loney, Chambers, Bennett, Roberts, and Stratford (1998) to evaluate the quality of epidemiological articles about prevalence or incidence. The instrument is composed of eight items, each of them is rated at one or zero, according to the adequacy of the methods and results presentation, with a maximum of eight points. This scale was chosen because it can be swiftly completed, given the short number of questions, and its score can be used in quantitative analysis. The instrument had to be adapted to the subject of PPD prevalence, as it had been originally intended for use in any article about prevalence or incidence (Loney et al., 1998).

Several databases were accessed in order to find the scientific articles which were analyzed in this study: Biblioteca Digital de Teses e Dissertações, Biblioteca Virtual em Saúde - Psicologia, CINAHL, EMBASE, ERIC, ISI Web of Science, LILACS, ProQuest, Psicodoc, PsycInfo, PubMed, Redalyc, SciELO, and Scopus. The specific search in each database was made with the following strategy: (depressão pós-parto or depressão pos-natal or postpartum depression or postnatal depression or depression postparto or depression postnatal) and (prevalência or prevalence or prevalencia).

Three steps were followed in the evaluation of the articles: title, abstract, and full text. In each step, it was verified whether or not the studies met with the inclusion criteria, which were: a);) Primary and quantitative study, with transversal or longitudinal cohort design, which evaluated the prevalence of postpartum depression in human mothers. b);) The sample did not undergo any PPD preventive intervention. c);) The diagnosis method was clearly mentioned and based on an objective instrument, or an evaluation made by a mental health professional. d);) At least one of the PPD evaluations occurred from 8 (included) to 365 days postpartum. e);) The study had general population as subjects. f);) At least one of the evaluations to identify PPD had a clear prevalence result report, with postpartum period data, instrument, sample size, and the absolute number or percentage of participants considered depressed. g); The full text was published in English, Portuguese or Spanish.

The inclusion or rejection of studies according to the established criteria and the quality evaluation using the Loney Scale was a decision made by two reviewers, independently. An agreement was sought when divergent assessments occurred.

The agreement level between reviewers was verified using the percentage of studies in which they agreed on the evaluation as a parameter. The measurement of reliability using the kappa index was discarded, because in situations where there is a large difference among the relative frequencies in the categories of the evaluated object (for example, in the evaluation of an inclusion criteria, there would be a higher number of studies classified as “true” than the ones classified as “false”), a high level of concordance by chance would be expected, resulting in a lower maximum value for k (value that relates to the perfect agreement between evaluators) (Eugenio, 2000). So, this index would be difficult to interpret in this study.

The reliability of the scoring, which is the sum of the total points achieved in the Loney Scale, was measured by obtaining the intraclass correlation coefficient, using the two-way mixed effects model (Shrout & Fleiss, 1979; McGraw & Wong, 1996). Identical evaluations in the comparison are represented by an intraclass coefficient of 1. Values above 0.7 were considered acceptable.

From the sample of studies included in the systematic review of PPD prevalence, two subsamples were taken for comparison: one of them made up by all Latin American articles, and the other, made up by randomly selected, non-Latin-American studies. Both samples had the same size. The randomization was done using a tool available with IBM SPSS software version 20.0. Non-parametric statistical tests were used for the hypotheses tests. The survey data were collected in the year 2013.

Results

A total of 3 909 entries were found on scientific papers about PPD prevalence. Excluding duplicate entries, there were 1 894 studies left, from which 337 met the inclusion criteria, according to the three steps selection (titles, abstracts, and full texts), as it is possible (see Figure 1). From the selected ones, 34 studies conducted data collection in Latin America, so, 34 other articles were randomly selected, which collected data in other regions.

From a total of 68 articles, 23 were published in Latin-American journals and 45 in other regions journals (see Figure 2).

The 115 articles where the full text was not evaluated arise from the fact that they are in a language other than Portuguese, English and Spanish, or it was not possible to obtain the full text on any of the bibliographic exchange system. Only the articles that did not have full or partial overlap of the sample were included. The concordance rate among the reviewers was over 70% in the analysis of titles, abstracts and full texts of articles.

According to Wikipedia (https://pt.wikipedia.org/wiki/Am%C3%A9rica_Latina) 24 countries are part of Latin America, six of them represented in this study. The researches were carried over in Brazil (18), Chile (8), Colombia (2), Mexico (2), Peru (2), Venezuela (1), and a non-identified Caribbean country (1). The non-Latin-American samples included participants from the United States (5), Sweden (4), China (3), Australia (2), Canada (2), Spain (2), Israel (2), the United Kingdom (2), South Korea (1), Denmark (1), Italy (1), Jamaica (1), Japan (1), Jordan (1), Holland (1), Lithuania (1), Norway (1), New Zealand (1), Singapore (1), and Turkey (1).

The main authors of all studies were from the same country as the analyzed population, with the exception of one study with Brazilian subjects, in which the main author was Australian, but the others were Brazilian. So, 98.5% of Latin American studies where written by Latin American authors. We considered the main corresponding author as the responsible, from whom we obtained full address to identify the country.

Methodological and results presentation in the articles quality

Non-parametrical tests were used to compare the differences between the scores in the Loney Scale of the Latin-American or non-Latin-American groups of articles, and Latin American or non-Latin American journals. Kolgomorov-Smirnov tests indicated that the distributions were not normal so non-parametrical tests were used. The median test considered the null hypothesis that both groups (Latin-American and non-Latin-American articles) would have the same median (p = 0.510). The Mann Whitney (U) distribution test also indicated the acceptance of the null hypothesis (p = 0.822) that both categories would have the same distribution. So, data confirmed the hypothesis that scientific articles that had their data gathered in Latin-American countries or in countries from different regions both had similar scores in the Loney Scale.

Table 1 presents the descriptive statistics of the Loney Scale items and the chi-square tests comparing Latin-American and non-Latin-American articles. Only on items three and seven significant statistical differences were found.

The methodological and article results presentation quality in non-Latin-American and Latin- American journals were compared. Comparing differences between the scores in Latin American and non-Latin American groups of journals, the median test resulted in the acceptance of the null hypothesis (p = 0.396). The Mann Whitney (U) distribution test also pointed to the acceptance of the null hypothesis (p = 0.144), meaning both categories would have the same distribution. So, the second hypothesis of this study was confirmed, since scientific articles published in non-Latin- American and in Latin American journals had equivalent scores in the Loney Scale.

Chi-square tests compared the frequencies of articles published in Latin-American journals with the ones from other regions in each item of the Loney Scale, showing significant statistical difference only in item 3, regarding the adequacy of size sample, which scored more frequently in non-Latin-American articles (see Table 1).

Discussion

Results showed that both groups, the Latin-American articles one and the one from other regions, presented low quality regarding methods and results presentation. Differences were also not noticeable between Latin-American and non-Latin-American journals. Data showed that the analyzed articles are not consistent enough and, therefore, health managers and politicians could not base their practices on such poor scientific evidence. Studies of prevalence are important to facilitate planning and the creation of guidelines for healthcare resource allocation and also for healthcare professionals to estimate the probability that patients will present certain symptoms (Reis, Ciconelli, & Faloppa, 2002). Besides, it is urgent to provide politicians, managers, and professionals strong and convincing evidence, as stated by Sánchez-Meca, Boruch, Petrosino and Rosa (2002). The epidemiological evidence about postpartum depression synthesized in this study must be used with caution, due to the weak methodology and results that cannot be fully used in an evidence-based management.

The articles used to make this comparison were a subsample of studies included in a systematic review, which obtained its data from primary sources of information. A distinguishing aspect of this study compared to other existing reviews (e.g. Sawyer, Ayers, & Smith, 2010; Vigod, Villegas, Dennis, & Ross, 2010; Melo Junior, 2011; Lobato, Moraes, & Reichenheim, 2012) was accessing fourteen databases in order to reach as many studies as possible. Another difference arises from the access to sources specialized in Psychology and developed to index Latin-American studies.

A high number of duplicate studies were found, whether due to the reuse of data in different articles or to different databases indexing the same article. This type of situation hampers the work of researchers, because even instruments developed to aid with the organization of references cannot solve the duplication of data and article repetition issues. Consequently, researchers have to spend a long-time filtering articles manually.

Selection bias must be controlled in systematic review studies with or without meta-analysis, since a single article can alter the results. The rate of agreement in article selection for systematic reviews is an important methodological aspect. Few studies discuss the agreement between evaluators in the initial steps of article selection (Oliveira, Oliveira, & Bergamaschi, 2006; Giannakopoulos, Rammelsberg, Eberhard, & Schmitter, 2012). Crossed evaluations by two specialists are hard to accomplish, due to the large amount of time required. This study counted with two reviewers: a specialist in Psychology and a specialist in Information Science, which may have contributed in some aspects but made it more difficult in others. The use of Artificial Intelligence, with natural and artificial language processing techniques, for example text mining, can be a viable way to substitute a second reviewer.

Could scientific articles written by authors who are Latin-Americans be good evidence? Could they aid in the decision making of public policy makers and healthcare workers? These are questions that concern authors and editors in the region. Editors state in editorials and published articles that biased attitudes towards papers submitted by authors from developing countries may exist. They argue that submissions made by Latin-American authors are sometimes dealt with prejudice or even editorial discrimination (Gibbs, 1995; Tyrer, 2005; Victora & Moreira, 2006; Sampaio & Sabadini, 2012). This is not a new discussion, but it continues to raise specialists' attention in the scientific publishing field.

Thirty-four articles from six Latin-American countries were compared with 34 articles from 20 countries from other regions. Brazil accounted for 26.5% of the Latin-American studies considered and Chile accounted for 11.8%. The 68 articles in this study were published in 54 journals, from which 23 were from Latin-American journals, so, 45% are from this region and indexed in LILACS, an important source of information that provides visibility to Latin American knowledge production on health. From 23 titles, ten were published in Brazilian journals. The Brazilian supremacy in the results of the study may be justified by the effort the country has made to provide visibility to its production, by creating and maintaining digital databases such as LILACS, SciELO, and BVS-Psi.

The eight items in the Loney Scale, used to evaluate the quality of methods and results presentation in studies about PPD prevalence, made it possible to compare Latin American to non-Latin American studies. It was observed that, statistically, there were no differences between both categories. The data prove that both Latin-American and non-Latin-American studies had the same performance. Irrespective of region, authors are delivering evidence of low methodological quality and inadequate results to support decision-making processes in public management.

Disagreements were observed in the average prevalence of PPD in Brazil, between studies carried out in the same city. For example: in Sao Paulo, prevalence rates of 33% (Coutinho, Baptista, & Moraes, 2002) and of 16% (Faisal-Cury, Tedesco, Kahhale, Menezes, & Zugaib, 2004) have been found. In Brasilia, 22% (Zaconeta, Motta, & França, 2004) and 11% (Santos, Martins, & Pasquali, 1999). With the accessed studies as reference, it is impossible to determine the actual prevalence in these regions and recommend a national prevention policy or a PPD treatment. The syntheses made with meta-analysis are little representative, due to a high heterogeneity, which happens in part because of methodological differences between studies. More information about meta-analysis and the prevalence found in this study can be analyzed in Silva (2013) doctorate thesis.

The exploration of the methodological heterogeneity of the articles is a highlight of this study. However, the methodology in this study is limited by the use of the Loney Scale that is made only by eight scoring items, with equal weightings. The results could have been different if a different scale was used, as the one developed by Giannakopoulos et al. (2012). This scale was not used because it was published after the data collection had been completed.

In order to consider an epidemiological study as scientific evidence, it is necessary the presentation of adequate methodological quality, which could allow including the articles in ystematic reviews and using them to support decisions. The combination of the results generated from different studies with good quality will certainly contribute to scientific knowledge reuse, avoiding the unnecessary duplication of efforts.

Without doubt, the methodological quality guarantees consistency and adequate results in the solution of research problems. Thus, the urgent need to improve researcher’s skills in creating consistent studies, and authors, in reporting them, is reinforced, as Witter (2006), Volpato (2013), Sabadini et al. (2009) and Trzesniak et al. (2012), Trzesniak (2014) pointed out. Problems in an article can only be corrected prior to its publication. Since the impact factor is measured after publishing, it is just an (somewhat frustrated) attempt of evaluating what is being published, with no effects in changing quality. The discussion about how important it is to create instruments able to evaluate the scientific quality of research with as much detail and strictness that science demands is recent. It is important to implement verification routines and provide orientation to researchers in the definition of the research methodology. Instruments to evaluate the quality of methods and results presentation before publishing, such as Boyle (1998), Loney & Stratford (1999), Giannakopoulos et al. (2012), and Vilas Boas and Silvany Neto (2012) scales, are relevant tools for suitable planning in epidemiological research.

It is important to evaluate published studies in order to include them in systematic reviews that support public managers' decision making. This topic also needs to be further expanded as lines of research in universities and research centers. Routines that guarantee the revision of articles prior to its publication are the priority. Reviewing articles after its publication cannot obviously improve the quality. Manuscripts evaluation routines used by the referees must be improved and specialized in order to advance towards methodological adequacy of studies, making them useful to systematic reviews and meta-analysis. The development of guidelines that increase the methodological quality should be a task for researchers. That will better allow science to increase its ability to improve people’s lives.

Conclusion

The hypothesis of the study was confirmed, since the Latin American studies and those of other regions presented similar scores according to the Loney Scale. However, the methodological quality and results presented in both groups is weak. The search for a sustainable science, that revisits and systematically reviews previously published knowledge will support decision-making processes in professional practices and public management. Beforehand, it is urgent to improve authors ability to write rigorously elaborated articles, with consistent methodology and reliable results.

References

Notes

* Research article. This article is a reduce version of the doctorate thesis of the first author: Sampaio, M. I. C. (2013). Qualidade de artigos incluídos em revisão sistemática: comparação entre latinoamericanos e de outras regiões. Doctorate thesis, Instituto de Psicología, Universidade de São Paulo, Brasil. Retrieved from http://www.teses.usp.br/teses/disponiveis/47/47132/tde-11122013-084214/pt-br.php

Author notes

^a Correspondence autor. Email: imaculadasampaio@gmail.com

Additional information

How to cite: Sampaio, M. I. C., Silva, G. A., & Meyer, S. B. (2018). Comparison of quality of scientific articles from Latin American and other regions. Universitas Psychologica, 17(4), xx-xx. https://doi.org/10.11144/Javeriana.upsy17-4.cqsa

Descargar