Reviewing Journal Rankings and Revisiting Peer Reviews: Editorial Perspectives*
两位《管理研究杂志》总编辑回应Macdonald和Kam对期刊质量和同行评审的批评，指出编辑可通过积极干预维护期刊声誉、缓解评审问题，并分享了《管理研究杂志》的实践做法。
In this article we respond to the key points made by Macdonald and Kam (2007) in relation to journal quality and the peer review process. Whilst we appreciate that their tone is intentionally provocative, the picture they present is one of unremitting gloom and reluctant acquiescence to a system out of control. It is as if the publication process has a series of self-supporting logics that separate it from any notion of publishing in order to benefit the discipline through the advance of knowledge and understanding. From this perspective the publishing process and the consequent content of management journals are presented as the outcome of a series of ‘games’ that put more emphasis on where someone publishes than on what they publish and its subsequent impact. Such criticisms are not new in that they have been vigorously discussed for decades across a range of disciplines. Furthermore, many of these issues are raised whenever academics get together and discuss their experiences of journal publishing. Given the frustrations and vagaries of the review and publication process, such complaints are understandable. But they deserve further scrutiny. We write this article as two of the General Editors of Journal of Management Studies. This is considered by the broad management studies community to be a ‘quality journal’ and during our time as General Editors, so far, we have overseen the reviewing of 1463 articles. We are therefore insiders. Our broad purpose is to show how journal editors need to intervene in order build and maintain a journal's reputation since it is not as impervious to change as Macdonald and Kam (2007) imply, and to mitigate some of the problems associated with peer review (see also Bedeian, 2004; Campanario, 1998; Miller, 2006; Starbuck, 2003, 2005). In doing so, we demonstrate that journal editors have actively to engage with these issues if they are to ensure that their journal continues to create value for the community at large. Macdonald and Kam's critique therefore reminds us why we have instituted certain practices at JMS. We begin with some general comments about journal quality before turning to the peer review process. Macdonald and Kam (2007) are right in pointing out that the importance of a journal's ranking on lists of journal quality, of which there are now a great number within countries and between countries (see, for example, the list available at Harzing's website, http://www.harzing.com), has assumed greater pre-eminence. Furthermore, whereas the ranking of journals in which people publish has always had implications for personal reputation, peer recognition and career advancement, increasingly it is impacting much more directly on institutional reputations and rankings and in turn on potential recruitment of staff and students. Individual publication choices and success are therefore assuming greater institutional significance. In part this is a result of the increasing importance and proliferation of a number of rankings of different types of Masters programmes (e.g. the Financial Times list of 40 journals used as part of their assessment of MBAs). These often contain an assessment of an institution's research output on the basis of articles that faculty publish in a limited and specified list of journals. These issues are also particularly marked in those countries where departmental funding is partly linked to an aggregate research quality score based on peer assessment of the published work submitted for each member of staff. We agree that a range of forces are making discussions of the relative quality of journals more numerous and heated than in the past. Furthermore, it is clear that the isomorphic pressures alluded to earlier are encouraging institutions around the world to give stronger guidance to faculty in terms of which journals they should submit their manuscripts to; publication in such journals has taken up a more prominent position in tenure and promotion decisions. Although incentive practices vary and can be considerable across institutions, the general outcome of this process is that more material is being submitted to a narrow group of journals that the community identifies as being of high quality. Macdonald and Kam are correct in arguing that this trend is reinforcing the status of these journals. However, they perhaps overestimate the solidity of a journal's prestige and standing within the community since the relative status of a journal can change over time, sometimes quite dramatically. Indeed, on the basis of Kuhn's (1962) approach to understanding how new scientific fields emerge, we would expect new journals to be created to publish work in these areas which may in turn become more prestigious as the quality and contribution of the work is recognized. For example, in the finance area, the Journal of Financial Economics (JFE) was established as a result of frustrations by some academics (Michael Jensen and others) that the Journal of Finance was not interested in publishing research in their areas (Jensen, 2006). Some time later, the Review of Financial Studies was established by those academics frustrated at the editorial policy of JFE under Jensen's tenure. Comparing the top ten journals in the ISI citation ranking in Management in 1995 and 2005, we can see that there have been some significant changes. Some new journals have appeared in the list and assumed a top position very rapidly, and there has been upwards and downwards movement of a number of established journals. Only the Academy of Management Review, Academy of Management Journal and Administrative Science Quarterly have remained in the top 10 during this ten year period. The point is that what are considered quality journals changes over time. It is also interesting to note that journals with a broader remit tend to be more stable than those with a narrower focus. Furthermore, over this period the number of broad-based management journals in the upper reaches of journal quality lists has increased. The polarization that Kam and Macdonald bemoan would be more worrying if the ‘quality’ journals were increasingly associated with particular domains, theoretical positions or methodological approaches. However, the reverse appears to have happened, with the consequence that there are a greater number of opportunities for a broad body of work. We as editors are very aware that we cannot take for granted the regard within which a journal is held. This is something that we have to constantly manage. For example, it does not take much of a slip in review decision times or time to publication for word to spread within the community and submissions to suffer. Whilst broad measures of journal quality, such as citation counts, may not change for some time, academic behaviour towards journals does in that word-of-mouth reports, particularly on certain aspects of the reviewing process, can greatly influence submission patterns. With this in mind we turn to discuss peer review. A quick immersion into the extensive literature on the peer review process indicates that the situation is possibly more serious than that identified by Macdonald and Kam. There are a number of failings that either individually or in combination may prevent or forestall innovative material from being published in highly regarded journals. In a much cited study, Peters and Ceci (1982) resubmitted 12 articles to the psychology journals in which they had already been published in the last 18–32 months. Prior to resubmitting the articles, the names and institutional affiliations of the authors were changed to fictitious ones (e.g. Tri-Valley Centre for Human Potential). To disguise the papers further a number of ‘cosmetic’ changes were made to the titles, abstracts and opening paragraphs in the introduction. Nine of the manuscripts were sent out for review (three were detected) and only one article was accepted, so that one journal made the same evaluation over the two periods. All that had changed was the authors' names and institutional affiliations. This article generated a voluminous response (over 70 commentaries in the same issue) and heated discussion over the nature of the peer review process. A review of this and other literature on the review process (e.g. Bedeian, 2004; Campanario, 1995, 1998; Miller, 2006; Starbuck, 2003) highlights three key flaws. First, reviewers are often accused of being overly harsh in their judgements. They approach manuscripts vigorously, looking to expose deficiencies, and fail to balance these with an equivalent appreciation of the merits. Reviewing thus becomes a fault finding procedure rather than one that seeks to engage constructively with the ideas, arguments and evidence in the paper. As Starbuck (2003) suggests, this approach may in part arise from reviewers seeing themselves as superior to the authors (in their role as reviewers). As he comments, ‘Occasionally reviewers seem arrogant, disrespectful, even nasty’ (p. 344). More generally, any system where people are evaluating each other, particularly anonymously, results in a hierarchical ordering that has to be carefully managed. Where editors privilege the reviewers' pronouncements, these change from being treated as suggestions to judgements that have to be attended to, no matter how carefully phrased. Failure to do otherwise may result in the rejection of the manuscript. This can lead to a situation where authors feel their ideas are unappreciated, undervalued and misunderstood. Taken to extremes, authors may feel that they are being forced to write the paper the reviewers want them to write, or respond to points that are superficial, misguided or just wrong. In these circumstances reviewers may become uninvited ghost-writers. Second, as Peters and Ceci (1982) suggest, a number of factors have been found to bias reviewer judgements (see Campanario (1998) for a full review). Where authors' identities are known, their institutional affiliations and social networks may influence judgements. However, these factors are reduced when a blind review process is used, although not eradicated because with the internet and electronic dissemination of manuscripts, an author's identity is very difficult to hide (see Hillman and Rynes, 2007). Here issues such as the apparent complexity of language, use of citations, appropriate jargon, sophisticated statistical procedures, and presentation of positive and significant results that do not merely replicate previous findings have all been found to influence or bias decisions. Third, and perhaps the area which has received greatest empirical attention, studies in a range of disciplines have consistently reported low overall inter-referee agreement (see Cicchetti, 1980, 1991; Gottfredson, 1978; Hendrick, 1977; Scar and Weber, 1978; Scott, 1974). Evidence within management studies is more scant. Starbuck (2003) reports a study he undertook whilst editor of Administrative Science Quarterly. An examination of 500 pairs of reviews revealed a correlation of 0.12. He concluded that ‘It was so low that knowing what one reviewer had said about a manuscript would tell me almost nothing about what a second reviewer had said or would say’ (p. 346). This review indicates that the points raised by Macdonald and Kam in relation to peer review have been identified in a number of disciplines. Peer review is not just a management problem; it is a problem for all academic disciplines. If we reject the arguments and findings of this body of literature then we close down any discussion as to how it might be improved. These criticisms of the review process therefore help remind us that as journal editors we have a responsibility to ensure that certain practices are instituted, that these are clearly communicated to all relevant parties and that we revisit them on a regular basis to ensure that they are working as intended. In the reflective spirit encouraged by this section of the Journal, we identify a number of areas where we have sought to mitigate the impact of some of these factors. Admittedly, our approach is designed to enhance the workings of the existing system, by addressing some key areas where actions can be taken, rather than seeking to replace it. This is only one possible approach. Easton (2007) examines more radical ones. Some commentators have argued that authors ought to be able to select, or at least nominate, potential referees (Bedeian, 2004; Campanario, 1998). This argument builds on the notion that, in writing their article, authors have become immersed in the field. They are therefore in the best position to determine their is able to the and nature of their Indeed, it may be that editors are not in area that is by the of their journal is a particular for journals that have a broad remit such as Journal of Management However, from the possible bias that this in that authors may people are to them and their it is not as if authors do not already have some influence on the of we all authors to write a note in which they identify not only their contribution in relation to work why the article is particularly relevant to the In doing authors clearly identify not only the broad community that is best able to the article also that of it which is relevant to the of the Some of these may or may not have for the Journal In a close of any article, if it is and in relation to should an editor to identify a number of the relevant knowledge We therefore to considerable to reviewers we are with the area that a paper is addressing and the that is being We that this for a review process than one or more reviewers as or reviewers only from the editorial In this it is possible to identify knowledge that is equivalent to that which was to the It is often the that they greater It should be that as editors we do not people to review articles. Indeed, reviewing is a that we all for a of we an to a potential reviewer we it clear that if they feel the article is their areas of they should to review. We would all that not article submitted to a journal should or be for Indeed, the for journals are 10 in some and as submissions This partly to a of also to that that a key role for journals is to The knowledge presented in journals certain and that it from other knowledge Editors of academic journals are not to a for in a journal therefore that an article is to certain about the of and empirical In an article is to advance knowledge in some In quality journals are on the basis of the of this contribution to many articles may fail to these Indeed, as we in an earlier on publishing in Journal of Management Studies in of the why a paper was was because it to a are therefore on evaluating the relative quality of a number of manuscripts since submissions have the potential to a significant In these circumstances a of the contribution of a paper is in areas that are and in more areas where the may be to a In the reviewers and editors want to see that a paper is doing more than the no matter how the paper may The may be of two broad First, a paper may to a new theoretical to an established it is to demonstrate the of existing and how the new us to see a quite Second, a paper may be opening up an new that has not been we might expect such work to be published in more prestigious journals only its has been For example, see the review of the of by There is some about what a a theoretical one and Some papers may in which people intentionally stronger 1995, and these out may in the of as are on how papers can innovative papers cannot many methodological or theoretical being considered or even innovative work has to to certain However, on the of a is a matter of that is the role of the since a decision often need to be made about the of a paper with the need for a paper to a of quality and with which the research has been In order to the of a paper that a significant has an process and also the review process, as we The of reviewer alluded to by Macdonald and Kam and in other studies may be Studies the of reviewer with some are based on the or overall of reviewers they with with and or These studies fail to two at a is in the comments with the reviews journal publishing is a matter of a reviewer It is our that the points made in reviews are more often than not identify and in the paper. Where they is in terms of the emphasis that they may on issues and then how they aggregate these into an overall The point is that reviewers may through the of (Bedeian, which result in rather than of These then their judgements as to the of the within a paper and the of Furthermore, whereas these studies are based on pairs of reviewers' Journal of Management management at least three In these it is the that at least two reviewers agree on the key issues and overall A second with the research on peer review is that it that the reviewers are and that they therefore the publication decisions. it is the responsibility to determine the reviewer reports an and evaluation of the and of a paper. It is also the role to as between reviewers' reports and authors' In this we to be editors rather than between authors and If our judgements are more than those of then low of agreement in the judgements by reviewers may not be as an as However, we do not how our judgements are and there is of evidence to that the in certain of But we do two that may mitigate our First, as an article the point of publication editor the paper and a of comments as to what they to be the and the a of between we to the If further is we then either the of General or if the is very we a member of the in that area to Second, where an particularly by a decision we have a one of the General Editors or Editors not in the paper and an editorial of the process, the reviews and editorial and a as to what should Given that we and our to authors in this is used very In the of times this has been in the decision has been in all one where the authors were a further to and the paper. As a further partly designed to the potential problems associated with reviewing and the section in in some other a different from regular papers in which to such We as editors may on such or may respond to put by This is an area with the of the for the of Management Studies which a for the Journal, we have sought to a number of that and for our work as The Editors of the Journal are for the in which they the Journal to the of the Indeed, all editorial positions in the Journal on its and in other general management journals. The Editors are an with of As Editors, our is in a number of The all from reviewers and on the nature of our general procedures, as as how they have been and in particular they or and so authors and reviewers submit to the comments based on their experiences of our These are all of the Editors, positive or and submitted to the as a standing are to the Editors on any of these In we with at a number of and their on what we are also on journal publishing at a range of in different of the our are to further and these opportunities for all to discuss any issues that may be of We also reviewers of a paper with a of the other reviewers' comments and of the to the such reviewers to see how their comments within the overall review process and them with an to on the decision possible we between reviewers and authors as as the editorial in the community is to us individually with regard to the we and the we We are all very aware that the standing of a journal is to the of the community at and in particular to the of to either submit their papers for possible publication or to as In list of at increasing the of the peer review process, he that should not be the on argument is that editors are not the to their Editors are for a of some of which their research standing in the to and so point is that these factors do not where might editors their and what of is Given the of article, on the for making judgements on manuscripts and A of editors of any journal within the management that to their they are people have often been in as of an editorial or editorial and have considerable as a reviewer for a range of journals. In other they have been making judgements on manuscripts in a number of to being an should also not that editors are increasingly part of a The of the editor are to a in submissions over from to we now have editors at the Whilst we may have responsibility for the reviewing of particular manuscripts, we are constantly papers and reviews with each other at different in the review process. Prior to a paper is by editor in order to ensure of decision making between the at editorial we have a standing where we discuss a number of studies so that we may from each or not of the community are interested in with to editorial is an attended by over one of the Editors in a the and a on an The was attended by over the by into writing papers for publication is and more for many people in the than an that are to discussions of journal and article quality more heated than before have to in the peer review process. However, these are issues that have in many not just We have to show in this article that as editors we are highly of the potential of the peer review process. A number of can be to mitigate these problems which we identify of journal with reviewer and greater in the of the journal that and on our work as and editorial and from other editorial as editors we to ensure that the peer review process for the outcome for authors at the same time our role as of the standing and reputation in the community at large. To the that is of and review practices more generally, we can Macdonald and Kam's as rather than for