Registered reports review for field experiments
为《运营管理杂志》推出注册报告评审新流程，旨在降低现场实验研究的发表风险，激励更多理论驱动的现场实验，提升运营管理研究的理论贡献与管理相关性。
In this editorial, we build upon the increased attention of the operations management (OM) community toward field experiments and the recent publication of the Pre-Approved Research Designs Special Issue that provided an initial test of Registered Reports as a novel review process for field experiments in OM. Addressing lingering concerns voiced by the editorial team and learning from the experiences of journals from other disciplines that implemented Registered Reports, we introduce a new initiative and outline a new review process in the Journal of Operations Management ( JOM $$ JOM $$ ) to motivate theory-focused field experiments in OM. Empirical research in OM strives to uphold two key virtues: providing managerially relevant insights and making meaningful theoretical contributions. Field experiments1—testing treatments in controlled digital and physical field settings—tend to possess the managerial relevance virtue because of their strong engagement with key stakeholders and the practical problems that they face. By contrast, the theoretical contributions of a field experiment depend on the extent to which its insights enrich our causal understanding of real-world phenomena. Critically, the degree to which scientific research based on field experiments in OM can be distinguished from consulting engagements and industrial experiments depends on the extent of its contributions to theory. Indeed, there are many different ways of contributing and forms of contribution to OM theory; however, they all serve a common purpose: to build a causal understanding of relevant OM phenomena. Field experiments can most decisively establish this causal understanding in the complexity of real-world operations (Ibanez & Staats, 2019). Thus, OM research that tests theoretically motivated treatments in field settings and provides sufficiently deep causal explanations for the observed effects, while also opening a door for future inquiry, possesses both virtues of empirical research. Consequently, we believe that the field of OM can benefit significantly from such research. Despite the fact that empirical research employing field experiments as the main methodological approach is on the rise in OM, it has traditionally lagged other disciplines such as economics, marketing, and information systems (Gao et al., 2023). For example, Simester (2017) reports that 37 field experiments were published in the top five marketing journals between 2010 and 2014, a period in which OM published a total of five field experiments in its top five journals (Gao et al., 2023). Between 2005 and 2021, only 31 research articles with field experiments were published in three top-tier OM journals: JOM, Manufacturing & Service Operations Management (MSOM), and Production and Operations Management (POM). These articles examine interesting research questions pertaining to operations, test causal impacts of managerial treatments on operational outcomes, and generate insights regarding how to improve operational processes. Few of these papers, however, are motivated by theoretical concerns, and the relative paucity of such field experiments in the OM literature signals missed opportunities for making theoretical contributions using field experiments. Indeed, when conducted, there is a risk that treatments in OM field experiments might be guided predominantly by a problem-solving focus, lacking theory-building, elaboration, or testing foci (Fisher & Aguinis, 2017). From our own experience, field experiment collaborations are more likely to be implemented when a research partner already has the intent to administer a specific treatment to improve organizational outcomes, and researchers contribute to testing this treatment in a rigorous way. The question becomes whether an experiment makes enough of a theoretical contribution when testing a treatment proposed by the partner organization that is motivated by achieving internally set goals—a concern of additionality (Luukkonen, 2000). This is not to suggest that testing interventions designed with a problem-solving focus cannot lead to significant theoretical contributions. Such cases are common in intervention-based research, where the researcher takes an active role supporting the change effort (Chandrasekaran et al., 2023; Oliva, 2019). However, we argue that if the problem-solving focus is prioritized in field experiment-based research, the theoretical contributions might be insufficient when little ex ante reflection has been given to the theoretical background and implications of the focal treatments, their causal impacts, and potential boundary conditions. Further, field experiments as a research design sacrifice contextual generalizability to achieve greater realism in studying the phenomena of interest (McGrath, 1981). Findings from the field experiment in one setting will be practically useful in other settings only to the extent that those settings share critical attributes with the original setting (i.e., the boundary conditions of a theory). Thus, similar to field research, insights from field experiments need to emphasize theoretic generalizability rather than generalizability to a population (Lee & Baskerville, 2003; Meredith, 1998, p. 450; Yin, 2003, p. 10). Ultimately, we attribute the general paucity of OM field experiments and the seemingly dominant problem-solving approach to conducting OM research using field experiments to the potential challenges and risks embedded in the overall process. It is extremely challenging to identify a theoretical gap or tension in the OM knowledge base, design a treatment that may fill this gap or resolve the tension, identify the right institutional setting, and develop relationships with a partner organization that is receptive to testing the proposed treatment. Researchers also need to design and execute the field experiment rigorously, and develop a high-quality manuscript. In addition to all this, and only after conducting all this work, authors might end up facing difficulties in publishing their manuscripts as a result of specific methodological choices that editors and reviewers might challenge with the benefit of hindsight. In short, the sheer amount of effort involved in each of these stages, and the uncertainties involved in the traditional peer review processes of OM journals can jointly explain the current state of research based on field experiments. Our thesis is that by reducing the “downstream” risks of OM researchers associated with the peer review process, we can stimulate the “upstream” activities in the process of conceptualizing research designs. In turn, this should ultimately lead to more field experiments that uphold the two virtues of empirical OM research: high managerial and theoretical relevance. To this end, JOM is introducing an initiative for a peer review process with registered reports, for handling manuscripts that involve field experiments in OM. The next section provides a brief background for registered reports and § $$ \S $$ 3 outlines the criteria for submission and the review process under the new initiative. Registered reports have been launched in some capacity by over 300 journals across different disciplines, including MIS Quarterly, Nature, PLoS One, Academy of Management Discoveries, Journal of Development Economics, and Human Resource Management Journal.2 Moreover, registered reports have been deployed successfully in the Pre-Approved Research Designs (PARD) Special Issue published by JOM (Chandrasekaran et al., 2024). These reports build on a similar premise as the now well-established preregistration movement: authors publicly outline their hypotheses and introduce the key methodological decisions before executing their deductive research project. However, registered reports take this idea one step further, by conducting the review process in two stages (Foster et al., 2019). In Stage 1, a proposal for the field experiment is refined through a rigorous peer review process. Then, after execution of this proposal, authors develop a full Stage 2 manuscript, which undergoes a final set of reviews. This two-stage process increases the transparency and credibility of deductive research (Timming et al., 2021). It mitigates the potential influence of questionable research practices and authors' biases throughout the research process (Briker & Gerpott, 2023). Moreover, authors are incentivized to produce theoretically and methodologically robust research instead of identifying a specific result, often one that may not generate a strong dissonance with established theoretical paradigms (Chambers, 2019). For further discourses on the merits of registered reports and authors' experiences with this format, we point the interested readers to Briker and Gerpott (2023), Foster et al. (2019), Chambers (2019) and, for advice that pertains specifically to field experiments, to Gerpott et al. (2024). The two stages of the review process work as follows. In Stage 1, authors submit a proposal with their intended research design and expected a priori theoretical contributions. This does not, of course, preclude the potential for a posteriori contributions, following the eventual execution of a design. However, both the theoretical arguments motivating the treatment, and the theoretical contributions must be sufficient in and of themselves to justify the work. In the assessment of the design and potential for contribution, this Stage 1 manuscript undergoes a formal peer review process, where reviewers and editors voice specific concerns and make formal suggestions to improve the quality of the design and its potential insights. Such feedback can be especially critical for field experiments, as there is often a single shot to get the design right (Ibanez & Staats, 2019). Crucially, this Stage 1 peer review lowers the risk to authors, who get a clear signal about the relevance of the research question and the potential validity of their experimental design. If the review team considers the motivation, theoretical contributions, and design to be strong enough, authors may then proceed to execute the field experiment and develop a Stage 2 manuscript. The Stage 2 manuscript is then assessed given the context, and in light of the plan provided by the Stage 1 review. This narrows down the scope of challenges that reviewers and editors make: feedback is only provided on any lingering methodological concerns from Stage 1, on the execution of the field experiment proposal, and on the extent to which the theoretical contribution achieved is sufficiently clear. Of course, we recognize that authors are the ones with direct access to the field setting and are in control of the experimental execution. Therefore, the author teams are ultimately responsible for ensuring that their design achieves unconfounded estimates and that any potential threats to internal validity (e.g., spillover effects, Hawthorne effects, etc.) are carefully controlled for. The review team contributes to these efforts by sharing recommendations on the proposed design and theoretical contributions; yet, authors are expected to reflect on whether such suggestions are enough to preempt any alternative explanations that might potentially compromise the theoretical contributions of the field experiment. Reflecting on the experience of the PARD Special Issue's authors and editors, Chandrasekaran et al. (2024) conclude that “the evidence seems to be quite strong that review of Stage 1 reports is a worthwhile endeavor” and further claim that “limiting the focus on field experiments was the appropriate call.” Yet, they acknowledge that a number of crucial challenges need to be addressed for this process to be successfully implemented at JOM. First, they point out that timing is critical for the Stage 1 review process to work. On the one hand, if the review cycles are too long, partner organizations might lose their patience and interrupt the collaboration. On the other hand, when assessing Stage 1 proposals, reviewers need to balance complex tradeoffs in an uncertain setting, which understandably takes time. In the Special Issue, editors and reviewers erred on the side of caution, which led to long review cycles. Second, Chandrasekaran et al. (2024) note that few reviewers and authors have the methodological and substantive experience required to evaluate field experiments. Finally, they point out that many field experiment proposals seemed to lack a strong theoretical contribution. In what follows, we propose a structure for a peer review process that builds on the successful implementation of the PARD Special Issue, along with specific elements that address these open questions. We build on the assessment guidelines and processes that have been implemented by leading journals,3 which we complement with the learnings from JOM's PARD Special Issue (Chandrasekaran et al., 2024). Below, we describe the submission and review processes for the new initiative. Given that Stage 1 reviews are time sensitive, we have created an expedited process that will be handled directly by the Editors-in-Chief. To make sure that a swift review process is maintained, authors are encouraged to be concise in developing their Stage 1 manuscript, which should be shorter than a regular paper. Stage 1 manuscripts will undergo a maximum of two review rounds. In round 1 of the Stage 1 review process, the submitted reports can be rejected, invited for revise-and-resubmit, or accepted in principle (AIP). In round 2, the decision will either be AIP or reject. Authors of promising rejected designs may be invited to submit their final manuscript (after execution of the field experiment) through the regular submission process, but will no longer be considered for the registered report review process. Authors will have 3 months to revise their Stage 1 proposal between consecutive review rounds. The target will be to complete the full review cycle within 6 months of initial submission. The authors of reports that are AIP will be invited to submit a completed manuscript within 6 months of the AIP decision, unless otherwise agreed as part of the Stage 1 review process. Reviewers are free to enter additional comments on the record, but they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, editors should not hold authors obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria. In this article, we motivate and introduce the 3R initiative as a new platform within JOM for stimulating, developing, and publishing theoretically relevant field experiment research in OM. We believe that field experiments as a research method hold significant potential to spur bold developments in OM theory. Recognizing the inherent challenges associated with designing, executing, and publishing research that uses field experiments, the 3R initiative offers a two-stage review process. This process aims to reduce the peer-review and publication risks perceived by aspiring field researchers, providing feedback on the theoretical and practical contributions that are sought as well as on the design of the field experiments prior to their actual execution.4 The 3R initiative will officially commence in January 2025 and JOM will be accepting Stage 1 submission for review under the process described above; please see https://www.jom-hub.com/departments for the mission statement of the initiative and guidelines for submission. We encourage the JOM community and the OM community in general to take advantage of and contribute to the 3R initiative. In line with the spirit of OM, we expect to learn much from the experiences of submitting authors and reviewers, and we remain committed to continuously improving the process for the initiative. The authors thank Elliot Bendoly, Tyson R. Browning, Aravind Chandrasekaran, Suzanne De Treville, Henrik Franke, Jan C. Fransoo, Bradley R. Staats, Kaitlin D. Wowak and JOM's Department Editors for providing developmental feedback regarding various aspects of the registered reports review initiative and the manuscript.