Registered reports in operations management: Lessons from an experimental trial
通过特刊试验，总结了运营管理领域采用预注册研究设计（注册报告）来鼓励实地实验的经验，包括流程、挑战和成功案例，对希望降低实地实验风险的研究者有用。
Field experiments involve the practice of conducting controlled interventions wherein researchers collaborate with practicing managers to study the effects of such interventions on a subset of subjects, processes or entities (Ibañez & Staats, 2019). In recent years, Operations Management (OM) as a field has seen significant interest to conducting field experiments as evidenced by studies in healthcare delivery (Anand et al., 2021; Staats et al., 2017), retail operations (Chuang et al., 2016; Craig et al., 2016) and recycling (McKie et al., 2024). While there are several benefits to conducting field experiment such as improved external validity, reduced observer bias and improved causal inference, field experiments require considerable relational investments, often require substantial time in data collection, and carry significant risks such as loss of access to participant sites through attrition. Given these points, OM researchers often shy away from field experiments as primary research method. By doing so, however, they miss an opportunity to ask and answer bold questions that can challenge existing OM theories and offer richer insights. As an illustrative example, consider the age-old question on why operational excellence initiatives (e.g., Lean/Six Sigma, Process Management) fail to sustain themselves over time. There have been many studies exploring the factors that influence the adoption and use of operational excellence in a variety of industry contexts (e.g., Anand et al., 2021; Anderson & Chandrasekaran, 2024; Shah & Ward, 2003; Sterman et al., 2002). These studies have capitalized on several research methods including case studies, surveys, analytical models, and econometric methods. Yet, the explanations delivered from these studies leave important questions unanswered. One way to address this gap would involve the use of carefully constructed field experiments, with specific sets of interventions supporting operational excellence initiatives adopted by organizations or their units, with some controls for otherwise confounding factors, and with monitoring in place to observe their impacts over time. Unfortunately, the challenges of recruiting enough firms to secure an adequate sample size, controlling for potential spillover effects and attrition, and ensuring compliance in the experimental protocol, renders a potential research project with lead time that could run into years and with significant risk and uncertainty. Accordingly, given time pressures on faculty publishing, the paucity of such rich studies into these complex settings is far from surprising. Instead, we continue to make incremental knowledge creation through alternative research designs. For this special issue, we were particularly motivated by the prospect of supporting authors interested in questions that required field experimental design, but who were otherwise worried about the risks in conducting them. To that end, we developed a process to encourage OM scholars to conduct experiments with interventions to advance our understanding of OM theories by reducing the risks and intrinsic cost of experiments through the pre-approval of their research designs. For this pre-approved research design (PARD), we borrowed the two-stage research approach, also known as Registered Reports, common in other fields that use experimental designs – for example, healthcare's use of randomized control trials – and is emerging as a regular practice in other outlets such as Nature, PLOS One, and Academy of Management Discoveries. During the first stage, the authors were invited to submit their experimental design with specifics on their interventions for review, the so-called Stage 1 report. These designs include proposed research questions, articulations of how addressing these research questions might contribute to theory, intended interventions, and experimental sample and protocol. They would also include the nature of the experimental design (randomized control trials, pre-post), measures collected in the study, power analyses with expected attrition rates, intended managerial contributions from the work, and, if available, partner identification and confirmation. These designs underwent a full review that focused on evaluating the need for such experiments (i.e., importance of research question), factors studied and controlled in the designs, power analyses to determine appropriate sample sizes, and the relevant analyses planned for the designs. If approved, these research designs were published in the JOM website, and the publication of the final paper, irrespective of the results obtained, was guaranteed. Associated Stage 1 documentation is often published as protocol papers in respected journals (e.g., Trials, Implementation Science) before beginning their data collection. Once a design has been approved, the authors engage on the second stage and conduct the experiment as approved. The authors then had the opportunity to submit the full paper with the pre-approved design and the results from the experiment, the Stage 2 report, which were then published in the SI after a quick round of reviews. We limited the submissions to the special issue to field experiments, as opposed to laboratory experiments, as ameliorating these risks and costs while working with a research partner, and collecting data from a real-world context, could arguably be viewed as having greater value. The special issue, and its unusual review and publication arrangements, were designed to encourage field experiments in the OM context as it creates opportunities to improve experimental design and thus reduce the risk of experimental failures and provide an opportunity to publish their results even if hypotheses are not confirmed. However, the editorial team was aware that supporting such process would challenge to the journal's existing reviewing and editing processes and capabilities. As such, the special issue was also conceived as a trial run for identifying the stress points and potential solutions to continuously support this process. In this editorial, in addition to introducing the papers that were part of this process, we present the learnings for both the authors and the handling editors of the special issue. We conclude with our reflections on the main insights gained from this trial and the remaining challenges for deploying pre-approved research designs for Operations Management research. The idea of PARD is somewhat new to OM. As a result, the number of papers submitted were smaller than a typical SI. Overall, we had 18 manuscripts submitted to this SI. One third of these submissions were desk rejected. Specifically, three proposals were rejected because researchers had already collected data from the experiment and were seeking for approval for the data analysis, thus defeating the purpose of the PARD. The other three proposals were rejected because experiments were testing technological options to improve processes – that is, the traditional design of experiment (DOE) approach (Montgomery, 2019) – and there was no OM theory behind the experimental design. Of the 12 papers that were sent for review, four of them were rejected because there was not enough of a theoretical contribution. Four additional papers were rejected because of issues with the research design, that is, inability to randomize or control for sample attributes to rule out alternative explanations. These two reasons for rejection accounted for almost 50% of the submitted papers and they correspond to the main reasons we see for rejections when reviewing OM empirical work: lack of contribution or relevance, and identification challenges that prevent causal interpretation of the estimates (Cunningham, 2021). We realized that working in the field limits the researchers' ability to control or randomize the sample or manipulate the timing and intensity of the treatment – these are the realities of field experimentation. Nevertheless, being aware of the implications of these limitations is an important part of the process of deciding whether the proposed experiment will be effective in addressing the research question. Three papers were rejected for what we called ‘inverted design’ process. These author groups attempted to leverage an existing experimental opportunity by operationalizing a treatment and constructs surrounding these events, that is, putting the experiment before the theory. Note that this is the context of a ‘natural experiment’ (Shadish et al., 2001) where researchers take advantage of a naturally occurring intervention to test elements of a theory. These proposals were rejected because there was no possibility to modify the experimental design, thus, again, defeating the purpose of the SI. Finally, one additional paper was withdrawn after the author team realized during the revision process that the theory they were attempting to test was not developed enough to have explicit causation mechanisms outlined, that is, the theory was still in its nascent or intermediate stage (Edmonson & McManus, 2007) and this ambiguity created problems in their measurement scales. Out of these 15 rejections to the special issue, five groups of the authors were encouraged to resubmit their work as a normal JOM submission. For three of the papers the setting and the interventions were intriguing enough that they offered a possibility of insight outside of the causality testing that could be achieved through an experiment. Two other papers were thought to be a better fit to the Intervention-based Research department (Chandrasekaran et al., 2020; Oliva, 2019), as either the intervention was not detached enough or there was a lack of control group. The three papers that survived the Stage 1 review process share the following characteristics. First, they all consider mature theories with explicit causal hypotheses and the experiment is designed to resolve paradoxes or empirical uncertainties. Second, the contributions of the research questions benefit the OM community and are not merely focused on the benefits of the individual sponsor of the research, that is, general OM theory. Third, there is a clear logical dependence where the research question is driving the research design and not the other way around. All accepted designs include a robust description of methods, protocols, measurements, and discussion of power. It was that methodological detail that allowed the review team to evaluate the research design and provide specific feedback and suggestions to improve it. Finally, all the successful author teams had the ability to work with sponsoring firms to identify proper controls and randomization of confounding factors during the review process. The three accepted designs all include randomized treatment designs with appropriate control groups. We present their designs and main findings in section 5. Managing the review process for the special issue provided several lessons that can help move the field forward. Some of these takeaways are specific to evaluating field experiments while others are important to reflect on as we consider accepting study designs, in contrast to final research papers. A first takeaway is that we currently have a limited resources in terms of the authors and reviewers within OM who are in deploying and evaluating field and it in the special issue. However, and the of researchers is there are resources to support the of in this For example, the field of on field experiments (e.g., & and both and Staats and and provide in the field of papers the of reviewers with to methods and this be within the however, as field experiment is still being it a team with and then having the it. and are important as it reviewers to our second the important of in conducting field The of experimental in OM from researchers conducting laboratory These share many with field experiments but are not conducting field experiments it is often to make and All research to some and studies evaluating them carefully with to a given question. Field experiments the to studies, and the potential theoretical might need to be For example, it not be to control for all factors, the way that one could in a laboratory In experiments, it is not to conduct of data and the design and the treatment be carefully thought experimental designs have to sample or identification to a often carefully some in for greater external The authors address however, the of the field experiments be than laboratory One that we to is that of To submit a field experiment for approval a for the study have been However, the has been they often to and run the study – not for a review process to be As a result, the timing and from can move than our review process The authors with but also this that the authors and editors be in to address the of The third takeaway the that the review process of a study before it can be a to the authors as it the opportunity to consider alternative explanations and design issues while there is still time to address them. It is not to a field experiment. are then in a place of deciding if a study is enough reviewers can engage on this and determine whether the is given the intended design and proposed theoretical to the the review process consider that has not been and evaluate it time we evaluate research we are about 1 and 2 we a paper that be published or are we accepting a paper that has a such that it be The review process carefully a paper, and the team its to address this challenge through with the In the Registered process where we have to the final paper, the reviewers are to about what out of a given with this process and to some still is, that we the The over that is to be for lack of a better that a review team for is one in all but we encourage reviewers and editors to carefully about this putting the authors through the and that if a design is then it is Finally, during the SI PARD process, we the review team and the authors to about the benefits of conducting a study that a is, the review team the in a that has theoretical and methodological a after collecting and the In our there is in OM to and scholars to the idea of from such such as have to encourage such as evidenced by journals such as Academy of Management A of is to publish questions that address and or unusual and that empirical that be by existing theory We that researchers these to make our field and some of the author teams that in the PARD process their and access with the sponsor the of a review after the design stage, but to the of the experiment, challenges for the author They often had to their experiments for the review process to their the experimental design to address the review In this section we what the author teams have to be their main insights in handling the design process and their with First, all the authors teams it clear that the individual experiment for the special issue was not an intervention but part of a working with their It was because of the existing that the author teams were by the to design a and, when to modify the proposed research design on the The working with the sponsor also created the of for the author teams to the of the experiment while it was through the review process, for example, working on other or of the Second, all the author teams that the with the sponsoring required two of and One the to the support for the research and its and one with the managers for or the experimental treatment and They all on the that these two required and Third, all the author teams their main as a to address the and with the design and required to address the research question. While to a research question required explicit with the they all a challenge in a the ability to the and measurement with the of the research design. it to for two of the teams how it was to to managers for suggestions as they have a understanding of the research context, and their were often for the Finally, all research teams on the work required after the of the experiment to make the results and to the example, to help the of or in on the to for by the example, work to address new or emerging from the experimental While some of these insights might in the of insights interventions and contexts is however, is the of all the work that behind the and after the experiment. to engage in this of and ability to them is one of the main reasons we not see enough field experiments in our 1 the papers that were accepted in this SI. The field experimental settings industry contexts that include and The research questions in these studies our existing theoretical understanding and the field experimental design was the appropriate methods in of these For et an important issue of bias that can operational in retail settings and required testing it in the field to a It is also to that the experiments were in that include and which that organizations all over the are to in studies that advance as as theoretical The papers also that were with their research questions and the time to For it was for the study by et to randomize the questions into one of the five experimental wherein the about the were or to the In the case of et randomization was the wherein the in one of the about the of and through their To confounding the authors a a as a control group. In the case of et the authors had two (i.e., and with two (i.e., in The authors carefully randomized their these four and adopted additional to For the access in the with the were for 1 to that the the as intended in the In terms of the intended it is also important to that not all the hypotheses proposed in the Stage 1 were in the experiment. For et for improved when are to take that their However, their experimental results that identification but not influence The authors also the reasons for the lack of support for their hypotheses the experimental design that our understanding on identification and Overall, the PARD approach to feedback on the designs before collecting data that the authors developed research questions that our existing theoretical on have we it make to support in OM are the challenges an the to be that review of Stage 1 is a All the successful author teams significant on their designs as of the review process and feedback to the field experiment. we were to for publication all Stage 2 after A the review process that the editorial and review team that the research design could answer the proposed research question while out potential alternative explanations. The questions could we a on this and there be an alternative for a the review teams The review process in for of the research additional controls to the experiments or of the or the of the power of the this of is that the author teams have for themselves or through a review from Nevertheless, we that the process of and the experimental protocol to the that it to review and improve is an important part of the process that not all the author teams are to engage we not have on the of proposals that were we that the feedback provided to proposals have given to the authors to either their experimental treatment or the intended of their research. If all the pre-approval process is to the to and the experimental protocol, we that on is a significant contribution. By reducing the risk of a the that a will be than by the we that the pre-approval process the purpose for the special issue to encourage experiments the theories or alternative explanations. risk and of these results encourage work with firms and for the field to the PARD approach authors the and & while ensuring that reviewers and editors are with these in a that not our theoretical knowledge but is also for the sites to the to the review process, that is, we given the costs and challenges of the Stage 1 the on field experiments was the appropriate The inability to field experiments and the risks in not having the appropriate research design the of the However, we see two substantial challenges for the field with this of reviews. First, as we were by the number of submissions with theoretical or contribution. We might have some bias as the for papers to the special issue proposals for field experiments, where having access to a research is Nevertheless, the main purpose of an experiment is to whether a causal is for a result, that is, testing a or testing be the main to an experiment, with access to a being also Yet, than of the submissions to the special issue being rejected because of a lack of explicit to a theory. We that these proposals be the of our in our to for methods the potential cost of a understanding of the theories of OM or the theory and testing processes As a we to take a of what it to have a theory of Operations Management & and for the community to than The second challenge in terms of potential from the that all successful teams had their experiments with sponsoring While we not see this as a we that the possibility to research designs after is in the context of and a we a of proposals when the author teams were not to the experimental context to address a challenge for the field as in are not to these for that have to them to share the We that the possibility of reducing the cost and risk of field experiments can that and encourage a with a process there are a of issues to sustain the review of experimental designs. First, it is not clear what be the for a Stage 1 review process. all design are by the author or we that it is not to address to be of a for reviewers and As we had to publish the results of design, in we had this all or in In this might have been as we might have the authors to a than given the context of the experiment and the for that of might have in in the review process. are the to a Stage 1 is that we need to before we can consider supporting in A challenge is whether these Stage 1 manuscripts can as an for other researchers to While we not publish these in but had them as such Stage 1 manuscripts as research design in other fields such as healthcare (i.e., such design is that OM as a field is not for such but on the authors and editors to these to improve and research designs. the process there are two design issues that need to be out for the process to be effective and A Stage 1 is not a full research and a such to be through a process and with than the processes we have in place for normal reviews. First, from the in and is clear that review time is in this a review of JOM review is often We currently lack the to reviewers to what they are doing and a review to provide feedback in of Given the of submissions in our PARD be thought of such of manuscripts that require feedback to with the The second challenge is how to this review process. some methodological is required to the of the research design. However, reviewers need also to be of the challenges of and a research design in the and they need to a of how to the the methodological and ability to the is through we the for the reviewers to also be to the theoretical contribution of the and the that the is from the design data is it is clear the potential of reviewers for this is that can be developed and through While work to be we that the from these can have a on the We are motivated by the results emerging from this trial and the of lessons and insights that we gained from the We the editors and reviewers that in this process and the of and their to experiment with this We that the results of this trial encourage JOM and the field to continue working to address the issues of this process.