Empirically grounding analytics (EGA) research in the Journal of Operations Management
本文是《运营管理期刊》的社论，阐述了经验基础分析（EGA）研究的概念、重要性及期刊对这类稿件的期望，旨在弥合分析模型与实证数据之间的鸿沟，提升研究的实践相关性。
Empirically grounding analytics (EGA) is an area of research that emerges at the intersection of empirical and analytical research. By “empirically grounding,” we mean both the empirical justification of model assumptions and parameters and the empirical assessment of model results and insights. EGA is a critical but largely missing aspect of operations management (OM) research. Spearman and Hopp (2021, p. 805) stated that “since empirical testing and refutation of operations models is not an accepted practice in the IE/OM research community, we are unlikely to leverage these to their full potential.” They named several “examples of overly simplistic building blocks leading to questionable representations of complex systems” (p. 805) and suggested that research using analytical tools like closed queuing network models and the Poisson model of demand processes could incorporate empirical experiments to improve understanding of where they do and do not fit reality, highlighting “the importance of making empirical tests of modeling assumptions, both to ensure the validity of the model for its proposed purpose and to identify opportunities for improving or extending our modeling capabilities. The fact that very few IE/OM papers make such empirical tests is an obstacle to progress in our field” (p. 808). They concluded that “Editors should push authors to compare mathematical models with empirical data. Showing that a result holds in one case but not another adds nuance and practicality to research results. It also provides stimulus for research progress” (p. 814). These arguments remind of Little's (1970) observation that many potentially useful analytical models are not widely adopted in practice. Thus, EGA research can help to close two major gaps between (1) the empirical and analytical subdivisions in the OM field and (2) scholarly output and practical relevance. As a journal focused on empirical research, the Journal of Operations Management (JOM) seeks to encourage EGA submissions and publications, but doing so requires our community of authors, reviewers, and editors to share an understanding of the expectations. While such contributions have been encouraged for some time in the verbiage on the JOM website, a more formal effort to draw out examples of EGA research was driven by an editorial call (Browning & de Treville, 2018), and we have since had many discussions, panels, webinars, and workshops to continue to develop and communicate the expectations. This editorial represents another step in that development. In a general sense, an EGA paper combines mathematical, stochastic, and/or economic modeling insights with empirical data. Modeling captures non-linearities and elements of distributions and allows these parameters to be incorporated into decision making, whereas empirical research transforms observations into knowledge. Analytical models are evaluated in terms of their results and insights, which might prompt further extensions to or modifications of the model, including new or different inputs and recalibrations. Most modeling papers stop there because the primary contribution is the analytical model. Although some realism is required, it falls short of empirical grounding, and a gap is often left between the model's insights and what implementation in practice will entail. Filling this gap by empirically grounding an analytic model creates knowledge by linking analytical insights to what has been observed using empirical methods (such as case studies, action research, field experiments, interviews, analysis of secondary data, etc.) to establish a theoretically and empirically relevant research question. Moreover, since analytical models tend to make many simplifying assumptions, EGA research can help tease out where these assumptions are valid and where they excessively bias results. Figure 1 situates two kinds of EGA research with traditional analytical models. Typically, publications with analytical models focus on the center of the figure: the model details and the insights derived from it. The left side of the figure refers to the empirical grounding of the model, that is, whether there is empirical evidence to justify the model's assumptions, parameters, and specific calibrations. The right side of the figure refers to empirical evidence of the impact of the model, that is, whether the model fits the problem situation, can be used in real time, and provides useful output. The concerns expressed above by Spearman and Hopp stem from the expectation that a single paper will present both the model and the empirical testing. This expectation leads to the situation in which empirical testing serves only to demonstrate the model in action, rather than preparing the way for the insights encapsulated in the model to be deployed in practice. Given the lack of openness (among some) to publishing further empirical testing, the model may be accepted by the research community based on its analytical strength—but the first question anyone from practice will ask is, “Who else has used this, and what were the results?” JOM is interested in papers that address questions related to both empirical sides of the development and use of analytical models—their grounding and their impact—that is, either side of Figure 1. Are data available for model parameters? How well do the results work in a variety of real situations? Are the results practically implementable? Are they useful to practitioners? Will managers actually use them? Figure 1 thus highlights important but often undervalued elements encountered in empirically grounding insights from analytical models. Both sides of Figure 1 require a significant amount of empirical research—and it is empirical work on either side of Figure 1 that is the primary contribution of an EGA paper in JOM. It is usually expecting too much of single paper for it to address both sides of Figure 1 sufficiently. On the left side of the figure, analytical models are linked to data and observations of reality: Their assumptions, parameters, and calibration should bear resemblance to a real situation. Here, an empirical contribution focuses on the empirical discovery of a new regularity (new assumption) that leads to the development or revision of analytical models to exploit that new-found regularity. Contributions on the left side of Figure 1 represent the “heavy lifting” of empirically grounding models, transforming mathematical insights into a form that permits measurement and application, and making existing mathematical and modeling insights available to address an observed problem. Finding, collecting, preparing, and analyzing data requires a substantial amount of work—especially when it is impossible to obtain data from the company or situation on which a model was developed. Key parameter values may be unobservable and require estimation from available data. Also, the assumptions that made the model tractable may not hold in key contexts: Empirical research needs to address this tradeoff between parsimony and accuracy. At JOM we want the value of such research to be recognized. Contributions on the right side of Figure 1 assess an existing model's performance in real contexts and address emerging issues. Experiments, field tests, and intervention-based research methods are likely candidates for this type of EGA research. These contributions typically build on the empirical insights from the left side of Figure 1 and the insights/results of prior analytical models, but they add the new knowledge created when the effect on decision making of the nonlinearities captured by analytical models is observed empirically. We classify these contributions as EGA as well, although one could also consider them as “analytically grounded empirics.” Engaging in either side of Figure 1 will trigger an improvement process where the model is revised based on new assumptions or the availability of new data, and/or its effectiveness (usefulness) and efficiency are increased in the real-world context. This will require a toggling back and forth between inductive reasoning to capture the new empirical evidence, deductive reasoning through the analytical model, and abductive reasoning (Josephson & Josephson, 1994) to reconcile the emerging insights and empirical regularities. The surprising and unexpected results that trigger the abduction logic indicate that both the model and its empirical grounding matter to creating actionable knowledge. Creating space for abduction is one of the reasons why successful EGA contributions are more likely to come from the sides than the center of Figure 1. Again, JOM encourages papers that tackle either side of Figure 1 and empirically motivate a significant revision to existing models (see examples below). The above-described empirical grounding is often replaced in the modeling literature (where the focus is the model formulation and insights) by either stylized assumptions (explicit simplifications that still capture key elements of the problem situation) or artificial (simulated) data to assess the model performance. Table 1 identifies four types of modeling efforts, depending on the source of assumptions and data for assessing model performance (cf. the left and right sides of Figure 1), together with the key contribution of each type of study (the italicized terms in each cell). The upper-left quadrant (a stylized model tested with artificial data) is common where an analytical insight is a paper's primary contribution. Empirical grounding can take the form of either moving to empirical data applied in an actual situation (lower-left quadrant) or observing areas in practice where the model requires extension (upper-right quadrant). Analytical Insight Parsimonious Causal Insights Explore implications Test of Value Quality of approximation Solutions/Designs Performance improvement An effective EGA process will encourage moving across the quadrants in Table 1: Progress made in any quadrant can open new doors in adjacent quadrants. As we gain fluency in managing the EGA research process, it will become easier to take analytical insights into the field, transforming them into effective interventions through a multi-stage process that links analytical and empirical publication outlets. All else being equal, JOM is more interested in empirical studies that assess the effectiveness and usefulness of a model (despite its simplifying assumptions), that is, the right side of Figure 1. However, it should be noted that items in the lower half of the table—referring to analytical models that are fit to empirical data but provide no insight into how implementation of the model increased knowledge, understanding, or improvements to the model—typically do not qualify as EGA even though the research is carried out in a real context. The contribution from EGA papers (in JOM) must be foremost empirical—even if some of the insights arise from the analytical model—but the use of the data must translate into model improvements that further improve the results derived from the model. This strategy, however, should not be confused with intervention-based research (IBR), where the outcome of the intervention is to improve existing theories or develop new theoretical insights as a result of the engagement with the problem situation (Chandrasekaran et al., 2020; Oliva, 2019). Tables 2 and 3 summarize aspects of several EGA papers that we will discuss further in this section. We begin with some example papers (in Table 2) that fit best on the left side of Figure 1, followed by papers (in Table 3) that fit best on the right side. Most of these papers have been published in JOM and exemplify the new space that we are seeking to develop, in which empirical work is done to improve the usability of a model. Serrano et al. (2018) began from the observation that financial risk propagates upstream in supply chains, the so-called “contagion effect,” similar to what has been observed with upstream propagation of order variability. They combined models that incorporate policies and constraints that drive agent behavior to reproduce the observed propagation behavior in terms of values and dynamics of variance of payments, and grounded the model on findings from the empirical finance literature to identify factors that drive such propagation and explore what payment variability propagation would look like when encountering rare but important states of nature. Chuang et al. (2022) began from the recognition that retail managers allocate auditing effort (i.e., inspections) for reducing inventory record inaccuracy (IRI) to groups of SKUs rather than to individual SKUs, as most models in the literature assume. This paper proposed the use of survival analysis to transform available data into an estimate of the data degradation rate in the group of SKUs and developed a simple model to allocate auditing effort optimally for groups of SKUs. The paper further grounds the model by empirically estimating the unobservable cost of IRI and testing the model with data from the research site, where it outperformed current managerial practices and the prevalent industry method for allocating auditing effort. Abbey et al. (2015) began from the observation that customers differ in their preference for new or remanufactured products, and wondered whether this observed heterogeneity might affect pricing decisions. They demonstrated the existence of these two customer groups empirically and then extended a pricing-demand model to incorporate them. When customers are assumed to be homogenous in their preference for new or remanufactured products, pricing-demand models indicate that the price for new products should decrease when remanufactured products are added to the mix. The extended model, however, indicates that the price for new products should increase when remanufactured products are added, with this price increase reducing cannibalization and increasing profit. Gray et al. (2017) began from the observation that companies offshored then reshored production. They examined nine offshoring cases, six of which were subsequently reshored. The reshoring cases were motivated by emergent problems such as intellectual-property protection, quality, and/or logistics. The authors observed that the offshoring decision was made on the basis of per-unit landed cost, with other factors not considered even though information was available. The reshoring decision, however, was based on much richer data. They modeled the decision process using system dynamics, with loops grounded in observations from the cases. The model reproduced the observed behavior and also suggested a risk of oscillation: Although the reshoring decision brought richer information into the decision process, there was no evidence that offshoring decisions were updated to consider more than per-unit landed cost. Craig et al. (2021) began from the observation that the majority of chargebacks used by retailers to penalize supplier fulfillment errors arose from execution errors rather than product-related in to and might not take into the type of Also, errors were by retailers and and the authors that such should be considered in the An that is for a supplier to and for the should be than one that is and for a supplier to and for the for The data fulfillment and from supplier They used the order model to exemplify a traditional OM tradeoff model to provide insight into how the decision is by how the rate with the order The used experiments to demonstrate first the of the errors and the for the in common use to or for cost. the authors a the importance of supplier and and in and to improve execution de Treville, et al. began from the observation that of the as at a were not with the as a lack of more been the would have evaluated that their How would the cost of the compare to the from more this question began with development of an analytical model that how the value of the value with increased for a of the of The insight was that the value from increased if values are by a that has a that of distributions to the however, the then the value of the best is to increase with an increase in It was not to the of values considered by the so the authors used the of above a of whether should be The value of the increase in the best was then to a estimate of the cost of increasing de Treville, et al. a that used to the value of the decision what to demand This work began from the in a & the value of reducing the time between and observation of demand could be as a real with this insight in a development of models from finance with the model. The analytical model as the applied to two cases to how of these real would affect the decisions companies made in Although this is at the of at the time it was that the contribution was more the model than it was the empirical insights, and it was not considered to fit well at JOM. et al. began from the in and that the to transform the of into an estimate of the of demand would be useful in the of demand data. and developed an analytic model to and to to demand in a The model, was too and also not to the needs of the company The authors then combined analytical tools from OM into a model to allocate between in This an estimate of the of demand for the but no demand data was available. The authors thus created a to transform of into a estimate of the of demand that was to them to out the at the company and obtain that the would the research fits best in the center of Figure 1 with the model as the primary et al. empirical testing of the insight that from practice the use of to estimate the of demand to the side of Figure 1. In the we noted that the developed in de Treville, et al. practical that was to an of how the but short of the empirical grounding that we are In de Treville, et al. the was into companies and to explore insights from the and to out the to This analysis of the supply through the of the to that the from their to the several in were than the from the increased time between when the decision was made which to and when actual demand was The to be extended to incorporate the fact that of the for from a The to be made but only 2 whether they had the this demand risk into the made to company decision that there was value in reducing the decision time to whether the had been was a of inventory for a with demand variability and a of the inventory through the of the extending the to incorporate demand it to the effect of demand risk on the of decision demand data was used to that the were to the observed et al. (2018) extended the to demand risk in the form of They then applied the extended model to data from also & the increased model from by the improvement in The rather result was that the simple form of the a when a was to however, a was to increase then the simple of the This combined a model extension with analytical grounding to provide a simple decision for when should be incorporated in decision et al. (2021) began from a for a that to fit of a The was to whether the extended supply for the was the of the They used the to estimate the cost, with the result that time was not for this and that an supply was The authors then a model to capture the full decision process in When the output not company data, it that because the supply was considered to be rather than no were made to the order was with the a key supplier made a The majority of the if the order was updated based on the key which was the The contribution was the that a of in an supply could make an important in and that a supply as could behavior in a way that a increase in EGA papers in JOM must have an empirical The analytical insights to be empirically are likely to from a model, or models, that have been published The JOM paper would be evaluated in terms of its empirical contribution rather than its modeling insights. make this in a we often authors to summarize the model in an its of When the development of an analytical model center in a that is a that it is not a fit for JOM the focus of the paper is on the center of Figure 1 rather than on either side of How much empirical grounding is paper will be to do this it is a matter of the is is a question of & and whether it is significant is largely on this below). How much the grounding add new insight or the must provide for its of grounding and the of the new insights, often by how and why a model's assumptions, and/or results should be It is authors to that grounding is and leads to The empirical grounding can be by a variety of both and should be grounded in empirical data, and assumptions that the model must be empirically As with research published in authors must a of not focus on a single of a problem. We encourage authors to make use of available data in empirical insights from the of the analytical model, that are not to this use of available should be to what they are doing and why their data provides for empirical The of a JOM paper also the paper should to OM This contribution a JOM EGA paper from an published in a journal such as the Journal of are and to the use of analytical models in practice. An EGA contribution in JOM new knowledge and understanding, a different space than and examples of model and As with other types of papers in the paper's contribution must also be significant rather than This is with each their on the of a paper's contribution. As a general OM JOM contributions to be rather than only to including the JOM is an open call for papers for a on This call will further example papers from other We this to provide opportunities to develop and what JOM from EGA