Advantages and Disadvantages
By Dejan Duzevik, February 3, 2017
Simulation is an approach that is used most commonly in two situations.
The first situation is when uncertainty is high due to sparse data. One such example is the simulation of an ancient Native American tribe, the Anasazi, which was a culture that lived between the 9th and 14th centuries. It is hard to run typical analytics on the limited available data about the Anasazi so researchers use simulation to try to understand what happened to the tribe.
A second common use of simulation is for experimentation in a low-cost, low-risk environment. Researchers at the European Organization for Nuclear Research (CERN) simulate particles colliding in the Large Hadron Collider before they validate their forecasts in the expensive real-world collider in Switzerland. More common examples include airline pilots practicing on flight simulators and doctors learning on test patients.
Both of these applications of simulation are helpful to scientists and researchers, but they come with a set of advantages and disadvantages. We have grouped these advantages and disadvantages into three broad areas related to technology, process, and socialization.
The following table gives a summary of the advantages and disadvantages of simulation, which we elaborate below.
Great forecasting power, but a good theory is needed
Data analysis methods such as regression are limited to forecasting the effects of events that are similar to what has already happened in the past. For example, if a brand has been investing in TV ads within a range of $50M to $100M in the past few years, a marketing mix model is excellent at forecasting what would happen if the spend is within those bounds.
However, the model is likely to produce nonsensical results once it extrapolates to forecast what would happen if TV spend is doubled or if a new marketing channel is deployed.
Simulation has an advantage over these methods in that it allows us to forecast things that have never happened before and to run scenarios outside of historical bounds. The caveat is that we need a good theory and causal hypotheses about how the system we are interested in analyzing works. Theories that have high predictive power, at least in social science, are hard to come by and may take years to develop.
Flexible, but not standardized
Simulations, and agent-based modeling in particular, provide highly flexible techniques for answering a wide range of research questions. These questions include what happened in the first moments of the universe, how wind turbulence around aircraft works, how the World Wide Web evolves, or how to better design hospitals.
Although simulation can be applied in a variety of contexts, a formalized set of rules and best practices is not always readily available. For this reason, simulation modeling (especially in social science) is incredibly creative, but may be daunting for new researchers who have no single reference to consult when starting out.
Building a simulation does not require data, but validation does
Simulation is an excellent approach to analyze problems when the available data is limited, since no data is necessary to construct a simulation. Validating a simulation, however, often requires multiple data sources to achieve a great degree of confidence in its representation of real-world dynamics. The process of validation is a disadvantage for simulation when comparing it to data analytics approaches since validating simulations is often more difficult.
For example, if we wanted to simulate traffic on a road, we would not need any data to start. We could construct a simulation that incorporates modeled cars, driver behaviors, and road conditions and voila: we have a traffic simulator. Analysis of this traffic simulation could provide surprising insights – such as the pattern of traffic jams migrating in the opposite direction that automobiles are traveling.
But to test whether such insights are valid, we would need to use various data. We would seek information about road conditions in a range of contexts – in cities, on highways, in the U.S. and other countries, and under different weather conditions. We could then recreate all of those scenarios within the simulation and see how well they match what actually happened in the real world. If the simulation data and the actual “validation” data match, then we have confidence that our simulated model of traffic is useful and valid.
To get the simulations to match real-world outcomes, we need to change the theoretical rules guiding the simulation or test different assumptions until they do. Simulations have the benefit of forecasting multiple metrics simultaneously, but this can make it challenging to get all of the assumptions synchronized. One change may improve the forecast for one metric but degrade the fit for another. Fortunately, expanding computing power and improving algorithms continue to reduce the time and effort to overcome the process barrier of calibrating and validating simulations.
Run any what-if scenario, but stay within budget
We build simulations with the goal of having a “Petrie dish” in which to experiment in a controlled and low-risk environment before taking action in the real world. At the outset of a project, a team can often list off a broad range of hypotheses to test within the simulator. Once a simulation is built and what-if scenarios can be run, the desire to keep testing more and more scenarios often grows.
Going back to the traffic simulation example, the initial goal of the simulation might be to determine whether to replace 4-way-stops with roundabout intersections in a particular section of town. The questions may compound from there: What will the impact be of traffic lights in other parts of town? How should the signage be placed? How will this impact traffic during the farmer’s market? It’s easy to see how the number of questions and scenarios can multiply very quickly.
Enabling a team to test and answer more questions is a great value-add that simulation provides. Projects may start by focusing on a single research question, but often grow to incorporate more complex ideas. This dynamic and creative process can build consensus by bringing more stakeholders to the table and ultimately lead to better decisions.
Yet, when time and resources are limited, it is important to ensure that the scope of a project does not expand beyond the available budget. Reaching agreement with the working team on the appropriate balance between focus on a specific deliverable and open-ended exploration is a good step to take early in the planning process.
Low cost, but high skepticism
Compared to the cost of experimenting in the real world, the use of simulation requires very little time and resources. Think about marketing: if we were to run various experiments in which we varied the amount we invest in different channels, we would have to go through dozens of budgets over as many years to gather enough data to answer a question with certainty. In the meantime, our brand and business may have gone in an undesirable direction.
The alternative to real-world experimentation is to run simulations to test different marketing plans. Within minutes we can test many ideas before acting on a plan and making decisions in the real world.
The disadvantage of this approach is that some audiences today are skeptical of simulation. Most of today’s analysis, especially in marketing, is based on reporting and building deterministic statistical models to describe what has happened in the past. Researchers often prefer these descriptive approaches to methods that test 31 theories about the future.
We believe that this skepticism is a result of the relative novelty of simulation in marketing analytics, and that with more success stories and validated forecasts, this skepticism will subside.
Innovative, but political
Simulation may be one of the most innovative approaches researchers engage in today. We have seen people advance in their careers for their intrapreneurial spirit in introducing simulation within their organization. Longstanding and thorny problems get tackled every day with social simulation, but the socialization of simulation results often presents organizational challenges. Because it identifies trade-offs between a range of metrics, simulated insights may bring conflicting interests to the forefront in organizations where stakeholder incentives are not aligned.
Building consensus around a simulation is done best when the process is organized from the start. All stakeholders should agree on the simulation’s framework, assumptions, and questions to be answered at the outset. In our experience, this is the best way to ensure that the simulation findings are impartial.