All Models Are Wrong?

"All models are wrong, some are useful." George Box

I've seen this quote used many times as seemingly a hedge to the usefulness of a model, as though saying, "don't expect a magic bullet."

There is something a bit discouraging about that saying. If all models are wrong, how do you know that the model you've built is useful? How have models been used that require this quote to constantly be referenced?

In this article, we will explore the context of this quote and ways that anyone can improve the usefulness of their models.

What is a Model?

Before we dive deeper into the context of the quote, perhaps we first need to understand what exactly is a model.

A scientific model seeks to represent empirical objects, phenomena, and physical processes logically and objectively. This is done by taking relevant aspects of a real-world process and using different modeling techniques to ascertain different aspects of what you are trying to better understand. This can be done by building either a physical model (something you can touch) or a conceptual model (a theoretical representation of a system).

The outcomes of a model provide new information that must then be analyzed to determine if it results in the desired outcome or if more iterations are needed.

Who was George Box?

George Box (1919 - 2013) was a British statistician who worked in the areas of quality control, time-series analysis, design of experiments and Bayesian inference. Over his career, he published more than 30 books and articles on these subjects.

During World War II, Box joined the Army. When the Germans were bombing London on a nightly basis, Box was tasked with figuring out what would happen if the Germans loaded their missiles with poisonous gas. Box observed that there was high variability in his results. Not knowing any statistics beforehand, Box began learning statistical models to better understand his experiment's outcomes.  

After the war, Box worked for Imperial Chemical Industries as a statistician until 1956. In 1960, he was invited to create the Department of Statistics at the University of Wisconsin-Madison, where he spent the rest of his life until he passed away in 2013 at the age of 93.

In several papers published in the 1970s, George Box made reference to the idea that "all models are wrong." He states that a model can not become correct merely by 'excessive elaboration' or 'overparameterization.' He pushed for the understanding that models provide useful approximations for the system they represent.

George Box collaborated with others to create several important statistical tools, including the Box-Jenkins models, the Box-Behnken designs and the Box-Cox transformation.

The Context of the Quote

George Box's quote is commonly used in statistical modeling but also applied to scientific models in general. The quote nicely sums up the challenge of creating a statistical representation of a real-life system with all the complexities involved.

Focusing more on conceptual models, these are abstractions of real-world systems. They are used when it is challenging to create experimental conditions where you can directly measure outcomes. Because of this, the experimenter must combine measurable phenomena with theoretical or random variables to observe what happens and then interpret the validity of the results.

For example, with weather forecasting, the meteorologist is attempting to predict the conditions of the atmosphere for a given location and time. They do this by combining current and historical quantitative data about the atmosphere, the land and the ocean and then predict how the atmosphere will change over time. These predictions help us in simple decisions on whether we need to bring an umbrella with us to work, to the more impactful decisions such as predicting weather conditions for crops, oceanic conditions for trans-Atlantic shipping, and temperature forecasts for predicting energy demand of utilities.

Because we are trying to model real-world systems that can require massive computational needs, we may be unable to measure and capture all applicable data. The system may have high volatility and/or contain aspects that we as humans have yet to fully understand. Therefore, the modeler must make hypotheses about what is unknown. Because of this, the results must be reviewed by a human who will take into consideration patterns, the model's past performance and their knowledge of the model's biases.

Take a simple example of building a model to predict the outcome of rolling a 6-sided die. There are some measurable aspects that we can use, such as the die has 6 numbers, one on each side. We can measure the size of the die used. But there are also many unknown variables. How high is the die being dropped, and at what angle? Is the die being placed in something like a dice tower? What is the makeup of the surface that the die is landing on? Does the roller randomly position the facings before the 'rolling' of the dice, or is that held constant? All of those unknowns could affect how many times the die bounces on a surface and its position in each of its states.

Since the outcome of the model can not be absolute, we must make a decision based on our confidence in the outcome. An outcome can (and most likely will) be wrong, but it could still guide us in our decision-making and understanding of what we are trying to measure. The weather forecast may not have been accurate enough to say that it will hit below freezing today, but it may very well be cold, and we would want to plan accordingly.

Therefore, to say that all models are wrong is not to say that models are something to be avoided. Some are useful because they provide us with additional information that we can scrutinize using our own human judgment to determine the usefulness of the outcomes.

How Can I Make My Model More Useful?

If some models are useful, then how do you make your model one of the useful ones? Here are a few areas of focus that will increase the scientific integrity of your models.

  1. Legibility: It is easier to share your ideas or even remember things you've tried before if everything is documented. You want to make sure that you keep a record of your lab notes as you build your models. This includes your background research, hypotheses, notes from your experiments, data sources, and conclusions regarding the outcomes of your experiments.
  2. Transparency: You want to be clear about your assumptions used in the model. Since a model is a combination of observable and unobservable parameters, be clear as to why you chose the values or random variables for your experiments. Also, be sure to document any known biases going into your model.
  3. Reproducibility: The ability for an independent party to be able to take what you did and produce the same results is crucial for scientific research. If your work is not reproducible, then it could never be something that could be productized. A second set of eyes that can achieve the same results under the same conditions strengthens the reliability of your models.

By focusing on these three areas, you can greatly improve the integrity of your work. Other people being able to read, understand and duplicate what you have done creates a collaborative environment where independent reviewers can pose their own questions against your model, testing it in ways that you may not have thought of. These new tests can increase the usefulness of your model by finding ways to increase the confidence of the outcomes, find areas that need improvement, and spread access to a larger audience to use your model. A model used in isolation will never be as useful as a model used globally.

Conclusion

We create models because there are complex systems in the real world that we hope to better understand. The outcomes of these models can affect our health, wealth, social interactions, and our understanding of how things work. An aphorism such as "All models are wrong, but some are useful" is a reminder that we should always scrutinize information presented to us and make sound judgments of the information's validity and applicability to the decisions we make. By making sure that the work we do in building models is legible, transparent, and reproducible, we've allowed independent review and scrutiny of our work which will only lead to a model that evolves in greater usefulness.

Additional Thought

In reading more about George Box, I learned of one of his innovations during his tenure at the University of Wisconsin.  Box would host a Monday night beer session where he would invite students and guest speakers to bring problems that they wanted to discuss, and as a group, they would have a general discussion about how the problem might be solved.  This is a great way to move out of the classroom and think about an individual problem and move from theory to practical application.

This is one of BlockScience Labs' goals with the Labs Platform.  We are creating a bundled set of tools that allow the user to follow a process for better understanding real-world business questions in a collaborative way.  We hope to onboard students who will use the Labs Platform in a way that will be directly applicable to their future careers outside of academics.  Learning the theoretical and the practical applications of those concepts will produce graduates who are better prepared to apply their skills to non-educational institutions.

Where can I learn more?

You can find out our latest updates by visiting our website at https://blocksciencelabs.com.

You can follow us on Linkedin at https://www.linkedin.com/company/blocksciencelabs.

You can also help fund BlockScience Labs and own a piece of its success. Check out our investment site to learn more at https://invest.blocksciencelabs.com.

About the Author

Chris Frazier has been an executive for over 10 years. He evolved from being a data analyst to becoming a BI and product leader, designing and implementing data-driven solutions to a successful media tech firm. Chris is now the CEO and Co-Founder of BlockScience Labs, a data science product company that builds solutions that support both System Engineers and Executive Scientists to make better business decisions through scientific processes.