When we work very hard to build a model using a computer there’s a modern-day bias to believe anything the computer tells us. But remember my maxim in lab this semester:
COMPUTERS ARE DUMB – YOU ARE SMART
Just because we develop a fancy GIS model, doesn’t mean that model is inherently trustworthy. How do we know if it deserves our trust?
We can compare our predictions to historical datasets, and see if, under historical conditions, the predictions match what unfolded. If we are attempting to predict wildfire risk in Virginia in 2025, maybe we look to 2023’s climate and environmental data as test inputs, to see if our model successfully predicts where fires broke out during the fire season of November 2023.
This process is called calibration
If we hold back a subset of calibration data for later testing, we can perform cross-validation. The idea is simple; we calibrate the modeler using most available historical data. Then after we build the model, we test to see if the small portion held back produces reliable, meaningful results that match observed pattern.
We could also design an experiment to test if predictions match reality.
While models have limitations, the following arguments outline what we can and can’t evaluate after completing a model:
- A model is often producing a normative condition against which you test future observations
- A model should be measured less against whether it perfectly matches reality, and instead focus on how it reduces uncertainty about future decision-making
- Often, models don’t create new knowledge. Rather they present existing knowledge in a novel way to aid decision making and/or planning
A final evaluation of model’s suitability is sensitivity testing. A model is said to be robust if a 5% change in some starting parameter value changes the result of the model < 5%. Thus, sensitivity analysis evaluates a model’s response to its own parameters and assumptions. Where outputs are extremely sensitive to a particular parameter, we must work our butts off to make sure that parameter is correct!