Forecasting gets an enormous amount of attention in decision systems. That makes sense: if demand, lead time, price, capacity, or another important input is uncertain, we want the best possible description of what might happen next.
But there is an easy mistake to make after building the forecast. We begin treating the forecast as if it were the product.
It is not. The business ultimately experiences a decision policy.
A forecast estimates uncertainty. A policy converts information into action. The economic outcome comes from the interaction between the two.
That distinction matters because improving a forecast metric does not guarantee a better decision. A model with lower MAE, MAPE, pinball loss, or another statistical score can still produce worse ordering, allocation, pricing, or capacity decisions once the forecast is passed through the decision logic. The reverse can also be true: a forecast that looks slightly worse statistically may lead to a materially better economic outcome because its errors occur in places where the policy is less sensitive.
The practical implication is simple: when the policy is what creates value, evaluate and tune the policy as close to the business objective as possible.
The policy is where the economics enter
Consider a simple inventory system. The forecast may describe a distribution of future demand, but the company does not earn money from the distribution itself. It earns or loses money after a policy decides how much to order.
That policy embeds economics, whether explicitly or implicitly. It determines how the system trades off excess inventory against missed demand, how it reacts to lead times, how it handles minimum order quantities, and how aggressively it responds to uncertainty.
If those tradeoffs are poorly specified, a more accurate forecast cannot rescue the system. It simply feeds better information into the wrong decision rule.
A useful way to view the system is:
uncertainty model → decision policy → realized operational outcomes → economics
Optimizing only the first link is often convenient because forecast loss functions are clean and easy to compute. Convenience, however, is not the same thing as alignment.
A forecast metric is a proxy
Forecast error is useful. It tells us something about the quality of the predictive model. The problem begins when a proxy quietly becomes the final objective.
Imagine two demand forecasts. Forecast A has lower average error across all weeks. Forecast B is slightly less accurate overall, but it is much better at representing the upper tail during the few periods when stockouts are extremely expensive. If the decision is an inventory order, Forecast B may generate substantially more profit even though a standard leaderboard ranks it second.
The decision cares about where the error occurs, what action the error changes, and what that changed action costs.
This means the relative value of forecast accuracy is endogenous to the policy and the economics around it. A one-unit error is not economically identical everywhere. The cost can depend on inventory position, timing, capacity, substitution, margin, lead time, and the asymmetry between overage and underage.
Once that is recognized, the natural unit of evaluation becomes less about forecast error in isolation and more about decision regret: how much value did the system leave on the table because of the decisions it made under uncertainty?
Simulation gives us a practical bridge
In many real systems, directly solving the full stochastic optimization problem is difficult. The operating logic may contain discrete rules, nonlinear business behavior, vendor constraints, black-box components, or a simulator that is much easier to execute than to differentiate.
That does not mean we have to fall back to tuning everything against forecast metrics.
A practical alternative is simulation-based policy optimization.
The workflow is straightforward:
- Define a parameterized policy that the business can actually operate.
- Generate plausible future scenarios for the uncertain quantities.
- Run the policy through those scenarios in a simulator.
- Score the resulting decisions using the economic objective.
- Search for policy parameters that improve that score.
- Validate the selected policy on held-out scenarios or later realized data.
The search itself does not have to be exotic. Grid search, random search, quasi-random designs, Bayesian optimization, evolutionary methods, or other derivative-free techniques can all be useful depending on dimensionality and simulation cost.
The important change is the objective. We are no longer asking only, “Which model predicts demand best?” We are asking, “Which combination of information and policy produces the best decisions under the uncertainty we actually face?”
This changes how models should be compared
Suppose a replenishment policy has three tunable parameters: an order quantity multiplier, a review cadence, and a risk adjustment. We could choose those parameters from historical heuristics and then spend months improving the forecast that feeds them.
Or we could hold the forecasting process fixed initially and optimize the three policy parameters against simulated profit, cost, or another business objective.
That experiment often reveals something important: some forecast improvements are economically consequential, while others barely move the decision outcome. Likewise, small changes to the policy can sometimes create more value than a large improvement in a headline forecasting metric.
This does not make forecasting less important. It makes the role of forecasting more precise. The forecast is an input to the decision problem, and its value should ultimately be judged by how it improves decisions.
The same principle applies beyond inventory. In pricing, the policy maps beliefs about demand response into price changes. In workforce planning, it maps uncertainty about workload into staffing. In transportation, it maps uncertain volume and travel conditions into routing or capacity commitments. In each case, prediction quality matters because it changes a decision—not because a statistical score is intrinsically valuable.
Stability can be part of the objective too
Direct policy optimization also makes operational considerations easier to represent.
A mathematically attractive policy can still be unusable if it changes orders dramatically every week, creates vendor whiplash, requires impossible data latency, or produces recommendations that operators cannot explain. Those costs are real even if they are absent from a forecast metric.
The simulator can include them.
For example, a policy objective might combine expected profit with penalties for order volatility, emergency actions, inventory aging, or capacity violations. This allows the optimization to search for policies that are not merely profitable in expectation, but operationally stable enough to deploy.
That is a much richer notion of “best” than selecting the forecast with the smallest average error and hoping the downstream rules translate it into value.
The architecture should follow the decision
A useful design principle follows from all of this: build the system backward from the decision.
Start with the action the organization must take and the economics of getting that action wrong. Then define the policy, identify the uncertain inputs that materially affect it, and build the forecasting machinery needed to represent those uncertainties.
This ordering prevents the predictive model from becoming the center of the architecture by default.
It also creates a cleaner research loop. If a new forecasting method is proposed, we can test not only whether it improves a statistical metric, but whether it changes the selected actions and improves economic performance under the same policy—or whether the policy itself should be retuned because the information structure changed.
That is a much stronger standard for a decision system.
The objective is better decisions
Organizations do not purchase forecasts. They purchase the consequences of decisions made with forecasts.
The closer our evaluation gets to those consequences, the less likely we are to optimize a proxy that has become detached from the business problem.
Forecasts remain essential. Better uncertainty models can create enormous value. But the last mile matters: information must pass through a policy before it becomes an action, and the action is where economics become real.
So when a simulator and an explicit business objective are available, use them. Measure the policy. Tune the policy. Challenge the policy. Then ask whether a better forecast makes that policy—and the decisions it produces—meaningfully better.
Optimize the policy, not just the forecast.