Post provided by Jody Reimer

One of the unifying themes in ecology may be the acknowledgement that we live in a world of finite resources, and so we also live in a world of tradeoffs. A diverse range of research questions can be distilled into a question about tradeoffs. For example, how should an animal forage in the presence of predation? Which selective forces determine the life history of a flowering perennial? How should we manage a population to maximize the sustainable harvest rate?

Jody

Questions as varied as these can all be addressed using the same method of stochastic programming[1] (SDP) (see McNamara and Houston, 1986; Rees et al. 1999; and Runge and Johnson, 2002, respectively). SDP has been used extensively to study optimal tradeoffs in a wide range of applications in ecology, evolutionary biology, and management. It is a flexible and powerful modelling framework that allows for simultaneous consideration of an individual’s state, how an optimal decision might explicitly depend on time, and for a probabilistic landscape of risks and rewards.

[1] Also known as Markov Decision Processes

“That doesn’t look right…”

When we first started working with SDP models, we started simple, planning to build complexity later. However, even the simplest of models was still yielding some nonintuitive behaviour. There were oscillating patterns that seemed inconsistent with animal behaviour. This resulted in a great deal of time spent looking for a suspected glitch in our code. When no error could be found, however, we started to suspect there was something else behind the oscillating patterns.

Capture — Figure 2. Model output describing in which patch an individual should forage, depending on the time (day) and the individual’s state (energy reserves). The predicted oscillation in time seem too complicated to describe the realized foraging behaviour of an animal.

By taking another look at how model results behaved with different parameters and over different time scales, a new way of thinking about these models emerged. Rather than the standard conceptual framework of thinking about a large set of equations—one describing an individual in each possible state at each possible time—we could think about individuals in all states simultaneously by using vectors and state transition matrices, and conceptualizing state changes using directed graphs. It also became clear that the convergence behaviour often taken for granted in applications of SDP models had a mathematical elegance that suggested underlying patterns that were, perhaps, knowable.

Surprising parallels with matrix population models

Reformulating classic SDP models in the language of matrices allowed us to look at model results through a different lens and to draw parallels to a seemingly unrelated method; matrix population models. Matrix population models are standard tools to study the behaviour of populations that are structured in some meaningful way (e.g., by age, stage, sex). The classical results used to understand these models rely on key properties of the model transition matrix, namely its eigenvalues and corresponding eigenvectors. In many applications of matrix population models, the long term behaviour of the population is determined entirely by the properties of the transition matrix and does not depend on the initial conditions at all.

As it turns out, the long time convergence behaviour of SDP models follows very similar rules, with the ultimate behaviour of the model determined by the properties of the relevant transition matrices. By formulating SDP models in the language of matrices, much of the intuition matrix population models could then be brought to bear on our understanding of SDP results.

Figure 3. Stochastic dynamic programming has been used to look at changes in optimal reproductive behaviour of polar bears in a warming Arctic (Reimer et al. 2019)

One of the most illuminating aspects of the parallels between these two methods lies in our understanding of qualitative convergence behaviour for matrix population models. There is a rich theory for those models that explains when a population will oscillate on its way to its long-time solution. This same theory was able to shed light on the curious oscillatory behaviour exhibited by SDP models, and provided confirmation that the oscillations were caused by the SDP model structure and parameters, rather than a coding error.

Reframing SDP models according to their matrix structure has implications for both implementing and interpreting model results, exploring series of realized states and decisions using Markov chains, and inspiring new types of analyses (e.g., exploring suboptimal decision rules that perform nearly as well as the optimal solution obtained numerically).