A review of optimization methods to solve adaptive management problems


Do you need to manage a system over time but are uncertain about what will happen in the future? Well! Adaptive management, or learning by doing, might provide a solution. The issue with adaptive management is that no one knows what it is or how to do it, because it has been used in many different fields of research for a different purpose and with a different meaning. In my research, I am interested in the decision theoretic approach of adaptive management (not the resilience approach). What does that mean? That means that I look for ways of providing adaptive decisions that will insure that you would make the best decisions possible over time. That’s right, our approaches account for all the things that could go wrong or right in our decisions before any humans do.


In our paper “Optimization methods to solve adaptive management problems” (Theoretical Ecology (2016). doi:10.1007/s12080-016-0313-0), we review the methods that would allow you to optimize your adaptive management strategies.

The fun bits: We provide decision trees that would help you choose between different type of approaches available (passive or active adaptive management); We explain the differences and similarities between these approaches;  And, we provide 8 algorithms that will help you understand and write your own code – Have I mentioned a bunch of reference to explore?

This paper concludes the 3 years of my CSIRO Julius Career Award on studying adaptive management problems. I am thankful to all my co-authors for their contributions, but also the adaptive management community more broadly.

Reflecting back, I really needed this paper when I started, and I hope this will help others. I studied adaptive management methods by accident 6 years ago. At the time I was overwhelmed. I couldn’t understand why there were so many different terminology to define a problem. I couldn’t understand why there were so many approaches available that did not seem to be efficient. Inspired by Darryl MacKenzie’s paper, I took a POMDP perspective, how would I model this problem? One thing, leading to another, and some precious collaborations, led to a best paper award at the 2012 AAAI conference. The power of using POMDP is illustrated in Sam’s Proc B paper (you can read about it here). In my opinion, much more could be done to exploit this result and we are just scratching the surface, so stay tuned!

The utlimate reference:

Chadès, I., Nicol, S., Rout, T.M., Peron M., Dujardin Y., Pichancourt JB., Hastings A., Hauser C. Optimization methods to solve adaptive management problems. Theoretical Ecology (2016). doi:10.1007/s12080-016-0313-0 (request PDF)

The ultimate abstract:

Determining the best management actions is challenging when critical information is missing. However, urgency and limited resources require that decisions must be made despite this uncertainty. The best practice method for managing uncertain systems is adaptive management, or learning by doing.
Adaptive management problems can be solved optimally using decision-theoretic methods; the challenge for these methods is to represent current and future knowledge using easy-to-optimize representations. Significant methodological  advances have been made since the seminal adaptive management work was published in the 1980s, but despite recent active-or-passive-adaptive-managementadvances, guidance for implementing these approaches has been piecemeal and study-specific. There is a need to collate and summarize new work. Here, we classify methods and update the literature with the latest optimal or near-optimal approaches for solving adaptive management problems. We review three mathematical concepts required to solve adaptive management problems: Markov decision processes, sufficient
statistics, and Bayes’ theorem.We provide a decision tree to determine whether
adaptive management is appropriate and then group adaptive management approaches based on whether they learn only from the past (passive) or anticipate future learning (active).We discuss the assumptions made when using existing models and provide solution algorithms for each approach. Finally, we propose new areas of development that could inspire future research. For a long time, limited by the efficiency of the solution methods, recent techniques to efficiently solve partially observable decision problems now allow us to solve more realistic adaptive management problems such as imperfect detection and non-stationarity in systems.

Timing of critical habitat protection matters (open access)



The latest addition to my research interest on how time influences our decision-making process just came out in Conservation Letters (Martin et al, 2016, Free access). We demonstrate once again, that time spent gathering more information to make better decision is beneficial to a point. Aside from the massive modelling effort we had to go through (see lessons learned below), our conclusion summarizes our main point:

It may be tempting to assume that more information is of value for its own sake, in a decision-making context information has value only when it leads to a change in actions taken, specifically, a change with enough benefit to species protection to outweigh the cost of obtaining the information. In the often contentious environment of endangered species decision making, parties who benefit from delay in taking action often lobby strategically for more information, not because they are concerned for the efficacy of protective actions but because their interests are best served by delaying protection as long as possible. In this environment, reminding everyone that more information does not always translate into more efficacious action may help strike a better balance between action and research. When it comes to species conservation, time is the resource that matters most. It is also the resource we cannot get more of.

Martin T.G., Camaclang A.E., Possingham H.P., Maguire L., Chadès I. (2016) Timing of critical habitat protection matters. Conservation Letters In Press, DOI: 10.1111/conl.12266 (OPEN ACCESS, PDF)

Lessons learned: 

This paper was about 5 years in the making, along the way I have learnt a big deal about using AI reinforcement learning tools for this problem. Once more I had to give up using RL and opted for an exhaustive search to find the optimal stopping time – which was really disappointing considering the amount of time I spent on it. As painful as it sounds, I was using the wrong approach. On top of my head, the hurdles were:

1) the matrix population model of the northern abalone species exhibit some time lag, making the process non-Markovian;

2) the Q-Learning approach took way too long to find the optimal stopping time considering the amount of different configurations I had to go through;

3) the near optimal strategies of the Q-Learning approach were not consistent due to lack of convergence;

4) it was way faster to perform an exhaustive search, and this should have been my first solution for a decision problem that was quite simple to solve.

I am glad this paper is out in Conservation Letters for everyone to enjoy. Well done to all my co-authors for their support and hard work on this piece – especially Tara, for pushing it through the line.

Comparing adaptive management and real options: slides and PDF

Adaptive management and real options approaches for sequential decisions making have undergone significant evolution over the last two decades. Both approaches are based on stochastic optimal control and Markov decision processes. They evolved independently from each other and their developments were motivated by different needs.

Adaptive management was specifically developed to handle decision problems with imperfect knowledge of the dynamics of the system, and is known as ‘learning by doing’. On the other hand, real options analysis was introduced specifically to value the flexibility to change actions over time in response to the evolution of uncertainty, and represents both optimal sequential decisions under uncertainty and a capital budgeting methodology. Because of these different purposes, different analytic and numerical methods were developed to solve these problems.

In our recent MODSIM paper (Chades et al, 2015), we review and compare the concepts, applications and recent advances in the numerical and analytic techniques in adaptive management and real options methodologies. A large body of knowledge accumulated in both fields makes a comprehensive review impractical in the context of this paper. Therefore, our review focuses on the most recent developments, with the purpose to identify potential areas of new developments that would address new challenges in the environmental decision area.

I. Chadès, T. Tarnopolskaya, S. Dunstall, J.Rhodes, and A.Tulloch (2015). A comparison of adaptive management and real options approaches for environmental decisions under uncertainty. In Weber, T., McPhee, M.J. and Anderssen, R.S. (eds) MODSIM2015, 21st International Congress on Modelling and Simulation. Modelling and Simulation Society of Australia and New Zealand, December 2015. ISBN: 978-0-9872143-5-5. (PDF)

Artificial Intelligence to the rescue of migratory shorebirds

Adaptive management or learning by doing, is praised as the best practice method to manage natural systems under uncertainty (see ESA’14 talk). Limited for a long time by our ability to solve adaptive management problems, our research now allows us to find the best adaptive management strategies when networks change over time. This was made possible thanks to our research in Artificial Intelligence (AI) and Conservation science.

What have we discovered in 2 steps?

1) Unlocking the beast. Being strategic about adaptive management means finding the best management strategy when we don’t know exactly what will happen in the future (structural uncertainty). Until very recently, finding the optimal strategy to such decision problems was possible for very small size problems, limiting the application of adaptive management principles. In 2012, we published a fundamental paper that demonstrates that adaptive management problems can be solved using a simplified POMDP (Partially Observable Markov Decision Process, see tiger paper). This is an important finding because modelling an adaptive management problem as a POMDP means we can use very fast algorithms from AI and solve very large adaptive management problems. On a side note, this paper was published at the top AI Conference (AAAI) and received “best paper award” (Computational Sustainability track, thanks for the support!).

2) Boldly go where no one has gone before. Our second step was to demonstrate the power of our findings on the most complex problem we could imagine. Thinking about it, the most difficult problems to solve in ecology are spatial problems (migratory networks) with changing dynamics over time (non stationarity, climate change) for which the consequences on species management are unknown (structural uncertainty, population dynamics). Well, we did it! Check our splendid paper in Proceedings B led by Sam Nicol that brings it all together. This work is amazing for so many good reasons: the shorebird application, the fundamental AI research, the writing, the figures, the authors, the journal and the 20-page supplementary information!

I hope you will enjoy it as much as we did.

Nicol, S., R.A. Fuller, T. Iwamura and I. Chadès (2015). Adapting environmental management to uncertain but inevitable change. Proceedings of the Royal Society B, 282(1808).

Sam also wrote a fantastic conversation article on the topic that explains the impact of our Proc B paper: We need to get smarter to save shorebirds from rising seas.


When do we need more data? Don’t miss the excel spreadsheet

Stefano Canessa and colleagues have recently published a much needed paper in Methods in Ecology and Evolution to help managers answer: when do we need more data?

Stefano provides an excel spreadsheet and also wrote a blog in MEE! Well done Stefano.

Canessa, S., Guillera-Arroita, G., Lahoz-Monfort, J. J., Southwell, D. M., Armstrong, D. P., Chadès, I., Lacy, R. C., Converse, S. J. (2015), When do we need more data? A primer on calculating the value of information for applied ecologists. Methods in Ecology and Evolution. doi: 10.1111/2041-210X.12423

I have found that thinking through what uncertainty really matters a rewarding process. For example, my last talk at ICCB 2015 also showed that critical uncertainty is often limited to a small number of unknown in Conservation. Go ahead fellow scientists, embrace uncertainty. If you were an economist you would say that uncertainty and associated decisions bring opportunities and perhaps flexibility!

On a side note, I’ve been recently thinking about a dynamic version of the Expected Value of Perfect Information – For example you can check out this paper.


Complementary strategies double the impact of cost-effective ranked strategies

Our manuscript on how complementarity can help saving more species per dollar spent is available online. If you are interested in cost-effectiveness analysis, PPP (Project Prioritisation Protocol), priority threat management, expert elicitations, or the Pilbara, have a look:

Chades, I., Nicol, S., van Leeuwen, S., Walters, B., Firn, J., Reeson, A., Martin, T. G. . and Carwardine, J. (2014), Benefits of integrating complementarity into priority threat management. Conservation Biology. doi: 10.1111/cobi.12413 (abstract) (request pdf)

In Tulloch et al (2013), we studied how complementarity could help choosing indicator species to improve our monitoring power, Chades et al (2014) explores further this idea in the context of priority threat management. Feel free to download the Pilbara report for details about the study.

Priority threat management process requires a strong collaborative team to be successful. This manuscript is a great example of a work that would not have been possible without the essential contribution of all authors.

Abstract:Figure2_manuscript_R1_with legend_rgb

Conservation decision tools based on cost-effectiveness analysis are used to assess threat management strategies for improving species persistence. These approaches rank alternative strategies by their benefit to cost ratio but may fail to identify the optimal sets of strategies to implement under limited budgets because they do not account for redundancies. We devised a multiobjective optimization approach in which the complementarity principle is applied to identify the sets of threat management strategies that protect the most species for any budget. We used our approach to prioritize threat management strategies for 53 species of conservation concern in the Pilbara, Australia. We followed a structured elicitation approach to collect information on the benefits and costs of implementing 17 different conservation strategies during a 3-day workshop with 49 stakeholders and experts in the biodiversity, conservation, and management of the Pilbara. We compared the performance of our complementarity priority threat management approach with a current cost-effectiveness ranking approach. A complementary set of 3 strategies: domestic herbivore management, fire management and research, and sanctuaries provided all species with >50% chance of persistence for $4.7 million/year over 20 years. Achieving the same result cost almost twice as much ($9.71 million/year) when strategies were selected by their cost-effectiveness ranks alone. Our results show that complementarity of management benefits has the potential to double the impact of priority threat management approaches.


A toolbox to solve stochastic dynamic programming problems in R, Matlab, SciLab or Octave

Our MDPToolbox is now published in Ecography. Thank you for supporting freely available programs. Please spread the word! The MDP/ SDP toolbox is now available in R, Matlab, SciLab and Octave. No excuses!

Stochastic dynamic programming (SDP) or Markov decision processes (MDP) are increasingly being used in ecology to find the best decisions over time and under uncertainty so that the chance of achieving an objective is maximised. To date, few programs are available to solve SDP/MDP. We present MDPtoolbox, a multi-platform set of functions to solve Markov decision problems (MATLAB, GNU Octave, Scilab and R). MDPtoolbox provides state-of-the-art and ready to use
algorithms to solve a wide range of MDPs. MDPtoolbox is easy to use, freely available and has been continuously improved since 2004. We illustrate how to use MDPtoolbox on a dynamic reserve design problem.

Chadès, I., Chapron, G., Cros, M.-J., Garcia, F. and Sabbadin, R. (2014), MDPtoolbox: a multi-platform toolbox to solve stochastic dynamic programming problems. Ecography. doi: 10.1111/ecog.00888