Post provided by Liliana Silva

Why we developed this automation framework

Observing animal behaviour is one of the most widely used methods in ecology. But anyone who has spent hours viewing video footage knows how quickly behavioural analysis becomes overwhelming. A single nest camera can generate hundreds of hours of recordings, and turning those videos into behavioural data often means endless manual annotation. As a former video technician myself, I have spent more than 2000 hours analysing footage and I, together with André Ferreira, often wondered whether there could be an easier way.

This challenge started us on a 5-year path to develop an automated framework for recognising nest behaviours from video recordings of the sociable weaver. Once we had a successful workflow, we brought together collaborators from two other long-term projects, on blue tits (Arlette Fauteux, France) and great tits (Irene Martínez-Baquero, United Kingdom) to make sure our results were useful for other systems. This resulted in our recent paper “From video to behaviour: An LSTM-based approach for automated nest behaviour recognition in birds”, where we describe a practical workflow that helps researchers move from video footage to behavioural data. I find it very rewarding that my previous struggles and understanding of the practical challenges involved in behavioural analysis could be turned into something useful for the research community.

As part of long-term monitoring projects of species in the wild, we all needed robust automated systems that could cope with environmental variation over time. Our goal was not simply to build a proof-of-concept modelling approach, but to create a deployment-focused framework that researchers could adapt to their own study systems and real-world field data.

What it does

The framework combines stored footage, previously annotated behaviours and deep learning using Long Short-Term Memory models to recognise behavioural sequences from video data. The system learns patterns of behaviour over time, rather than relying on single images alone. This is especially important for behaviours that unfold over time, since the model can better distinguish similar-looking behaviours, such as differentiating a bird entering a specific nest from a bird simply passing by.

Using this approach, we built three automated systems, one for each species. These models provide second-by-second behavioural classifications, including nest entering, leaving, building, aggression and sanitation, depending on the species. Importantly, model performance was comparable to human observer agreement while being much faster.

What this contributes to the field

Automated behavioural recognition has enormous potential for ecology and evolution. Tools like this can help researchers process much larger datasets than would be possible manually, opening the door to large-scale behavioural studies. For example, the sociable weaver project collects more than 2,000 hours of video per year. Our models increased analysis speed eightfold, from 40 to over 300 videos per week. Additionally, by keeping only footage containing behaviours of interest, the system reduced storage requirements by over 90%. Together, this automation helped researchers collect more data faster, while saving both time and storage space to explore new research questions.

One of the key strengths of the workflow is its flexibility. Researchers can train the system using their own annotated videos, making it easily adaptable to different species and ecological contexts. We also focused heavily on practical implementation by walking through data preparation, annotation strategies, model training and deployment considerations, and providing a roadmap that other researchers can realistically follow.

Importantly, artificial intelligence can sometimes feel inaccessible, but by sharing a practical open-source workflow, we hope more researchers will feel confident exploring these approaches.

Paper link: https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.70325

Websites: https://sociableweaverproject.com , https://mesangecefe1.wixsite.com/mesangecefe , https://www.wythamwoods.ox.ac.uk/wytham-tit-project