Post provided by Nick Leseberg

Presenting the new MEE article ‘Using intrinsic and contextual information associated with automated signal detections to improve call recognizer performance: A case study using the cryptic and critically endangered Night Parrot Pezoporus occidentalis‘, Nick Leseberg shares the methods behind the hunt for the elusive night parrot.

If you’ve not heard the story of the night parrot (Pezoporus occidentalis), it’s worth a recap. First seen by Europeans in the mid-19^th century, it was observed occasionally by explorers and pastoralists between 1845 and 1880. However, by the late-19^th century, they had all disappeared, and the night parrot became something of a legend. There were no certain sightings after around 1880, only a trickle of unconfirmed reports suggesting the night parrot had probably persisted somewhere in central Australia. With no specimens, no photo, and no real idea what the bird sounded like, searching for it was like searching for the proverbial needle in the haystack. It wasn’t until 1990 that conclusive proof of the species’ continued existence arrived, when a dead night parrot was found by the side of a road in western Queensland. In 2006 another dead bird was found in the same region, then finally in 2013, the discovery of a small population of live birds was announced.

Detecting Night Parrots

The discovery of a live population of night parrots provided the first opportunity in more than 130 years to study the species in the field. Early research, led by Dr Steve Murphy, focused on which techniques might be effective for detecting the species. I was lucky enough to be asked to help with this research as a PhD student in the Green Fire Science lab at the University of Queensland.

Night parrots are extremely secretive. They spend the day roosting in tunnels built in dense hummocks of spinifex, and emerge at dusk to feed, sometimes flying considerable distances to their feeding areas, before returning to their roosts before sunrise. The chances of detecting the bird by seeing it are infinitesimal. However, the research team discovered that the birds call for a short period when they emerge after sunset, and again for a short period when they return to their roosts before sunrise.

This vocal behaviour made it clear that acoustic detection was going to be critical, and automated recording units (ARUs) were immediately deployed at the sites where the birds had been found. Analysis of the first datasets showed that not only were the birds predictably vocal at sunset and sunrise, the birds were also fairly sedentary, and this vocal behaviour was a daily occurrence. This key finding was quickly put into practice – multiple ARUs were deployed in suitable habitat in an effort to find more birds.

Building a Recogniser

Given the amount of data being collected, it was soon clear that an acoustic recogniser would be an important research tool. Naïvely, I thought it would be relatively straightforward to build one! I wanted a process I could easily understand and manipulate, something some of the advanced recognition algorithms did not offer, and a requirement I soon realised I had in common with other field ecologists dipping their toe into acoustic recognition for the first time.

I set about building a recogniser, and as the literature suggested, focused mainly on how the recogniser was trained, while trialling a few different algorithms. I settled on the R package ‘monitoR’, because it was simple to understand. Some other algorithms were too much like a ‘black box’, where the relationship between what goes in and what comes out is unclear. Speaking to colleagues I soon realised this was a common complaint.

While the recogniser worked, I continued trying to improve its performance. A recogniser returns a number of ‘detections’, each with an associated score. The proportion of these that are actual, in our case night parrot calls, is determined by a score threshold you set. Setting the threshold high means that few detections are returned, but that most are likely to be night parrot calls. This often results in calls, particularly faint or soft calls, being missed. To extract these faint detections you lower the threshold, which results in more detections being returned, but a smaller proportion of these are actual calls, hugely increasing the work required to check detections.  

Ultimately, the necessary threshold setting is determined by the question you’re trying to answer, and how much time you have to check your detections. In my case, if an ARU is positioned near a roosting site, and we’re simply interested in knowing whether the birds are still present, detecting a single call within a dataset will be enough. I can afford to prioritise precision and set the threshold high, knowing I will reduce the number of detections returned. If the ARU is in a region where birds have never been detected, extracting the single call that might be in that dataset is critical, so I must lower the threshold to give myself the best chance of detecting that single call, which may be faint. This decision requires me to check the large number of detections returned by the recogniser to make sure I don’t miss that important call.

Improving Recogniser Performance

A good recogniser will simultaneously maximise both precision (the proportion of detections that are actual calls) and recall (the proportion of actual calls within a dataset that are extracted). While examining the recogniser’s outputs with my colleagues, we noticed some patterns that could be exploited. We saw that false positive detections were particularly prevalent at certain times of night, and that particular recogniser templates created the majority of false positives. We also found that a number of false positives were caused by non-target calls made in close proximity to the ARU.

We decided to investigate if we could exploit these patterns, by creating a model that used the recogniser’s outputs and some contextual information to try and predict the probability that any given detection was a night parrot call. This was novel, as it was a post-processing approach. Most of the literature focuses on aspects of pre-processing, such as how you train the recogniser, or which algorithm is used. We found that substituting this model-derived probability for the recogniser’s score output greatly increased the recogniser’s overall performance, improving precision against recall.

Summary

Our approach achieves good results using relatively simple techniques. It relies on knowledge of a species’ ecology, and could be developed further using other contextual variables, or more advanced recognition algorithms. Most importantly, it is accessible to field ecologists and conservation scientists for whom building an acoustic recogniser is an important tool, yet do not have the time or expertise to immerse themselves in this rapidly growing field of study.

To read the full study, check out the Methods in Ecology and Evolution article, ‘Using intrinsic and contextual information associated with automated signal detections to improve call recognizer performance: A case study using the cryptic and critically endangered Night Parrot Pezoporus occidentalis’.