Post provided by András Zábó

Just imagine…

You’re all sitting excitedly around the monitor, watching the video captured by the drone. This is the first time you’ve tested the complete monitoring system in the national park… Your drones had already been capable of autonomously finding, detecting, and tracking zebras, but you had never flown drones that were also capable of identifying individual zebras… And both you and many colleagues are fascinated by uncovering and understanding individual behaviour.

You watch as, in the footage, the detected zebras slowly wander along grazing, while the yellow bounding boxes drawn around them move with them, indicating successful detection of the animals. Only one individual has a green bounding box, but for good reason: your enhanced system is now capable of identifying and distinguishing individual animals as well, allowing you to collect an individual behavioural profile for Marty.

Suddenly, something startles the herd in the footage. They break into a run and bunch together more tightly. The tension rises in all of you as the bounding boxes become tangled due to the proximity of the running zebras, and the green bounding box, along with Marty’s name, disappears… The zebras abruptly split into two groups, one moving right and the other left, both rapidly approaching the edge of the drone’s field of view. Your autonomous drone must immediately decide which group to continue following. You see one of the reappearing bounding boxes suddenly turn green, and Marty’s name pops up again with a re-identification confidence of 92%. To your relief, the drone immediately moves towards that group and follows them, never losing sight of Marty.

Exciting, Exciting… But what for?

As I mentioned in the example above, individual identification is essential if we want to study the behaviour of animals on an individual basis. Beyond that, however, it is also valuable for applications ranging from population monitoring to tracking the health condition of animals. And if you’ve seen Madagascar 2, you’ll know that a great deal of confusion can be avoided if you’re able to tell your best friend apart from everyone else…

Rapid: Finding the “one in a million”

To address the challenges outlined above, we developed an individual identification algorithm that is freely available for anyone. It is not only accurate but also fast, making it suitable for deployment on computationally limited computers (so called edge-devices). In addition, whenever it predicts an individual’s identity, it assigns a confidence score to the prediction, providing an indication of how certain the algorithm is. We named the resulting software RAPID (Real-time Animal Pattern re-Identification on edge Devices). Additional key design objective was modularity, allowing RAPID to be easily integrated into existing monitoring systems (remember the example at the beginning).

To avoid any misunderstandings: RAPID is equally well suited to more general individual-identification tasks where processing time or computational capacity is not a limiting factor (e.g. analysis of manually collected footage or camera trap data). So, if drones are not your area of interest, there’s no need to feel left out—you are very much part of the intended audience!

But how?

You might be wondering what exactly the identification is based on. Is a zebra’s mane unique enough? Or perhaps body shape is the key? The answer lies in their patterns. The stripes and spots of patterned animals are as unique as our fingerprints. RAPID recognises these patterns and matches them against an image database containing individuals that have already been identified.

To make all of this happen within a fraction of a second, we make use of an algorithm originally developed by Spotify to provide lightning-fast recommendations of similar songs. The only difference is that, instead of recommending similar music, we use it to “recommend” similar zebras (or other patterned animals).

When does it work and when doesn’t it?

Based on our tests, we have good news for another character from Madagascar: Melman the giraffe—RAPID works not only on zebras, but also on giraffes! In fact, when we tested the algorithm on tigers, jaguars (and even cattle), it was also able to identify individuals with high accuracy.

It is important to note that for species without patterns, the algorithm cannot be applied, as they are, in this sense, “fingerprint-less”. Beyond different types of patterns, RAPID also performs well across a variety of environments (dense rainforest, open savannah), with different camera systems (drones, camera traps, manually collected footage), and under varying lighting conditions (sunlight, mixed sun and shade, nighttime infrared images).

However, alongside the presence of distinctive patterns, one key limitation is that RAPID can only identify individuals that are already present in its database of known individuals (a so-called closed-set scenario), and only when similar viewpoints are available in the database compared to the unidentified image.

What’s next?

By distinguishing previously unseen individuals from those already observed (a so-called open-set scenario), and by characterising viewpoints, RAPID can become applicable in situations where its current performance is not yet reliable.

If you are interested, try RAPID (https://github.com/robot-perception-group/RAPID-animal-reidentification), watch the video abstract (https://www.youtube.com/watch?v=O6NWzLEivr8), or read the full paper (https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.70332).