Post provided by Antica Čulina

Hi, I am Antica. I work as a senior researcher at the Ruđer Bošković Institute in Croatia. This is a story about how something that bugged me 15 years ago was finally formalised in a paper on a Risk of Bias assessment in ecological and evolutionary meta-analysis. The take away massage – it is never too late to do the right thing.

I am one of these people that do not pay attention to the ‘important’ dates in my life. So, I had to check, but it seems I started my PhD at the end of 2010. I guess it was 2011 when my supervisor suggested that I should consider doing a meta-analysis instead of a classical review for my first chapter. He had heard about this ‘new’ thing to synthetise results from existing papers, and I was the first of his students to try it out. It was about the fitness causes and consequences of divorce in monogamous birds (which you can find here, Culina et al. 2015)

Let’s just say it was a challenging task, as no-one in the group knew how to do it. Finally (and 3 years after starting the work on it) my meta-analysis was published. The long-term outcome was the whole research line I am working on today – how to make EcoEvo research more reliable, and impactful. Here, I will keep to the part related to the Risk of Bias work.

When you do a meta-analysis, you need to extract information from primary studies on the effects that interest you – for me this was data on the strength of the relationship between breeding success of pairs in one season, and the likelihood that they divorce (or stay faithful) in the next season; and on breeding success post-divorce or fidelity. While reading primary studies, I noticed that some are just ‘better’ done and I trusted them more. I came up with a system that consisted of three questions and used these to score each study. I called it ‘data quality assessment’, and you can see exactly what I have done in the Supplement of Culina et al. 2015. I used this variable as a covariate in a meta-regression model. The results showed that pairs that will divorce had lower breeding success than faithful and widowed pairs, and that this effect decreases with the increase in data quality. In other words, studies that I trusted more showed less pronounced effect of low breeding success on divorce.

I started to think more about this problem. Meta-analysis uses primary studies to make some general conclusions about biological effects. However, what if some of these primary studies cannot be trusted, i.e. are not that well conducted? How much can we trust a synthesis of such studies? As a reviewer of many meta-analyses, I have never noticed these aspects being considered. Then, quite some time later I learned that there are formal ways to assess study quality that are used in medicine and in environmental evidence synthesis. Without conducting these, your meta-analysis will not (likely) be published.

While there was no consistent definition on what ‘quality’ constitutes, something that seemed quite consistent was a term ‘Risk of Bias’. RoB is a term used to describe threats to a study’s internal validity. It depends on the extent of any trend or systematic deviation in data collection, data analysis, or data interpretation. You should care about it when you design your primary study, and when you conduct evidence synthesis of primary studies. RoB tools help you to assess RoB in primary studies, and are commonly used in medicine. However, use of RoB assessment in EcoEvo is extremely low (only 2-4% of evidence synthesis include it). At this point, I think it was probably 2023, I decided the time had finally come to tackle this interesting topic for EcoEvo evidence synthesis.

However, I lacked a deeper expertise in this kind of the assessment, and also wanted to get opinions from other meta-analyses experts on how to approach the topic. Thus, I invited some great people on board: Dugald Foster, Matthew Grainger, Rose E. O’Dea, Oliver L. Pescott, Alfredo Sánchez-Tójar, Robin J. Boyd, Julia Koricheva, Shinichi Nakagawa and Gavin Stewart. And then things happened… slowly, as this was a side project for all of us, but they happened.

At the end of 2023 we sent out a survey to the authors of EcoEvo meta-analyses to learn more about their familiarity with the RoB assessment. We received 232 responses. In 2024 we conducted a survey of 209 EcoEvo journals that solicit evidence synthesis for the guidelines on conducting RoB. Our surveys show that most journals accepting reviews don’t mention RoB assessment at all, and that many researchers still confuse it with publication bias assessment. Only 12% of respondents had a correct interpretation of the RoB.

To help researchers to familiarise themselves with the Risk of Bias, we suggest them to consider five key questions: Was randomisation applied?; Was selection bias avoided?; Was confounding considered?; Were measurements consistent?; Was the observer/analyst blind?

However, to achieve substantial improvements in application of this essential evidence synthesis standard, we also suggest actions relevant to researchers, publishers, and funders: raise the awareness and provide guidelines on RoB and RoB assessment, facilitate RoB assessment by providing expertise and easier to use RoB tools, and make RoB assessment possible by improving reporting standards of primary studies. Without this, RoB will be very difficult to assess.

A co-author and a friend Matt wrote: ‘Better meta-analysis does not start with better statistics, it starts with better evidence.’  This means that we both need to make sure that we create better evidence base (i.e. primary studies), and that synthesis of evidence must consider the RoB of primary studies. We believe that the community, including researchers, funders, and publishers, should come together and put some effort into introducing higher standards for both primary research and evidence synthesis in ecology and evolution – it is never too late to do the right thing.

We are now considering our next steps in the work on RoB in EcoEvo. If you are interested in joining these, please reach out.

Read the full paper here.

References

Culina, A., Radersma, R. and Sheldon, B.C. (2015), Trading up: the fitness consequences of divorce in monogamous birds. Biol Rev, 90: 1015-1034. https://doi.org/10.1111/brv.12143