How do we know if something works?

There was an interesting news story in Nature recently about using clinical techniques to analyse international aid projects and see whether they actually work. Obviously this is important because if you’re spending a huge amount of money to uplift people you want to know that it’s actually doing that. If it’s not then you’re wasting your money and aren’t helping anyone. So how do we know if something is working?

Most people would probably say you do something and see if that solves your problem. To use an example from the Nature article, the World Bank wanted to reduce malnutrition in Bangladesh. To do this they taught mothers about nutrition and saw that the rate of malnutrition did fall. Success. Or so it seemed. The problem with this sort of intervention is you actually have nothing to compare it with and you can’t be sure that your intervention was responsible for the change. The World Bank knew this and they also monitored malnutrition in areas where they did not teach mothers about nutrition. In those areas, which are called “controls,” they also saw a drop in malnutrition. That means malnutrition was decreasing everywhere and their intervention was not helping, or at least wasn’t the only reason. As it turns out their intervention didn’t work because although they had taught the mothers about nutrition it was the fathers who decided what food the family ate.

To know if something works we definitely need controls. This is why it’s almost impossible to tell if something you did cured your illness. It’s hard to face but if you had the flu, had acupuncture and got better you wouldn’t be able to say that you got better because of the acupuncture or because you were going to get better anyway. What we need, to see if something really works, is a comparison between people receiving an intervention or people who are not. Also very important is to assign intervention and control groups randomly to avoid any bias. If someone wants something to succeed then they can bias the results by assigning the best subjects to that group, making the trial useless. There are more details than that but randomly assigning people to intervention or control groups and comparing what happens is the basic idea of the randomised controlled trial (RCT). What’s really cool is that RCTs can be applied to almost anything, from health to policies.

Jeffrey Sachs, a sustainable-development economist at Columbia University, worries that RCTs are not an ethical way to assess development projects, because they withhold aid intervention from control groups.

Above you can see a criticism of RCTs, one that doesn’t actually hold. RCTs are the best tool for learning if something works. The problem with aid projects is that they haven’t been analysed in such a way, so we only think that they work. If the aid project is genuinely helpful and we knew that then it would be unethical to withhold it from people. As these projects haven’t been subjected to trials before we don’t know that they work and, like in the nutrition example, it may be that the aid interventions are no better than doing nothing and are just wasting money. This is why it’s important to do such trials whenever we do not know whether something works.

In South Africa a possible target for a RCT is the youth wage subsidy. Government tells us that it will help with the country’s unemployment situation but COSATU claims that the research is more ambiguous. I don’t know enough about it to say but if the research is unclear then the best thing to do would be a RCT. We could randomly assign towns around South Africa to implement the youth wage subsidy, others to continue as normal and then look at what happens with regards to employment in the different towns. This would give us a clear answer of whether it will work in our situation or not.

In fact RCTs are how we should be deciding all our policy decisions. To that end I highly reading Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. The paper was written for the UK government and, despite being reasonably long, is very easy to read. It goes into far more detail here about why we need RCTs, how to do RCTs and provides a number of examples of how RCTs can improve policies. It’s probably too much to hope that the South African government reads and implements it.

So the message here is that we know if something works by comparing an intervention against no intervention (or the current standard where there is one) and the best way to do this is a randomised controlled trial. It’s important that we do these whenever we don’t know if something works because if it doesn’t then we are wasting our time and harming people by not using effective interventions. RCTs are commonly used in medicine but they are not limited to medicine. They are used in other fields and it’s important that they are. We need to know whether aid interventions are working and we need to know what sort of government policies work best.

Advertisement