In March 2022, my LinkedIn feed lit up with people sharing this article:
It is by Reuters, a highly-respected news source, on a report by Arabesque, an asset manager and data provider I greatly admire and wrote about in Grow the Pie. It was taken at face value and shared uncritically by LinkedIn Top Voices, such as the post below with 415 Likes and 92 comments:
One of these comments, by another LinkedIn Top Voice, claimed that “Diverse boards do better period” and that this evidence could be used against the “non converted”. The “period” implies that the issue is open-and-shut, ironically the opposite of diversity, and the “non converted” suggests that this issue is about ideology rather than openness to evidence. In reality, most people are neither converted nor non-converted, but willing to change their views based on new facts.
I was immediately intrigued, given my research on both diversity and sustainability, and wanted to read the paper. But there was no link to it in the Reuters article, nor on the Arabesque’s website (and someone I know contacted Arabesque and came up empty). So I asked for a copy in my above comment, presuming that someone endorsing it so enthusiastically would have read it, but got no reply. Someone else on LinkedIn shared it with the headline “Boards with more women on them deliver more value. Period” and I similarly asked for the study; the poster replied that she hadn’t actually seen the study but took the journalist’s word for it as journalists are always trustworthy. (The post has now disappeared).
So there was no such study. Reuters wrote about a study that never existed, only interviewing Arabesque about what they did and taking their word for it, and people shared the Reuters article similarly without checking. This is a classic example of confirmation bias – if a newspaper article claims a result that people don’t want to be true, people would demand to see it to try to rip it apart.
The “study” eventually came out three months later, and it is here. It is just over three pages, with not a single table of numbers. The only analysis to back up the claim was the following bar chart:
Unfortunately, there are many issues.
1. Measure of Diversity
The report is very transparent on how it measures diversity: “Our ESG Diversity Feature … feature assesses a company’s performance on various diversity metrics, including the representation of and equal opportunity for women and minorities in the workforce and on the board. Inputs to the feature also include the number of discrimination lawsuits as well as corporate commitment to supplier diversity”.
But this is is very different to what the Reuters article claimed, which was about “Boardrooms with more women”, and many people on LinkedIn claimed that the study proved the value of gender diversity in the boardroom. Instead, Arabesque studied many other aspects of diversity: ethnic diversity, diversity outside the boardroom, discrimination lawsuits, and supplier diversity. This is not the fault of Arabesque, who should be commended for having such a broad measure of diversity, but those misportraying what the study actually measured or, worse, injecting what they want to be true into their interpretation of the study. We have no idea as to whether it’s gender diversity in the boardroom, or the many other aspects of diversity captured by Arabesque, which is driving the results. (Indeed, one of the goals of Arabesque is to provide investors with more comprehensive measures than what’s captured by standard metrics).
2. Measure of Climate Performance
The report is also very transparent on how it measures climate performance: “the ESG Book Temperature Score … measures the extent to which corporations across the world are contributing to the rise in global temperature. Companies’ publicly disclosed Scope 1 and Scope 2 emissions are translated to a score based on sector-specific emissions
pathways, ranging from 1.5°C to >2.7°C. A score of 3°C, however, indicates that the company does not disclose emissions in line with the Greenhouse Gas (GHG) Protocol.”
So the Temperature Score does not actually measure climate performance. A company is automatically in the worst possible bucket (3°C) if it does not disclose – regardless of how it’s actually performing. And a company’s Scope 1 and 2 emissions will depend much more on whatever industry it happens to be in rather than whether it actually “deliver[s] more on climate”. A cement company that’s best-in-class in emissions reduction (see Nili Gilbert’s excellent TED talk) will have a worse score than a tech company that takes no climate action, even though it is arguably delivering much more on climate. The GHG Protocol also allows for carbon offsets, despite numerous problems with them.
3. Correlation vs. Causation
The chart contains no test of statistical significance. But even if there were statistically significant differences, this would only be a correlation. There could be reverse causality: perhaps emitting companies (such as oil and gas) are more likely to attract non-diverse employees; in contrast, diverse employees might care more about climate and be attracted to less-emitting firms. Alternatively, there could be omitted variables: more profitable companies can invest more in both climate and workforce diversity. There is not a single control variable in the analysis. Everybody claims to know that correlation is not causation, but we suddenly forget this when we like the causal story being paraded.
What are the broader lessons that we can learn from this episode?
- Pause Before Sharing. We should not share a study without checking that it actually exists, otherwise we spread misinformation. Journalists should not write about a study unless they’ve actually seen and scrutinised it (rather than basing their article on an interview or a press release), and users of social media should not share a conclusion without checking that there’s actually a study that delivers that conclusion.
- Check What’s Measured. Studies claim to find a relationship between an input variable (e.g. diversity) and an output variable (e.g. climate performance). But, as I explain in Chapter 3 of May Contain Lies, often what the data actually measures is different from what’s claimed. This is a particular issue in sustainability when issues such as purpose, conscious capitalism, ethical leadership etc. are very difficult to measure.
- Explore Alternative Explanations. If we see a conclusion that we don’t like (e.g. “diverse companies perform worse on climate”), we’d try to appeal to alternative explanations such as reverse causality and omitted variables. Perhaps companies in crisis perform worse on climate since they need to focus on financial survival, and perhaps companies in crisis are shaken into looking outside the old boys’ network when recruiting. We need to apply the same discernment to a study we like as to one we don’t.