Be Careful What You Report: Why Data Collection Matters

العربية | Português 

Photo: Pixabay

Census data tells us that, in 2011, Canada’s Jewish population was around 309,000. Just five years later, in 2016, it was listed at a little more than 143,000. Half the population, gone. What’s happening to Jews in Canada to cause such a dramatic drop?

In fact, nothing has happened to Canada’s Jewish community. What changed were the data collection methods. The 2016 Census no longer offered respondents “Jewish” as an option to identify their ancestors. And the result was staggering.

This is a perfect example of how careful we must be about data collection methods in the social sector. Even the smallest changes can make massive differences to public policy, the way tax dollars are spent and so much more.

Does It Really Matter?

What’s the big deal? The Canadian government changed its data collection methods, some numbers plummeted and we all noticed. It made national news. It’s easy enough to go back and fix the question (as Statistics Canada intends to do for the 2021 Census), so really, where’s the harm?

The problem is that we only noticed this change because the result was huge and appeared catastrophic. But what if the change was more subtle? In reality, this issue is likely occurring — and going unnoticed — in the social sector all the time. And if the results from our different data collection methods don’t change enough for us to examine what’s happening, we won’t stop and take notice.

It Can Change Everything

Canada’s neighbors south of the border are also considering changes to their national Census. The US Department of Justice wants to add a citizenship question, which they say will help them protect against discrimination among voters.

Of course, if you’ve been paying any attention at all to the news lately, you can understand why asking Census respondents about their citizenship could cause a lot of problems. The obvious effects are that either:

•  Fewer non-citizen populations will respond to the Census, depriving those groups of representation, or

•  People won’t answer honestly, making Census data less accurate.

That’s a big problem. A considerable number of resources are dedicated to collecting and analyzing Census data. Making changes to the data collection methods that are likely to reduce its accuracy is a problem in and of itself. But even if you’re not that worried about accuracy for accuracy’s sake, consider the points raised by The Atlantic: “The census is used for allocating nearly $700 billion a year in federal money, electoral votes, as well as for the apportionment of House districts — that is, deciding how many representatives a state sends to Congress each year.”

All of those things are pretty important. It seems like a good idea to allocate them based on data that is accurate.

Check Your Data Collection Methods

It’s unlikely either Canada or the United States has set out to make their Census data less accurate. And both countries employ teams of experts to develop and deploy the Census. So if they’re struggling to get it right, what does that mean for you?

To ensure your data is as representative as possible of the community you’re studying, review your data collection methods.

•  Check your survey (and your privilege).

What question is most likely to be producing biased data? Remember, just because you’d have no problem answering a specific question, that doesn’t mean everyone will feel the same. Watch for questions that could marginalize or questions that might put people off answering at all.

•  Check your physical data collection methods.

How are you actually gathering the data? In person? Online? Using data generated by the day-to-day operations of your program? Think about who you might be inadvertently excluding.

•  Check your results.

Sometimes changing your data collection methods will result in an obvious change in your results. Sometimes it won’t. But examining your results carefully is a good way to guard against bias or error. Datassist published a Data Integrity Checklist with some tips on what to look for.

•  Get expert assistance.

There’s no shame in asking for help. If you’re concerned your survey might be producing biased or inaccurate results, talk to an expert. The team at Datassist is proud to help government agencies, social sector organizations and journalists with data collection, analysis and visualization. Get in touch today.

For more on data collection methods, you can also check out We All Count, Datassist’s new project on equity in data science.

This article first appeared on Datassist’s website and is reproduced here with permission.

Heather Krause is a data scientist and statistician. She founded Datassist, an international team of data professionals that provides data consulting to journalists, nonprofits and policymakers worldwide. Among the groups she has worked with are the World Bank, Bill and Melinda Gates Foundation, CARE, USAID, and the Status of Women in Canada.