Apr 8, 2022 7:00 AM

The Census Is Broken. Can AI Fix It?

Machine learning can help count people to deliver government funding and decide political representation, but the technology still makes mistakes.

Orange ping pong balls arranged in a grid on a green background one ball is smashed

The AI Database →

Application

Prediction

End User

Government

Source Data

Geolocation

Images

Technology

Machine learning

Machine vision

Getting a census count wrong can cost communities big. A March 10 report from the US Census Bureau showed an overcount of white and Asian people and an undercount of people who identify as Black, Hispanic or Latino, or multiracial in 2020, a failure that has led to renewed calls to modernize the census.

Progress reaching historically undercounted groups has been slow, and the stakes are high. The once-a-decade endeavor informs the distribution of federal tax dollars and apportions members of the House of Representatives for each state, potentially redrawing the political map. According to emails obtained through a records request, Trump administration officials interfered in the population count to produce outcomes beneficial to Republicans, but problems with the census go back much further.

The US Census Bureau began its latest effort to modernize the population count in 2010, largely through automation and technology. According to a summarization of approaches that influenced the 2020 census, part of the goal was to keep costs comparable to 2010 levels, at about $12 billion. Despite millions more people to count, the US Census Bureau almost halved the number of people hired for door-to-door visits—350,000 people were deployed in 2020, down from 516,000 in 2010.

Counting hundreds of millions of people is no simple task, and governments are turning to digital solutions ranging from simple online forms to sprawling AIs and even pictures from space to lighten the load.

2020 marked the first time most US census forms were filed online instead of on paper. Information from federal agencies like the IRS and location data from private brokers helped update addresses or add new ones. Automation helped with field staff recruitment, hiring, training, and payroll. In 2020, the Census Bureau also used software to classify new construction projects seen in satellite imagery for the first time, an approach that added 5 million new addresses to a list of households in the country. In contrast, virtually every address recorded in the 2010 census was verified by a person who physically went to that location. By 2020, that had shrunk to just 35 percent.

And despite applications of new technology, the census still got plenty wrong. Estimates suggest Native Americans were undercounted by 5.6 percent, while the Hispanic or Latino census category had the second-highest undercount, at an estimated 4.9 percent. Children ages 0 to 4 were undercounted by an estimated 2.79 percent. The US Census Bureau recently assembled a team to research why it’s so hard to count young children, and a breakdown of housing units undercounted in 2020 is due out this summer.

The Census Bureau is now working on a machine learning algorithm that detects new construction, using satellite imagery to track projects from start to finish. Developed with input from Statistics Canada and an automation and analytics consultancy, the initial prototype achieved accuracy rates of about 90 percent. In the year ahead, the US Census Bureau will test its AI in different parts of the country to see how the model performs in various climates and regions. Designers of the system use building permit data to validate initial results, an approach they suspect will make the model more accurate in places where new construction projects don’t require a building permit. If all goes according to plan, the Census Bureau wants to eventually make construction activity data available as a means of tracking local economic activity down to the zip code.

Those who support using machine learning to analyze satellite imagery believe the two systems can work together to help solve big problems, like protecting the Amazon rainforest, combating poverty, or more accurately estimating the number of people in a given area. But experts building these models say that when they’re used to count people, they’re still prone to mistakes.

Before becoming the director of the US Census Bureau, statistician Robert Santos accused the Trump administration of sabotaging the population count. Upon release of data last month, Santos told reporters he wasn’t surprised by the Hispanic or Latino undercount. Census Bureau focus groups showed similar reservations before the start of the count. Santos said the pandemic—which put people out of work and exacerbated hunger and housing issues—impacted everyone, but especially Latinos. Census Bureau staff also blamed the pandemic, wildfires, and a record-breaking year of hurricanes for interrupting the process. Santos said the undercount can be taken into account to adjust population estimates that impact funding, but it’s too late to change apportionment, the process that determines the number of House of Representatives seats assigned to each state. That will have to wait for the next census in 2030.

The undercount in 2020 may have contributed to states with large Latino populations— like Arizona, Texas, and Florida—receiving fewer seats in the House of Representatives. In testimony at the state capital in February 2021—nearly a year before the release of official undercount data—Texas lawmakers expressed concerns about funding and representation shortfalls in heavily Latino areas like the Rio Grande River Valley. “There are probably scores of legislative and congressional districts that are Latino majority that are probably malapportioned because they really have more people in those districts than what the census numbers show,” says Arturo Vargas, executive director of the National Association of Latino Elected and Appointed Officials, a nonprofit group. “That undermines representative democracy,” he adds.

Vargas attributes the undercount of Latinos to insufficient funding to prepare and administer the census, as well as significant political interference by the Trump administration, most obviously through a last-minute attempt to add a citizenship question. The Census Bureau's increased use of technology is not bad in itself, he says, but he fears relying on such solutions alone could continue historical trends. In a statement issued after the undercount came to light, Vargas repeatedly called for more resources to reach groups historically undercounted in the US Census and for the continuation of modernization efforts. But he also expressed concern that some tech was being applied to make the census less expensive to conduct but not necessarily more accurate.

The problem Vargas has with using software and satellite imagery to update the master address file used to send census forms or reminders is that it can’t reveal what’s happening within individual buildings. Beneath one roof, he says, owners or occupants can subdivide two units into four and leave no paper trail. “Unless you actually have somebody walking the street and looking for those addresses, you may not capture that easily, which is why we have a real concern about the bureau relying so much on technology to try and get the most complete address file as possible,” he adds.

Greg Yetman is codirector of the Center for International Earth Science Information Network (CIESIN), a part of the Climate School at Columbia University. As part of a NASA contract, CIESIN has been exploring ways to deliver socioeconomic data by observing the Earth since the early 1990s. Yetman says things like understanding that it’s common for people to live in basement apartments in the Queens borough of New York City, for example, are “always hard to capture and really difficult to measure from space.” Apartment conversions, sublets by an owner or occupant, or unregistered settlements—all likely to increase as the cost of living climbs—aren’t often captured by the census or satellites either. And if a person is unhoused or has few financial records, they may not show up in location-sharing data collected by private brokers.

There’s room to improve on the census in the US, but the Constitution requires that one be conducted every decade, and Yetman says the country is “data rich.” By comparison, some countries haven’t carried out detailed household surveys in decades. Obstacles such as cost, conflicts, or difficulty reaching remote locations can make some communities harder to count.

In 2017, the Nigerian government, CIESIN, and others working with funds from the Bill & Melinda Gates Foundation used satellite imagery and machine learning to map the country’s population to deliver measles vaccinations. Since then, Gates Foundation senior program officer Vince Seaman says, the effort has expanded to five other African countries, a project known as Grid3. That work, he adds, demonstrates that the tech is only part of the solution. After applying machine learning to photos from satellites, community surveys were carried out to reach thousands of people in person and to verify results.

In research published last month, satellite imagery and machine learning were used to automatically identify housing plots and predict population, age, and sex in five provinces in the western half of the Democratic Republic of Congo (DRC). The project brought Grid3 participants like the University of Southampton in the UK together with groups like the DRC’s National Bureau for Statistics. Anonymous surveys of nearly 80,000 people were carried out by the Kinshasa School of Public Health and University of California, Los Angeles School of Public Health to validate the performance of a deep learning model that achieved about 80 percent accuracy. Coauthors say their method is no replacement for a true attempt to count the entire population, but it can supply a predictive snapshot of society in places with little or poor-quality data. No national census has taken place in the DRC since 1984.

Yetman has spent more than 20 years working with satellite images. He works with Pop Grid, a data collaborative for a diverse group of organizations that count populations, including the European Commission, Facebook, the German Aerospace Center, and NASA. He says deep learning models for identifying buildings can’t always tell where one roof ends and another begins, and he warns there’s no such thing as a model that works everywhere in the world.

In the US, he explains, applying an AI model trained using images of roofs from the western US is problematic if it's applied to homes on the East Coast because the western expansion of the country follows a grid-based system, while cities like Boston developed with less uniformity. Equally, a roof in South Africa looks different from one in Zambia. AI can easily mistake the roof of a stall at a commercial market in Accra, Ghana with the roof of an unregistered home or struggle to accurately predict the number of people in urban settlements or rural villages. “Without the on-the-ground survey that says there’s a slum or informal settlement here, it’s really difficult to know just from the structure of the roof patterns,” Yetman says. He adds that obtaining high-quality data for training models to detect buildings or home plots based on local conditions is the hardest part of the job.

Deep learning models that make population predictions from satellite imagery tend to overestimate the size of rural populations and underestimate the size of urban populations, likely due to an inability to recognize the height of buildings, according to a 2021 analysis by researchers from MIT, the University of Minnesota, and the University of Arkansas.

The issue of building detection is being tackled by a number of other companies and organizations. Radiant Earth Foundation works with geospatial and machine learning practitioners on sustainable development goals like the eradication of hunger and extreme poverty by 2030. The group is also developing building detection with machine learning in Dakar, Senegal. Hamed Alemohammad, chief data scientist at the foundation, explains that his company’s building detection models can sometimes mistake three different buildings for one home. “Those models have uncertainties and errors, nothing is perfect,” he says. “If I'm making a building detection model for every region in Nigeria, I need training data from that region because the model learns the features of that region. We lack geographic diversity these days,” he says. To supply benchmark data representative of different communities, the nonprofit recently launched the Radiant ML Hub.

While some initiatives aim to improve population counts, others are developing AI to generate the kind of data typically collected in door-to-door community surveys. A University of California, Berkeley project now underway is developing a generalizable AI capable of solving a diverse range of tasks, from identifying forest cover or the length of roads to predicting elevation or home prices.

To test the limits of that model, last summer researchers tried to recreate parts of the American Community Survey, which the US Census Bureau conducts every year. The model predicted population density and household income with an accuracy above 50 percent. That might not sound impressive, but the results took a single graduate student one week to produce, while the last US census cost $14 billion dollars and required the labor of hundreds of thousands of people. The researchers found some other limitations—the experiment couldn't, for example, predict the percentage of household income dedicated to rent. Census data can also be influenced by factors you can’t see from space, like local tax law or the quality of local schools, says UC Berkeley Global Policy Lab director Solomon Hsiang.

But the lofty dream of the generalizable model, Hsiang says, is to make it possible to address any environmental and socioeconomic problem on a global scale and empower any policymaker with a computer to use satellite imagery and machine learning to make informed decisions. “Ours is not as good, the quality is a lot lower,” he says of the AI model he worked on. “Can you ever get it to a point where you're really saving hundreds of millions of dollars by doing things a totally different way? The question is how far can you push it?”