As part of Black History Month, CR is highlighting some of the consumer issues that disproportionately affect Black communities. We have witnessed the troubling ways in which discrimination and bias from the analog world have migrated to the digital space.
Algorithmic discrimination occurs when an automated decision system repeatedly creates unfair or inaccurate outcomes for a particular group. CR’s investigative reporters and the Digital Lab have identified many ways that businesses, such as auto insurance companies, use factors such as education, job status, and the neighborhood you live in to charge you higher rates which can disproportionately affect marginalized communities.
The Ranking Digital Rights Corporate Accountability Index released this week included for the first time a review of algorithms and what they found was striking: “Many of the world’s most powerful algorithms are accountable to no one — not even the companies that build and deploy them.”
The biases that exist in algorithmic data processing are of serious concern because they have the potential to further perpetuate and exacerbate existing systemic inequalities as these technologies become more entrenched in society. We have already begun to see how discrepancies in their creation and execution can lead to severe consequences as the use of algorithmic decision making becomes more prominent in government, law enforcement, and even consumer products.
We must first understand how these algorithms work and how the biases arise in order to understand the negative implications for people of color and Black people in particular. From there we can begin to explore what policy solutions can help mitigate these problems.
What are AI and algorithmic biases, and how do biases arise?
Artificial intelligence (AI) is a broad term that means using data to make predictions or classifications about future data points, while an algorithm is simply a set of instructions to make these predictions and classifications. Data is used to train an algorithm so that it can make more accurate decisions, and the algorithm is only as good as the quality of the data it is fed. Certain kinds of algorithms have been around for decades and have been commonly used for statistical modeling. However, newer types of algorithms tend to be less explainable to the public and even to the engineers who design them, particularly when the task is more complicated — like trying to classify an image as containing a certain object or even predicting human behavior.
There are many sources of bias when designing an algorithm to complete a certain task, but many of them revolve around human error. Non-inclusive datasets (datasets that may not fully represent the populations the algorithm is trying to make decisions for) or biased data collection methods can lead to poor outcomes in algorithmic decision making for those who are underrepresented in the training data. Other types of error can arise from the specific type of model being used as well as the attributes of the data the engineer chooses as being important to the final outcome. Overall, while many companies claim that their algorithms lack bias and are significantly better than humans for decision making, there are still many steps in the process of designing algorithms that require human intervention. While many algorithmic technologies are improving, they will likely never be perfect.
How are Black communities disproportionately affected?
While there are many sources of bias in algorithms, a major reason why algorithms can perpetuate discrimination against minorities is due to biases that often stem from societal inequities. For example, some police departments have begun to use predictive policing algorithms, which aim to predict where and when a crime is going to occur (or even who is likely to have committed a crime), with the goal of better allocating policing resources to these predicted areas. These algorithms use historical data from crime reports on where and when crimes take place to make predictions about future occurrences of crime. However, this historical data tends to be skewed, since Black communities tend to be overpoliced, so alleged crimes are reported more often than they are in whiter areas. If algorithms use data from sources like past arrests or crime reports, it is likely that these algorithms will point police officers to locations that are already being heavily policed, which reinforces the already biased decisions about where officers should patrol.
While the previous example discussed overrepresentation in datasets, underrepresentation of Blacks and minorities in training data can be equally harmful. Facial recognition algorithms are becoming more common in everyday life, being used in anything from security systems to identifying potential suspects in alleged crimes by law enforcement. Studies have shown that many facial recognition algorithms perform worse for those with darker skin. A well-known study by Joy Buolamwini and Timnit Gebru tested facial recognition algorithms from three different companies and found that they all consistently performed best when identifying lighter-skinned males and worst on darker-skinned females, by significant percentages. Darker-skinned men also had higher error rates compared to lighter-skinned males. As these technologies become more embedded into our society, we should consider the consequences of discrepancies in error rates of people with different skin colors.
Some of these algorithms are already being used in law enforcement to identify people suspected of crime, and false positives tend to arise more often for Black individuals. Due to potentially heightened surveillance of Black communities as well as jeopardizing the lives of those mis-identified by facial recognition technology, many have argued that the use of facial recognition by government should be banned. As problems with biases and inaccuracies in facial recognition continue to surface, some local governments in the US are banning the use of facial recognition by public agencies, including law enforcement. Fight for the Future has a petition in support of federal legislation that would ban law enforcement use of facial recognition. Color of Change has a petition urging people to break up with Amazon in part because of the harmful impact of its facial recognition software on Black and brown communities. Companies like Microsoft have placed moratoriums on their facial recognition technology by use of law enforcement until more testing or a national law is put in place to create guardrails for the technology. We must ensure that these technologies do not contribute to a loss of liberty or other rights for particular groups of people, and also consider whether these types of technologies are actually contributing to a more equitable society before implementing them in routine processes.
The use of algorithms has also exposed some systemic biases in our healthcare system. One study found that an algorithm being used in hospitals across the country has been discriminating against Black patients. The algorithm was assigning risk scores to individuals based on how sick they were in order to refer them to programs that could provide further care. The study claims that the algorithm was assigning Black patients a lower risk score than white patients, even when they were equally sick. The scientists claim that these discrepancies are due to a range of issues in the historical data related to systemic racism in healthcare from distrust of the healthcare system to explicit racial discrimination by health-care providers. For example, more care, associated with higher costs, had been allocated to white patients compared to Black patients — even when they had the same chronic conditions.
What can we do to address algorithmic bias?
Algorithmic bias will likely be a crucial area of policy in the near future as these technologies become more common in everyday life. CR is committed to racial justice, fairness and greater transparency in addressing bias in algorithms. The current lack of regulation surrounding algorithms has created a “Wild West” for many companies using AI and has the ability to do major damage to marginalized communities and consumers in general. We support legislation that promotes transparency, fairness, and accountability for algorithms. Impact assessments, which require disclosure about how an algorithm works, the types of data it uses, and the purposes it is used for and potential discriminatory impacts, are commonly brought up in proposed legislation. We have supported a bill currently moving through the Washington State Legislature on public agency use of algorithms that requires impact assessments from public agencies that want to implement automated decision making, and that also prohibits the use of algorithms for determining access to basic resources that the government provides. Hopefully, similar legislation will start cropping up as states and the federal government realize the serious potential impacts of algorithmic discrimination. In addition, CR has a petition urging Congress to convene hearings on algorithmic bias that can help establish smart guardrails to prevent discrimination.
Furthermore, we advocate for agencies that enforce antidiscrimination law to reassess how algorithms could impact people’s ability to file a lawsuit alleging discriminatory conduct, and also hold companies accountable for discriminatory impacts. Agencies like the Equal Employment Opportunity Commission, the Department of Housing and Urban Development, and the Consumer Financial Protection Bureau should update their guidelines to prohibit algorithmic discrimination to the extent possible and make sure that automation does not further hinder those claiming discrimination to receive compensation. The Federal Trade Commission should also take enforcement action against purveyors of “snake oil” in AI, since companies sometimes claim their technology is capable of doing things that are likely unattainable, often with discriminatory outcomes. We also urge lawmakers to close gaps in existing antidiscrimination law that may need to be filled with the rise of automated decision making.
We also want to highlight other groups working to develop innovative ideas to address algorithmic bias issues. Organizations like the Algorithmic Justice League and Upturn are focusing on centering equity and racial justice in their research, making the public more aware of how AI can be disproportionately harmful to marginalized communities. Other groups like the Data Nutrition Project are making tools for computer scientists to better assess the completeness and quality of their data before feeding it to an algorithm. As algorithmic bias becomes more central to legislative discussion, it will take innovations from various stakeholders in both the private and public sectors to mitigate harm while leaving space for potential benefits that could promote more equitable outcomes for Black individuals and other minorities.
