AI World - Biases in Algorithms

Introduction

In the world of AI, a new challenge that is being increasingly discussed is the biases that creep into algorithms. Due to these biases, when we leave it to algorithms to take decisions, there can be unintended consequences. 

In the past, developing an algorithm involved writing a series of steps that a machine could implement again and again without getting tired or without making a mistake. But today’s algorithms, based on machine learning, do not follow a programmed sequence of instructions. Rather they ingest data and figure out for themselves the most logical sequence of steps. They then keep improving this sequence as they consume more and more data.

 

Unintended consequences: Some examples

In March 2016,  Microsoft developed a Twitter chatbot named Tay, built by mining public conversations. The bot would learn and adapt based on its conversations on Twitter.  But in less than 24 hours, Tay began to spew out offensive tweets. What began as a fun chat bot designed to engage in “casual and playful” conversation turned into a PR nightmare.

Facebook developed a feature called “Memories”, highlighting to users what happened on this date in previous years.  It could remind people of memorable vacations, friends’ weddings, or other joyful occasions. However, it also reminded people of painful events, like the anniversary of the death of a family member, or it might ask someone to say happy birthday to a deceased friend.

Google Translate uses AI and deep learning to provide automated translation service for dozens of languages. But in November 2017, there were complaints that the algorithm was sexist. That was because in case of languages like Turkish, there is a single third-person pronoun “o” with no gender while in English, we typically use either “he” or “she,” depending on the gender. When translating Turkish to English, the translation algorithm would decide a gender when translating the gender-neutral “o,” producing translations like “he is a doctor,” “she is a nurse,” “he is hard working” or “she is lazy.” The problem was identified not only for Turkish but also for many other languages that mark gender differently. While Google quickly fixed the problem, the incident was an embarrassment to the tech giant.

Needing to deal with   millions of resumes, Amazon developed an algorithm that could screen out potential applicants. The algorithm looked at patterns in the resumes of previous successful hires and began applying those characteristics to new applicants. Unfortunately, the algorithm reinforced the biases of hiring for male-dominated roles like software engineering. The algorithm taught itself that resumes that included phrases like “Society for Women Scientists” were less preferred because they contained the word “women.” In October 2018, the company scrapped the project.

Princeton University researchers used off-the-shelf machine learning AI software to analyze and link 2.2 million words. They found that European names were perceived as more pleasant than those of African-Americans. The words “woman” and “girl” were more likely to be associated with the arts while science and maths were most likely connected to males. In analyzing these word-associations in the training data, the machine learning algorithm picked up on existing racial and gender biases shown by humans.

Latanya Sweeney, Harvard researcher and former chief technology officer at the Federal Trade Commission (FTC), found that online search queries for African-American names were more likely to return ads to that person from a service that renders arrest records, as compared to the ad results for white names. The same differential treatment occurred in the micro-targeting of higher-interest credit cards and other financial products when the computer inferred that the subjects were African-Americans, despite having the same backgrounds as whites.

MIT researcher Joy Buolamwini found that facial recognition software systems failed to recognize darker-skinned complexions. Generally, most facial recognition training data sets are estimated to be more than 75% male and more than 80% white. When the person in the photo was a white man, the software was accurate 99% of the time at identifying the person as male. According to Buolamwini’s research, the product error rates for the three popularly used systems were less than 1% overall, but increased to more than 20% in one product and 34% in the other two in the identification of darker-skinned women as female.

The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) algorithm is used by judges to decide whether defendants should be detained or released on bail pending trial. The algorithm was found to be biased against African-Americans. Compared to whites who were equally likely to re-offend, African-Americans were more likely to be subjected to longer periods of detention while awaiting trial.

The Predictability Paradox

We can thus create intelligent algorithms in highly controlled environments, to ensure they are highly predictable in behavior. However, these algorithms will encounter problems they were not prepared for. Alternatively, we can use machine learning algorithms to create resilient but also somewhat unpredictable algorithms. This is the predictability-resilience paradox. While we prefer fully explainable and interpretable algorithms, the balance between predictability and resilience inevitably seems to be tilting in the latter direction.

One solution to resolve the predictability resilience paradox is to use multiple approaches. Thus, in a self-driving car, machine learning might drive the show but in case of confusion about a road sign, a set of rules might kick in. Another solution being discussed is the development of explainable or interpretable systems.

What Can We Learn From these AI Failures?

Part of the power of AI and deep learning is that AI training can indiscriminately learn all nuances of language, even if we do not explicitly instruct it to do so.  Unfortunately, it can pick up trends that we would not like it to pick up — such as the inherent gender bias in our use of language. Thus, AI reinforces biases and stereotypes we may inherently have, but do not even realize.

AI itself is not fundamentally biased. The biases in these algorithms are the result of biased training data built by humans. It is simplistic to suggest that we must collect unbiased data.  Almost all human data is fundamentally biased in some way.

Companies need to incorporate anti-bias training alongside their AI and ML training, spot potential for bias in what they are doing, and actively correct for it. And along with the usual testing processes for software, AI needs to undergo an additional layer of social testing so that problems can be caught before they reach the consumer and result in a massive backlash. Additionally, the data scientists and AI engineers developing the models must be formally trained on the risks of AI.

Most importantly, business leaders need specialized AI training to understand both the possibilities and the risks. Executives need to understand data science and AI enough to manage AI centric products and services and  to appreciate the potential shortcomings of AI.

Understanding these dangers is the responsibility of not just those leading AI initiatives, but all executives. A PR leader who understands social media dynamics and the vicious troll culture can avert the dangers of a self-learning AI Twitter bot. An executive familiar with HR and employment discrimination law can help flag potential dangers of resume screening bots. And a manager with operating experience across multiple countries might be able to spot the pitfalls in translating genderless pronouns.

No single manager has the contextual knowledge to spot everything that can go wrong. So companies need to train all their business leaders on AI’s potential and risks, so that every line of business can spot opportunities and flag concerns.

Concluding notes

Algorithms have the potential to change the world for the better. To leverage that potential, it is important that confidence in algorithms is not undermined. We lose confidence in algorithms much more quickly than we do in human forecasters when both make the same mistake. We are unforgiving of algorithms when they fail, but we accept that humans are fallible. Take the case of autonomous cars. According to some estimates, driverless cars would save up to 1.5 million lives just in the US and close to 50 million lives globally in the next 50 years. Yet, in a poll conducted in April 2018, 50% of the respondents said they considered autonomous cars less safe than cars driven by human beings.

When it comes to trust, two factors are very important: control and transparency. Research indicates that as long as we have some control, even minimal, trust is significantly enhanced. We should introduce simple features that give people a feeling that they are in control. When automated elevators were introduced, people were scared to use them in the absence of an operator. What reassured lift users was the introduction of a red button that said “Stop”. The button was no big deal. In fact, if someone pressed the button, they were directed to use the phone and speak to a remote operator. Yet, the Stop button made users feel as if they were in control.

Transparency can improve trust in an algorithm. When we know how the algorithm works, we feel more in control. But again, the reality is nuanced. When we meet new people, we expect a degree of transparency. We tend to distrust people who are too cagey and seem to be concealing their thoughts and intentions. At the same time, we get suspicious when people try to be too transparent. We start wondering what they are trying to prove. Therefore, for human beings, there is a right level of transparency. The same logic applies to algorithms. Too much information about the algorithm can undermine user trust just as too little.

 

There are different dimensions of trust that we must keep in mind when we design an algorithm. In general, customers should be convinced that the algorithm knows what it is doing, has good motives and upholds basic human values such as honesty and fairness. That is users must know how the algorithm works, why it takes decisions in particular ways and what tradeoffs it makes while making a recommendation. However, not all these aspects may be important for everyone and for all situations. Different aspects of trust may become important in different contexts. So, explanations about the algorithm have to be carefully prepared and calibrated to address the specific trust issues.

While designing and deploying algorithms, companies must consider the role of diversity within their work teams, training data, and the level of cultural sensitivity within their decision-making processes. Greater diversity can potentially avoid harmful discriminatory effects on certain protected groups, especially racial and ethnic minorities. The use of cross functional teams in developing algorithms is emerging as a best practice. Rather than trying to automate everything, more human involvement is desirable for better algorithms. Last but not the least, the formal and regular auditing of algorithms can also help in detecting and mitigating bias.

 

Hope you found this useful.