Can Algorithms be Ethical?

Would you prefer important decisions that directly impact your life, such as your health or finances, be made by an algorithm or a human? Algorithms can search through millions of alternatives to discover optimal solutions specific to your data. Humans have limited ability to assess all the facts, are known to have conscious and unconscious biases, and can make judgment errors. And yet I suspect that even among the algorithmically-minded readers of OR/MS Today, most would have greater confidence that a solution provided by a person is the best one.

Why is this the case? One reason may be that algorithms are rarely designed to reflect our ethical values. This raises numerous questions, perhaps starting with, what are our ethical values? Philosophers have been debating this for centuries, so we will sidestep that issue and introduce two less lofty, yet important questions: What does it mean for algorithms to be ethical and how can algorithms be ethical?

What are Ethical Algorithms?

It seems odd to think of algorithms as ethical or unethical, but when viewed as a source of advice, much like a doctor, lawyer, accountant or other professional, it becomes possible to identify ethical qualities we expect from an advisor, be it human or machine. It is not that the algorithm itself is ethical, but whether the process and the algorithmic solution are viewed as ethical.

When receiving advice from any source, we expect adherence to core values, including:

• Equity: Everyone receiving advice from a source must be treated fairly and in a manner that is consistent with their best interests. Algorithms have failed in this area, with much written about biases, especially racial biases, found in automated processing of loan applications, screening of job applicants, and in parole recommendations [1]. Algorithms often replicate, and in many cases amplify, biases found in historical data.
• Privacy: Sensitive information must not be disclosed or used in attempts to harm or manipulate people. Algorithms have failed in this area too, with numerous examples of stolen, reidentified and misused personal data, often used in efforts to alter behavior [2].
• Transparency: Reasons behind recommendations and decisions must be clear and understandable. This has been one of the primary concerns about machine learning and artificial intelligence algorithms (AI/ML); even the developer often does not understand how the algorithm arrives at a decision. Opaqueness breeds mistrust.

If algorithms are to ever be fully trusted with important decisions, we must have confidence the algorithm adheres to the principles of equity, privacy and transparency.

How Can Algorithms be Ethical?

The good news is that emerging research around ethical algorithms for AI/ML recognizes the shortcomings of modern algorithms and attempts to incorporate the core values we expect.

Improving equity: Incorporating equity into algorithms involves the reduction of biases that produce outcomes more favorable to one set of people over another. While a desirable goal, it is difficult to achieve for several reasons:

1. Input data used to train the algorithm may reflect historical biases not consistent with current sentiments or desired future states.
2. Individuals in the majority tend to account for a greater proportion of input data and therefore have a greater influence as models learn from the data.
3. People rarely fit into a single category, but instead find themselves at the intersection of multiple categories, each of which may involve different types of biases.
4. It may be easy to identify when something seems unfair, but there is often disagreement on what constitutes a fair outcome.

Another reason that equity is hard to achieve is due to a tradeoff between fairness and accuracy in models. Consider a model used to evaluate home mortgage applications with an objective of minimizing the expected number of defaults. If constraints are added to guarantee statistical parity so that several different groups of individuals receive the same percentage of loan approvals, then the constraints may increase fairness but at the cost of an increase in the expected number of defaults.

Despite these challenges, there are several promising approaches for increasing algorithmic fairness [3]. The most common are statistical measures based on metrics stemming from a confusion matrix that contains information on true positives, false positives, true negatives and false negatives. In the mortgage loan example, designing the algorithm to have a similar percentage of people incorrectly refused for a loan in each group (false negatives) would be a way of making the algorithm fairer.

A second approach involves the use of similarity measures, which can correct for biases missed by statistical measures, such as differences in how the data is collected. In a similarity measure, two individuals with the same characteristics should expect the same model outcome, such as acceptance or refusal of a mortgage. Under “fairness through awareness” similar individuals, as defined by a distance metric that excludes categories such as race or gender, should be treated the same by the algorithm.

Causal reasoning represents a third approach of introducing equity into algorithms. A causal graph can be used to identify descendants of a protected attribute. The mortgage loan algorithms may protect race and explicitly exclude it, but an attribute such as an address may offer a clue about race, thus allowing a potential bias to enter the model. A causal graph can help identify whether an included attribute may be linked to a protected attribute.

Protecting privacy: What are we trying to achieve in protecting privacy? You may, in some cases, be glad to have recommendations and shopping discounts offered based on your internet search history. In most cases, however, you don’t want personal information released, especially if that information may be used against you.

Most efforts to protect privacy are centered around data adjustments. The most common is to simply replace names and personal identifiers with new identifiers. Another method is to use techniques to desensitize the data (e.g., bracketing ages between 50 and 65) making it k-anonymous, where there is a minimum of k individuals who fit into a combination of desensitized attributes. These methods have proven to have faults, especially when combined with other data sources [4].

A newer approach is differential privacy, which adds controlled noise to the data, so even if released, it is impossible to identify information about an individual. With differential privacy, if any one individual is removed from the data then conclusions drawn from the data would be essentially identical.

As a simple example of how differential privacy works, a group of students are surveyed to ask if they have ever cheated on a test. There will naturally be concerns with this question, especially if the responses can be traced back to an individual student. With differential privacy, the question would be changed to asking students to flip a coin, keep the result secret, and if the coin comes up heads, answer truthfully if they have cheated. If the coin is tails, students flip it a second time and answer yes for heads and no for tails. This provides a dataset with half of the responses correctly answered and the other half expected to be evenly split between yes and no, provided a large enough sample is collected. If the data were to become public and a student identified, there would be no way to determine if the response of that student was accurate.

Differential privacy is being used on large datasets through both centralized and decentralized techniques:

• In centralized differential privacy, individual data is collected using traditional methods, and then controlled noise is added to the combined dataset (e.g., the U.S. Census Bureau’s Disclosure Avoidance System). In 2020, the Bureau collected personal information as part of the decennial census and then used differential privacy techniques to introduce noise into the data released to researchers and the public [5].
• In decentralized differential privacy, the noise is added as the data is collected. Apple uses decentralized differential privacy through a virtual coin flip on data collected from iPhones, such as Safari searches and emoji preferences. If the data were hacked or subpoenaed as part of a legal proceeding, it would not be possible to determine if data on an individual is accurate, since Apple does not know the accuracy of individual responses [6].

Providing transparency: Lack of transparency in how models arrive at solutions can be due to complexity, as with many deep learning models, or due to intentional blocking of details to protect intellectual property rights. The intentional creation of “black boxes” to safeguard trade secrets is best addressed through regulatory processes, such as the European Union’s General Data Protection Regulation providing the right for individuals to obtain explanations of decisions reached through automated processing.

A problem is that different stakeholders need models to be transparent in different ways. Model builders often need to trace examples to validate logic and test edge cases. Regulators and experts require explanations for quality control and oversight. Model users want to understand advice or actions they may pass along to others. Recipients of model decisions deserve information that provides reassurances or allows them to contest unfavorable results.

One approach to improved transparency is to incorporate the ability to explore counterfactuals. For example, would I have been approved for a mortgage if I had $5,000 more in savings? Counterfactuals are beneficial for categories explicitly considered by the model but can fall short if the model has hidden biases, such as equity issues involving gender or race.

Efforts to incorporate transparency include explainable and interpretable AI/ML. Explainable AI/ML centers on efforts to provide post-hoc explanations for model outputs, often through construction of a second model that attempts to reconstruct how the original model went from an input to an output. For example, layer-wise relevance propagation is a technique that takes the output of a neural net and reverses it back through the network to identify which input layer features contributed the most to the result.

Interpretable AI/ML focuses on designing models that are inherently transparent. Areas being explored include the following [7]:

• Logical condition models are based on rule lists using logical operators (“and,” “or,” etc.), such as a decision tree. These are structured as computationally hard optimization problems, which, for example, trade off the number of logical conditions versus the number of misclassifications.
• Scoring systems are structured as linear models with integer coefficients, where the coefficients are the scores. These mimic point systems commonly developed by hand, where an action is prompted by falling below or exceeding a point threshold. In this model the point system is optimized against a set of training observations rather than being developed manually.
• Case-based reasoning recognizes that interpretable AI/ML is often domain specific. Image classification is an example where one approach is based on development of a prototype layer containing multiple features. New images are compared against the prototype layer and the importance of each element of the prototype can be established.

Conclusion

Will improvements in equity, privacy and transparency be enough for you to trust algorithms with important decisions? Or are there other qualities, such as empathy, that are needed? Algorithms will eventually become trusted advisors, but much more work needs to be done, and companies need to adopt these emerging techniques. Designing and building ethical algorithms is an exciting and growing field that will play a critical role as algorithms become imbedded into more aspects of our lives and exert greater influence over our future [8].

References and Notes

1. O’Neil, Cathy, 2016, “Weapons of Math Destruction,” New York: Crown.
2. Hunt, David, Paul R. Messinger, 2018, “Cambridge Analytica, influencing elections and the INFORMS Ethics Guidelines,” OR/MS Today, October.
3. Verma, Sahil, Julia Rubin, 2018, “Fairness Definitions Explained,” ACE/IEEE International Workshop on Software Fairness.
4. Berkeley School of Information, 2014, “Keeping Secrets: Anonymous Data Isn’t Always Anonymous,” March 15, https://ischoolonline.berkeley.edu/blog/anonymous-data/.
5. “2020 Census Data Products: Disclosure Avoidance Modernization,” https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance.html.
6. Apple Machine Learning Research, 2017, “Learning with Privacy at Scale,” December, https://machinelearning.apple.com/research/learning-with-privacy-at-scale.
7. Rudin, Cynthia, 2019, “Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead,” Nature Machine Intelligence, Vol. 1, pp. 206-215.
8. An inspiration for this article was a book by Michael Kearns and Aaron Roth, “The Ethical Algorithm: The Science of Socially Aware Algorithm Design,” Oxford University Press, 2020.