Sven Peterson (sven.peterson@premera.com) is Vice President of Compliance, Ethics and Regulatory Services | Corporate Compliance and Ethics Officer at Premera. Rose Riojas (rosa.riojas@premera.com) is Ethics Program Manager at Premera. The views expressed herein are theirs alone and are not intended to represent those of Premera.
Marie’s phone buzzed as she was getting ready for work. It was a text message from a number she didn’t recognize. Figuring it was a work colleague, she opened the text. To her surprise, it was a message inviting her to join group therapy sessions at a new studio located only a few blocks away. The message explained that based upon her recent healthcare experiences and purchases, Marie would likely benefit from addressing unresolved mental health issues before they became pressing. Furthermore, the new studio was covered by Marie’s health insurance.
Marie was stunned. She wondered who was behind the outreach to her and how they had obtained her information. What did they know about her medical history? Wasn’t that supposed to be protected information? Did they know her purchasing habits? The whole situation was creepy. She decided to reach out to her state’s attorney general’s office. She also posted on social media and reached out to her local television station.
While this example is fictional, it is inspired by true events, and it raises numerous ethical, reputational, and legal/regulatory issues. As a starting point, Marie is of course correct that there are many protections in place regarding personal data and privacy, including the Health Insurance Portability and Accountability Act (HIPAA) regarding certain healthcare data, the Gramm-Leach-Bliley Act regarding certain financial data, and other federal and state laws and case law.
But the law in this area is changing and is often unclear, with policy-makers racing to catch up with technological developments. In the meantime, companies need to consider what the right thing to do is from an ethical perspective, and how their actions will be considered by stakeholders such as customers, citizens, the press, legislators, regulators, and their own employees—all of whom may have differing and sometimes conflicting views. The absence of clear legal standards and widely accepted social norms makes it much more difficult for companies to navigate this terrain.
Ethical issues to consider
In our story, Marie’s first reaction centered around the perceived invasion of privacy. She wondered who was really contacting her and how they had obtained her data. Legally, there are different rules governing use and sharing of data by a healthcare provider or a health plan, or by a vendor who may be seeking to enter the market. However, even if those rules are followed to the letter of the current law, they may leave consumers and the public unsatisfied and concerned, leading to reputational harm for companies as well as possible investigations and legal action, which will be expensive and harmful to reputation even if ultimately successfully defended. Finally, these types of narratives often drive policymaking that sets future standards.
In the current case, artificial intelligence (AI) and machine learning (ML) were likely used to combine various data sets in order to determine that Marie would likely benefit from the program and, perhaps, had the means to pay. These data sets could have involved commercially available information regarding Marie’s purchasing habits, in addition to some healthcare data (though how healthcare data could be used is legally restricted depending upon the user and source). In order to better understand the issues at stake, let’s clarify how we define AI and ML. We will follow Microsoft in defining them as follows: “Artificial intelligence is the capability of a computer system to mimic human cognitive functions such as learning and problem-solving . . . Machine learning is an application of AI. It’s the process of using mathematical models of data to help a computer learn without direct instruction. This enables a computer system to continue learning and improving on its own, based on experience.”[1]
While Marie first focused on whether the offer was an invasion of her privacy, it could also be seen as an unfair distribution of services benefitting her. To see why, consider who all has been offered the service and why—and who it was not offered to. Are all similarly situated people being reached out to? While Marie may not want the service, it is certainly possible that others would want it but have not been targeted for outreach. This could be because the underlying data sources were themselves biased, which often occurs as the result of systemic bias in our society. While we frequently think of AI systems as neutral or without bias, when ML is applied to data, the underlying bias is replicated and reflected in the results. Consider Optum, which developed an algorithm to assist in identifying high-risk patients who may require additional medical care. Optum did not intend for the algorithm it used to be biased, and in fact, it explicitly excluded race and used a seemingly neutral variable: predicted future healthcare costs.[2] Unfortunately, this variable was a proxy for race due to racial differences in access to and usage of healthcare for similar health conditions. As a result, more care was directed to white patients than to Black patients with similar conditions. Optum modified the algorithm once the bias was discovered by changing it to predict future patient health conditions rather than costs, which significantly reduced biases.[3]
Even technology firms have inadvertently implemented biased algorithms. For example, in 2015, Amazon realized that a system it had developed to review résumés preferred male over female candidates.[4] In this case, the data set of applications over the last ten years were largely male, and therefore the application of the data set led to a preference for male applicants. Amazon eventually ended the project.
In addition to concerns regarding privacy, bias, and fairness in provision of services (possible discrimination), another issue to consider is accuracy. In Marie’s story, we don’t know if Marie really could benefit from the services in question, whether the underlying data is accurate, or if the way that ML may have combined data has resulted in accurate conclusions. If it hasn’t, this could create additional reputational or even legal risks for the company in question, particularly in a sensitive space.
As these examples and many others show, the use of AI and ML on potentially biased data sets can result in discriminatory outcomes. Often this occurs because a seemingly neutral variable (e.g., health spending for a particular condition) acts as a proxy for a protected variable such as race. Unfortunately, removing all such proxies can be difficult, particularly in an unequal society. This may result in outcomes that are illegal or unethical, for example, if they result in disparities in rates charged for members of different races for the same services. In a society that is structurally unfair, there can also be a trade-off between accuracy and fairness.
Ongoing developments in legal standards and expectations
Companies and industry leaders need to be aware that their business decisions will affect not only how they are perceived by the public and government but also how rules will evolve in the coming years. At both the federal and state levels, legislators and regulatory bodies are evaluating what new standards and rules are needed to govern emerging uses of data and AI. Among the efforts taking place at the federal level, one of the most important is being developed by the U.S. Department of Commerce’s National Institute of Standards and Technology.[5]
At a state level, several states have moved forward with legislation or regulation, or are considering doing so. For example, the National Association of Insurance Commissioners (NAIC) unanimously adopted AI guiding principles in August 2020 intended to offer guidance to the insurance industry.[6] As NAIC work on the topic continues, it is likely that model laws or regulations will be proposed, which could set national standards. Meanwhile, Colorado enacted legislation in 2021 governing the use of data and AI by insurance companies that not only bans direct, indirect, and proxy discrimination by insurers on the basis of race, religion, color, sex, national or ethnic origin, sexual orientation, or gender, but also requires insurers to document their predictive modeling programs and the steps taken to mitigate the risk of discrimination from using these models.[7] And, in response to concerns over possible uses or misuses in hiring, California is considering proposed regulatory changes that would address employers’ and third parties’ use of AI in employment practices.[8]
These are just a few of the efforts occurring across the country to adapt legal and regulatory regimes to the technological developments in AI and ML. As these efforts proceed, and as courts increasingly adjudicate disputes in this area, the contours of new requirements and expectations will become clear. Unfortunately for many companies, such clarity will come too late, as they will have suffered significant to catastrophic reputational and business losses, lost key court cases and suffered damages, and/or seen their business models prohibited. In the face of such a volatile, ever-changing and highly ambiguous world, what is a company to do?
Best practices
The good news is that there are concrete steps that a company can take right now to mitigate risk by intentionally and transparently considering the ethical implications of their uses of AI and ML. We suggest that companies consider adopting the following practices:
-
Establish a cross-functional governance structure for AI and data use. Forming a cross-functional oversight/governance committee over the use of data and AI can help ensure that informed and carefully considered decisions are made, consistent with your company’s values and applicable legal and regulatory requirements. It is advisable to include representatives from across your company, including sales, operations, legal, and, if possible, someone with expertise in ethics. Finding a way to bring in external perspectives from outside the company is also advisable.
-
Establish clear policies and principles. Principles might include transparency, accountability, fairness, privacy, security, safety, reliability, and human control in the use of data and AI (including ML). These will make clear to customers, employees, and the public where your company stands so that they are not left confused and suspicious as Marie was in our story. Policies need to translate these principles into concrete and measurable action.
-
Build diverse teams (in every sense). None of us can truly walk in the shoes of another. Teams that are diverse in multiple ways—race, ethnicity, gender, gender expression, sexual orientation, religion, skill sets and training, education, and career background—will be more able to anticipate possible problems and solutions.
-
Monitor and document attempts to reduce algorithmic unfairness. You and your company will make mistakes and will learn from them. By monitoring and documenting efforts, that learning can become more intentional and effective. Such documentation can also prove useful if a major problem occurs and you need to explain yourself to regulators, in court, or in the court of public opinion.
-
Generate clear, good faith justifications for models deployed. If a model is being deployed, your company needs to be able to justify it ahead of time, not after problems emerge. The act of generating clear, good faith justifications will often expose problems itself. Many of us have discovered that we don’t know something as well as we thought we did when we go to explain it to someone else. And, if problems do emerge, as they will, you will be in position to show that the choices your company made were principled and made in good faith.
Conclusion
As our opening story demonstrated, things can quickly go wrong in the use of data and AI, particularly when ML is involved and has gone beyond where humans might have. We all can benefit from putting ourselves in Marie’s shoes and asking how we would want to be treated and what we would want to know about how our data was used. The world we find ourselves in is volatile, uncertain, ever-changing, and highly ambiguous, but with the humility to listen to stakeholders and diverse voices—and the foresight to adopt best practices and governing structures—companies can thrive and earn the respect of their customers, employees, and other stakeholders.
Takeaways
-
Artificial intelligence (AI) and machine learning (ML) offer enormous promise. However, the absence of clear legal standards and widely accepted social norms pose difficult challenges for companies.
-
Consider how stakeholders such as customers, citizens, the press, legislators and regulators, and employees will react. Listen with humility.
-
Examine source data for bias. In a society that is structurally unfair, there can also be a trade-off between accuracy and fairness.
-
Understand how business decisions may interact with ongoing legislative, regulatory, and industry debates that will set the terms for future use of AI and ML.
-
Develop a clear and transparent philosophy and governance structure regarding use of AI and ML, and ensure teams are diverse.