Designing Ethical Technology Requires Systems for Anticipation and Resilience

Concerns about the responsible use of technology are growing as use cases and applications become more sophisticated and ubiquitous across organizations and society. Within the context of AI, recruiting technology company HireVue began experimenting with facial recognition algorithms to filter job applicants based on their facial movements and tone of voice in 2019. This year, the company dropped the technology amid concerns of bias and discrimination against applicants, but organizations will continue to pursue emerging technologies that can help them address persistent managerial questions about hiring, performance, and resource allocation. The scale and speed at which such technologies offer answers to those questions thrust organizational leaders into new territory that has the potential for both higher rewards and higher risks.

More efficient and cheaper computer processing and storage, distributed networks, and cloud-based computing allow for faster, more complex calculations, reducing the amount of time and human intervention needed for a company to evaluate a worker, a credit score, or even a medical diagnosis. In addition to speed, AI impacts the scale of outcomes by standardizing the decisions and actions of multiple individuals across an organization, potentially scaling both consistency and bias. AI also affects second- or third-order stakeholders. For example, the use of cheating-detection software affects not only the company’s direct customers (the school system purchasing the service) but also students and their families who are subject to the AI system’s decisions.

Together, the increased speed and scale of emerging technologies can make ethical lapses more likely, more costly, and harder to recover from. To reduce ethical lapses, organizations need two kinds of systems: systems for anticipation and systems for resilience. In this article, we will examine frameworks for introducing these systems in organizations. We will draw on AI and data-driven examples from the real world, but these principles can be applied to a variety of emerging technologies.

Creating Systems for Anticipation

Anticipating unintended consequences and outcomes requires that leaders take the time to learn about and imagine likely sources of error before the technology is deployed. When ethical lapses are prevented or remedied in the design phase, organizations can avoid reputational and financial damages. Effective anticipation requires managers to do the following:

Define success in broader terms.