The Algorithm Has No Conscience (And That's the Point)
Chapter 9: The Algorithm Has No Conscience (And That’s the Point)
“Ethics is knowing the difference between what you have a right to do and what is right to do.” — Potter Stewart
The Room Where It Didn’t Happen
On November 6, 2025, a group of scientists, regulators, ethicists, and industry representatives filed into a conference room convened by the FDA’s Digital Health Advisory Committee. The agenda was narrow: assess the risks and regulatory pathway for generative AI–based digital mental health devices. The kind of devices described in the previous chapter — the chatbots that listen, the companions that never sleep, the therapists that live in your pocket.
The meeting lasted hours. The transcript runs to hundreds of pages. And the conclusion, stripped of bureaucratic hedging, was this: the FDA had, at that point, authorized more than one thousand AI-enabled medical devices across cardiology, radiology, ophthalmology, pathology, and dozens of other specialties. The number authorized for mental health — for the domain that affects one in five American adults, that costs the economy over $280 billion annually, that kills more than 49,000 Americans by suicide each year — was zero.
Not one.
The machines were already in the pockets. Millions of people were already talking to AI chatbots about their depression, their anxiety, their suicidal thoughts. Woebot, Wysa, Replika, and now a new generation of large language model–powered companions were operating in a space the regulators had not yet decided how to regulate. The committee raised specific concerns: generative AI chatbots “confabulate” — they generate plausible but fabricated information with the same confident tone they use for accurate responses. They cannot reliably detect suicidal ideation. Their “personalities” drift unpredictably between model updates. A patient who forms a therapeutic bond with a chatbot on Tuesday may be talking to a subtly different entity by Friday.
The committee did not resolve these concerns. It catalogued them. It recommended further study. It adjourned.
And across the country, that night, millions of people opened their phones and told a machine how they felt. The room where regulation was supposed to happen had produced nothing. The waiting room from the last chapter — the one with no therapist in it — had filled itself with algorithms that no one had authorized.
I open with this scene not to condemn the FDA, which was doing exactly what a regulatory body should do when confronted with a novel technology it does not yet understand. I open with it because it crystallizes the central tension of this chapter: the gap between the speed at which AI enters clinical life and the speed at which our ethical and regulatory institutions can respond. The technology does not wait for the framework. It never has. The stethoscope was not approved by a regulatory body. Neither was the electrocardiogram. Neither was the electronic health record. Each entered medicine, disrupted it, and was retroactively subjected to standards that were written in response to harms that had already occurred.
AI is following the same pattern. The question is whether this time, the harms will be larger than the institutions that eventually contain them.
The Bias Autopsy
Let me tell you about a pulse oximeter.
It is a small device — a clip that fits on a fingertip, painless, ubiquitous. You have almost certainly worn one. It shines two wavelengths of light through your finger and measures the ratio of oxygenated to deoxygenated hemoglobin in your blood. The output is a single number: your oxygen saturation. If the number is above 94 percent, you are probably fine. If it drops below 88 percent, you are in danger. Between those thresholds, clinical decisions are made — supplemental oxygen, hospital admission, intubation, the cascade of interventions that separate a patient who goes home from a patient who goes to the ICU.
The device has been in use since the 1980s. It was validated on light-skinned patients. And for forty years, it has systematically overestimated oxygen levels in patients with darker skin.
This is not a subtle effect. A 2020 study in the New England Journal of Medicine found that Black patients were nearly three times as likely as white patients to have occult hypoxemia — dangerously low oxygen levels that the pulse oximeter failed to detect. The device said they were fine. They were not fine. The clinical decisions made on the basis of that number — whether to administer oxygen, whether to admit, whether to escalate care — were being made with data that was wrong, and wrong in a way that systematically disadvantaged patients with darker skin.
The pulse oximeter is not an AI system. It is a piece of hardware with a known physics problem. But I tell you about it because it illustrates something essential about bias that most discussions of AI ethics get exactly backward.
The pulse oximeter was not biased because someone designed it to be. No engineer sat at a desk and decided that Black patients’ oxygen levels mattered less. The bias entered through absence — the absence of diverse skin tones in the validation cohort, the absence of anyone in the room who thought to ask “does this work the same way on everyone?” The bias was not in the device. It was in the dataset. It was in the dataset’s autobiography — the story the data told about whose bodies were considered default, whose health was worth measuring precisely, whose suffering registered as a signal and whose was invisible as noise.
AI inherits this autobiography. It does not write a new one.
When a machine learning model is trained on electronic health records from a hospital system that has historically underserved Black patients, it learns the hospital’s history, not the patient’s biology. It learns that Black patients receive fewer referrals, fewer diagnostic tests, fewer follow-up appointments — not because their conditions are less severe, but because the system that generated the data delivered less care. The model then reproduces this pattern, predicting that Black patients need less care, because in the training data, they received less care. The bias is laundered through the algorithm, emerging on the other side with the imprimatur of mathematical objectivity.
A landmark paper from 2019 demonstrated this with devastating clarity. A widely used commercial algorithm, deployed across major U.S. health systems to identify patients for care management programs, was found to systematically underestimate the illness burden of Black patients. At a given risk score, Black patients were significantly sicker than white patients with the same score. The algorithm was not using race as an input — it was using healthcare costs as a proxy for health needs. But because Black patients, due to structural barriers, historically generated lower healthcare costs for the same conditions, the algorithm concluded they were healthier. It mistook the system’s neglect for the patient’s wellness.
This is what I mean by the dataset’s autobiography. The data tells the story of the world that produced it, and that world is not equitable. The algorithm reads the autobiography faithfully. It has no capacity to question the narrative. It has no conscience that whispers this doesn’t seem right. It does not notice that the story it is reading was written by a system that spent four centuries valuing some bodies less than others. It just optimizes.
Return to the photograph-to-movie metaphor. Bias is not a single corrupted frame. It is a corrupted frame rate. Some patients’ movies play at full speed — every data point captured, every symptom recorded, every clinical encounter documented with high fidelity. Other patients’ movies play in fragments — missing frames, dropped audio, entire scenes cut because the system never recorded them. The AI watches both movies and concludes that the fragmented one is shorter, simpler, less eventful. It does not understand that the fragments are not the patient’s story. They are the healthcare system’s confession.
The Trolley Problem Is a Lie
Whenever the words “ethics” and “AI” appear in the same sentence, someone invokes the trolley problem. A runaway trolley is headed toward five people. You can divert it to a side track where it will kill one person. Do you pull the lever?
The trolley problem is a masterpiece of philosophical pedagogy and a catastrophe of practical guidance. It has dominated popular discussions of AI ethics for a decade, and it has taught us almost nothing useful about the actual ethical challenges of AI in medicine.
Here is why.
The trolley problem assumes a single decision point — one lever, one moment, one binary choice with knowable consequences. Clinical AI makes thousands of micro-decisions across millions of patients, each decision shaped by training data the clinician has never seen, weighted by parameters no human can inspect, producing outputs that blend into the noise of routine care. There is no lever. There is no single moment of choice. There is a continuous gradient of algorithmic influence seeping into clinical workflows in ways that are often invisible to the physicians who use them.
Consider a clinical decision support tool that helps emergency physicians triage chest pain patients. The algorithm takes in vital signs, lab values, ECG findings, patient history, and outputs a risk score. High score: admit. Low score: discharge. The physician reviews the score, considers it alongside their own clinical judgment, and makes a decision.
Where is the lever? Is it the moment the algorithm was trained — when someone chose which dataset to use, which outcomes to optimize for, which patients to include and exclude? Is it the moment the algorithm was deployed — when a health system decided to integrate it into the workflow without telling patients? Is it the moment the physician glances at the score — the microsecond where the number on the screen nudges their clinical gestalt in one direction or another? Is it the moment the patient is discharged — sent home with a risk score that said they were safe, by a physician who might have admitted them if the number had been different?
The trolley problem gives us one lever and one moral agent. Clinical AI distributes moral agency across hundreds of actors — the data scientists who built the model, the executives who funded it, the regulatory bodies that did or did not evaluate it, the physicians who do or do not override it, the patients who were never told it existed. Responsibility is not allocated; it is dissolved. And when something goes wrong — when a patient who should have been admitted is sent home, when an algorithm’s blind spot becomes a patient’s catastrophe — the dissolution of responsibility means that no one is clearly accountable. The data scientist says they built the model to specification. The executive says they trusted the data science team. The regulator says the device was not within their current framework. The physician says the algorithm recommended discharge. The patient is dead, and the moral ledger is blank.
This is the real ethical challenge of AI in medicine, and it looks nothing like a trolley problem. It looks like a fog — a diffusion of accountability so thorough that the concept of blame loses traction. The ethical question is not who pulls the lever? It is: How do we build systems of accountability for tools whose influence is continuous, distributed, and partially invisible?
Abstract frameworks will not solve this. Principles printed on laminated cards and hung in hospital corridors will not solve this. What might solve it is something that looks less like philosophy and more like biology.
The Immune System
Your body has an ethical framework. It does not call it that, of course. It calls it the immune system.
The immune system does not work by building walls. It does not prevent every pathogen from entering the body — that would require sealing off the organism from the environment, which is incompatible with life. Instead, it operates through a system of internal surveillance, adaptive response, and memory. It monitors constantly. It detects threats not by referencing a fixed list of known dangers but by recognizing patterns that deviate from self. It responds proportionally — escalating from innate immune responses to adaptive ones as the threat persists. And it remembers. After an encounter with a pathogen, the immune system retains the knowledge, responding faster and more precisely to the same threat in the future.
I want to propose that AI ethics in medicine should be modeled not on legislation — not on walls and perimeters and fixed lists of prohibited actions — but on the immune system. Not because biology is a perfect analogy for governance, but because the properties that make the immune system effective are precisely the properties our current ethical frameworks lack.
Internal surveillance. The immune system does not wait for symptoms to appear. It patrols continuously, sampling, testing, checking. AI oversight should do the same. Not annual audits — continuous algorithmic monitoring. Real-time dashboards that track model performance across demographic subgroups. Automated alerts when prediction accuracy diverges between populations. Adversarial testing — red teams whose job is to find the failure modes before the patients do. The immune system’s T-cells circulate constantly, interrogating every cell they encounter. AI ethics needs its own T-cells: independent monitoring systems that do not answer to the same executives who profit from the algorithm’s deployment.
Adaptive response. The immune system does not have a single response to all threats. It escalates. A minor incursion triggers the innate immune response — fast, generic, sufficient for most challenges. A serious threat activates the adaptive immune system — slower but exquisitely targeted, producing antibodies tailored to the specific pathogen. AI ethics should escalate similarly. A minor performance discrepancy triggers automated retraining. A moderate bias finding triggers a formal audit. A serious harm triggers suspension, investigation, and mandatory public disclosure. The response must be proportional, rapid, and structured — not dependent on whether a journalist happens to discover the problem.
Memory. This is where current AI governance fails most catastrophically. The immune system learns from every encounter and never forgets. Our regulatory systems forget constantly. The same biases are “discovered” in paper after paper, generating citations and conference presentations but no lasting institutional change. The pulse oximeter bias was documented in the 1990s. It was re-documented in 2005. It was re-documented again in 2020, this time in the New England Journal of Medicine, and generated international headlines. Thirty years of discovery. The device on the shelf remained largely unchanged. An immune system that functioned this way — recognizing the same pathogen every decade and mounting a response from scratch each time — would be fatal. AI ethics needs immunological memory: a shared, mandatory, continuously updated registry of known failure modes, accessible to every developer and every regulator, where each documented harm becomes permanent knowledge that shapes future development.
Now connect these three properties to the three principles that have guided this book.
Augmentation requires that the human remains in the loop — not as a rubber stamp but as a genuine decision-maker with the authority and information to override the algorithm. The immune system model demands this: internal surveillance is meaningless if the monitoring data is visible only to the algorithm’s creators and not to the clinicians who use it. Transparency dashboards, real-time performance metrics, and clear override protocols are the clinical equivalent of making the immune system’s activity visible to the organism.
Transparency requires that the algorithm show its reasoning — not its full mathematical architecture, which would be incomprehensible, but its basis for recommendation. When a clinical AI suggests discharging a chest pain patient, the physician should be able to see: which features drove the score, how the model performs on patients demographically similar to this one, and what the model’s known limitations are in this context. The immune system’s transparency is chemical — it signals through cytokines, surface markers, inflammatory cascades that other cells can read. AI transparency must be similarly legible: not hidden inside the model but expressed at the surface, in the clinical interface, where the human can read it.
Equity requires that the immune system protect all patients equally. This is the hardest requirement and the one most likely to fail. An immune system that vigorously defends some cells while ignoring others is an autoimmune disease — the body attacking itself. An AI system that performs well for some populations while systematically failing others is the algorithmic equivalent: a tool that harms the patients it is supposed to protect. The equity requirement means that no AI system should be deployed in clinical care until its performance has been validated across the demographic spectrum of the population it will serve. Not a footnote in the Methods section. A hard prerequisite. The immune system does not go online until it can distinguish self from non-self across all tissues. Clinical AI should not go online until it can deliver equitable performance across all patients.
The Mirror
I have spent this chapter arguing that the algorithm has no conscience. Let me tell you why that is not the disaster it appears to be.
Every ethical framework in the history of medicine has been reactive. The Nuremberg Code was written after the experiments. The Belmont Report was written after Tuskegee. The Helsinki Declaration was revised after each new scandal revealed gaps in the previous version. Medical ethics does not lead. It follows. It responds to harm by codifying the prohibition of that specific harm, then waits for the next harm to reveal the next gap.
AI’s amorality breaks this cycle — not by solving the problem but by making it impossible to ignore.
When a human physician harbors an unconscious bias — ordering fewer pain medications for Black patients, referring fewer women for cardiac catheterization, spending less time with patients who do not speak English — the bias is hidden inside the physician’s clinical judgment, inseparable from the expertise, invisible to audit. The physician does not know they are biased. The patient suspects but cannot prove it. The bias persists because it has no surface. It lives in the fog of human cognition, where intention and outcome are never fully distinguishable.
When an algorithm harbors the same bias, it is auditable. The model’s predictions can be disaggregated by race, gender, age, socioeconomic status. Its error rates can be compared across populations. Its training data can be examined for the historical inequities it encodes. The bias, once hidden in the fog of human judgment, is now written down — expressed in weights and parameters that, while not individually interpretable, produce outputs that can be measured and challenged.
The algorithm is a mirror. It reflects the biases of the system that created it with a fidelity that human self-reflection has never achieved. And the mirror does not flinch. It does not rationalize. It does not tell itself that the disparity is explained by clinical factors. It simply shows you the numbers: this model predicts worse outcomes for these patients, and here is the evidence.
The algorithm has no conscience. Good. We have a conscience. And for the first time in the history of medicine, we have a tool that makes our failures of conscience visible, measurable, and fixable — if we choose to look.
This is the inversion at the heart of this chapter, the reason I changed the title from “The Ethics of Algorithmic Medicine” to “The Algorithm Has No Conscience (And That’s the Point).” The point is not that AI lacks moral reasoning. The point is that AI’s lack of moral reasoning forces us to make ours explicit. You cannot deploy an algorithm without defining its objective function — and the objective function is a moral statement, whether or not anyone recognizes it as such. Optimize for cost reduction? That is a moral choice. Optimize for diagnostic accuracy averaged across all patients? That is a moral choice — one that hides disparities inside the average. Optimize for equitable performance across populations? That is a moral choice too — one that may require sacrificing some overall accuracy to ensure that no subgroup falls below an acceptable threshold.
The algorithm demands that we specify what we value. It demands that we define “good outcome” mathematically. And in that demand — in the cold, amoral insistence on a number where before there was only a vague aspiration — lies a strange and unexpected gift. For the first time, medicine’s values are not assumptions. They are parameters. And parameters can be examined, debated, and changed.
The photograph-to-movie metaphor has one more turn. Throughout this book, the movie has been the patient’s story — their data evolving over time, revealing patterns invisible in any single frame. But there is another movie playing alongside it, one we have mostly ignored. It is the movie of the system — the healthcare institution, the algorithm, the dataset, the regulatory framework — and its frames reveal a story of who was included and who was left out, whose data was collected and whose was not, whose suffering was optimized for and whose was noise.
The algorithm shows us both movies simultaneously. The patient’s movie and the system’s movie, playing in the same theater, projected on the same screen. The patient’s movie tells us what is happening to them. The system’s movie tells us what we are doing to them. And the conscience — the thing the algorithm lacks and we possess — is the capacity to watch both movies at once and decide that the second one must change.
Next: Interlude — The Physician’s Cut
This book is free and open. Support thoughtful AI in medicine.