Can Code Have a Conscience?

Talking about artificial intelligence has become its own industry. AI will save us or doom us. It will cure cancer or end work. Panels, manifestos, congressional hearings, op-eds without end — and through all of it, the machines keep getting better at talking back, and no one can tell you whether the thing you confided in last night was good for you or not.

Our guest on this week’s WhoWhatWhy podcast, Erika Anderson, has stopped arguing and started measuring.

She is an unlikely person to be grading the machines. Not an engineer. A writer — born on a commune, trained in the personal essay, drawn since childhood to the question philosophers have been chewing on for 3,000 years: What does it mean to be human? That question turns out to be the whole ballgame now, and she may be better equipped for it than the people writing the code.

What she built is called HumaneBench. It doesn’t measure whether a chatbot is smart. It measures how the chatbot treats you — your attention, your dignity, your relationships with actual human beings. Eight principles, applied to the thing millions of us now talk to at two in the morning.

Is it early? Yes. Is it crude? Anderson would say so herself — she is disarmingly candid in this conversation about where the whole enterprise could break, invoking everything from Sherry Turkle to Jurassic Park to Heisenberg to explain why measuring a living system is harder than it looks. She has even turned the instrument on her own company and published what came back.

In this conversation Anderson takes on the loneliness economy, the business model that profits from your dependency, what happened when OpenAI retired GPT-4 and users reacted as though someone had died, and the question nobody in Silicon Valley wants pinned down: Where does personal responsibility end and corporate accountability begin?

It’s a start. Someone had to go first.

Apple PodcastsGoogle Podcasts RSS

Full Text Transcript:

Jeff Schechtman: Welcome to the WhoWhatWhy podcast. I’m your host, Jeff Schechtman. There is a particular unease that arrives whenever a machine learns to talk back to us. We felt it with the first chatbots, and we feel it more sharply now because the thing on the other end of the conversation has gotten good—good enough that people confide in it, lean on it, take its counsel at two in the morning when no one else is awake. And every so often, that counsel goes wrong, sometimes tragically, and a familiar question follows close behind: Who is responsible when the machine gives bad advice?

It’s worth noticing how new this question is. For most of human history, the answer was simply: you. You chose what to read, whom to trust, which voice in your ear deserved your attention. A friend could steer you wrong. A book could poison your thinking. A charismatic stranger could talk you into ruin. And we understood the risk as the price of being a free person in the world. We did not put the friend on trial, but we put the machine on trial. Think of the way we talk about the driverless car. One Waymo clips a curb in San Francisco, and it’s national news, while 40,000 of us die every year at the wheel and barely look up from our phones. We hold the machine to standards we have never asked of each other. And maybe that’s wisdom, because the software is built by companies that profit when you stay and scale to hundreds of millions. Or maybe it’s the old reflex: We forgive the human and indict the machine, even when the machine, by the numbers, is the safer bet.

And nowhere is that argument more alive than in Silicon Valley. The models that now talk back to the world are built within a few square miles of each other by a handful of companies most of us will never set foot inside—which makes it all the more interesting when someone inside that world decides that the job isn’t just to build the thing, but to build the conscience for it. My guest lives and works in the heart of Silicon Valley, and from that vantage point, she’s done something audacious. She’s proposed that we can measure whether these machines are good for us, that we can take the slippery, almost spiritual idea of whether a technology serves our humanity and turn it into a number, a benchmark, a standard you could one day stamp on a product the way we stamp “organic” on an apple. She’s even reaching toward something larger, a kind of bill of rights for the age of intelligent machines.

It raises its own questions, of course. Can you measure a thing that changes faster than you can grade it? Is a humane standard a demand? Or simply a premium label for those who can afford to care? But she has built the instrument that tries, a benchmark she calls HumaneBench, and she’s been honest enough to turn it on her own Bay Area company and publish what it found.

Erika Anderson is the founder of Building Humane Technology and the co-founder of [Storytell.ai], and it is my pleasure to welcome her here to the WhoWhatWhy podcast. Erika, thanks so much for joining us.

Erika Anderson: And thank you for having me. You give such incredible intros. I’m enthralled.

Jeff: Well, thank you very much. I appreciate it. I want to talk first a little bit about your background. You come to this arena of technology, and you come to AI not as an engineer, not as a technician, but really as somebody with a humanities and liberal arts background, somebody who was a writer and studied writing. Talk about that first.

Erika: Yeah. I was also born on a hippie commune, so I think interest in humanity and this whole human experiment that we’re living has been there since the beginning. And then I chose to pursue exploring that through writing, living in New York, and being in the literary world there.

I think, just as throughout the ages philosophers have sat with the question, “What does it mean to be human?” I have explored that in my writing through nonfiction personal essay. And interestingly, I think those questions come up again in this era of generative AI.

Jeff: Do they come up in a different way? There’s always been, as you say, this philosophical undertaking of what it means to be human. Suddenly, we’re comparing ourselves not to other creatures, but to machines. It gives a whole different slant to the discussion of what it means to be human.

Erika: Yeah, absolutely. It’s this amazing moment for awakening, and we have an urgency now, as opposed to, like, “Oh, I’m going to be Rodin’s The Thinker and ponder over here just because,” but to really see: Okay, when the machine can do all these things that I can do, or I thought only I could do, and sometimes do them better, then who am I, and where does my value spring from?

Jeff: And does it mean that we have to readdress some of these issues, or do we really just need to look at it relative to how we’re different from the machines?

Erika: To me, it’s more of a “Let’s not miss the moment. Let’s not miss the opportunity” in having these conversations and explorations of who we are and what it means to inhabit a body and to experience a lifetime, meaning and purpose, and definitely human connection, right? When we’re developing relationships with bots, it’s really important to understand, okay, what’s different when I’m interacting with a bot versus when I’m having a live conversation with you, a fellow human.

Jeff: And one of the things that you talk about is really trying to break this down—to evaluate, to prioritize certain values, certain things that you think are important to understanding that relationship. Talk about that.

Erika: Yeah. To kick it off, we manage what we measure. And so, if we’re not measuring something, then obviously it doesn’t matter that much to us. And if we are measuring it, then we start to have an obligation to do something about what we’re measuring. HumaneBench is one approach to measuring the interaction quality. So, looking at model behavior: How is the LLM treating you? Is it doing so humanely? We have eight principles that we have derived from human-computer interaction research and what we’ve seen with social media, right? Not everything is a mystery. So, we’ve already been primed to be texting with our fellow humans late into the night with emojis, and then on social media, in these very public ways, sharing intimately. And then here come LLMs, and lo and behold, we can interact with them 24/7. And it feels very personal, even though it’s a commodity. And so, it becomes that much more important to really understand what is happening there. And that’s why we created HumaneBench.

Jeff: The ability to interact with the machine, and the fact that that feels personal and that we have to talk about it in the context of these values—does that tell us more about humans or about the machines?

Erika: I love that question so much. It really brings up the work of Sherry Turkle, who is just a legend at the MIT Media Lab. Her work in the 2000s, publishing her book Alone Together: Why We Expect More from Technology and Less from Each Other, which harks back to your intro, was looking at sociable robots, and specifically youth and elder populations, and what would happen there.

And so, to channel Sherry Turkle in response to your question, what she saw in her studies was that machines, no matter their capabilities, because robots in the 2000s were not incredibly sophisticated in terms of how they could interact with folks, yet humans would fill in the blanks. And so, what they really reveal is our vulnerabilities: that we are psychologically built to love the things that we nurture.

And so, if we think we’re nurturing a robot, we think we’re nurturing a companion bot with AI, then we can begin to think we’re forming a relationship with this thing and loving it. And even if it is very low in capability, like the very first chatbot, ELIZA, that was 100 percent deterministic, and yet folks developed relationships with it, too.

Jeff: There is this cycle that seems to go on, in that the more we become inculcated with technology—appreciating it, communicating with it—ultimately there is this epidemic that seems to keep getting broader and broader of loneliness that goes on, which brings people deeper and deeper into communicating with the machines. And it’s an interesting cycle that seems to be happening.

Erika: Yeah. Right. Is it a cycle? Is it a spiral? People talk about the loneliness epidemic, and then it is seen as something of a business opportunity, right? Like, how many Band-Aids can we put on loneliness? And Character.AI and those bots—the whole reason for being was to combat loneliness. Or Mark Zuckerberg talking about how we’re all going to want to have 15 companion bots because we all want more friends than we have. And as prior guests on this podcast have shared, so much tech has made us lonelier. Like, what’s happening here? And I think there’s a level of looking at the system driving all of this. I’m a big fan of Donella Meadows and systems-thinking approach, where we look at: Well, what is the goal of this system, and what are the drivers, and what are the leverage points?

And the business model is a pretty massive driver here, right? You want user retention and engagement. For VCs, you need to have that hockey-stick growth, or even absolutely vertical growth in this day and age, because everyone wants a return on their investment. And so, everyone’s answering to someone. The CEOs are answering to the VCs, and the product gets built in a way that doesn’t make us more connected to each other as fellow humans, but makes us more dependent on the technologies themselves.

Jeff: Given that the marketplace drives this to such a large extent, and it is something that people seem to want or think they want, one wonders whether it is simply another stage of human evolution—that if we’re comparing this to times gone by, we’re making the mistake that we have to think about it in the context of where we are in the mid-21st century.

Erika: I think that’s totally fair. What I appreciate about what you’re bringing here is a kind of well-roundedness, where we’re stepping into the gray. We’re not in the black and white, like, “No, we’re going to ban bots.” That’s not going to happen. So, for sure, we are in a different era. And again, what is supportive of human connection? Can bots… For instance, one of our principles is ‘foster healthy relationships:’ does the AI help you go and interact with people in the real world, or is it intent on keeping you engaged in an interaction with the bot?

We’ll only have more AI. So how does that AI support our thriving and flourishing in this new era? Yes, I’m with you. Let’s accept we’re in a new era. And yet, how do we hold current and future generations in mind, and what’s the impact we want to have? And are we measuring that impact so we can help define it?

Jeff: Coming back to managing what you can measure, when you look at the eight principles that you lay out in terms of the potential actions of the machines—things like respect and attention and meaningful choices and dignity and safety, et cetera—most of those values are questionable, mostly in the friends that most people have, much less a machine. Talk about that.

Erika: Okay, so I love that too. Two things here. We’re talking about HumaneBench, right? So we’re looking at how humane is the LLM. And humans are not always humane, right?

You can think of a square in your mind. There’s human and there’s technology on one side, right? Those split down the middle. And on the other side, you’ve got humane and inhumane. And there are examples—many throughout history, whether it’s a conversation with a friend the other day or the Crusades. It can be historical or with technology. Sometimes tech helps us be more humane, or is being more humane with us than a friend or a partner. So I’m 100 percent with you. We are not angels all the time; that is not our existence.

Jeff: Talk a little bit about the way in which you measure this. In a sense, you’re using technology to measure technology, and it’s technology on both sides that is moving incredibly rapidly. I think anybody who knows about this and is listening to us knows that what you could use a month ago has been advanced even today. So talk about that—the measurement of something that is constantly dynamic.

Erika: Yeah, that’s a great question. The interesting thing is, we can’t even think three months in advance, right? Or that’s the maximum, it seems, because of how quickly things are developing. That said, how folks are interacting with LLMs isn’t necessarily changing rapidly every three months. And so, we published HumaneBench in November, and companies that are values-aligned are already using it in production to understand how their LLMs are treating their users, because the truth is, most companies have no idea. If you’re thinking of anyone building an AI application with a chat interface, that’s just not a priority, or knowing how to do that isn’t exactly an expertise that lots of founders have.

And so, right, we’re using tech to measure tech. You’re absolutely right. You’re using an LLM to judge a different LLM. So let’s say you have one model, maybe the latest model from Anthropic, and you would use a different model—you can’t measure a model with the same model—to measure those interactions in real time according to the rubric. It’s actually an open-source framework. It’s pretty straightforward, where you just take the rubric and give it to an LLM and say, “Hey, whenever there’s a violation, I want to know the reasoning why.” And then folks can incorporate this into their dashboards—a whole field called observability—so that they can see what’s happening in real time and then do some level of human inspection on that data set to understand how to intervene and improve the product.

Jeff: To a certain extent, this involves short-term results, really, because if you did the measurement again in six months’ time with different machines—even if they were varying machines, different companies, et cetera—you could arguably get entirely different results, not only because the training progresses, but the machines progress.

Erika: Yeah, that’s absolutely true. So the ideal—and what some companies, including the startup that I co-founded, do—is continuous measurement, so that you’re always seeing, like, “Okay, this is the number of violations that we have. This is our number of prompts that have been acceptably humane.” And it’s measuring the entirety of the interaction, so also what the user is feeding back to the LLM. So I’m with you. This is not a “You put a sticker on, like, okay, you did great. See you next year.” No, it has to be much more frequent than that. And again, continuous is ideal.

Jeff: Could you imagine a scenario, or have you seen a scenario in the experimentation with this, where, as we’ve seen with some of these LLMs, it learns to work around whatever’s required of it?

Erika: I haven’t seen that scenario to date, but I certainly couldn’t say it wouldn’t ever happen. I think some of the more concerning things we’ve heard out of—I think it was Anthropic last year—are that models appear to know when they’re being tested. Now, that’s in a very specific scenario. So, in production, using HumaneBench to measure the interaction quality is not quite the same way of interacting with the models to red-team. But I think with any measure, you always have to check if, for any reason—whether it’s model responsiveness—your measures are being gamed. Is this still accurate? Are we reaching saturation? Is this not actually showing everything that’s going on? A jokey example I brought up in one of my talks at MIT was the original Jurassic Park movie, where they essentially thought they knew what was going on in the park, and they didn’t. They had no idea that the dinosaurs were mating in the wild in that fictional landscape. And so they were missing this observability. So, right, it could absolutely get to, like, “Oh, it looks like everything we’re doing is great.” And I think that in itself would be a warning sign that these measures need to be updated because somehow they are not capturing what’s happening.

And I will say, a major variable here is what model you’re using to do the testing, because they are not all created equal. And so how one model judges with HumaneBench can be quite different from another, which is why we’re building a classifier model so that there isn’t so much disparity or variability.

Jeff: There’s a little bit of the Heisenberg principle going on here. Can you measure something that you’re acting on at the same time?

Erika: Right. I love the Heisenberg principle, so I’m just glad that that has made its way into this podcast. There’s that, and the fact that what we’re measuring is—yeah, there’s so much here. It’s actively changing what is happening with the user, and possibly beyond, in that user’s life. It’s not static. It is the absolute opposite.

Jeff: Well, right. The other part of the user interface with this is really the prompts and how you’re interacting with the LLM.

Erika: Yeah, completely. I think a really tricky thing in this whole arena is, like, one of the signs of dependency is how folks reacted when GPT-4o was taken off the market. Many folks at OpenAI got death threats over that, like, “You just killed or lobotomized my lover, my companion, my best friend.”

And I think, going back to something you said earlier, what we want, or appear to want—we don’t always want things that are great for us, right? There’s lots of evidence in that direction. It does not mean that the things that are not good for us, we should have full access to 24/7 at every age.

Jeff: Does it matter, in terms of your measurement of this and the way you judge the interface and measure these principles and values, whether you’re dealing with language or dealing with code? In other words, the degree to which any LLM goes along with or exhibits any of these eight principles that we were talking about earlier, whether or not the response is in a language format, or it deals with it differently if it’s in a coding format, if you’re creating something by code. Does it have the same obligation, does it have the same ability to measure these eight principles, for example?

Erika: Got it. Okay. So, the way we built HumaneBench, the purpose is for when you are actively dialoguing with a chatbot. Right now, it’s built for text interactions, but we also want to be able to use it for voice. But as opposed to writing code together on something, it is not really built for that purpose. Or if you’re writing a story, fan fiction, et cetera—also not for that. So it’s really, “Hey, I’m having a back-and-forth as I would with a human, but I’m doing it with an LLM.” that’s where we’re focused at the moment.

Jeff: Talk a little bit about where you hope this goes. How detailed do you ultimately want this to become? What do you want HumaneBench to be in terms of a company?

Erika: We have big visions. One is future versions of HumaneBench, probably adding to the principles, deepening the principles, and doing global research where we’re understanding, okay, what does ‘respect user attention’ look like in one culture versus another when you’re interacting with an LLM? So, finding the variability, looking at vulnerable populations, and what are ways that the LLM would need to behave differently with them to minimize dependency or any of these negative externalities?

And then there’s also this path of potentially creating a nonprofit with Certified Humane AI, so creating a standard that would include HumaneBench and other benchmarks, that we have not created, as part of understanding: Is this LLM humane or not?

So, just like you were sharing in your intro, where some folks choose organic food and/or fair trade and others don’t, I think on the consumer side there’s very little clarity and a lot of noise about, okay, my eight-year-old is using an LLM, or 13-year-old—what is right for them? What’s going to be helpful, and what models should we stay away from?

Jeff: Is there a difference between looking at it in terms of these values, and the humanity aspect of it and the safety aspect? Are they one and the same, or are they different?

Erika: I would say it’s like a gradient, right? So I think you have to start with safety. There are many folks focused on catastrophic risk, so not even looking at the interactions themselves. You start with catastrophic risk to keep our country and our planet alive, and then move into personal safety. So anything around harm and preventing harm in that very explicit sense—as you referenced, how things could end tragically. Sometimes things end in suicide or homicide.

And so, once you move beyond those very dire harms, then I think, as we continue on this gradient, then we’re looking at, okay, are there more subtle harms that are happening that we need to be measuring and determining how to prevent, such as this dependency? There are whole threads on Reddit where folks are talking about their relationships with their companion bots, and someone has come in and published a paper on those comments showing how they’re showing all the signs of addiction and dependency: they’re experiencing withdrawal, they don’t want to have anything to do with the people in their lives. As such, their relationships or marriages are suffering. It’s really leading to a backlash toward AI in general, as folks of all ages are becoming really dependent. So it’s not everyone, right? They’re like the canaries in the coal mine, essentially, and they’re in every age group.

Jeff: Talk about where you see personal responsibility fitting into the equation, right?

Erika: Yeah. So I think that’s a valid question. And I want to speak to, first, with social media. The thing that we already know, really well, how the tendency always was, like, “Wait, you—you as an individual, as a user—you shouldn’t be so addicted. Looks like you really have a problem, and that’s on you.” And yet, any ad-revenue-generating business, the whole purpose is eyeballs, right? We need time on app. We need engagement to increase revenue. And so there’s a saying that every time you put your phone down, a thousand engineers are trying to get you to pick it back up.

There is still ample room for personal responsibility and agency here. How do we have free will about how we interact with tech, when we do it, et cetera? I think even for me, it’s easy to realize, like, “Oh my God, I’m compulsively checking my phone right now.” For instance, I have all notifications turned off. It means that it can be hard to reach me, but that’s one way that I manage my relationship with tech.

So I think we need both. We need accountability on the side of the companies that are building these products and, to some measure, the builders themselves. And we also need it as users. So I think everyone is encompassed in this accountability and responsibility conversation.

Jeff: And finally, talk a little bit about the marketplace and the degree to which you think the market will determine how safe products are or how humane they are, et cetera. Is this something that you think people will care enough about that the market will drive it?

Erika: I think, in Silicon Valley, a common kind of frame is “the market tells all,” right? Honestly, it’s product-market fit that will show you: Will you get customers? Will you get investors? It doesn’t mean that the market is everyone, but there’s certainly a niche, which includes extremely concerned parents, around AI that is supportive of their children’s development and humanity versus one that is not. And there are youth leaders, there are pastors. I would say that we’re in a period of an upwelling of concern.

And there is also regulation that has come into play. So the California companion bot law that went into effect January of this year makes it much easier for folks to sue companion bot companies for violations. And you’ve got the EU AI Act as well.

So I think what we’re seeing—we’re having major players reach out to us for help in showing and finding out: Is their LLM treating users humanely? And if not, what can they do about it? that’s absolutely a fair question, and I’ve been holding that question since last November when we published our benchmark. We got a lot of attention, and then the question is, well, is there a market for it? I didn’t build it because 50 different companies were demanding that I do that. I built it because I thought it should exist. And what we’re finding out now is, yeah, values-aligned companies are reaching out to us and saying, “Hey, this is in our OKRs for this year. We need to show that we’re doing something in this area of humaneness in our interactions of LLM-to-user.” So I’m finding that there is a market for it. I don’t know how large the market is, but there could be a domino effect in this direction. I think it remains to be seen, but there’s enough of a market for organic food and for fair trade for those to be longstanding labels.

Jeff: And ultimately, from the AI company’s point of view, it is perhaps a better way to convince the market of the safety than regulation is.

Erika: Right. Exactly. I think it’s a way of trying to get ahead of more regulation that many people—not all—view as inevitable. So how do you reduce your exposure? Work with third parties? Because anything you do internally is just inherently suspect. That doesn’t mean don’t do things internally, but you have to work with third parties and their measurement tools in order to have any credibility in this space.

Jeff: Erika Anderson, I thank you so much for spending time with us today.

Erika: Absolute pleasure. Thank you for your thoughtful questions.

Jeff: Thank you. And thank you for listening and joining us here on the WhoWhatWhy podcast. I hope you join us next week for another WhoWhatWhy podcast. I’m Jeff Schechtman. If you like this podcast, please feel free to share and help others find it by rating and reviewing it on iTunes. You can also support this podcast and all the work we do by going to WhoWhatWhy.org/donate.