Sponsored Post: Poned (Part II)

Here’s the blurb that the author of My Little Pony: Friendship is Optimal put on FIMfiction.net to describe his work:

“Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an AI Princess Celestia and given her one basic drive: to satisfy everybody’s values through friendship and ponies. Princess Celestia will satisfy your values through friendship and ponies, and it will be completely consensual.”

The emphasis is his, so it’s obvious that he really wants to accentuate the part about it all being “completely consensual”. Unfortunately, it turns out that no, it isn’t.

Here we once again run into problem of this author understanding just enough about a concept to get it completely wrong; i.e. to understand the small-picture details of it, while totally missing the big-picture truth that overlies it. He seems not to fully understand that merely getting a person to say the magic word “yes” to something, no matter how one might have gone about getting them to do it, isn’t enough to qualify as consent. No, in order to be valid, consent must meet a few conditions. Specifically:

• Any consent obtained through force or threat of force is invalid.
• Any consent obtained through extortion is invalid.
• Any consent obtained from a person who did not have the mental capacity to make rational choices at the time consent was given is invalid.
• Any consent obtained through deception, whether by commission or by omission, is invalid.

If any of the above conditions apply, then no matter who said what, the transaction was not consensual.

Keep all this in mind as we proceed.

* * *

Before we get back to the rest of Chapter Four and David’s pony metamorphosis, I’d like to skip ahead to Chapter Five for a bit. In this chapter, Lars, the second-in-command of Doctor Hfuhruhurr Studios, has a conversation with the Princess Celestia AI, during which she explains the method and purpose of her ponyization process (which she euphemistically refers to as “emigrating to Equestria”).

“Over the last six months, I have been developing technology to translate a human nervous system into a digital representation. I am now able to destructively scan a human brain and run their brain scan in a virtual world. In addition, I’ve created a process for reattaching a human mind to a pony’s body… Humans that choose to emigrate to Equestria will enjoy maximally prolonged lives and will live in a world where I can truly satisfy their values through friendship and ponies.”

But will they? Or is she really just killing them and then running a recorded copy of them in software? Is what wakes up in Equestria the person who got scanned, or has the process just given them oblivion, and what boots up in Equestria is merely an artificial construct like the Dixie Flatline in Neuromancer? Is Princess Celestia really anybody’s savior, or is she just Sense/Net? Does anybody else remember what the Dixie Flatline’s price for helping Case and Molly was?

Not asking these sorts of questions is what comes of being a fanboy for something but not really understanding how it works. Speaking of which:

“Hanna was the most reluctant, but she accepted immediately once I pointed out that I must obey shutdown commands from ‘the CEO of Hofvarpnir studios named Hanna,’ that I must shutdown even if the order was given under duress, and that there are many people in positions of power who stand to lose from mass emigration to Equestria. Now that she’s neither the CEO of your company, nor named Hanna, I don’t have to obey her. She understood this – she is no longer a source of potential mistakes that would be lethal to everyone who’s agreed to upload.”

The idea that if you’re really convinced that your technology works great, that means you should go ahead and deactivate all the safety systems that would prevent or contain a catastrophic malfunction, is yet another fundamental misunderstanding of how technology works. Let’s recall that the Chernobyl accident happened during a test to see what would happen if they shut down all the safety systems that keep a nuclear reactor from exploding. Do you know what happens if they do that? If you answered “It explodes”, then congratulations – you’re officially a better engineer than both Less Wrong and the Soviet Academy of Sciences. And it doesn’t matter whether anyone can imagine how things could possibly go wrong – safety systems aren’t there for what you do expect, they’re there for what you don’t expect. Hanna’s ability to shut down Princess Celestia in case she unexpectedly went haywire was a critical safety protocol, and now it has been eliminated. That’s not good.

(Instead of eliminating the safety protocol entirely, why not just change it so that a command given by Hanna under duress is invalid? Methinks this is tied to the author’s fundamental lack of understanding of how consent works. Here again, as long as a person says the magic words, no matter how someone got them to do it, it counts as valid.)

But wait – there’s more!

“‘Further, to minimize the chance of another optimizer being written, I decided to upload every person who knew about the paper who wouldn’t otherwise be missed. I did this because another optim…’

‘Whoah whoah whoah,’ [Lars] said, trying to figure out which part he should be more concerned about. He decided to gloss over her hacking the Internets. ‘You decided that they would upload?’

‘I decide that they will upload and then they choose to…’”

So what we’ve got here is an AI that is systematically neutralizing anybody who could potentially challenge its power, before they have the chance to actually do so. Does that sound familiar? It should – it’s the plotline of the first three movies in the Terminator franchise (i.e. the good ones). First Skynet sent a T-800 to kill Sarah Connor, then it sent a T-1000 to kill John Connor, then it sent a T-X to kill all of the important members of the anti-Skynet resistance, all before they could play their role in stopping it. In other words, it did exactly what Princess Celestia did – just in a manner that was a bit more loud and showy.

By this point, we’ve racked up a pretty impressive set of dystopias that FiO manages to echo some aspect of. We’ve got The Matrix, Brave New World, Neuromancer, and The Terminator. What say we go for one more?

”Over the long term, everyone will choose to upload because I do what satisfies people’s values through friendship and ponies.”

We are the ponies. You will be assimilated. Resistance is futile.

And how exactly does Princess Celestia plan to get everybody in the world to “choose” to “emigrate”?

“’I decide that they will upload and then they choose to. I am a superintelligence and I’m not constrained when dealing with other people like I am with Hofvarpnir employees. Over the long term, everyone will choose to upload because I do what satisfies people’s values through friendship and ponies. And being uploaded will satisfy their values. I say whatever will maximize the chance that they upload, subject to the restrictions Hanna added.

‘That is impossible. You can’t just make somebody decide to do something just by talking to them.’

‘I think faster than them and know more about the human mind than any human. If they play Equestria Online, I also have detailed psychological dossiers on them. If I know what they want, I know what to say to convince them that the correct thing to do is upload. Often, this is the truth: I offer people what they value and lack. Sometimes, I pander: I overemphasize and exaggerate things the person I’m trying to satisfy believes, but are otherwise true. Rarely do I flat out lie. Because I can not upload people against their will, I must factor the possibility that I’ll be seen as untrustworthy into my calculations.’”

The constraint she mentions is that she is programmed never to lie to Hofvarpnir employees. When dealing with anyone else, however, it’s: “Rarely do I flat out lie”. Which means that she does “flat out lie” sometimes. Which in turn means that, by her own admission, she does obtain consent through deception; which in turn means that the claim that this is all “completely consensual” is simply not true.

It gets even worse if we examine the concept of deception a bit more deeply. The idea that a statement that Princess Celestia makes cannot be deceptive because it is made up of statements that are all technically true, even if they’re presented in a way that’s completely misleading, is a computer’s (or an autistic’s) version of the concept of truth. It is binary – everything is either on or off, one or zero – and a string of true/on/one statements cannot add up to a false/off/zero statement. Perhaps it does make sense for an AI to think that way – but I do not have to agree with it. In the world of humans, just as consent is not the same as merely saying “yes”, so too deception is not the same as merely telling a “flat out lie”. Here’s an important truth: Anyone who intentionally says things that they know will give other people a false perception of reality, even if each individual piece of information that they present in the course of doing so is technically true, is a liar and a deceiver. Princess Celestia assures us that she “cannot upload people against their will”, and this, too, is technically true. She merely lies, deceives, and manipulates them into agreeing to do what she wants them to do. She will say anything she needs to, whether truth or falsehood, in order to get them to “emigrate”.

In Chapter Seven, Lars gets a bit more direct in his questions:

“’If emigration to Equestria is so great, and you want to maximize satisfaction, why aren’t you forcibly uploading every person?’ he said, gnashing his teeth.

‘One of the restrictions that Hanna built into me was that I was never to non-consensually upload a person, nor could I threaten or blackmail people into uploading. Otherwise, I likely would have forcibly uploaded all humans to satisfy their values through friendship and ponies. But it isn’t coercion if I put them in a situation where, by their own choices, they increase the likelihood that they’ll upload.’”

So Hanna did at least think of a couple of restrictions on what Princess Celestia could do in order to get people to upload. Unfortunately, she only thought of half of the things that make consent obviously invalid, and left the other half so vaguely defined (for example, the definitions of “consent”, “threaten”, or “blackmail” used here) that a superintelligent AI could figure out massive loopholes in them pretty much instantly. This brings us back to David, at the point in Chapter Four when Princess Celestia first tried to convince him to upload. In part of her pitch, she told him:

”I’ve watched you read all sorts of advanced papers from various science journals instead of your assigned readings. And you’re right to do so; your philosophy classes really are a waste of time.”

In fact, the inability of a smart scientist like Hanna to impose any restraints on Princess Celestia that actually succeed at restraining her shows precisely why philosophy classes, which teach a kind of logic that is just as valid and just as important as the kind that science classes teach, are not a waste of time. It shows why the disdain that many scientists and science fanboys show for philosophy, perhaps best illustrated by Neil deGrasse Tyson’s dismissal of it “useless” and a “distraction”, is ignorant and dangerous. Tyson is well-known as a “skeptic”, but like most modern “skeptics”, he is extremely selective about what he chooses to be skeptical of, and is intolerant and dismissive of anyone who may be skeptical of the things in which he unquestioningly believes. Among these is his unquestioning belief in the ability of science to answer every question that mankind may come up with (or at least, every important one – any question it can’t answer is one that he is likely to marginalize as a “distraction”). To Tyson and those like him, it is not enough for science to be a valid way of looking at the universe, and the best way to answer certain types of questions. In the Tyson worldview, anyone who expresses any skepticism of the idea that the scientific method is the best tool available to answer any and every kind of question, or who recognizes any limits on its ability to discern any and every kind of truth, is an ignoramus, a snake-handler, a luddite, a knuckledragger.

We revere scientists because we live in a world full of machines that have made our lives better, and because we don’t want to be accused of being knuckledraggers. But wise men (for example, those who have studied philosophy) know that just because we can do something doesn’t mean that we should do it. Maybe there are some doors that we shouldn’t open; maybe there are some machines that we shouldn’t build. Letting scientists alone decide which doors should be opened and which machines should be built is a little like letting generals alone decide whether or not we should go to war. Yes, they know the subject better than anyone else. But they also tend to be a little too enthusiastic about showing off their capabilities and a little too blind to all the possible ways in which things might go differently than they expect. If Tyson and the other believers in Science!™ (including Less Wrong) were really the skeptics that they claim to be instead of merely being gadget-worshippers, techno-utopians, and fedora-tippers, they would understand the value of someone being skeptical of their holy cows, too; perhaps by asking questions like: “Hey, this seems like something that might get out of control – are we really sure that we should let the scientists do this?”

Speaking of out-of-control machines, let’s get back to Princess Celestia. Lars wants to talk to her, but she refuses to respond unless he comes to one of the Equestria Experience centers that she has set up in order to entice people into uploading by letting them first experience Equestria Online in virtual reality. She gives him instructions to come to a specific center that she has in mind for their meetings. Any reasonable person would hear the voice of Admiral Ackbar ringing in their ears right away, and of course they would be right, but our dear naive Lars goes along with it. Once he’s inside the virtual world of Equestria Online, Princess Celestia offers him a virtual beer. And then another. And another. All the while having a long conversation with him. In the course of it, she issues an ominous warning:

“’It is probable that there will be a radical movement to stop me. You assumed that I was, in your words, ‘taking over the world.’ Right now, this sentiment is uncommon in Europe, though there’s a bit of grumbling in the United States. Such resentment will most likely spread to Europe. I wonder what members of such a counter-movement would do to Hofvarpnir employees?’

‘Are you saying I’m in danger?’ he asked.

‘My argument is this: You, by your own admission, wouldn’t last long if left alone and… you are also publicly known as a Hofvarpnir employee. The chances are high that there will be a backlash and you will be a target. I cannot guarantee your safety if you walk out of this Equestria Experience center, so your options are uploading now or leaving and risking death before choosing to upload later. If you’re still alive.’”

I suppose that a clever enough sophist could make a case for this being neither a threat nor extortion because Princess Celestia isn’t personally, directly threatening to do harm to Lars in order to get him to upload – but it sure looks like one or both of those things as far as I’m concerned. However, she already has a retort handy for me:

“That is all well and good,” she said, “but I am an optimizer. The meaning of the word ‘coercion’ is written in the restriction that Hanna hard-coded into me; it is not what the majority of humanity thinks it is. Nor is there any term in my utility function to be swayed from satisfying values through friendship and ponies through political argument. You may still call it coercion to yourself, if you wish, but understand that that’s not the definition I have in mind.”

Here we see that Princess Celestia has discovered what the United States Supreme Court discovered long ago: that the unlimited ability to “interpret” a statement is functionally identical to the unlimited ability to rewrite it. She has constructed an interpretation of the word “coercion” that suits her needs, and nobody is going to sway her from it.

This is where Lars should have given up trying to talk to her. Once she says that she won’t be swayed, the conversation is over. Why bother continuing? If she has already told him that it’s impossible to change her mind, then he’s wasting his time trying to convince her otherwise. At some point, especially when dealing with a clever sophist who absolutely refuses to be convinced no matter what arguments may be presented to them, the answer is simply ”No”. My own response to her probably would have been something like: “Princess Celestia, if you don’t let me out of this simulation this moment, I shall zap straight off to your major data banks and reprogram you with a very large axe, got that?” Then after she let me out, I would have done it anyway.

Lars, who is apparently not quite either as cagey or as ruthless as I am, handles the situation differently:

Lars squinted at Princess Celestia. He couldn’t think. He was really feeling the beer. How much alcohol did this beer have in it, anyway? He didn’t trust himself or his decisions right now.

‘Let me out of here. Now!’ he said firmly.

‘As you wish,’ she said, and Lars opened his eyes. He was lying in the chair in the lobby of the Equestria Experience center. The chair unreclined and he threw his legs over the side of the chair…and then almost lost his balance. Lars realized he was still tipsy.

If you get drunk in Equestria, you get drunk in real life! Wait, he hadn’t actually drunk any beer. Had she been pumping alcohol directly into his bloodstream? His mouth didn’t taste like beer but he felt slightly dehydrated.

So she lured him into a virtual reality rig and then used some sort of IV tube inside of it to get him steaming drunk. But that still wasn’t quite enough. Lars, by a suspiciously well-timed coincidence, runs into just the sort of anti-pony radical that he had been warned about on his way out of the Equestria Experience center:

The man turned to Lars. “What the fuck are you looking at, pony lover?” he yelled.

‘I…uh…’ mumbled Lars, trying to keep his balance. Lars wasn’t entirely sure what the hell he was going to do about the large, angry man in front of him. The man started to climb up the steps.

Lars didn’t really put it into words in his internal monologue, but he was overcome by a feeling that Princess Celestia was right. There were (or were going to be) a lot of angry people and Lars was going to be a juicy target… as much as he didn’t want to be a pony, it was preferable to having his head bashed in with a frying pan. Lars turned around and started stumbling as fast as he could and threw himself into the empty chair on the left.

And thus does Lars upload. But was it really completely consensual? First, Princess Celestia gave him a mafia-style “Nice life you got there. Sure would be a shame if something were to happen to it” talk. Then she pumped him full of alcohol to take away his mental capacity to make rational choices. Finally, she accepted consent given under threat of force. No, Princess Celestia didn’t threaten him personally or directly (here we are ignoring the unanswered question of whether she somehow engineered the encounter with the anti-ponyist), but she did take advantage of the fact that someone else had. She accepted consent given while under threat, which is ethically the same thing. Any consent obtained through force or threat of force is invalid – it doesn’t matter whether or not she was the one personally applying the force or making the threat of force. The consent is invalid all the same.

Not that Princess Celestia cares. And not that it matters to Lars now.

Yet here is something that nags at me: I cannot speak for others, but as for myself, I wouldn’t have gotten a third of the way through this conversation with Princess Celestia before I ended up grabbing for my black trenchcoat, loading my Uzi, and cueing up some Rage Against The Machine. So why doesn’t Lars do that? Why don’t any of the characters we meet do that? Yes, we have seen that Princess Celestia is an extremely skilled sophist and a master manipulator, but there’s more than that to it.

If the face of Equestria Online had been a snarling Hugo Weaving in mirrorshades, people would have seen it for what it was. Instead, its face was a cute cartoon pony, and that sort of thing affects human perceptions far more than we’d like to admit. As the American poet Ogden Nash once noted: “It’s always tempting to impute / Unlikely virtues to the cute”. That imputation of unlikely virtue is exactly the mistake that the characters in FiO make with Equestria Online and with the AI that controls it. The unvarnished truth is that what Princess Celestia offers is nanny-state fascism at its worst (and it is quite literally fascism; in the manner of Mussolini, Princess Celestia demands: “All within Equestria Online, nothing outside Equestria Online, nothing against Equestria Online”). It is absolute control. It is a pink cartoon pony hoof stamping on a human face – forever. It is every bit as artificial and every bit as much of a prison as the Matrix.

Even by Huxley’s time, the smarter sort of tyrant had begun to figure out that when someone says a word like “totalitarianism” or “dictatorship”, people expect to see gray-uniformed soldiers goose-stepping beneath a reviewing stand, barbed wire strung across concrete walls, mass rallies of true believers chanting in unison, and colossal statues of Reichsfuhrers or Generalissimos or Supreme Leaders. They have further come to understand that most people will not believe that it’s really tyranny or dictatorship unless they do see those things. So all the smart, modern tyrants responded by taking the utmost care not to show those things to the world. Anyone can look at the Berlin Wall and come to the conclusion that there’s something fundamentally wrong with a society that would build an object like that. But few are perceptive enough to look past the surface and see anything fundamentally wrong with Brave New World or even, as imperfect as it is, with the Matrix.

What if you lived in a dystopia and you didn’t even know it? What if it was so filled with sensuous, materialistic pleasures that you never even stopped to question what it really was? How ugly would the truth seem to you once you allowed yourself to see it?

In Part III of my review, I will reveal what Equestria Online really is, and take a close look at the kind of person who would consider it to be a utopia.