front.tex

\mygraphics{images/img2.jpg}

\begin{center}
Eliezer Yudkowsky is a Research Fellow at the Machine
Intelligence Research Institute.
\end{center}


\begin{center}
 Written by Eliezer Yudkowsky
\end{center}


\begin{center}
 Published in 2015\\
 Machine Intelligence Research Institute\\
 Berkeley 94704\\
 United States of America\\
 intelligence.org\\
\end{center}

\begin{center}
  Released under the\\
  Creative Commons
Attribution-NonCommercial-ShareAlike 3.0 Unported license.\\
 CC BY-NC-SA 3.0
\end{center}

\begin{center}
  \input{version.tex}
\end{center}

\begin{center}
The original MIRI version is available from:\\
\url{https://intelligence.org/rationality-ai-zombies/}\\
\ \\
The changes since then are listed on:\\
\url{https://github.com/jrincayc/rationality-ai-zombies}
\end{center}

\begin{center}
 The Machine Intelligence Research Institute gratefully
acknowledges the generous support of all those involved in the
publication of this book, and the donors who support
MIRI's work in general.
\end{center}

\begin{center}
 Cover by Tim Woolley.
\end{center}


\tableofcontents


\mysectionnn{Preface}


 You hold in your hands a compilation of two years of daily blog
posts. In retrospect, I look back on that project and see a large
number of things I did completely wrong. I'm fine with
that. Looking back and \textit{not} seeing a huge number of things I
did wrong would mean that neither my writing nor my understanding had
improved since 2009. \textit{Oops} is the sound we make when we improve
our beliefs and strategies; so to look back at a time and not see
anything you did wrong means that you haven't learned
anything or changed your mind since then.


 It was a mistake that I didn't write my two years
of blog posts with the intention of helping people do better in their
everyday lives. I wrote it with the intention of helping people solve
big, difficult, important problems, and I chose impressive-sounding,
abstract problems as my examples.


 In retrospect, this was the second-largest mistake in my approach.
It ties in to the \textit{first}{}-largest mistake in my writing, which
was that I didn't realize that the big problem in
learning this valuable way of thinking was figuring out how to practice
it, not knowing the theory. I didn't realize that part
was the priority; and regarding this I can only say
``Oops'' and
``Duh.''


 Yes, sometimes those big issues really are big and really are
important; but that doesn't change the basic truth that
to master skills you need to practice them and it's
harder to practice on things that are further away. (Today the Center
for Applied Rationality is working on repairing this huge mistake of
mine in a more systematic fashion.)


 A third huge mistake I made was to focus too much on rational
belief, too little on rational action.


 The fourth-largest mistake I made was that I should have better
organized the content I was presenting in the sequences. In particular,
I should have created a wiki much earlier, and made it easier to read
the posts in sequence.


 \textit{That} mistake at least is correctable. In the present work
Rob Bensinger has reordered the posts and reorganized them as much as
he can without trying to rewrite all the actual material (though
he's rewritten a bit of it).


 My fifth huge mistake was that I---as I saw it---tried to speak
plainly about the stupidity of what appeared to me to be stupid ideas.
I did try to avoid the fallacy known as Bulverism, which is where you
\textit{open} your discussion by talking about how stupid people are
for believing something; I would always discuss the issue first, and
only afterwards say, ``And so this is
stupid.'' But in 2009 it was an open question in my
mind whether it might be important to have some people around who
expressed contempt for homeopathy. I thought, and still do think, that
there is an unfortunate problem wherein treating ideas courteously is
processed by many people on some level as ``Nothing
bad will happen to me if I say I believe this; I won't
lose status if I say I believe in homeopathy,'' and
that derisive laughter by comedians can help people wake up from the
dream.


 Today I would write more courteously, I think. The discourtesy did
serve a function, and I think there were people who were helped by
reading it; but I now take more seriously the risk of building
communities where the normal and expected reaction to low-status
outsider views is open mockery and contempt.


 Despite my mistake, I am happy to say that my readership has so
far been amazingly good about \textit{not} using my rhetoric as an
excuse to bully or belittle others. (I want to single out Scott
Alexander in particular here, who is a nicer person than I am and an
increasingly amazing writer on these topics, and may deserve part of
the credit for making the culture of \textit{Less Wrong} a healthy
one.)


 To be able to look backwards and say that you've
``failed'' implies that you had
goals. So what was it that I was trying to do?


 There is a certain valuable way of thinking, which is not yet
taught in schools, in this present day. This certain way of thinking is
not taught systematically at all. It is just absorbed by people who
grow up reading books like \textit{Surely You're
Joking, Mr. Feynman} or who have an unusually great teacher in high
school.


 Most famously, this certain way of thinking has to do with
science, and with the experimental method. The part of science where
you go out and look at the universe instead of just making things up.
The part where you say ``Oops'' and
give up on a bad theory when the experiments don't
support it.


 But this certain way of thinking extends beyond that. It is deeper
and more universal than a pair of goggles you put on when you enter a
laboratory and take off when you leave. It applies to daily life,
though this part is subtler and more difficult. But if you
can't say ``Oops''
and give up when it looks like something isn't working,
you have no choice but to keep shooting yourself in the foot. You have
to keep reloading the shotgun and you have to keep pulling the trigger.
You know people like this. And somewhere, someplace in your life
you'd rather not think about, you \textit{are} people
like this. It would be nice if there was a certain way of thinking that
could help us stop doing that.


 In spite of how large my mistakes were, those two years of blog
posting appeared to help a surprising number of people a surprising
amount. It didn't work reliably, but it worked
sometimes.


 In modern society so little is taught of the skills of rational
belief and decision-making, so little of the mathematics and sciences
underlying them\,\ldots that it turns out that just reading through a
massive brain-dump full of problems in philosophy and science can, yes,
be surprisingly good for you. Walking through all of that, from a dozen
different angles, can sometimes convey a glimpse of the central
rhythm.


 Because it is all, in the end, one thing. I talked about big
important distant problems and neglected immediate life, but the laws
governing them aren't actually different. There are
huge gaps in which parts I focused on, and I picked all the wrong
examples; but it is all in the end one thing. I am proud to look back
and say that, even after all the mistakes I made, and all the other
times I said ``Oops'' \ldots


 Even five years later, it still appears to me that this is better
than nothing.


 ~

{\raggedleft
 {}---Eliezer Yudkowsky, February 2015
\par}

\mysectiontwo{Biases: An Introduction}{Biases: An Introduction\newline by Rob Bensinger}


 It's not a secret. For some reason, though, it
rarely comes up in conversation, and few people are asking what we
should do about it. It's a pattern, hidden unseen
behind all our triumphs and failures, unseen behind our eyes. What is
it?


 Imagine reaching into an urn that contains seventy white balls and
thirty red ones, and plucking out ten mystery balls. Perhaps three of
the ten balls will be red, and you'll correctly guess
how many red balls total were in the urn. Or perhaps
you'll happen to grab four red balls, or some other
number. Then you'll probably get the total number
wrong.


 This random error is the cost of incomplete knowledge, and as
errors go, it's not so bad. Your estimates
won't be incorrect \textit{on average}, and the more
you learn, the smaller your error will tend to be.


 On the other hand, suppose that the white balls are heavier, and
sink to the bottom of the urn. Then your sample may be unrepresentative
\textit{in a consistent direction}.


 \textit{That} sort of error is called
``statistical bias.'' When your
method of learning about the world is biased, learning more may not
help. Acquiring more data can even consistently \textit{worsen} a
biased prediction.


 If you're used to holding knowledge and inquiry in
high esteem, this is a scary prospect. If we want to be sure that
learning more will help us, rather than making us worse off than we
were before, we need to discover and correct for biases in our data.


 The idea of \textit{cognitive bias} in psychology works in an
analogous way. A cognitive bias is a systematic error in \textit{how we
think}, as opposed to a random error or one that's
merely caused by our ignorance. Whereas statistical bias skews a sample
so that it less closely resembles a larger population, cognitive biases
skew our \textit{beliefs} so that they less accurately represent the
facts, and they skew our \textit{decision-making} so that it less
reliably achieves our goals.


 Maybe you have an optimism bias, and you find out that the red
balls can be used to treat a rare tropical disease besetting your
brother. You may then overestimate how many red balls the urn contains
because you \textit{wish} the balls were mostly red. Here, your sample
isn't what's biased.
\textit{You're} what's biased.

{
 Now that we're talking about biased
\textit{people}, however, we have to be careful. Usually, when we call
individuals or groups ``biased,'' we
do it to chastise them for being unfair or partial. \textit{Cognitive}
bias is a different beast altogether. Cognitive biases are a basic part
of how humans in general think, not the sort of defect we could blame
on a terrible upbringing or a rotten personality.\footnote{The idea of personal bias, media bias, etc.\ resembles
statistical bias in that it's an \textit{error}. Other
ways of generalizing the idea of
``bias'' focus instead on its
association with nonrandomness. In machine learning, for example, an
\textit{inductive} bias is just the set of assumptions a learner uses
to derive predictions from a data set. Here, the learner is
``biased'' in the sense that
it's pointed in a specific direction; but since that
direction might be \textit{truth}, it isn't a bad thing
for an agent to have an inductive bias. It's valuable
and necessary. This distinguishes inductive
``bias'' quite clearly from the
other kinds of bias.\comment{1}}}


 A cognitive bias is a systematic way that your innate patterns of
thought fall short of truth (or some other attainable goal, such as
happiness). Like statistical biases, cognitive biases can distort our
view of reality, they can't always be fixed by just
gathering more data, and their effects can add up over time. But when
the miscalibrated measuring instrument you're trying to
fix is \textit{you}, debiasing is a unique challenge.


 Still, this is an obvious place to start. For if you
can't trust your brain, how can you trust anything
else?


 It would be useful to have a name for this project of overcoming
cognitive bias, and of overcoming all species of error where our minds
can come to undermine themselves.


 We could call this project whatever we'd like. For
the moment, though, I suppose
``rationality'' is as good a name as
any.


\subsection{Rational Feelings}

{
 In a Hollywood movie, being
``rational'' usually means that
you're a stern, hyperintellectual stoic. Think Spock
from \textit{Star Trek}, who
``rationally'' suppresses his
emotions, ``rationally'' refuses to
rely on intuitions or impulses, and is easily dumbfounded and
outmaneuvered upon encountering an erratic or
``irrational''
opponent.\footnote{A sad coincidence: Leonard Nimoy, the actor who played Spock,
passed away just a few days before the release of this book. Though we
cite his character as a classic example of fake
``Hollywood rationality,'' we mean
no disrespect to Nimoy's memory.\comment{2}}}


 There's a completely different notion of
``rationality'' studied by
mathematicians, psychologists, and social scientists. Roughly,
it's the idea of \textit{doing the best you can with
what you've got}. A rational person, no matter how out
of their depth they are, forms the best beliefs they can with the
evidence they've got. A rational person, no matter how
terrible a situation they're stuck in, makes the best
choices they can to improve their odds of success.


 Real-world rationality isn't about ignoring your
emotions and intuitions. For a human, rationality often means becoming
more self-aware about your feelings, so you can factor them into your
decisions.


 Rationality can even be about knowing when \textit{not} to
overthink things. When selecting a poster to put on their wall, or
predicting the outcome of a basketball game, experimental subjects have
been found to perform \textit{worse} if they carefully analyzed their
reasons.\footnote{Timothy D. Wilson et al., ``Introspecting
About Reasons Can Reduce Post-choice Satisfaction,''
\textit{Personality and Social Psychology Bulletin} 19 (1993):
331--331.\comment{3}}\supercomma\footnote{Jamin Brett Halberstadt and Gary M. Levine,
``Effects of Reasons Analysis on the Accuracy of
Predicting Basketball Games,'' \textit{Journal of
Applied Social Psychology} 29, no. 3 (1999): 517--530.\comment{4}} There are some problems where conscious
deliberation serves us better, and others where snap judgments serve us
better.


 Psychologists who work on dual process theories distinguish the
brain's ``System 1''
processes (fast, implicit, associative, automatic cognition) from its
``System 2'' processes (slow,
explicit, intellectual, controlled cognition).\footnote{Keith E. Stanovich and Richard F. West,
``Individual Differences in Reasoning: Implications
for the Rationality Debate?,'' \textit{Behavioral and
Brain Sciences} 23, no. 5 (2000): 645--665,
\url{http://journals.cambridge.org/abstract_S0140525X00003435}.\comment{5}} The
\textit{stereotype} is for rationalists to rely entirely on System 2,
disregarding their feelings and impulses. Looking past the stereotype,
someone who is actually being rational---actually achieving their
goals, actually mitigating the harm from their cognitive biases---would
rely heavily on System-1 habits and intuitions where
they're reliable.

{
 Unfortunately, System 1 on its own seems to be a \textit{terrible}
guide to ``when should I trust System
1?'' Our untrained intuitions don't
tell us when we ought to stop relying on them. Being biased and being
unbiased \textit{feel} the same.\footnote{Timothy D. Wilson, David B. Centerbar, and Nancy Brekke,
``Mental Contamination and the Debiasing
Problem,'' in \textit{Heuristics and Biases: The
Psychology of Intuitive Judgment}, ed. Thomas Gilovich, Dale Griffin,
and Daniel Kahneman (Cambridge University Press, 2002).\comment{6}}}


 On the other hand, as behavioral economist Dan Ariely notes:
we're \textit{predictably} irrational. We screw up in
the same ways, again and again, systematically.


 If we can't use our gut to figure out when
we're succumbing to a cognitive bias, we may still be
able to use the sciences of mind.


\subsection{The Many Faces of Bias}


 To solve problems, our brains have evolved to employ cognitive
heuristics---rough shortcuts that get the right answer often, but not
all the time. Cognitive biases arise when the corners cut by these
heuristics result in a relatively consistent and discrete mistake.


 The representativeness heuristic, for example, is our tendency to
assess phenomena by how representative they seem of various categories.
This can lead to biases like the \textit{conjunction fallacy}. Tversky
and Kahneman found that experimental subjects considered it less likely
that a strong tennis player would ``lose the first
set'' than that he would ``lose the
first set but win the match.''\footnote{Amos Tversky and Daniel Kahneman,
``Extensional Versus Intuitive Reasoning: The
Conjunction Fallacy in Probability Judgment,''
\textit{Psychological Review} 90, no. 4 (1983): 293--315,
doi:10.1037/0033-295X.90.4.293.\comment{7}}
Making a comeback seems more \textit{typical} of a strong player, so we
overestimate the probability of this complicated-but-sensible-sounding
narrative compared to the probability of a strictly simpler scenario.

{
 The representativeness heuristic can also contribute to
\textit{base rate neglect}, where we ground our judgments in how
intuitively ``normal'' a combination
of attributes is, neglecting how common each attribute is in the
population at large.\footnote{Richards J. Heuer, \textit{Psychology of Intelligence Analysis}
(Center for the Study of Intelligence, Central Intelligence Agency,
1999).\comment{8}} Is it more likely that Steve is
a shy librarian, or that he's a shy salesperson? Most
people answer this kind of question by thinking about whether
``shy'' matches their stereotypes of
those professions. They fail to take into consideration how much more
common salespeople are than librarians---seventy-five times as common,
in the United States.\footnote{Wayne Weiten, \textit{Psychology: Themes and Variations,
Briefer Version, Eighth Edition} (Cengage Learning, 2010).\comment{9}}}

{
 Other examples of biases include \textit{duration neglect}
(evaluating experiences without regard to how long they lasted), the
\textit{sunk cost fallacy} (feeling committed to things
you've spent resources on in the past, when you should
be cutting your losses and moving on), and \textit{confirmation bias}
(giving more weight to evidence that confirms what we already
believe).\footnote{Raymond S. Nickerson, ``Confirmation Bias: A
Ubiquitous Phenomenon in Many Guises,''
\textit{Review of General Psychology} 2, no. 2 (1998): 175.\comment{10}}\supercomma\footnote{\textit{Probability neglect} is another cognitive bias. In the
months and years following the September 11 attacks, many people chose
to drive long distances rather than fly. Hijacking
wasn't \textit{likely}, but it now felt like it was on
the table; the mere possibility of hijacking hugely impacted decisions.
By relying on black-and-white reasoning (cars and planes are either
``safe'' or
``unsafe,'' full stop), people
actually put themselves in much more danger. Where they should have
weighed the probability of dying on a cross-country car trip against
the probability of dying on a cross-country flight---the former is
hundreds of times more likely---they instead relied on their general
feeling of worry and anxiety (the affect heuristic). We can see the
same pattern of behavior in children who, hearing arguments for and
against the safety of seat belts, hop back and forth between thinking
seat belts are a completely good idea or a completely bad one, instead
of trying to compare the strengths of the pro and con
considerations.\footnotemark%

 Some more examples of biases are: the \textit{peak/end rule}
(evaluating remembered events based on their most intense moment, and
how they ended); \textit{anchoring} (basing decisions on recently
encountered information, even when it's
irrelevant)\footnotemark and \textit{self-anchoring} (using
yourself as a model for others' likely characteristics,
without giving enough thought to ways you're
atypical);\footnotemark and \textit{status quo bias}
(excessively favoring what's normal and expected over
what's new and different).\footnotemark\comment{11}}
\footback{3}
\footnotetext{Cass R. Sunstein, ``Probability Neglect:
Emotions, Worst Cases, and Law,'' \textit{Yale Law
Journal} (2002): 61--107.\comment{21}}
\footnext
\footnotetext{Dan Ariely, \textit{Predictably Irrational: The Hidden Forces
That Shape Our Decisions} (HarperCollins, 2008).\comment{22}}
}
\footnext
\footnotetext{Boaz Keysar and Dale J. Barr,
``Self-Anchoring in Conversation: Why Language Users
Do Not Do What They
`Should,'\,'' in
\textit{Heuristics and Biases: The Psychology of Intuitive Judgment},
ed. Thomas Gilovich, Dale Griffin, and Daniel Kahneman (New York:
Cambridge University Press, 2002), 150--166, doi:10.2277/0521796792.\comment{23}}
\footnext
\footnotetext{Scott Eidelman and Christian S. Crandall,
``Bias in Favor of the Status Quo,''
\textit{Social and Personality Psychology Compass} 6, no. 3 (2012):
270--281.\comment{24}}

{
 Knowing about a bias, however, is rarely enough to protect you
from it. In a study of \textit{bias blindness}, experimental subjects
predicted that if they learned a painting was the work of a famous
artist, they'd have a harder time neutrally assessing
the quality of the painting. And, indeed, subjects who were told a
painting's author and were asked to evaluate its
quality exhibited the very bias they had predicted, relative to a
control group. When asked \textit{afterward}, however, the very same
subjects claimed that their assessments of the paintings had been
objective and unaffected by the bias---in all
groups!\footnote{Katherine Hansen et al., ``People Claim
Objectivity After Knowingly Using Biased
Strategies,'' \textit{Personality and Social
Psychology Bulletin} 40, no. 6 (2014): 691--699.\comment{12}}\supercomma\footnote{Similarly, Pronin writes of gender bias blindness:

 \begin{quote}
 In one study, participants considered a male and a female
candidate for a police-chief job and then assessed whether being
``streetwise'' or
``formally educated'' was more
important for the job. The result was that participants favored
whichever background they were told the male candidate possessed (e.g.,
if told he was ``streetwise,'' they
viewed that as more important). Participants were completely blind to
this gender bias; indeed, the more objective they believed they had
been, the more bias they actually showed.\footnotemark
\end{quote}

 Even when we know about biases, Pronin notes, we remain
``naive realists'' about our own
beliefs. We reliably fall back into treating our beliefs as
distortion-free representations of how things actually
are.\footnotemark\comment{13}}
\footback{1}
\footnotetext{Eric Luis Uhlmann and Geoffrey L. Cohen,
```I think it, therefore
it's true': Effects of Self-perceived
Objectivity on Hiring Discrimination,''
\textit{Organizational Behavior and Human Decision Processes} 104, no.
2 (2007): 207--223.\comment{25}}%
\footnext
\footnotetext{Emily Pronin, ``How We See Ourselves and How
We See Others,'' \textit{Science} 320 (2008):
1177--1180,
\url{http://psych.princeton.edu/psychology/research/pronin/pubs/2008\%20Self\%20and\%20Other.pdf}.\comment{26}}

}

{
 We're especially loathe to think of our views as
inaccurate compared to the views of others. Even when we correctly
identify others' biases, we have a special \textit{bias
blind spot} when it comes to our own flaws.\footnote{In a survey of 76 people waiting in airports, individuals
rated themselves much less susceptible to cognitive biases on average
than a typical person in the airport. In particular, people think of
themselves as unusually unbiased when the bias is socially undesirable
or has difficult-to-notice consequences.\footnotemark Other
studies find that people with personal ties to an issue see those ties
as enhancing their insight and objectivity; but when they see
\textit{other people} exhibiting the \textit{same} ties, they infer
that those people are overly attached and biased.\comment{14}}\footnotetext{Emily Pronin, Daniel Y. Lin, and Lee Ross,
``The Bias Blind Spot: Perceptions of Bias in Self
versus Others,'' \textit{Personality and Social
Psychology Bulletin} 28, no. 3 (2002): 369--381.\comment{27}} We fail
to detect any ``biased-feeling
thoughts'' when we introspect, and so draw the
conclusion that we must just be more objective than everyone
else.\footnote{Joyce Ehrlinger, Thomas Gilovich, and Lee Ross,
``Peering Into the Bias Blind Spot:
People's Assessments of Bias in Themselves and
Others,'' \textit{Personality and Social Psychology
Bulletin} 31, no. 5 (2005): 680--692.\comment{15}}}

{
 Studying biases can in fact make you \textit{more} vulnerable to
overconfidence and confirmation bias, as you come to see the influence
of cognitive biases all around you---in everyone but yourself. And the
bias blind spot, unlike many biases, is \textit{especially severe}
among people who are \textit{especially intelligent, thoughtful, and
open-minded}.\footnote{Richard F. West, Russell J. Meserve, and Keith E. Stanovich,
``Cognitive Sophistication Does Not Attenuate the Bias
Blind Spot,'' \textit{Journal of Personality and
Social Psychology} 103, no. 3 (2012): 506.\comment{16}}\supercomma\footnote{\ldots Not to be confused with people who think
they're unusually intelligent, thoughtful, etc.\ because
of the illusory superiority bias.\comment{17}}}


 This is cause for concern.


 Still\,\ldots it does seem like we should be able to do better.
It's known that we can reduce base rate neglect by
thinking of probabilities as frequencies of objects or events. We can
minimize duration neglect by directing more attention to duration and
depicting it graphically.\footnote{Michael J. Liersch and Craig R. M. McKenzie,
``Duration Neglect by Numbers and Its Elimination by
Graphs,'' \textit{Organizational Behavior and Human
Decision Processes} 108, no. 2 (2009): 303--314.\comment{18}} People vary in how
strongly they exhibit different biases, so there should be a host of
yet-unknown ways to influence how biased we are.


 If we want to improve, however, it's not enough
for us to pore over lists of cognitive biases. The approach to
debiasing in \textit{Rationality: From AI to Zombies} is to communicate
a systematic understanding of why good reasoning works, and of how the
brain falls short of it. To the extent this volume does its job, its
approach can be compared to the one described in Serfas, who notes that
``years of financially related work
experience'' didn't affect
people's susceptibility to the sunk cost bias, whereas
``the number of accounting courses
attended'' did help.

\begin{quote}
 As a consequence, it might be necessary to distinguish between
experience and expertise, with expertise meaning ``the
development of a schematic principle that involves conceptual
understanding of the problem,'' which in turn enables
the decision maker to recognize particular biases. However, using
expertise as countermeasure requires more than just being familiar with
the situational content or being an expert in a particular domain. It
requires that one fully understand the underlying rationale of the
respective bias, is able to spot it in the particular setting, and also
has the appropriate tools at hand to counteract the
bias.\footnote{Sebastian Serfas, \textit{Cognitive Biases in the Capital
Investment Context: Theoretical Considerations and Empirical
Experiments on Violations of Normative Rationality} (Springer, 2010).\comment{19}}
\end{quote}


 The goal of this book is to lay the groundwork for creating
rationality ``expertise.'' That
means acquiring a deep understanding of the structure of a very general
problem: human bias, self-deception, and the thousand paths by which
sophisticated thought can defeat itself.

\subsection{A Word About This Text}


 \textit{Rationality: From AI to Zombies} began its life as a
series of essays by Eliezer Yudkowsky, published between 2006 and 2009
on the economics blog \textit{Overcoming Bias} and its spin-off
community blog \textit{Less Wrong}. I've worked with
Yudkowsky for the last year at the Machine Intelligence Research
Institute (MIRI), a nonprofit he founded in 2000 to study the
theoretical requirements for smarter-than-human artificial intelligence
(AI).


 Reading his blog posts got me interested in his work. He impressed
me with his ability to concisely communicate insights it had taken me
years of studying analytic philosophy to internalize. In seeking to
reconcile science's anarchic and skeptical spirit with
a rigorous and systematic approach to inquiry, Yudkowsky tries not just
to refute but to \textit{understand} the many false steps and blind
alleys bad philosophy (and bad lack-of-philosophy) can produce. My hope
in helping organize these essays into a book is to make it easier to
dive in to them, and easier to appreciate them as a coherent whole.


 The resultant rationality primer is frequently personal and
irreverent---drawing, for example, from Yudkowsky's
experiences with his Orthodox Jewish mother (a psychiatrist) and father
(a physicist), and from conversations on chat rooms and mailing lists.
Readers who are familiar with Yudkowsky from \textit{Harry Potter and
the Methods of Rationality}, his science-oriented take-off of J.K.
Rowling's \textit{Harry Potter} books, will recognize
the same irreverent iconoclasm, and many of the same core concepts.


 Stylistically, the essays in this book run the gamut from
``lively textbook'' to
``compendium of thoughtful
vignettes'' to ``riotous
manifesto,'' and the content is correspondingly
varied. \textit{Rationality: From AI to Zombies} collects hundreds of
Yudkowsky's blog posts into twenty-six
``sequences,'' chapter-like series
of thematically linked posts. The sequences in turn are grouped into
six books, covering the following topics:


 Book 1---\textbf{Map and Territory}. What is a belief, and what
makes some beliefs work better than others? These four sequences
explain the \textit{Bayesian} notions of rationality, belief, and
evidence. A running theme: the things we call
``explanations'' or
``theories'' may not always function
like \textit{maps} for navigating the world. As a result, we risk
mixing up our mental maps with the other objects in our toolbox.


 Book 2---\textbf{How to Actually Change Your Mind}. This truth
thing seems pretty handy. Why, then, do we keep jumping to conclusions,
digging our heels in, and recapitulating the same mistakes? Why are we
so \textit{bad} at acquiring accurate beliefs, and how can we do
better? These seven sequences discuss motivated reasoning and
confirmation bias, with a special focus on hard-to-spot species of
self-deception and the trap of ``using arguments as
soldiers.''


 Book 3---\textbf{The Machine in the Ghost}. Why
\textit{haven't} we evolved to be more rational? Even
taking into account resource constraints, it seems like we could be
getting a lot more epistemic bang for our evidential buck. To get a
realistic picture of how and why our minds execute their biological
functions, we need to crack open the hood and see how evolution works,
and how our brains work, with more precision. These three sequences
illustrate how even philosophers and scientists can be led astray when
they rely on intuitive, non-technical evolutionary or psychological
accounts. By locating our minds within a larger space of goal-directed
systems, we can identify some of the peculiarities of human reasoning
and appreciate how such systems can ``lose their
purpose.''


 Book 4---\textbf{Mere Reality}. What kind of world do we live in?
What is our place in that world? Building on the previous
sequences' examples of how evolutionary and cognitive
models work, these six sequences explore the nature of mind and the
character of physical law. In addition to applying and generalizing
past lessons on scientific mysteries and parsimony, these essays raise
new questions about the role science should play in individual
rationality.


 Book 5---\textbf{Mere Goodness}. What makes something
\textit{valuable}{}---morally, or aesthetically, or prudentially? These
three sequences ask how we can justify, revise, and naturalize our
values and desires. The aim will be to find a way to understand our
goals without compromising our efforts to actually achieve them. Here
the biggest challenge is knowing when to trust your messy, complicated
case-by-case impulses about what's right and wrong, and
when to replace them with simple exceptionless principles.


 Book 6---\textbf{Becoming Stronger}. How can individuals and
communities put all this into practice? These three sequences begin
with an autobiographical account of Yudkowsky's own
biggest philosophical blunders, with advice on how he thinks others
might do better. The book closes with recommendations for developing
evidence-based applied rationality curricula, and for forming groups
and institutions to support interested students, educators,
researchers, and friends.


 The sequences are also supplemented with
``interludes,'' essays taken from
Yudkowsky's personal website, \url{http://www.yudkowsky.net}.
These tie in to the sequences in various ways; e.g., The Twelve Virtues
of Rationality poetically summarizes many of the lessons of
\textit{Rationality: From AI to Zombies}, and is often quoted in other
essays.


The original versions of the essays were on \textit{Less
Wrong}\footnote{\url{http://lesswrong.com/}} (where you can leave
comments) or on Yudkowsky's website. You can also find a glossary
for \textit{Rationality: From AI to Zombies} terminology online,
at \url{http://wiki.lesswrong.com/wiki/RAZ\_Glossary}.


\subsection{Map and Territory}


 This, the first book, begins with a sequence on cognitive bias:
``Predictably Wrong.'' The rest of
the book won't stick to just this topic; bad habits and
bad ideas matter, even when they arise from our minds'
contents as opposed to our minds' structure. Thus
evolved and invented errors will both be on display in subsequent
sequences, beginning with a discussion in ``Fake
Beliefs'' of ways that one's
expectations can come apart from one's professed
beliefs.


 An account of irrationality would also be incomplete if it
provided no theory about how \textit{rationality} works---or if its
``theory'' only consisted of vague
truisms, with no precise explanatory mechanism. The
``Noticing Confusion'' sequence asks
why it's useful to base one's behavior
on ``rational'' expectations, and
what it feels like to do so.


 ``Mysterious Answers'' next
asks whether science resolves these problems for us. Scientists base
their models on repeatable experiments, not speculation or hearsay. And
science has an excellent track record compared to anecdote, religion,
and\,\ldots pretty much everything else. Do we still need to worry about
``fake'' beliefs, confirmation bias,
hindsight bias, and the like when we're working with a
community of people who want to \textit{explain} phenomena, not just
tell appealing stories?


 This is then followed by The Simple Truth, a stand-alone allegory
on the nature of knowledge and belief.


 It is cognitive bias, however, that provides the clearest and most
direct glimpse into the stuff of our psychology, into the shape of our
heuristics and the logic of our limitations. It is with bias that we
will begin.

{
 There is a passage in the \textit{Zhuangzi}, a proto-Daoist
philosophical text, that says: ``The fish trap exists
because of the fish; once you've gotten the fish, you
can forget the trap.''\footnote{Zhuangzi and Burton Watson, \textit{The Complete Works of
Zhuangzi} (Columbia University Press, 1968).\comment{20}}}


 I invite you to explore this book in that spirit. Use it like
you'd use a fish trap, ever mindful of the purpose you
have for it. Carry with you what you can use, so long as it continues
to have use; discard the rest. And may your purpose serve you well.


\subsection{Acknowledgments}


 I am stupendously grateful to Nate Soares, Elizabeth Tarleton,
Paul Crowley, Brienne Strohl, Adam Freese, Helen Toner, and dozens of
volunteers for proofreading portions of this book.


 Special and sincere thanks to Alex Vermeer, who steered this book
to completion, and Tsvi Benson-Tilsen, who combed through the entire
book to ensure its readability and consistency.

\myendsectiontext


\bigskip